VXVM Recovery

Veritas Volume Manager
(Recovery techniques and Troubleshooting)
Author
Name
Project
Murali Thangavel
Applied Materials
Reviewed by
Veritas Volume Manager (Recovery techniques and Troubleshooting)
S.No.
1
2
3
4
5
6
7
8
Topics
Veritas Disk Headers
Veritas Configuration Database
Configuration Backup
Restoring configuration
Recovering from path failure
Recovering deleted volume
Veritas Explorer
Troubleshooting
This document will provide an introduction to important and often overlooked

recovery features, configuration database backups, VrtsExplorer and
Troubleshooting. The document will also provide two disaster-recovery case
studies to show how these recovery features can be used to aide in recovering
from disasters when they strike.
Veritas Disk Headers

When a device is initialized or encapsulated with Veritas Volume Manager, a
disk header is written to the device's private region. The disk header contains a
disk id to uniquely identify the device, a disk group identifier to indicate which
disk group the device is associated with, a set of flags to indicate the status and
intended use (e.g., hot spare) of the device, and a hostid to indicate which host
(if any) has the device imported. The disk header can be printed by passing the
Veritas device name to vxdisk "list" option:
# vxdisk list c1t3d0s2
Device: c1t3d0s2
devicetag: c1t3d0
type:
auto
hostid: ossa
disk:
name=disk02 id=1265408080.9.ossa
group: name=datadg id=1265408259.16.ossa
info:
format=sliced,privoffset=1,pubslice=4,privslice=3
flags: online ready private autoconfig autoimport imported
pubpaths: block=/dev/vx/dmp/c1t3d0s4 char=/dev/vx/rdmp/c1t3d0s4
privpaths: block=/dev/vx/dmp/c1t3d0s3 char=/dev/vx/rdmp/c1t3d0s3
guid:
udid:
SEAGATE%5FST914602SSUN146G%5FDISKS%5F30383232393633425A5A0000
site:
version: 2.1
iosize: min=512 (bytes) max=2048 (blocks)
public: slice=4 offset=0 len=286596864 disk_offset=101760
private: slice=3 offset=1 len=81151 disk_offset=20352
update: time=1273843696 seqno=0.10
ssb:
actual_seqno=0.0
headers: 0 248
configs: count=1 len=59873
logs:
count=1 len=9071
Defined regions:
config priv 000017-000247[000231]: copy=01 offset=000000 enabled
config priv 000249-059890[059642]: copy=01 offset=000231 enabled
log
priv 059891-068961[009071]: copy=01 offset=000000 enabled
Multipathing information:
numpaths: 1
c1t3d0s2
state=enabled
#
Due to the critical nature of the configuration information stored in the disk
headers, it is important to periodically back up this information. As I will discuss in
the section Configuration Backups, Veritas introduced new features in Veritas
Volume Manager 4.X to ease the backup process.
Veritas Configuration Database

When new objects (e.g., subdisks, plexes, volumes) are created with the Veritas
CLI or GUI, Veritas will write the configuration data to describe these objects to a
configuration database. The configuration database is stored in the private
region of several devices in each disk group for redundancy. To list the devices in
a disk group with a copy of the configuration database, the vxdg utility can be
invoked with the "list" option and a disk group to query:
bash-3.00# vxdg list datadg | egrep "config disk.*clean online"

config disk c1t2d0s2 copy 1 len=59873 state=clean online
config disk c1t3d0s2 copy 1 len=59873 state=clean online
bash-3.00#
To print the contents of the configuration database, the vxprint utility can be
used. The following example uses the vxprint "-h" (list record hierarchies) and "-t"
(use one-line format tailored for each record type) options to display the
configuration database with an informational header and descriptive records
bash-3.00# vxprint -ht -g datadg
DG NAME
NCONFIG
NLOG MINORS GROUP-ID
ST NAME
STATE
DM_CNT SPARE_CNT
APPVOL_CNT
DM NAME
DEVICE
TYPE PRIVLEN PUBLEN STATE
RV NAME
RLINK_CNT KSTATE STATE PRIMARY DATAVOLS SRL
RL NAME
RVG
KSTATE STATE REM_HOST REM_DG REM_RLNK
CO NAME
CACHEVOL KSTATE STATE
VT NAME
RVG
KSTATE STATE NVOLUME
V NAME
RVG/VSET/CO KSTATE STATE LENGTH READPOL PREFPLEX UTYPE
PL NAME
VOLUME
KSTATE STATE LENGTH LAYOUT NCOL/WID MODE
SD NAME
PLEX
DISK DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME
PLEX
VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM MODE
SC NAME
PLEX
CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
DC NAME
PARENTVOL LOGVOL
SP NAME
SNAPVOL
DCO
EX NAME
ASSOC
VC
PERMS MODE STATE
SR NAME
KSTATE
dg datadg
default
dm disk01
dm disk02
c1t2d0s2
c1t3d0s2
v vbr
default 35000
auto
auto
81151
81151
1265408259.16.ossa
286596864 286596864 -
ENABLED ACTIVE 20971520 SELECT
fsgen
pl vbr-01
vbr
ENABLED ACTIVE 20971520 CONCAT
sd disk01-01 vbr-01
disk01 0
20971520 0
c1t2d0
pl vbr-02
vbr
ENABLED ACTIVE 20971520 CONCAT
sd disk02-01 vbr-02
disk02 0
20971520 0
c1t3d0
bash-3.00#
RW
ENA
RW
ENA
The object types (e.g., subdisks, disk groups, plexes, volumes, etc.), block offsets,
object names, and locations can be useful for understanding the Veritas Volume
Manager layout and reassembling the configuration layout during a disaster. Due
to the importance of this information, backing up the configuration database is
imperative! As you will see in the next section, Veritas Volume Manager makes
backing up the configuration information a snap.
Configuration Backups
The vxconfigbackupd daemon was introduced in Veritas Volume Manager 4.X
to provide an automated way to back up disk headers and the configuration
database. Vxconfigbackupd is started automatically at system boot time and
actively monitors the configuration database and disk headers. When
vxconfigbackupd detects a change to the active configuration,
vxconfigbackupd will write the configuration to a set of files in the
/etc/vx/cbr/bk/<dgname>.<dgid> directory. ("<dgname>" identifies the disk
group, and "<dgid>" identifies the disk group id.) The vxconfigbackupd man
page provides the following descriptions for the files in the
/etc/vx/cbr/bk/<dgname>.<dgid> directory:
/etc/vx/cbr/bk/<dgname>.<dgid>/<dgid>.diskinfo
Location of backup file for disk attributes.
/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.binconfig
Location of backup file for binary configuration copy.
/etc/vx/cbr/bk/<dgname>>.<dgid>/<dgid>.cfgrec
Location of backup file for configuration records in vxprint -m format.
In addition to the automated backups that are performed by
vxconfigbackupd, manual backups can be performed with the
vxconfigbackup utility. This is valuable for backing up specific versions of the
configuration and archiving the configuration to a user-supplied destination.
The following example uses the vxconfigbackup "-l" (location to store backup)
to back up the current configuration to the /etc/dgbackups directory:
$ /usr/lib/vxvm/bin/vxconfigbackup -l /etc/dgbackups
Start backing up diskgroup oradg to \
/etc/dgbackups/oradg.1127240283.19.ossa ...
VxVM NOTICE V-5-2-3100 Backup complete for diskgroup: oradg
If you are using a version of Veritas Volume Manager prior to 4.X, you will be
unable to use the vxconfigbackupd and vxconfigbackup utilities to manage
backups. Don't fear -- the sample shell script in Listing 1 can be used to archive
the configuration to a text file, which can be passed to vxmake "-d" option to
re-create the configuration if disaster were to strike.
Restoring Configuration
When configuration information is lost due to a system outage or a mistake at
the keyboard, the configuration can often be recovered to a point in time
(preferably to a point in time when a known working backup was taken with
vxconfigbackup ) with the vxconfigrestore utility. The following example shows
how to restore the contents on the oradg disk group using the configuration
backup in the /etc/dgbackups directory:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups oradg
Diskgroup oradg configuration restoration started ......
Installing volume manager disk header for c1t1d0s2 ...

oradg's diskgroup configuration is restored (in a precommitted state).
Diskgroup can be accessed in read only and can be examined using
vxprint in this state.
Run:
vxconfigrestore -l /etc/dgbackups/ -c oradg ==> to commit the restoration.
vxconfigrestore -l /etc/dgbackups/ -d oradg ==> to abort the restoration.
When vxconfigrestore is invoked, the configuration will be staged to a transient
location, allowing the administrator to validate the configuration with the
vxprint utility. If the configuration looks sound, the configuration can be
committed to the disk headers and the configuration database by invoking
vxconfigrestore with the "-c" (commit) option:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/ -c oradg

Committing configuration restoration for diskgroup oradg ....
oradg's diskgroup configuration restoration is committed.
Once the configuration is restored, the volumes can be started, and the file
systems that reside on those volumes can be mounted.
Case Study #1: Recovering from a Power Failure

While sitting at my desk on a Friday (around 03am), DCO team got the High
priority ticket and they called me and informed that the below message:
Volume Manager failures on host ossa
Content-Length: 240
Failures have been detected by the VERITAS Volume Manager:
failed volumes:
oravol01
I immediately logged into ossa and received the following error when trying
to list the directory that was on the failed volume oravol01:
$ ls -la /oracle/admin
.: I/O error
A quick check of the system logs revealed numerous SCSI error messages:
$ tail /var/adm/messages
Jul 26 17:49:37 ossa scsi: [ID 107833 kern.warning] WARNING:
/pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:37 ossa SCSI transport failed: reason 'incomplete': retrying command
Jul 26 17:49:42 ossa scsi: [ID 107833 kern.warning] WARNING:
/pci@1f,0/pci@1/scsi@4/sd@1,0 (sd1):
Jul 26 17:49:42 ossa disk not responding to selection
.
The Veritas vxprint utility was also unable to display the disk group
configuration (since the configuration database was unavailable):
$ vxprint -g oradg -ht
$
10
Got the information from HnE, that the storage array was powered off, Based
on their findings that a power cords failed. Once they replaced the cord, the
storage array came back to life, but the oradg disk group I needed to access
was in the disabled state:
$ vxdg list
NAME
STATE
oradg
disabled
ID
1123603158.13. ossa
A quick check of the disks showed that they were online and associated with
the disabled disk group:
$ vxdisk list
DEVICE
TYPE
DISK
GROUP
STATUS
c0t0d0s2
auto:none
online invalid
c0t1d0s2
auto:none
online invalid
c1t1d0s2
auto:cdsdisk
c1t1d0
oradg
online dgdisabled
c1t2d0s2
auto:cdsdisk
c1t2d0
oradg
online dgdisabled
c1t3d0s2
auto:cdsdisk
c1t3d0
oradg
online dgdisabled
c1t4d0s2
auto:cdsdisk
c1t4d0
oradg
online dgdisabled
c1t5d0s2
auto:cdsdisk
c1t5d0
oradg
online dgdisabled
c1t6d0s2
auto:cdsdisk
c1t6d0
oradg
online dgdisabled
In some situations, Veritas may report offline devices as "failed was: cXtXdXs2".
When this happens, the vxreattach command can reconnect Veritas Volume
Manager to "lost" devices. Luckily, in our case, Veritas was able to reconnect
to the devices so I unmounted the file system, deported the disk group, and
imported the disk group to enable the oradg disk group:
$ umount /oracle/admin
$ vxdg deport oradg
$ vxdg import oradg
11
Deport and Import operations are required to fix a disabled disk group and will
validate that the disk group configuration records are consistent. Once the
disk group was imported, I ran vxinfo to view the volume and plex status.
$ vxinfo -g oradg p
vol oravol01
fsgen Startable
plex oravol01-03 ACTIVE
vol oravol01-L01 fsgen Startable
plex oravol01-P01 ACTIVE
The "Startable" flag indicates that the volume and plexes are in a startable
state, so I executed the vxvol utility to start the volume:
$ vxvol -g oradg start oravol01
Once the volume came online, I ran fsck to replay the transactions in the VxFS
journal:
$ fsck -F vxfs /dev/vx/dsk/oof/oravol01
log replay in progress
replay complete - marking super-block as CLEAN
After fsck finished the consistency check, I mounted the file system. Due to the
recovery features built into Veritas volume manager and Veritas file system, we
were able to avoid a full file system restore! Since I received the failure
notification immediately after Veritas detected a problem with the volume, I
was able to reduce the time it took to recover the faulted system.
12
Case Study #2: Recovering a Deleted Disk group (Volumes)

While writing shell script on a Wednesday morning (around 4am), one of my
colleagues came to my desk and told me he/she had accidentally destroyed
the production disk group instead of the test disk group he/she thought he/she
was working on. A quick check of the Veritas configuration verified this:
$ vxprint -hft
$
Since we used vxconfigbackup to take regular backups of the disk group
configuration, we located the last known good working backup and passed
this location to vxconfigrestore 's "-l" (location) option:
$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/oradg_0612205 oradg
Diskgroup oradg configuration restoration started ......

oradg's diskgroup configuration is restored (in precommitted state).

Diskgroup can be accessed in read only and can be examined using
vxprint in this state.
Run:
vxconfigrestore -l /etc/dgbackups/oradg_0612205 -c oradg ==> to commit
the restoration.
13
vxconfigrestore -l /etc/dgbackups/oradg_0612205 -d oradg ==> to abort the

restoration.
I verified the configuration was consistent by running vxprint , and committed
the configuration to the disk headers and configuration database by invoking
vxconfigrestore with the "-c" option:
$ /usr/lib/vxvm/bin/vxconfigrestore -c -l /var/tmp/back oradg
Committing configuration restoration for diskgroup oradg ....
oradg's diskgroup configuration restoration is committed.
Once we committed the configuration, the layout matched our last known
good configuration backup:
$ vxprint -ht
Disk group: oradg
DG NAME NCONFIG
ST NAME STATE
NLOG
MINORS GROUP-ID
DM_CNT SPARE_CNT
DM NAME DEVICE
TYPE
PRIVLEN PUBLEN STATE
RV NAME RLINK_CNT KSTATE STATE

RL NAME RVG
KSTATE STATE
CO NAME CACHEVOL
VT NAME NVOLUME
PRIMARY DATAVOLS SRL
REM_HOST REM_DG
REM_RLNK
KSTATE STATE
KSTATE STATE
V NAME RVG/VSET/CO KSTATE STATE

PL NAME VOLUME
APPVOL_CNT
KSTATE STATE
LENGTH READPOL PREFPLEX UTYPE
LENGTH LAYOUT
NCOL/WID MODE
SD NAME PLEX
DISK
DISKOFFS LENGTH [COL/]OFF DEVICE MODE
SV NAME PLEX
VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM
SC NAME PLEX
CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE
MODE
DC NAME PARENTVOL LOGVOL

SP NAME SNAPVOL
DCO
14
dg oradg
default
default 10000
1127240283.19. ossa
v oravol01 - ENABLED ACTIVE 41943040 SELECT

pl oravol01-03 oravol01
oravol01-03 fsgen
ENABLED ACTIVE 41943168 STRIPE 3/128 RW
sv oravol01-S01 oravol01-03 oravol01-L01 1 13981056 0/0
2/2
ENA
2/2
ENA
2/2
ENA
..
.
v oravol01-L03 -
ENABLED SYNC 13981056 SELECT -
fsgen
pl oravol01-P05 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT sd c1t3d0-02
oravol01-P05 c1t3d0 0
13981056 0
c1t3d0 ENA
pl oravol01-P06 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT sd c1t6d0-02
oravol01-P06 c1t6d0 0
13981056 0
RW
RW
c1t6d0 ENA
To ensure that no damage had occurred to the file system, we used fsck to
consistency check the file system:
$ fsck -F vxfs /dev/vx/rdsk/oradg/oravol01
and got our Oracle DBA to verify the integrity of each data file with Oracle's
database verification utilities. Because accidents and disasters can strike at
any time, it is important to perform regular backups of the Veritas
configuration and store this data in a safe.
15
Veritas Explorer
VRTSexplorer , also known as Veritas Explorer, VxExplorer, vxexplorer, and
vxexplore, is a tool provided by Symantec Corporation Technical Support that is
executed on a host to gather some of the information that may be needed by a
Support Engineer to troubleshoot an issue. The output of this utility is a compressed
tar file that must be sent to Support. The information collected may not be
sufficient to resolve the issue, but it is likely to give very relevant information about
it.
The use of VRTSexplorer utility involves three different steps:
1.
2.
3.
Obtaining VRTSexplorer
Executing the VRTSexplorer binary
Sending the output to Symantec Technical Support (Vendor)
These steps are detailed below:

1. Obtaining the package that contains the VRTSexplorer utility:
To download the VRTSexplorer utility, follow the below instructions:
Connect to the FTP site:
# ftp ftp.veritas.com
The below message appears with a prompt for a login name. Log in as the
user anonymous.
Connected to ftp.veritas.com
220-ftp
220-***********************************************************************
220- This system is for the use of authorized users only. Individuals
220- using this computer system without authority, or in excess of their
220- authority, are subject to having all of their activities on this
220- system monitored and recorded.
220- In the course of monitoring individuals improperly using this
220- system, or in the course of system maintenance, the activities of
220- authorized users also are monitored.
220- Anyone using this system expressly consents to such monitoring
220- and recording. Please be advised that unauthorized access to this
220- system is a violation of State and Federal law. If monitoring
220- reveals possible evidence of criminal activity, system personnel
220- may provide that evidence to law enforcement officials.
220-***********************************************************************
16
220
Name (ftp.veritas.com <ftp://ftp.veritas.com/>): anonymous
The below message appears with a prompt for a password. Use a full email
address as the password:
331 Guest login ok, send your complete e-mail address as password.
Password: customer@email.com <mailto:customer@email.com>
While logged into the FTP server as the user anonymous, it is not possible to
list any files or directories. It is important that the below steps are followed
exactly:
Change directories to /pub/support:
ftp> cd /pub/support
Set FTP mode to binary:
ftp> bin
Download the file that contains utility:
ftp> get vxexplore.tar.Z
Close the ftp connection:
ftp> bye
Untar the download:
Untar the downloaded file (only if the VRTSexplorer utility has been
downloaded and not installed with the VRTSspt package) :
# zcat vxexplore.tar.Z | tar xvf The above command will create a VRTSexplorer directory containing
everything that is needed. Change into this directory:
# cd VRTSexplorer
2. Execute the VRTSexplorer utility:
Some notes on the execution of VRTSexplorer:
To determine the version of VRTSexplorer that is on the host, along with

other options, execute # ./VRTSexplorer -help:
# ./VRTSexplorer -help
VRTSexplorer: version 5.5o
To run the VRTSexplorer utility for only one particular product, such as
Veritas Volume Manager, execute:
# ./VRTSexplorer vxvm
To run the VRTSexplorer utility to exclude a particular product, such as

Veritas NetBackup, execute:
# ./VRTSexplorer -nbu
17
Refer to the README file for additional options that are available with the
VRTSexplorer utility.
The below messages will appear requesting input for the case number,
destination directory, restart of vxconfigd (do not choose to do this if
PowerPath is installed or if this is a node in a cluster), etc. The program will
output several messages and finish by tarring and compressing the
collected information. Please note that this operation can take some time
to complete.
Below is partial example of what might be seen (the messages below are
from VRTSexplorer version 5.5o and will be different from earlier versions of
the utility) when executing the VRTSexplorer utility to gather information
about a host, output is dependant on which product packages are
installed:
# ./VRTSexplorer
VRTSexplorer: Initializing.
VRTSexplorer: Please enter case number, or just hit enter: 150-175-206
VRTSexplorer: Please select a destination directory (default: /tmp): /tmp
VRTSexplorer: Using /tmp as destination directory.
VRTSexplorer: Collecting system configuration information for SunOS system.
VRTSexplorer: Collecting VERITAS package version information.
VRTSexplorer: Collecting loadable module information.
VRTSexplorer: Collecting SLIM information.
VRTSexplorer: Collecting SLIM Agent installer logs.
VRTSexplorer: Collecting SLIM Server installer logs.
VRTSexplorer: Collecting Cross Product/Platform Installation (CPI)
information.
VRTSexplorer: Collecting ISIS configuration information.
VRTSexplorer: Collecting VERITAS SANPoint Control Console configuration
information.
##### All information and files will be placed under
/tmp/VRTSexplorer_150-175-206_server1/spc
Logfile is /tmp/VRTSexplorer_150-175-206_server1/spc/LOG
##### Capturing SAL/VAIL/VEA information from host server1'
##### Saving VERITAS Package Information..........
##### Logging VEA Information #####
+Copying vxisis.log files.
+Copying /var/vx/isis vxsvc corefiles
+Copying /opt/VRTSob/bin vxsvc corefiles
##### Logging VRTSvail Information #####
Checksum /opt/VRTSvail/providers/vx*:...............
18
##### Saving Detailed information for all VERITAS

packages..........................
##### Saving Host syslogs #####
+Copying syslog files.....
VRTSexplorer: Collecting SIG licensing information.
VRTSexplorer: Collecting VRW configuration information.
VRTSexplorer: Collecting VxFS configuration information.
VRTSexplorer: Determining current VxVM operating mode.
VRTSexplorer: Collecting VxVM configuration information.
VRTSexplorer: Collecting DMP configuration information.
NOTICE: This section will stop and restart the VxVM Configuration Daemon,
vxconfigd. This may cause your VxVA, VMSA and/or VEA session to exit.
This may also cause a momentary stoppage of any VxVM configuration
actions. This should not harm any data; however, it may cause some
configuration operations (e.g. moving subdisks, plex
resynchronization) to abort unexpectedly. Any VxVM configuration
changes should be completed before running this section.
If you are using EMC PowerPath devices with Veritas Volume Manager,
you must run the EMC command(s) 'powervxvm setup' (or 'safevxvm
setup') and/or 'powervxvm online' (or 'safevxvm online') if this
script terminates abnormally.
Restart VxVM Configuration Daemon? [y,n] (default: n) n
VRTSexplorer: Collecting VRAS configuration information.
VRTSexplorer: Collecting Web GUI Engine information.
VRTSexplorer: Collecting Cross Product/Platform Installation (CPI)
information.
VRTSexplorer: Script finished.
VRTSexplorer: The cksum for the tarfile is:
2113302371 2569248 /tmp/VRTSexplorer_150-175-206_server1.tar.gz
VRTSexplorer: Please ftp /tmp/VRTSexplorer_150-175-206_server1.tar.gz
VRTSexplorer: to ftp.veritas.com:/incoming or work with your
VRTSexplorer: support representative for other upload options.
#
19
3. Send the file created above to Symantec Technical Support:

Example:
Change directory to the destination directory (default: /tmp):
# cd /tmp
Connect to the FTP site:
# ftp ftp.veritas.com
The below message appears with a prompt for a login name. Log in as the
user anonymous.
Connected to ftp.veritas.com
220-ftp
220-***********************************************************************
220- This system is for the use of authorized users only. Individuals
220- using this computer system without authority, or in excess of their
220220220220220220220-
authority, are subject to having all of their activities on this

system monitored and recorded.
In the course of monitoring individuals improperly using this
system, or in the course of system maintenance, the activities of
authorized users also are monitored.
Anyone using this system expressly consents to such monitoring
and recording. Please be advised that unauthorized access to this
220- system is a violation of State and Federal law. If monitoring

220- reveals possible evidence of criminal activity, system personnel
220- may provide that evidence to law enforcement officials.
220-***********************************************************************
220
Name (ftp.veritas.com <ftp://ftp.veritas.com/>): anonymous
The below message appears with a prompt for a password. Use a full email
address as the password:
331 Guest login ok, send your complete e-mail address as password.
Password: customer@email.com <mailto:customer@email.com>
While logged into the FTP server as the user anonymous, it is not possible to
list any files or directories. It is important that the below steps are followed
exactly:
Change into the /incoming directory:
ftp> cd /incoming
Set FTP mode to binary:
ftp> bin
Upload the file that was created above:
ftp> put VRTSexplorer_150-175-206_server1.tar.Z
Close the ftp connection:
ftp> bye
20
Troubleshooting
-Troubleshooting the Boot Process
-Recovering the Boot Disk Group
Files Used in the Boot Process
* /etc/system
Contains VxVM entries
* /etc/vfstab
Maps mount points to devices
* /etc/vx/volboot
Contains disk ownership data
* /etc/vx/licenses/lic, /etc/vx/elm
Contains license files
* /var/vxvm/tempdb
Stores data about diskgroups
* /etc/vx/reconfig.d/state.d/install-db
Indicates VxVM is not initialized
* /VXVM#.#.#-UPGRADE/.start_runed
Indicates that the VxVM upgrade is not complete
21
Troubleshooting: The Boot Device Cannot be Opened

Possible causes:
* Boot disk is not powered on
* Boot disk has failed
* SCSI bus is not terminated
* Controller failure has occurred
* Disk is failing and locking the bus
To resolve:
* Check SCSI bus connections:
- Use probe-scsi-all
* Boot from an alternate boot disk
Troubleshooting: Startup Scripts Exit Without Initialization

Possible causes:
Either one of the following files is present:
* /etc/vx/reconfig.d/state.d/install-db
This file indicates that VxVM software packages have been added, but VxVM has
not been initialized with vxinstall. Therefore, vxconfigd is not started.
* / VXVM#.#.#-UPGRADE/.start_runed
This file indicates that a VxVM upgrade has been started but not completed.
Therefore, vxconfigd is not started.
Troubleshooting: Conflicting Host ID in volboot
The volboot file contains the host ID that was on the system when you installed
VxVM.
If you manually edit this file, VxVM does not function.
* To change the hostname in the volboot file:
vxdctl hosted newhostname
* To re-create the volboot file:
vxdctl init [hostname]
22
Troubleshooting: License Problems (keys corrupted, missing, or expired)

Save /etc/vx/licenses/lic/* to a backup device. If the license files are removed or
corrupted, you can copy the files back.
License problems can occur if:
* The /etc/vx/licenses/lic files become corrupted
* An evaluation license was installed and not updated to a full license.
To resolve license issues:
* vxlicinst (installs a new license)
* vxiod set 10 (starts the I/O daemons)
* vxconfigd (starts the configuration daemon)
Troubleshooting: Missing /var/vxvm/tempdb (missing, misnamed, or corrupted)

This directory stores configuration information about imported diskgroups. The
contents are recreated after a reboot. If this directory is missing, misnamed, or
corrupted, vxconfigd does not start.
To remove and recreate this directory:
# vxconfigd k x cleartempdir
Troubleshooting: Debugging with vxconfigd

Running vxconfigd in debug mode:
# vxconfigd k m enable x debug_level
* debug_level = 0 No debugging (default)
* debug_level = 9 Highest debug level
Some debugging options:
* -x log
Logs all console output to the /var/vxvm/vxconfigd.log file
* -x logfile=name
Use the specified log file instead
* -x syslog
Direct all console output through the syslog() interface
* -x timestamp
Attach a timestamp to all messages
* -x tracefile=name
23
Log all possible tracing information in the given file.
Troubleshooting: Invalid or Missing /etc/system File

The /etc/system file is used in the kernel initialization and /sbin/init phases of the
boot process.
This file is a standard Sun system file to which VxVM add entries to:
* Specify drivers to be loaded
* Specify root encapsulation
If the file or these entries are missing, you encounter problems in the boot process.
When booting from an alternate system file, do not go past the maint mode. Boot
up on the alternate system file, fix the VxVM problem, and then reboot with the
original system file.
ok> boot a
When prompted specify the /etc/hosts file for the system file. You will get many
errors but youll get far enough in order to fix the original system file.
Temporarily Importing the Boot Diskgroup

Through a temporary import, you can bring the boot diskgroup to a working
system and repair it there:
1) Obtain the diskgroup ID (dgid) of the boot diskgroup:
# vxdisk s list
2) On the importing host, import and temporarily rename the diskgroup:
# vxdg tC n tmpdg import dgid
3) Fix and replace the files and volumes as necessary.
4) Deport the diskgroup back to the original host:
# vxdg h orig_hostname deport tmpdg
24
Conclusion
This document provided an introduction to important Veritas Volume Manager
recovery techniques and discussed ways to ensure recoverability after a
disaster. When dealing with difficult Veritas situations, it is imperative to know
what function a command performs before executing it. It is also helpful to
engage Veritas support or the folks on the Veritas mailing lists when major
disasters occur. This will ensure that a second set of eyes is available to review
the problem, and will ensure the quickest and safest methods are used to
recover a faulted system. I tested all commands on a Solaris 10 server running
Veritas Volume Manager 4.1, and I highly recommend testing all configuration
changes on non-production systems.
25

VXVM Recovery

Diunggah oleh

Informasi Dokumen

Judul Asli

Hak Cipta

Format Tersedia

Bagikan dokumen Ini

Bagikan atau Tanam Dokumen

Opsi Berbagi

Apakah menurut Anda dokumen ini bermanfaat?

Apakah konten ini tidak pantas?

Hak Cipta:

Format Tersedia

VXVM Recovery

Diunggah oleh

Hak Cipta:

Format Tersedia

Veritas Volume Manager

(Recovery techniques and Troubleshooting)

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Veritas Volume Manager (Recovery techniques and Troubleshooting)

This document will provide an introduction to important and often overlooked

Veritas Disk Headers

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Veritas Configuration Database

bash-3.00# vxdg list datadg | egrep "config disk.*clean online"

ENABLED ACTIVE 20971520 SELECT

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Installing volume manager disk header for c1t1d0s2 ...

$ /usr/lib/vxvm/bin/vxconfigrestore -l /etc/dgbackups/ -c oradg

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Case Study #1: Recovering from a Power Failure

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Case Study #2: Recovering a Deleted Disk group (Volumes)

Installing volume manager disk header for c1t1d0s2 ...

oradg's diskgroup configuration is restored (in precommitted state).

Veritas Volume Manager (Recovery techniques and Troubleshooting)

vxconfigrestore -l /etc/dgbackups/oradg_0612205 -d oradg ==> to abort the

PRIVLEN PUBLEN STATE

RV NAME RLINK_CNT KSTATE STATE

PRIMARY DATAVOLS SRL

V NAME RVG/VSET/CO KSTATE STATE

LENGTH READPOL PREFPLEX UTYPE

DISKOFFS LENGTH [COL/]OFF DEVICE MODE

VOLNAME NVOLLAYR LENGTH [COL/]OFF AM/NM

CACHE DISKOFFS LENGTH [COL/]OFF DEVICE MODE

DC NAME PARENTVOL LOGVOL

Veritas Volume Manager (Recovery techniques and Troubleshooting)

v oravol01 - ENABLED ACTIVE 41943040 SELECT

ENABLED ACTIVE 41943168 STRIPE 3/128 RW

sv oravol01-S01 oravol01-03 oravol01-L01 1 13981056 0/0

sv oravol01-S02 oravol01-03 oravol01-L02 1 13981056 1/0

sv oravol01-S03 oravol01-03 oravol01-L03 1 13981056 2/0

ENABLED SYNC 13981056 SELECT -

pl oravol01-P05 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT sd c1t3d0-02

pl oravol01-P06 oravol01-L03 ENABLED ACTIVE 13981056 CONCAT sd c1t6d0-02

Veritas Volume Manager (Recovery techniques and Troubleshooting)

These steps are detailed below:

To determine the version of VRTSexplorer that is on the host, along with

To run the VRTSexplorer utility to exclude a particular product, such as

##### Saving Detailed information for all VERITAS

Veritas Volume Manager (Recovery techniques and Troubleshooting)

3. Send the file created above to Symantec Technical Support:

authority, are subject to having all of their activities on this

220- system is a violation of State and Federal law. If monitoring

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Troubleshooting: The Boot Device Cannot be Opened

Troubleshooting: Startup Scripts Exit Without Initialization

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Troubleshooting: License Problems (keys corrupted, missing, or expired)

Troubleshooting: Missing /var/vxvm/tempdb (missing, misnamed, or corrupted)

Troubleshooting: Debugging with vxconfigd

Log all possible tracing information in the given file.

Troubleshooting: Invalid or Missing /etc/system File

Temporarily Importing the Boot Diskgroup

Veritas Volume Manager (Recovery techniques and Troubleshooting)

Veritas Volume Manager (Recovery techniques and Troubleshooting)