AIX - Logical Volume Manager

----------------------------------------------------------------------
Table of Contents
What is LVM Really?
High Level Commands
Intermediate Commands
Library Calls - API
LVM Device Driver
Disk Device Driver
SCSI Device Driver
Terms
Physical Volumes
PVIDs
VGDA
VGSA
Volume Groups
VGDA expansion
Case A, VGDA expansion
Case B, VGDA expansion
Case C, VGDA Expansion
Logical Volumes
Mirrors
What Does LVM Give You?
Flexibility
Data Integrity
Write Verify
Speed
exportvg
importvg
reducevg
varyonvg
Problem Determination
Problem Recovery
Failure/Recovery Examples
Example1: Resynchronize Good PV Gone Offline
Example2: Replace Failing Physical Volume
Example3: Bad Magic Number in Superblock
Example4: Fix LVM/ODM Inconsistency on non-rootvg VG
Example5: Fix LVM/ODM Inconsistency on rootvg
Example6: Phantom ODM entries
Summary
APPENDIX: LVM Data Recovery Example
----------------------------------------------------------------------
What is LVM Really?
• High Level Commands
 • Intermediate Commands
• Library Commands
• LVM Device Driver
• Disk Device Driver
• SCSI Device Driver
• Actual Disk
High Level Commands
• High Level Commands
• Commands used by SMIT
• Commands used by users in command line or shell scripts
• High levels of error checking
Notes: High Level Commands
Most of the LVM commands reside in /usr/sbin. There are exceptions such as the library commands which
reside in /usr/lib/liblvm.a and /usr/lib/libsm.a and /etc/cfgvg. The High Level commands are a
combination of shell scripts and C binaries:
chlv , chpv, chvg, cplv, exporting, extendlv**, extendvg, importvg, lslv*,
lsvg*, lsvgfs*, migratepv, mklv, mkvg, reducevg, reorgvg, rmlv, rmlvcopy,
syncvg, synclvodm, varyonvg, varyonvg*
• indicates that this command is a binary
• ** this command also is used for the mklvcopy command
These commands are those that SMIT and experienced users are meant to execute directly.
Intermediate Commands
• Commands used by High Level commands
• Commands provide usage statements - unfortunately
• Some level of error checking still provided, no documentation on output
Notes: Intermediate Commands
The Intermediate commands are all C binaries:
getlvcb, getlvname, getlvodm, getvgname, lvgenmajor, lvgenminor, lvrelmajor, lvrelminor, putlvcb,
putlvodm, lchangelv, lcreatelv, ldeletelv, lextendlv, lquerylv, lreducelv, lresynclv, lchangepv, ldeletepv,
linstallpv, lquerypv, lresyncpv, lcreatevg, lqueryvg, lqueryvgs, lvaryonvg, lvaryoffvg, lresynclp,
lmigratepp.
The commands are called by the High Level command shell scripts. These commands unfortunately give
usage statements when queried from the command line, which has given users the incorrect impression
that these commands are for general use in LVM. This is not true. These commands have been "tuned" for
use by the High Level shell scripts. Because it is known that only high level shell scripts will be calling
these intermediate commands, certain error checking is not performed by the intermediate commands.
The High Level commands are responsible for screening potentially dangerous errors or conditions out.
These binaries also contain some functionality that are not used and even some bugs that have not been
fixed because the function isn't used!
Library Calls - API
• Library calls to be used as Application Program Interface
• Error checking provided
• Calls documented in Infoexplorer or books
• Very rarely used
Notes: Library Calls - API
The LVM library calls are part of the LVM Applications Programmer Interface (API) which lets
programmers access the LVM layer directly through the library layer provided by /usr/lib/liblvm.a. In our
experience, very few people use this API because of the complexity of issues within LVM. Users mostly
rely on shell scripts that call the High Level commands. The API library functions begin with lvm_ and
are documented within Infoexplorer. These commands are only called in situations where the user is
writing C code and does not want to access the High Level shell scripts from within a C program. Or, they
require a faster response from LVM than they can achieve with shell scripts. Writing an LVM specific
program using the API certainly does give a performance improvement to some applications. Note, the
LVM library API is not a thread-safe library. Thus, user programs written for threads that reference the
LVM API may not work correctly.
LVM Device Driver
• Access same as those for typical IBM device driver
• Checking provided, but assumes user has basic device driver knowledge
• Rarely used. If used wrong, disastrous results!
Notes: LVM Device Driver
The LVM device driver comes in two portions, the pinned portion and the non-pinned portion. The pinned
portion is called /usr/lib/drivers/hd_pin and /usr/lib/drivers/hd_pin_bot. Before AIX 4.1, the driver was
just called hd_pin and the entire driver was pinned into memory (not pageable). With AIX 4.1, the driver's
true non-pageable portion is in hd_pin_bot and the hd_pin is now pageable. The LVM device driver is
either called by the jfs filesystem or the lvm library routines. When a request is received by the LVM
device driver, it calls the disk device driver.
Disk Device Driver
• Access same as those for typical IBM device driver
• Sometimes other vendors use the disk device driver as pseudo driver for their products
Notes: Disk Device Driver
There are various disk device drivers in AIX. The most common disk device driver is the one for scsi
device drivers, /usr/lib/drivers/scdisk and /usr/lib/drivers/scdiskpin. The second most common disk device
driver is probably the one for the 9333 serial dasd device. This is found in the binaries /usr/lib/drivers/sd
and /usr/lib/drivers/sdpin. In both device drivers, the portion called "pin" is that which is pinned into
memory and cannot be paged out of memory. When a request from the LVM device driver is received by
the disk device driver, the request is packaged and then transferred down to the scsi device driver.
SCSI Device Driver
• Access same as those for typical IBM device driver
• Very useful to access devices directly through this interface, although
• people prefer not to use this access point
Notes: SCSI Device Driver
There are several scsi device drivers for AIX and they have the common name
string of "scsi" residing as part of their name. The most common of them is
the original SCSI-1 device driver /usr/lib/drivers/hscsidd. The scsi device
driver takes a command from a scsi device such as tape, disk, scanner,
etc...and processes it to be sent to the scsi device connected onto the scsi
bus. The scsi device is device neutral, it does not know or care which
device sent it a command or even what that command is. It treats all
requests the same and puts them into the same type of scsi command packaging
required by all scsi devices.
[Image]
Terms
• Physical Volumes
• Volume Groups
• Logical Volumes
• Mirror
Notes: Terms
[Image]
On one system, many disks can make up a volume group. Disks cannot be shared
between volume groups. The entire disk is dedicated to being part of one
volume group.
[Image]
Logical volumes reside only on volume groups. A logical volume can span
multiple disks within a volume group (but cannot span multiple volume
groups): lv00. Or, a logical volume can just reside on one disk: biglv.
Finally, a logical volume can have multiple copies of itself: mirrlv.
Physical Volumes
* Disk Independence
o Physical Type of disk is transparent to the LVM.
* Portability of volume groups
o Volume groups can be easily imported into another system, the disks under AIX are portable
* Limit of 1016 physical partitions per disk maximum
o tracked by VGSA
* Verify the true ID of a disk
o each disk has a PVID (Physical Volume ID) - or soft serial number
o can be accidentally destroyed with a dd command, or intentionally with disk formatting
Notes: Physical Volumes
The PVID of the disk is stored in ODM, use "odmget -q "name=hdisk<#>" CuAt" to get it.
Also can check the PVID on teh disk with "lquerypv -h /dev/hdisk<#>", this command will print out in
HEX format the first 128 byest from the beginning of the disk platter. The PVID is held in the first two
columns of hex row 0x80.
PVIDs
• PVID's and how they configure
o New disk doesn't have a PVID, until used in a Volume Group
o PVID is a combination of the machine's serial number and the date when the PVID is
generated.
• It is not recommended to use the "dd" command to copy the entire contents of one disk to another
o this will also copy the PVID, and each disk should have a unique PVID or the system
could become confused.
VGDA
• Volume Group Descriptor Area - VGDA
o at the front of each disk
o contains info about the volume group, the logical volumes that reside on the volume group,
and the disks that make up the volume group.
 o VGDA also used in quorum voting.
Notes: VGDA
Maximum of 32 disks per volume group.
The VGDA can be thought of like a suitcase of information, it can be packed up and unloaded at another
AIX system, and all the information about that volume group is available to the new system.
VGSA
• Volume Group Status Area - VGSA
o comprised of 127 bytes, where each bit represents one of the1016 Physical Partitions (PP)
that reside on each disk.
o Used in Mirroring as a bit-mask to determine which PPs have become stale.

Volume Groups
• LVM Requires Quorum of disks in the volume group
o Sanity check of VGDA to prevent data corruption.
o 51% or more votes must be available to a volume group before LVM can continue.
• Volume Group Identifier - VGID
o VGID is a soft serial number for the volume group, similar to PVID for the physical
volume.
Notes: Volume Groups
For a single disk, there are 2 VGDAs on disk, with a quorum of 2.
For a second disk, disk 1 retains 2 VGDAs and disk 2 gets 1 VGDA, for a quorum of 3.
Adding a third disk results in the extra VGDA from the first disk moving to the third disk, for a quorum of
3 with each disk having one vote.
Adding each additional disk adds one new VGDA per disk.
VGDA expansion
* The following pictures are a graphical view of VGDA expansion.
* Shows how the VGDA grows from one disk to another
o shows how the information is carried in the VGDA which cross-references information among the disks
in a volume group.
Case A, VGDA expansion
[Image]
Notes: Case A
In these examples, the large box in the upper left is the VGDA. You can see how the VGDA is used to
monitor disks and logical volumes that belong to a volume group. In Case A, the volume group,
samplevg, contains only hdisk0. The example also shows how the VGDA holds the map for the logical
volume lv00. There is no VGDA represented on hdisk1 since it is not part of the samplevg. However, it
may be the case that hdisk1 contains some long-forgotten VGDA from a previous volume group. But
since the VGDA for samplevg has no record of hdisk1 belonging to the volume group, hdisk1 is ignored
in samplevg's eyes.
Case B, VGDA expansion
[Image]
Notes: Case B
Case B is an example of the aftermath of an extendvg of samplevg into hdisk1. A copy of the VGDA from
hdisk0 is placed onto hdisk1. But, notice that the VGDA has also been updated on hdisk0 to reflect the
inclusion of a new disk into the volume group. Although lv00 does not exist on hdisk1, the VGDA on
hdisk1 will have a record of the lv00 mapping sitting on another disk since it's part of the same volume
group.
Case C, VGDA Expansion
[Image]
Notes: Case C, VGDA
Case C shows the result of the creation of two logical volumes, lv01 and lv02, onto hdisk1. This is to
show how the VGDA sitting on hdisk0 is updated with the activity that is being conducted on another disk
in the volume group.
Logical Volumes
• Members of specific volume groups
o cannot be spanned from one volume group to another.
• Can control location of logical volumes being created
o control which disk or partition where logical volume is placed
• Can view location of logical volumes
o lspv -p physical_volume
• Non-typical logical volumes o log logical volume used by JFS
o dump device logical volume
o boot logical volume
o paging device logical volume
• Logical Volume Control Block - lvcb
o consists of the first 512 bytes of a logical volume
o holds important information
 + creation date of logical volume
 + information about mirrored copies
 + possible mount points into a JFS
• Logical Volume Identifier - LVID
o Soft serial number used to represent the logical volume to the LVM libraries and low level
commands.
o Made up of VGID.<num>, where num is the order in which it was created in the volume
group.
Mirrors
• Sequential vs. Parallel
• Sequential Mirroring
+ Sequential mirroring based on concept of an order within mirrors.
+ All read-write requests go through a PRIMARY copy, and if the request was a write, it is propagated to
the secondary copies.
+ Write request is not complete until the secondary copies have finished the write request.
• Parallel Mirroring
+ A read request will be serviced by the first available copy.
+ A write request will be broadcast to the copies and finishes when all are done.
• What good is Sequential Mirroring?
o There is a definite PRIMARY copy which will be the source mirror copy for mirror
resync.
• Parallel Mirroring is the most commonly used type.
 • Staleness of Mirrors
o If a copy has become unavailable, it is referred to as "stale".
o Tracked in the VGSA in terms of physical partitions.
Notes: Mirrors
Sequential Mirroring
[Image]
Parallel Mirroring
[Image]
What Does LVM Give You?
* Flexibility
* Data Integrity
* Speed
Flexibility
* Real-time Volume Group and Logical Volume expansion/deletion
* Use of Logical Volume under filesystem
o LVM provides a physical to logical mapping layer, filesystem is defined on top of LVM.
* Use of Logical Volume as raw data storage
o It is not required to have a JFS on a logical volume.
* User customized logical volumes
* Ability to customize data integrity check
Notes: Flexibility
User can change mirror write consistency check, create mirroring and change the requirements for a
quorum in a volume group.
User can create logical volumes using a map file which allows them to specify the exact disks the logical
volume will inhabit, and the order within that disk of the logical volume.
Data Integrity
* Mirrors
o AIX allows up to three copies of a logical volume in sequential or parallel organization.
* Mirror Write Consistency Check - MWCC
o Method of tracking the last 62 writes to a mirrored LV.
o If crash occurs, upon reboot the last 62 writes to mirrors are examined and one of the mirrors is used as
a source to synchronize the mirrors.
+ For sequential mirroring, the latest write that occurred is guaranteed to be completed, since there is a
definite PRIMARY disk which is used as the source for synchronization.
+ For parallel mirroring, all mirrors are guaranteed to be consistent, but the latest write is not guaranteed
to be completed after the crash.
* Ability to detect staleness and to correct
o Stale PPs are tracked in VGSA.
o No further writes are sent to stale PP, saving time.
o When problem is corrected, partitions can be updated from other mirrors.
* Quorum checking
o LVM requires a quorum before working in a Volume Group.
* Data relocation - due to physical problems with disk
o Type 1: Internal to disk
o Type 2: hardware relocate ordered by LVM
o Type 3: Software relocate
Notes: Data Integrity
There is a command used to back up mirrored logical volumes which essentially allows you to back up a
mirrored logical volume and freeze the copy so that changes are not occuring while the backup is going
on. This command is called splitlvcopy
There are three types of data relocation: internal to the disk, hardware relocate ordered by LVM, and
software relocation. Relocation typically occurs when the system fails to perform a read or a write due to
physical problems with the disk platter. In some cases, the data i/o request completes, but with warnings.
Depending on the type of recovered error, we may be so wary of the success of the next request to that
physical location that LVM orders a relocation to be on the safe side. The lowest logical layer of
relocation is the one that is internal to the disk. These types of relocations are typical private to the disk
and there is no notification to the user that a relocation occurred. The next level up in terms of relocation
complexity is a hardware relocation called for by the LVM device driver. This type of relocation will
instruct the disk to relocate the data on one physical partition to another portion (reserved) of the disk.
The disk takes the data in physical location A and copies it to a reserved portion of the disk, location B.
However, after this is complete the LVM device driver will continue to reference physical location A, with
the understanding that the disk itself will handle the true i/o to the real location B. The top layer of data
relocation is the "soft" relocation handled by the LVM device driver. In this case, the LVM device driver
maintains a bad block directory and whenever it receives a request to access a logical location A, the LVM
device driver will look up the bad block table and translate it to actually send the request to the disk drive
at physical
location B.
Write Verify
* Every write to a physical portion of a disk that is part of a logical volume causes the disk device driver
to issue the Write and Verify scsi command.
* After each write, the disk will re-read the data to check that what was written was correct.
* This obviously can have a performance impact.
Speed
* Mirrors
o Main objective of Mirroring is Reliability of Data.
o Parallel disk mirroring can improve the read performance of the system, but at a cost to the write
performance.
* Striping
o Main objective of Striping is speed improvements for sequential access.
o Striping is the concept of spreading sequential data across more than one disk to improve disk I/O.
Notes: Speed
[Image]
We present an example to show the benefit of striping: A piece of data that is stored of the disk is 100
bytes. The physical cache of the system is only 25 bytes. Thus, it takes 4 read requests to the same disk to
complete the reading of 100 bytes:
As you can see, since the data is on the same disk, four sequential reads must be required. If this logical
volume were created with a stripe width of 4 (how many disks) and a stripe size of 25 (how many
consecutive bytes before going to the next disk), then you would see:
[Image]
As you can see, each disk only requires one read request and the time to gather all 100 bytes has been
reduced 4-fold. However, there is still a bottleneck of having the four independent data disks channel
through one adapater card. But, this can be remedied with the expensive option of having each disk on an
independent adapter card. Note the effect of using striping:
the user has now lost the usage of 3 disks that could have been used for other volume groups.
exportvg
* The exportvg command is used to remove all knowledge of a volume group from a system.
* exportvg deletes information about a volume group from ODM.
* exportvg also cleans up some special files related to the volume group from the system.
* Exportvg command does not affect the structures on disk, such as VGDA.
* Before the disks associated with a volume group are removed, exportvg should be performed.
Notes: exportvg
Exportvg first makes sure the volume group is varied off. Then, it deletes /etc/filesystems entries
associating logical volumes and filesystems. The /dev/special files are removed, and then ODM entries
associated with the volume group are deleted from ODM.
No actoin is taken on the physical disk platter data structures, such as VGDA.
importvg
* used to bring a set of disks composing a volume group into a new system.
* importvg reads information from the on-disk VGDA to understand the new volume group
configuration.
Notes: importvg
AIX makes moving disks composing volume groups from one system to another easy. importvg is used to
bring a set of disks composing a volume group into a new system.
importvg allows the system to learn which disks constitute the data disks, what logical volumes exist on
the data disks, what mount points exist for logical volumes, PVIDs for the disks, LVIDs for the logical
volumes, and preferred logical volume names.
The system will prevent name collisions between LVM entries in the new and existing system, and LVM
may rename the logical volume without user approval to avoid name collisions.
reducevg
* reducevg is used to remove a disk from a volume group.
* reducevg is totally different from exportvg.
* reducevg modifies both ODM and the on disk VGDA.
Notes: reducevg
Reducevg removes information about the disk and any logical volumes residing on that disk. This
includes logical volumes that may straddle that disk and other disks. If any part of the logical volume is
on the disk to be removed from the volume group, the logical volume is removed. reducevg removes
information about the disk from the VGDA on the remaining disks in the volume group, as well as the
target disk of the reducevg. If you wish to later import a volume group into a new system, use exportvg to
remove information from the original system. Using reducevg on all the disks in the volume group is not
equivalent to using exportvg. If you use reducevg on all the disks in the volume group, and then try to
import that volume group into a new system, all the VGDA information needed by the new system will
have been removed by the reducevg.
varyonvg
* varyonvg activates a volume group & logical volumes.
* checks for a quorum of physical volumes.
* checks for discrepancy between on-disk structures and ODM.
Notes: varyonvg
varyonvg can be run on an already varied-on volume group.
Use with the -b flag (4.2 and later) to break a lock on disks locked as a result of some other command.
There exists an ODM object known as the vg-lock. Whenever an LVM
modification command is started, the LVM command will lock the vg-lock for the volume group. If for
some reason the lock is inadvertently left behind, the volume group can be unlocked by running the
varyonvg -b command.
Problem Determination
* Questions to ask...
* What is the root cause of the error?
* Can this problem be distilled to the simplest case?
o Try to find a simple test case to cause the problem.
* What has been lost and what is left behind?
* Is this situation repairable?
o Sometimes may not be.
Problem Recovery
* Warn the user of the consequences
o Sometimes failed recovery attempts can leave the system worse off than before.
* Gather all possible data
o get info from the VGDA, ODM and errpt.
* Save off what can be saved
o Save any files that have not been corrupted, such as ODM database files in /etc/objrepos. Save any
other important files.
* Each case is different, so must be the solution
Failure/Recovery Examples
* Example1: Resynchronize failing physical volume
* Example2: Replace failing physical volume
* Example3: Bad magic number in superblock
* Example4: Fix LVM/ODM inconsistency on non-rootvg volume group
* Example5: Fix LVM/ODM inconsistency on rootvg volume group
* Example6: Fix Phantom ODM entries
Example1: Resynchronize Good PV Gone Offline
* Use these steps to reintroduce a physical volume
o went offline for some reason without loss of quorum (if mirrored)
or data integrity. The cause of problem is a bad cable, etc.
* The data on the physical volume must be intact and the disk itself must not be damaged.
o If data on the physical volume is not intact, use the next recovery procedure.
Example 1 Steps to Repair
* Remove the device from the system configuration in the ODM Device Configuration Database.
rmdev -l PVname
* Repair the problem.
(loose or bad cable, unseated adapter etc.)
* Add the disk back into the ODM Device Configuration Database.
mkdev -l PVname
* Synchronize the stale physical partitions. This synchronizes logical volume copies that are not current.
syncvg -p PVname
Example 1 assumes that the logical volumes in the volume group were mirrored.
Example2: Replace Failing Physical Volume
 * Use the following steps to remove a failing physical volume from the system and replace the physical
volume.
Example 2 Steps
* Back up all single-copy logical volumes on the failing device, if possible.
* Unmount all single copy filesystems from the disk. Mirrored filesystems do not have to be unmounted.
unmount /DIRECTORY1
* Remove all single copy filesystems from the failed physical volume.
rmfs /DIRECTORY1
* Remove remaining physical page copies from the physical volume. rmlvcopy PVname
#REMAINING_PPs
* Remove the physical volume from the volume group reducevg -df VGname PVname
* Remove the device from the Device Configuration Database in ODM. rmdev -l PVname
* Remove the device and replace it with a working device.
Configure the new device into the system using smit or cfgmgr.
* Make the device available.
chdev -l PVname -a pv=yes
* Add the new physical volume to the volume group.
extendvg VGname PVname
* Remake the logical volumes on the physical volume.
mklv -OPTIONS VGname #PPs PVname
* Recreate the filesystems on the logical volume.
crfs -v jfs -d LVname -m /DIRECTORY1
* Extend multiple-copy logical volumes onto the disk again.
mklvcopy LVname #LPCOPIES PVname
* Resync copied physical partitions to the new physical volume.
syncvg -p PVname
* Restore single copy filesystem data from backup media to logical volumes using utilities such as tar.
Notes: Example 2
How do you tell if it is a single copy filesystem (i.e. not mirrored)?
Single copy filesystems have the same amount of logical partitions as physical partitions on the output
from the lspv command.
Pages that remain on the physical volume after rmfs are mirrored on other physical volumes.
Example3: Bad Magic Number in Superblock
* The primary superblock of a filesystem can be corrupted.
* Most damage to the superblock is catastrophic.
* One simple and correctable problem is a bad magic number.
o Generally this problem shows up when using fsck on a filesystem.
The fsck command returns fsck: Not an AIXV4 filesystem.
Example 3 Steps to Repair
1. Determine the logical volume associated with the failing filesystem.
2. Display the primary superblock using the od utility.
3. Determine what data is corrupted in the primary superblock by comparing the data with the superblock
structure found in /usr/include/jfs/filsys.h. If the magic number is corrupted, the value will be something
other than the contents of kernel variable fsv3magic. Currently, this value is 0x43218765.
4. Display secondary superblock using the od utility (see next page for example).
5. Determine what data is corrupted in the secondary superblock by comparing the data with the
superblock structure found in /usr/include/jfs/filsys.h.
6. If the magic number is correct (0x43218765 for an AIX V3 filesystem or 0x65872143 for an AIX V4
filesystem), the secondary superblock is probably correct. If the secondary superblock is intact, use the dd
command to copy the secondary superblock to the primary superblock.
* Step 2: Display primary superblock using od.
Note the bad magic value at byte 0x1000.
$ od -x -N 64 /dev/lv00 +0x1000
0001000 1234 0234 0000 0000 0000 4000 0000 000a
0001010 0001 8000 1000 0000 2f6c 7633 0000 6c76
0001020 3300 0000 000a 0003 0100 0000 2f28 0383
0001030 0000 0001 0000 0200 0000 2000 0000 0000
0001040
* Step 4: Display secondary superblock. Note correct magic value 0x65872143 at 0x1f000.
$ od -x -N 64 /dev/lv00 +0x1f000
001f000 6587 2143 0000 0000 0000 4000 0000 000a
001f010 0001 8000 1000 0000 2f6c 7633 0000 6c76
001f020 3300 0000 000a 0003 0100 0000 2f28 0383
001f030 0000 0001 0000 0200 0000 2000 0000 0000
001f040
You must be superuser to do this recovery.
The example output assumes that lv00 is the logical volume containing the filesystem with the bad magic
number and that it is in rootvg. If the filesystem is an AIX Version 4 filesystem (i.e., created with non-
default values of fragment size and nbpi or compressed), the magic number will be the value of
fsv3pmagic in /usr/include/jfs/filsys.h and that is 0x65872143.
The s_time field of the secondary superblock can be checked to determine the last time the secondary
superblock was updated. This time is given in seconds since epoch.
Note: The hdf utility has been removed in AIX V4. However, the same functionality can be got by using
the -h flag with lquerypv as - "lquerypv -h /dev/lv00 0x1000 60" or the hdf executable from an AIX V3
system can be
run on an AIX V4 system as "hdf /dev/lv00 0x1000 60". These are alternatives
to using the od command as shown on the foil.
* Step 6: Copy the secondary superblock to the primary superblock.
dd count=1 bs=4k skip=31 seek=1 if=/dev/lv00 of=/dev/lv00
dd: 1+0 records in.
dd: 1+0 records out.
* Step 7: Use logform to clear out the log for the filesystem.
/usr/sbin/logform /dev/hd8
* Step 8: Unmount the filesystem.
unmount /dev/lv00
* Step 9: Use fsck to clean up inconsistent files caused by using
secondary superblock.
fsck /dev/lv00 2>&1 | tee /tmp/fsck.errs
Example4: Fix LVM/ODM Inconsistency on non-rootvg VG
* If data in the ODM becomes inconsistent with data on the physical disk
in the VGDA, VGSA or LVCB, the volume group or portions thereof may be
 made unusable.
* This example damages the ODM data base, shows the symptoms of the
damage, then fixes the damage. The system does not have to be in
maintenance mode to perform these recovery steps.
* This procedure does not work with the rootvg volume group.
Notes: Example 4 LVM/ODM Inconsistency in nonrootvg
Damage to the ODM was done by typing:
$ odmdelete -q "name=testvg" -o CuAt
( Do not actually try to damage the ODM (especially rootvg)).
Then tried:
$ mklv -a ie -y lra3 -x 30 testvg 20
0516-328 putlvodm: Identifier not entered.
Usage:
putlvodm [-a IntraPolicy] [-B label] [-c Copies] [-e InterPolicy] [-L LVid]
[-l LVname] [-n NewLVName] [-r Relocate] [-s Strict]
[-t Type] [-u UpperBound] [-y Copyflag]
[-z Size] LVid
putlvodm [-o Auto-on] [-k] [-K] [-q VGstate]
[-v VGname -m majornum] [-V] VGid
putlvodm [-p VGid] [-P] PVid
0516-822 mklv: Unable to create logical volume.
Example 4 Recovery Steps
* Export the broken volume group to remove entries from the Device
Configuration Database of the ODM
exportvg testvg
* Reassociate the volume group with the physical volumes. In this case
the volume group testvg was located on physical volume hdisk1. Any disk
that was previously in the volume group can be specified.
importvg -y testvg hdisk1
* Vary online the volume group . If data on the disk is intact, the
volume group is restored.
varyonvg -f testvg
Example5: Fix LVM/ODM Inconsistency on rootvg
* These steps recover the rootvg volume group.
* First try using redefinevg while system is running.
o If that doesn't fix the problem must go into maintenance mode.
Notes: Example 5
In order to perform the following repair, the system must be in the RAM Disk
Maintenance Mode .
The steps can also be used to recover other volume groups by substituting
the appropriate physical volume and volume group information. It is
suggested that this example be made a script.
Service boot menu:
----------------------------------------------------------------------------------------
Select One Option:
1. Boot to Diagnostics mode
2. Boot to Single User mode
3. RAM disk maintenance mode
Choice [1-3] :
-----------------------------------------------------------------------------------------
* Delete all of the ODM information about logical volumes.
o Get the list of logical volumes from the VGDA of the physical
volume.
* Delete the volume group information from ODM.
Assume that hdisk0 is part of rootvg. In CuDvDr, value1 = major number,
value2=minor number, value3= object name for major/minor number. rootvg
typically has value1 = 10, due to the order of it's creation.
1. lqueryvg -p hdisk0 -L | awk '{print $2 }' |
while read LVname; do
odmdelete -q "name=$LVname" -o CuAt
odmdelete -q "name=$LVname" -o CuDv
odmdelete -q "value3=$LVname" -o CuDvDr
done
2. odmdelete -q "name=rootvg" -o CuAt
odmdelete -q "parent=rootvg" -o CuDv
odmdelete -q "name=rootvg" -o CuDv
odmdelete -q "name=rootvg" -o CuDep
odmdelete -q "dependency=rootvg" -o CuDep
odmdelete -q "value1=10" -o CuDvDr
odmdelete -q "value3=rootvg" -o CuDvDr
* Add the volume group associated with the physical volume back to the
ODM.
* Recreate the Device Configuration Database in the ODM from information
on the physical volume.
3. importvg -y rootvg hdisk0
4. varyonvg -f rootvg
Example6: Phantom ODM entries
* Problem: a disk is removed from a system, leaving behind phantom ODM
entries. How to clean up?
* Try to do rmlv of the phantom lvs, reducevg on the volume group, and
rmdev of the disk.
* If this doesn't work you must delete all the information about the
volume group from ODM by hand.
Notes: Example 6
The most common problem seen at customer locations is that of an external
disk being yanked off without doing the necessary pre-yanking operations
such as reducevg, rmdev etc. How best can you clean up the system once the
damage has been done? In other words, what operations will purge the ODM of
the entries pertaining to the removed disk, it's volume group, it's logical
volumes and filesystems?
You will see that the lsvg and lspv commands still show you that a volume group (ex. newvg) that used to
exist on the disk is present. However, it is not really there, because the disk has been removed.
If the first commands don't succeed, the only way to completely clean up the ODM is by doing the
following -
1. First, backup the Cu* files just in case you end up deleting more than you should
mkdir /tmp/Cu.bak
cp /etc/objrepos/Cu* /tmp/Cu.bak
2. Search for all instances of the volume group, hdisk#, lv#, loglv# in each of the customized classes.
For example,
odmget CuAt | grep newvg
odmget CuAt | grep hdisk#
will show you the attribute of these objects in this class.
Search in all the customized classes - CuAt, CuDv, CuDep, CuDvDr and CuVPD.
Each of them needs to be cleaned up. Generally CuAt will handle the names,
lvids, pvids and vgids of the volume groups and logical volumes. CuDvDr
controls major/minor number generation. In CuDvDr, value1 = major number,
value2=minor number, value3= object name for major/minor number.
3. Next, delete what you find in 2. as follows -
If the output of 2. says -
name = "newvg"
You can clean this by using
odmdelete -o CuAt -q "name=newvg"
If the output of "odmget CuDvDr | grep hdisk3" says -
value3 = hdisk3
type
odmdelete -o CuDvDr -q "value3=hdisk3"