.vmdk Snapshots and the Importance of CID Chains

Introduction

I spent half of yesterday troubleshooting a problem that arose from an un-synchronized VMware VI3 snapshot. It was an unpleasant experience, but I learned about the importance of CID chains!

After several hours of poking and prodding a dead VM, my colleague Dan and I finally stumbled across CID chains.

We came to the conclusion that snapshots, and specifically the delta .vmdk files VI3 creates each time it generates a new snapshot, are connected to each other in a chain via randomly assigned dynamic CID values. VI3 assigns each new snapshot file a CID, and that value changes each time the VM reboots.

If the CID chain is broken for any reason, the VM cannot mount the virtual disks assigned to it.

Problem

Your VM’s virtual disks will not mount and you receive the following error message:

Cannot open the disk ‘/vmfs/volumes/INSERT SPECIFIC VALUE HERE.vmdk’ or one of the snapshot disks it depends on.
Reason: The parent virtual disk has been modified since the child was created.

Yikes!

What do you do?!

1. Don’t panic (yet).
2. Shut down the VM completely, ASAP.
3. Don’t change anything!
4. Read this article!

You should be able to correct the problem, assuming you haven’t made any changes to the .vmdk files.

Analysis

This problem likely arose because you made one or more snapshots of one or more of the virtual disks associated with a VM, and the snapshots have become unsynchronized; you are probably also running in an ESX/VI3 environment.

The root cause of the problem relates to how VI3 manages snapshots and snapshot delta changes to the original virtual disk (.vmdk). Most likely, the snapshot delta hierarchy has become unsynchronized and the various child snapshot .vmdk files associated with the original .vmdk are no longer referencing the correct parent .vmdk.

The key to piecing the snapshot delta files back together is in the CID chain. The CID chain is the glue behind the snapshot delta hierachy.

Each .vmdk file in the chain is assigned a CID value. The .vmdk also points to a parent CID value and parent .vmdk file. The parent CID value must point to the parent .vmdk file created immediately before the child .vmdk.

Each .vmdk file's CID is dynamically changed to a random hex value every time the VM boots. If a VM boots and the CID chain is not in the same state as per the previous boot process, the child CID-->parent CID relationships become unsynchronized. This is how VI3 determines authenticity of snapshot delta files.

The only way to re-synchronize a CID chain is to edit the .vmdk files in each chain manually.

Example CID chain:

1. Parent .vmdk (original .vmdk) = vdisk.vmdk
2. Child .vmdk created after 1st snapshot = vdisk-000001.vmdk
3. Child .vmdk created after 2nd snapshot = vdisk-000002.vmdk

If this chain is broken:

1. manually point vdisk-000002.vmdk to vdisk-000001.vmdk, and
2. manually point vdisk-000001.vmdk to vdisk.vmdk.

Three fields deserve attention in each .vmdk file, for the purposes of this discussion:

1. The CID field,
2. The parentCID field, and
3. The parentNameHint field.

Note: The parent .vmdk does not contain a parentNameHint field and its parentCID field always equals "ffffffff".

Samples

Sample output of parent vdisk.vmdk file:

[root@myvi3server]# cat vdisk.vmdk
# Disk DescriptorFile
version=1
CID=7f81b951
parentCID=ffffffff
createType="vmfs"
# Extent description
RW 50331648 VMFS "vdisk-flat.vmdk"
# The Disk Data Base
#DDB
ddb.virtualHWVersion = "4"
ddb.geometry.cylinders = "3133"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.adapterType = "lsilogic"
ddb.toolsVersion = "7202"

Sample output of child vdisk-000001.vmdk file:

[root@myvi3server]# cat vdisk-000001.vmdk
# Disk DescriptorFile
version=1
CID=8eb633b8
parentCID=7f81b951
createType="vmfsSparse"
parentFileNameHint="/vmfs/volumes/478b9802-ce7ed955-96a4-0015c5fd9308/servername/vdisk.vmdk"
# Extent description
RW 50331648 VMFSSPARSE "vdisk-000001-delta.vmdk"
# The Disk Data Base
#DDB
ddb.toolsVersion = "7202"

Note: The file path in the parentNameHint field in this sample is pointing to an iSCSI SAN volume. The long hex guid number is a directory pointer.

Sample output of child vdisk-000002.vmdk file:

[root@myvi3server]# cat vdisk-000002.vmdk
# Disk DescriptorFile
version=1
CID=249e6aff
parentCID=8eb633b8
createType="vmfsSparse"
parentFileNameHint="sryulris0cogp01-000001.vmdk"
# Extent description
RW 50331648 VMFSSPARSE "vdisk-000002-delta.vmdk"
# The Disk Data Base
#DDB
ddb.toolsVersion = "7202"

Note: The file path in the parentNameHint field is pointing to a file that resides in the same local directory as vdisk-000002.vmdk. If the file is located in a different directory, the path must indicate this accordingly.

Solution

The following steps synchronize the snapshot delta files so that VI3 recognizes their relationship with each other, as well as their authenticity:

Note: Perform the following steps in a text editor. Backup the original files prior to making any changes.

Edit vdisk-000002.vmdk:

1. Take note of the CID value in vdisk-000001.vmdk (=8eb633b8)
2. Take note of the file path of vdisk-000001.vmdk (=local directory)
3. Point the parentCID field to the CID value of vdisk-000001.vmdk
   --> parentCID=8eb633b8
4. Point the parentNameHint field to the vdisk-000001.vmdk file
   --> parentFileNameHint="sryulris0cogp01-000001.vmdk"

Edit vdisk-000001.vmdk:

1. Take note of the CID value in vdisk.vmdk (=7f81b951)
2. Take note of the file path of vdisk.vmdk (="/vmfs/volumes/478b9802-ce7ed955-96a4-0015c5fd9308/servername/)
3. Point the parentCID value to the CID of vdisk.vmdk
   --> parentCID=7f81b951
4. Point the parentNameHint field to the vdisk.vmdk file
   --> parentFileNameHint="/vmfs/volumes/478b9802-ce7ed955-96a4-0015c5fd9308/servername/vdisk.vmdk"

If the virtual disks assigned to the VM were not changed or altered in any way, the solution ends here. If the virtual disks were altered, or removed and then re-added, manually edit the appropriate virtual SCSI controller’s settings in the VM’s .vmx configuration file to point to the last child .vmdk in the snapshot chain.

Note: Perform the following steps in a text editor. Backup the original file prior to making any changes.

Edit servername.vmx:

1. Locate the SCSI controller associated with the virtual disk that must be repaired. The first virtual disk will normally be assiged to virtual SCSI controller scsi0:0, the second virtual disk will normally be assigned to virtual SCSI controller scsi0:1, and the third virtual disk will normally be assigned to virtual SCSI controller scsi0:2 etc…
2. Assuming that the second virtual disk must be repaired, locate the section of the file that references scsi0:1.
3. Point the following field to the correct location of the last child .vmdk in the snapshot chain:
   scsi0:1.fileName="/vmfs/volumes/46fd62d479749c9697e30015c5fd9308/servername/vdisk-000002.vmdk - Note: The file path may vary, though the snapshots are stored in the same directory as the parent .vmdk (vdisk.vmdk) by default. Despite this, enter the complete path to the child .vmdk (vdisk-000002.vmdk), including any iSCSI volume guid’s etc...
4. Save the file and restart the VM.
5. If the VM does not boot, or if it complains that "The parent virtual disk has been modified since the child was created", verify the CID chain hierarchy and file paths, then retry. If the VM still does not boot, and the CID chains and file paths are all correct, the parent or child .vmdk files were probably altered or changed in an unrecoverable way, or they may not be the correct files!