Hallo Folks,
I'm a software guy more than a hardware guru - so please be gentle. I've been mainly following your tips in these forums.
We had/have several VMs on a dedicated hetzner machine using ESXi (5 ish free version as I remember).
The machines are e.g. a git repo, owncloud and wiki plus a collection of media files etc.
Recently I saw some sector errors in the syslog - but didn't think to close down the system.
In the end it shut down itself - "overrun" - something like that.
Hetzner tell me the drive is faulty and wish to replace it.
Background - ESXi - "should" be a simple setup - no snapshots, and no raid etc.
I'm trying to save data as much as I can!
Hetzner have cold booted the machine with their rescue system and I was able to mount the HDD with vmfs-fuse.
I have rsynced the VM directories to another machine. (the flat-vmdk seems to have slightly different sizes on each machine though):
# ls -al --block-size=k
Original:
-rw------- 1 root root 2147483647K Nov 18 15:51 Ubuntu_12.04.2_LTS-flat.vmdk
New machine:
-rw------- 1 root root 2147440280K Dec 11 15:52 Ubuntu_12.04.2_LTS-flat.vmdk
On the new machine I can mount most VMs (with vmware-mount) and save all interesting files.
But one super large (2TB) VM does not mount.
# vmware-mount -p Ubuntu_12.04.2_LTS.vmdk
Failed to open disk: The file specified is not a virtual disk (64424525503)
The above command fails on BOTH machines - The Original AND New.
"ls -al" shows:
# ls -al
total 2155591808
drwxr-xr-x 2 root root 4096 Dec 12 12:19 .
drwx------ 23 root root 4096 Dec 11 10:14 ..
-rw------- 1 root root 8246001664 May 8 2017 Ubuntu_12.04.2_LTS-22649c02.vswp
-rw------- 1 root root 2198978846084 Dec 11 15:52 Ubuntu_12.04.2_LTS-flat.vmdk
-rw------- 1 root root 8684 Nov 17 10:38 Ubuntu_12.04.2_LTS.nvram
-rw------- 1 root root 535 Jul 13 11:22 Ubuntu_12.04.2_LTS.vmdk
-rw-r--r-- 1 root root 43 Feb 13 2015 Ubuntu_12.04.2_LTS.vmsd
-rwxr-xr-x 1 root root 3304 Jul 13 11:22 Ubuntu_12.04.2_LTS.vmx
-rw-r--r-- 1 root root 3439 May 8 2017 Ubuntu_12.04.2_LTS.vmxf
-rw-r--r-- 1 root root 161590 Oct 14 2015 vmware-67.log
-rw-r--r-- 1 root root 251266 Nov 26 2015 vmware-68.log
-rw-r--r-- 1 root root 341136 Jan 5 2016 vmware-69.log
-rw-r--r-- 1 root root 627483 May 7 2017 vmware-70.log
-rw-r--r-- 1 root root 182378 May 8 2017 vmware-71.log
-rw-r--r-- 1 root root 161839 May 8 2017 vmware-72.log
-rw-r--r-- 1 root root 608614 Nov 17 10:39 vmware.log
-rw------- 1 root root 98566144 May 8 2017 vmx-Ubuntu_12.04.2_LTS-577018882-1.vswp
But note that I have removed these files from that directory (according to advice I read somewhere):
# ls -al
total 2308
drwxr-xr-x 2 root root 4096 Dec 11 10:13 .
drwx------ 23 root root 4096 Dec 11 10:14 ..
-rwxr-xr-x 1 root root 3304 Jul 13 11:22 Ubuntu_12.04.2_LTS.vmx~
-rw------- 1 root root 0 May 8 2017 Ubuntu_12.04.2_LTS.vmx.lck
The descriptor file "looks" ok:
# Disk DescriptorFile
version=1
encoding="UTF-8"
CID=42cac430
parentCID=ffffffff
isNativeSnapshot="no"
createType="vmfs"
# Extent description
RW 4294967294 VMFS "Ubuntu_12.04.2_LTS-flat.vmdk"
# The Disk Data Base
#DDB
ddb.toolsVersion = "9216"
ddb.virtualHWVersion = "8"
ddb.longContentID = "6a08d231843d35c7f119a42f42cac430"
ddb.uuid = "60 00 C2 97 9f ae 75 fc-00 5e f5 22 e9 2a 8f e0"
ddb.geometry.cylinders = "267349"
ddb.geometry.heads = "255"
ddb.geometry.sectors = "63"
ddb.thinProvisioned = "1"
ddb.adapterType = "lsilogic"
A snippet of the log file:
2017-07-13T09:22:35.051Z| Worker#0| I120: DISK: OPEN '/vmfs/volumes/5170e62a-2e6b7f78-9797-50465d4d2279/Ubuntu_12.04.2_LTS/Ubuntu_12.04.2_LTS.vmdk' Geo (267349/255/63) BIOS Geo (0/0/0)
:
:
2017-07-13T09:22:35.056Z| vmx| I120: DISKUTIL: ide1:0 : capacity=0
2017-07-13T09:22:35.056Z| vmx| I120: DISKUTIL: ide1:0 : geometry=0/0/0
2017-07-13T09:22:35.056Z| vmx| I120: SCSI: scsi0: intr coalescing: on period=50msec cifTh=4 iopsTh=2000 hlt=0
2017-07-13T09:22:35.056Z| vmx| I120: SCSI0: UNTAGGED commands will be converted to ORDER tags.
2017-07-13T09:22:35.056Z| vmx| I120: SCSI DEVICE (scsi0:0): Computed value of scsi0:0.useBounceBuffers: default
2017-07-13T09:22:35.056Z| vmx| I120: Creating virtual dev for scsi0:0
2017-07-13T09:22:35.056Z| vmx| I120: DumpDiskInfo: scsi0:0 createType=11, capacity = 4294967294, numLinks = 1, allocationType = 2
2017-07-13T09:22:35.056Z| vmx| I120: StateLogger::VM has thin disk scsi0:0; FT enable will be disallowedSCSIDiskESXPopulateVDevDesc: Using FS backend
2017-07-13T09:22:35.056Z| vmx| I120: DISKUTIL: scsi0:0 : geometry=267349/255/63
2017-07-13T09:22:35.056Z| vmx| I120: SVGA: Device capabilities 0x00ff83e2
2017-07-13T09:22:35.056Z| vmx| I120: SVGA: Maximum display topology 2560x1600.
2017-07-13T09:22:35.056Z| vmx| I120: SVGA: Maximum display topology 2560x1600 with 1 displays.
2017-07-13T09:22:35.056Z| vmx| I120: SVGA: Truncated maximum resolution to VRAM size: 4194304 bytes VRAM, 1176x885 Max Resolution
2017-07-13T09:22:35.056Z| vmx| I120: SVGA: FIFO capabilities 0x0000077f
2017-07-13T09:22:35.057Z| svga| I120: VTHREAD start thread 9 "svga" pid 6029
2017-07-13T09:22:35.057Z| svga| I120: Async SVGA thread is alive
2017-07-13T09:22:35.058Z| vmx| I120: Ethernet0 MAC Address: 00:50:56:00:03:34
2017-07-13T09:22:35.059Z| vmx| I120: Ethernet1 MAC Address: 00:0c:29:bd:f2:f4
2017-07-13T09:22:35.064Z| vmx| I120: GuestLib Generated SessionId 14935145101610005353
2017-07-13T09:22:35.064Z| vmx| I120: MStat: Creating Stat vm.heartbeat
2017-07-13T09:22:35.064Z| vmx| I120: VMXVmdbGuest_GetToolsVersion did nothing; tools version has not yet been initialized
2017-07-13T09:22:35.064Z| vmx| I120: DISKUTIL: scsi0:0 : max toolsVersion = 9216
The vmx file (the file holds DELETE and NEW lines - which are edits to be made when trying to start this VM in VMplayer):
.encoding = "UTF-8"
config.version = "8"
virtualHW.version = "8"
pciBridge0.present = "TRUE"
pciBridge4.present = "TRUE"
pciBridge4.virtualDev = "pcieRootPort"
pciBridge4.functions = "8"
pciBridge5.present = "TRUE"
pciBridge5.virtualDev = "pcieRootPort"
pciBridge5.functions = "8"
pciBridge6.present = "TRUE"
pciBridge6.virtualDev = "pcieRootPort"
pciBridge6.functions = "8"
pciBridge7.present = "TRUE"
pciBridge7.virtualDev = "pcieRootPort"
pciBridge7.functions = "8"
vmci0.present = "TRUE"
hpet0.present = "TRUE"
nvram = "Ubuntu_12.04.2_LTS.nvram"
virtualHW.productCompatibility = "hosted"
powerType.powerOff = "soft"
powerType.powerOn = "hard"
powerType.suspend = "hard"
powerType.reset = "soft"
displayName = "Ubuntu_12.04.2_LTS"
extendedConfigFile = "Ubuntu_12.04.2_LTS.vmxf"
floppy0.present = "TRUE"
numvcpus = "4"
cpuid.coresPerSocket = "4"
scsi0.present = "TRUE"
scsi0.sharedBus = "none"
scsi0.virtualDev = "lsilogic"
memsize = "7864"
scsi0:0.present = "TRUE"
scsi0:0.fileName = "Ubuntu_12.04.2_LTS.vmdk"
scsi0:0.deviceType = "scsi-hardDisk"
ide1:0.present = "TRUE"
ide1:0.clientDevice = "FALSE"
ide1:0.deviceType = "cdrom-image"
ide1:0.startConnected = "FALSE"
floppy0.startConnected = "FALSE"
floppy0.fileName = ""
floppy0.clientDevice = "TRUE"
ethernet0.present = "TRUE"
ethernet0.virtualDev = "e1000"
DELETE-ethernet0.networkName = "VM Network"
ethernet0.addressType = "static"
ethernet1.present = "TRUE"
ethernet1.virtualDev = "e1000"
D-ethernet1.networkName = "VM Internal"
ethernet1.addressType = "generated"
guestOS = "ubuntu-64"
uuid.location = "56 4d da 20 f9 04 7d 31-ef 24 19 40 c4 bd f2 ea"
uuid.bios = "56 4d da 20 f9 04 7d 31-ef 24 19 40 c4 bd f2 ea"
vc.uuid = "52 03 26 12 bb b8 cf d1-60 d3 b8 71 78 fe 27 5d"
ide1:0.fileName = "/vmfs/volumes/5170e62a-2e6b7f78-9797-50465d4d2279/iso/gparted-live-0.21.0-1-amd64.iso"
bios.forceSetupOnce = "FALSE"
scsi0.pciSlotNumber = "16"
ethernet0.pciSlotNumber = "32"
ethernet1.generatedAddress = "00:0c:29:bd:f2:f4"
ethernet1.pciSlotNumber = "33"
NEW-ethernet0.vnet=”VMnet0”
NEW-ethernet0.generatedAddressOffset=”0”
vmci0.id = "-994184470"
vmci0.pciSlotNumber = "34"
tools.syncTime = "FALSE"
cleanShutdown = "FALSE"
replay.supported = "FALSE"
sched.swap.derivedName = "/vmfs/volumes/5170e62a-2e6b7f78-9797-50465d4d2279/Ubuntu_12.04.2_LTS/Ubuntu_12.04.2_LTS-22649c02.vswp"
replay.filename = ""
scsi0:0.redo = ""
pciBridge0.pciSlotNumber = "17"
pciBridge4.pciSlotNumber = "21"
pciBridge5.pciSlotNumber = "22"
pciBridge6.pciSlotNumber = "23"
pciBridge7.pciSlotNumber = "24"
ethernet0.generatedAddressOffset = "0"
ethernet1.generatedAddressOffset = "10"
hostCPUID.0 = "0000000d756e65476c65746e49656e69"
hostCPUID.1 = "000306a90010080077bae3ffbfebfbff"
hostCPUID.80000001 = "00000000000000000000000128100800"
guestCPUID.0 = "0000000d756e65476c65746e49656e69"
guestCPUID.1 = "000306a900040800969822031fabfbff"
guestCPUID.80000001 = "00000000000000000000000128100800"
userCPUID.0 = "0000000d756e65476c65746e49656e69"
userCPUID.1 = "000306a900040800969822031fabfbff"
userCPUID.80000001 = "00000000000000000000000128100800"
evcCompatibilityMode = "FALSE"
vmotion.checkpointFBSize = "4194304"
softPowerOff = "FALSE"
ethernet0.address = "00:50:56:00:03:34"
unity.wasCapable = "FALSE"
tools.remindInstall = "FALSE"
sched.cpu.units = "mhz"
sched.cpu.shares = "high"
sched.scsi0:0.shares = "normal"
sched.scsi0:0.throughputCap = "off"
But the flat-vmdk itself appears to be corrupted - i.e. I tried continuum's advice and installed sgdisk etc.:
sgdisk -p Ubuntu_12.04.2_LTS-flat.vmdk
Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
Warning: File size is not a multiple of 512 bytes! Misbehavior is likely!
Creating new GPT entries.
Disk Ubuntu_12.04.2_LTS-flat.vmdk: 4294756728 sectors, 2.0 TiB
Logical sector size: 512 bytes
Disk identifier (GUID): ED2FF2F1-FC2F-4496-A3EB-5E80920BB2FF
Partition table holds up to 128 entries
First usable sector is 34, last usable sector is 4294756694
Partitions will be aligned on 2048-sector boundaries
Total free space is 4294756661 sectors (2.0 TiB)
Number Start (sector) End (sector) Size Code Name
<END OF OUTPUT>
i.e. the whole disk is free space. (it is/was a 2TB VM)
I also tried guestmount, this "finds" only one partition /dev/sda - but crashes out if I attempt to mount it.
I don't intend to set up ESXi all over again. I just want to extract as much data as possible.
Because of the size of this flat-vmdk I am pretty much limited to the unix world and command line tools
- I can't currently download a file of that size (slow connections) because I am away from home.
These machines are dedicated machines at a distant computer center - I have no physical access.
I would greatly welcome any hints on how to proceed!
Thanks
J