Cyclone Create OS Image File

=Windows 7= Create a new VM in virtualbox, try and make as generic as possible so take off USB and audio devices.

Keep default settings, suggest about 20Gb HDD (.vdi format), 30Gb if installing office.

Suggest keeping the initial hidden boot partition in case want to bitlocker later or put Windows recovery (WinRE?) into it. Set it to 500Mb in Autounattend.xml so you have some overhead. (This has a type id of 27 which appears to be so Windows cant see it, but is just normal NTFS formatted)


 * Attach Win7 ISO as CD drive (SW_DVD5_SA_Win_Ent_7w_SP1_64BIT_English_-2_MLF_X17-58882.ISO)
 * Attach floppy drive image (W7.IMG) containing Autounattend.xml (auto install/setup file), unattend.xml (for sysprep) and sys.bat (for starting sysprep without having to type anything)


 * Note xml works on 32 bit and 64 bit systems.
 * Note, 500 in Primary partition size corresponds to 524,284,416 bytes, which equates to ~ 499.999 Mb. The Boot folder ends up here if using standard 2 partition layout.
 * Note this example adds some accounts specific to my situation, which you may well not want. It would be better to add them later, but have left them in here because the rest of the docs are written around this image which happened to have them in.
 * Note this will wipe the hdd without any questions.

Sysprep image
When you are happy it is setup how you need for your situation, run sys.bat from the floppy image (W7.IMG), which simply automates copying unattend.xml and running sysprep.

copy /y unattend.xml c:\Windows\System32\sysprep\ c: cd c:\Windows\System32\sysprep\ sysprep /generalize /oobe /shutdown /unattend:unattend.xml

unattend.xml, (it will get modified later, this is just enough for testing so it will get through an install automatically.)

=Win XP=

Sysprep image
TODO

=Creating the Image= Once the image is finished, shutdown the machine if it isn't already.

Clone/Merge the snapshot out
You can either user the GUI Snapshots > Clone > Full Clone > Current Machine State (current versions doesn't seem to let you select a path though so it will be slow writing to the same drive)

Or Goto the Snapshots sub-folder of the VM and look at the newest .vdi, if it were {eade675a-8362-473d-91aa-f5c1c248e801}.vdi then the UUID is eade675a-8362-473d-91aa-f5c1c248e801 so this is the command to clone it (dont worry it does snapshots merge!)

$ vboxmanage clonehd eade675a-8362-473d-91aa-f5c1c248e801 /media/7200.12/test.vdi

Mount the .vdi snapshot as block device
Now mount the image as a block device with qemu-nbd (first need to add the device driver module with modprobe). qemu-nbd which has the added bonus it will mount each partition in the image, without you needing to work any offset out.

Despite being in the disk group you still need sudo to mount it, if it worked you should see nbd0pX representing each partition (depending on how many partitions it has). In the below example it happens to be Windows 7 so has two by default.


 * Note If you dont use sudo it will fail but be stuck in a state where you then cannot mount it with sudo unless you unmount first with sudo.

$ sudo modprobe nbd $ sudo qemu-nbd -c /dev/nbd0 "/media/7200.12/test.vdi" $ ls -al /dev/nb* brw-rw 1 root disk 43,  0 2012-01-07 22:54 /dev/nbd0 brw-rw 1 root disk 43,  1 2012-01-07 22:54 /dev/nbd0p1 brw-rw 1 root disk 43,  2 2012-01-07 22:54 /dev/nbd0p2 brw-rw 1 root disk 43, 16 2012-01-07 20:14 /dev/nbd1

Mount the block device to cleanup
Mount the operating system partition to remove uneeded files, this will probably be the second one with Windows 7 and the only one with XP. Probably only worth deleting the main windows fluff. The unmount.

Windows 7 removal list
ntfs.log hiberfil.sys pagefile.sys "System Volume Information" RECYCLER Windows\CSC

Windows XP removal list
TODO

Example removal
$ udisks --mount /dev/nbd0p2 Mounted /org/freedesktop/UDisks/devices/nbd0p2 at /media/disk $ rm -rf '/media/disk/pagefile.sys' $ rm -rf '/media/disk/System Volume Information' $ rm -rf '/media/disk/$Recycle.bin' $ udisks --unmount /dev/nbd0p2

Save the NTFS partitions out
Save the NTFS image of partition 1 using ntfsclone, using its special image format, pipe it to std out, compress it through gzip and output that on stdout directed to the image archive file test.p1.gz.

Ideally this wants to be on a different hdd to speed it up.

Gzip although not offering the best compression it is a lot faster and more importantly faster to decompress. It's choice is really a balancing act of your limiting factors. With the assumption most of your imaging will be over gigabit, the limiting factor is the write speed of the hdd or the image decompression time. Gzip will work multithreaded with pigz, which on most machines is faster than the hdd write speed. If you are imaging across the internet you may well be better using bzip2. Bear in mind it decompresses as piped output as it goes so it needs to be a streamed format, so 7z or rar isn't possible as they need to see the start and end of the file (seeking) before they can do anything (they are archivers, not compressors). Note, Could modify to use other compression programs, lzop would be fastest but larget image, although if hdd limiting then of no consequence, xz (LZMA2) better compression but slower to decompress. $ ntfsclone --save-image -o - /dev/nbd0p1 | gzip -9 -c > /media/7200.11/test.p1.gz

You could have the output on an NFS drive share to your server so it writes it directly too it (or std out to an FTP client that writes it directly to the FTP server, on your server)

However you do it you eventually need this outputted file on the server in the /cyclone/img/ folder, then update the database with the new image.

Both partitions in this case will need cloning, the second perhaps as test.p2.gz with /dev/nbd0p2 as the source.

Complete Example: Mount, remove cruft, image
Remember you need to be in the disks group.

Assume first partition is Windows Boot partition, second partition contains Windows proper. Pigz is used to speedup the full partition ntfsclone operation, it makes no difference for the boot.

vboxmanage clonehd eade675a-8362-473d-91aa-f5c1c248e801 /media/7200.12/test.vdi sudo modprobe nbd sudo qemu-nbd -c /dev/nbd0 /media/7200.12/test.vdi udisks --mount /dev/nbd0p2 rm -rf '/media/disk/pagefile.sys' rm -rf '/media/disk/hiberfil.sys' rm -rf '/media/disk/System Volume Information' rm -rf '/media/disk/$Recycle.bin' rm -rf '/media/disk/Windows/CSC' udisks --unmount /dev/nbd0p2 ntfsclone --save-image -o - /dev/nbd0p1 | gzip -9 -c > /media/7200.11/test.p1.gz ntfsclone --save-image -o - /dev/nbd0p2 | pigz -9 -c > /media/7200.11/test.p2.gz sudo qemu-nbd -d /dev/nbd0

The partition images can then be put in the cyclone/images folder.

The cyclone/db.sqlite database needs changing at various points to fill in the new image.

Update Ubuntu 15.10
udisks no longer exists, it is now udisksctl, which doesnt appear to work. Also now using KVM/qemu instead of virtualbox.

$ udisksctl mount -b /dev/nbd0p2 Object /org/freedesktop/UDisks2/block_devices/nbd0p2 is not a mountable filesystem.

Have to use mount with root instead, thus in full

sudo modprobe nbd sudo qemu-nbd -c /dev/nbd0 /media/3/VMM/win7_x64.qcow2 sudo mkdir -p /tmp/ntfs sudo mount /dev/nbd0p2 /tmp/ntfs rm -rf '/tmp/ntfs/pagefile.sys' rm -rf '/tmp/ntfs/hiberfil.sys' rm -rf '/tmp/ntfs/System Volume Information' rm -rf '/tmp/ntfs/$Recycle.Bin' rm -rf '/tmp/ntfs/Windows/CSC' rm -rf '/tmp/ntfs/Windows/SoftwareDistribution' sudo umount /tmp/ntfs sudo ntfsclone --save-image -o - /dev/nbd0p1 | pigz -9 -c > /nfs/xxx.xxx.xxx.xxx/cyclone/img/w7-x64-standalone-20151214-p1.gz sudo ntfsclone --save-image -o - /dev/nbd0p2 | pigz -9 -c > /nfs/xxx.xxx.xxx.xxx/cyclone/img/w7-x64-standalone-20151214-p2.gz sudo qemu-nbd -d /dev/nbd0

Update Ubuntu 16.04 kernel > 4.4
Need to add max_part=16 otherwise you cant mount the partitions! sudo modprobe nbd max_part=16

Update Ubuntu 18.10
Win10 image with Tinycore UEFI x86_64 firmware (/usr/share/OVMF/OVMF_CODE.ms.fd), needs external snapshots as UEFI not supported with internal ones.

virsh snapshot-create-as --domain Windows10_1803 "w10_office_installed" --diskspec vda,file=/media/1/VMM/Win10_1803/snapshot/office2,snapshot=external --disk-only --atomic
 * 1) create snapshot, boot snapshot


 * 1) add WIN10.IMG as floppy in VirtManager, open admin cmd prompt and run a:\sys.bat - [note no A: will appear in Explorer]. This will sysprep it at the end of the batch file.

sudo modprobe nbd sudo qemu-nbd -c /dev/nbd0 /media/1/VMM/Win10_1803/snapshot/office2
 * 1) mount image

sudo mkdir /tmp/ntfs sudo mount /dev/nbd0p4 /tmp/ntfs rm -rf '/tmp/ntfs/System Volume Information' rm -rf '/tmp/ntfs/$Recycle.Bin' rm -rf '/tmp/ntfs/pagefile.sys' rm -rf '/tmp/ntfs/swapfile.sys' rm -rf '/tmp/ntfs/$RECYCLE.BIN' sudo umount /tmp/ntfs
 * 1) cleanup


 * 1) create images

sudo ntfsclone --save-image -o - /dev/nbd0p1 | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p1.gz sudo ntfsclone --save-image -o - /dev/nbd0p4 | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p4.gz
 * 1) P1 Windows recovery Environment (NTFS),, P4 Microsoft Basic Data (NTFS, is resized on restore to fit disk)

sudo dd if=/dev/nbd0p2 | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p2.gz sudo dd if=/dev/nbd0p3  | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p3.gz
 * 1) P2 EFI System (raw), P3 Microsoft Reserved (raw)

sudo fdisk -l /dev/nbd0 > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-original_partition_table.txt

sudo qemu-nbd -d /dev/nbd0

Update Ubuntu 19.10
change --diskspec vda to --diskspec sda