Cyclone Create OS Image File

From richud.com
Jump to navigation Jump to search


Windows 7

Create a new VM in virtualbox, try and make as generic as possible so take off USB and audio devices.

Keep default settings, suggest about 20Gb HDD (.vdi format), 30Gb if installing office.

Suggest keeping the initial hidden boot partition in case want to bitlocker later or put Windows recovery (WinRE?) into it. Set it to 500Mb in Autounattend.xml so you have some overhead. (This has a type id of 27 which appears to be so Windows cant see it, but is just normal NTFS formatted)

  • Attach Win7 ISO as CD drive (SW_DVD5_SA_Win_Ent_7w_SP1_64BIT_English_-2_MLF_X17-58882.ISO)
  • Attach floppy drive image (W7.IMG) containing Autounattend.xml (auto install/setup file), unattend.xml (for sysprep) and sys.bat (for starting sysprep without having to type anything)
  • Note xml works on 32 bit and 64 bit systems.
  • Note, 500 in Primary partition size corresponds to 524,284,416 bytes, which equates to ~ 499.999 Mb. The Boot folder ends up here if using standard 2 partition layout.
  • Note this example adds some accounts specific to my situation, which you may well not want. It would be better to add them later, but have left them in here because the rest of the docs are written around this image which happened to have them in.
  • Note this will wipe the hdd without any questions.
<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="windowsPE">
       <component name="Microsoft-Windows-International-Core-WinPE" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <InputLocale>en-gb</InputLocale>
            <SystemLocale>en-gb</SystemLocale>
            <UILanguage>en-us</UILanguage>
            <UserLocale>en-gb</UserLocale>
        </component>
       <component name="Microsoft-Windows-International-Core-WinPE" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <InputLocale>en-gb</InputLocale>
            <SystemLocale>en-gb</SystemLocale>
            <UILanguage>en-us</UILanguage>
            <UserLocale>en-gb</UserLocale>
        </component>
         <component name="Microsoft-Windows-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
         <DiskConfiguration>
                <Disk wcm:action="add">
                    <CreatePartitions>
                        <CreatePartition wcm:action="add">
                            <Extend>false</Extend>
                            <Order>1</Order>
                            <Type>Primary</Type>
                            <Size>500</Size>
                        </CreatePartition>
                        <CreatePartition wcm:action="add">
                            <Order>2</Order>
                            <Type>Primary</Type>
                            <Extend>true</Extend>
                        </CreatePartition>
                    </CreatePartitions>
                    <ModifyPartitions>
                        <ModifyPartition wcm:action="add">
                            <Active>true</Active>
                            <Format>NTFS</Format>
                            <Order>1</Order>
                            <PartitionID>1</PartitionID>
                            <TypeID>0x27</TypeID>
                        </ModifyPartition>
                        <ModifyPartition wcm:action="add">
                            <Format>NTFS</Format>
                            <Order>2</Order>
                            <PartitionID>2</PartitionID>
                        </ModifyPartition>
                    </ModifyPartitions>
                    <DiskID>0</DiskID>
                    <WillWipeDisk>true</WillWipeDisk>
                </Disk>
            </DiskConfiguration>
            <ImageInstall>
                <OSImage>
 			<InstallToAvailablePartition>true</InstallToAvailablePartition>
                    <WillShowUI>OnError</WillShowUI>
                </OSImage>
            </ImageInstall>
            <UserData>
                <AcceptEula>true</AcceptEula>
            </UserData>
        </component>
        <component name="Microsoft-Windows-Setup" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
         <DiskConfiguration>
                <Disk wcm:action="add">
                    <CreatePartitions>
                        <CreatePartition wcm:action="add">
                            <Extend>false</Extend>
                            <Order>1</Order>
                            <Type>Primary</Type>
                            <Size>500</Size>
                        </CreatePartition>
                        <CreatePartition wcm:action="add">
                            <Order>2</Order>
                            <Type>Primary</Type>
                            <Extend>true</Extend>
                        </CreatePartition>
                    </CreatePartitions>
                    <ModifyPartitions>
                        <ModifyPartition wcm:action="add">
                            <Active>true</Active>
                            <Format>NTFS</Format>
                            <Order>1</Order>
                            <PartitionID>1</PartitionID>
                            <TypeID>0x27</TypeID>
                        </ModifyPartition>
                        <ModifyPartition wcm:action="add">
                            <Format>NTFS</Format>
                            <Order>2</Order>
                            <PartitionID>2</PartitionID>
                        </ModifyPartition>
                    </ModifyPartitions>
                    <DiskID>0</DiskID>
                    <WillWipeDisk>true</WillWipeDisk>
                </Disk>
            </DiskConfiguration>
            <ImageInstall>
                <OSImage>
 			<InstallToAvailablePartition>true</InstallToAvailablePartition>
                    <WillShowUI>OnError</WillShowUI>
                </OSImage>
            </ImageInstall>
            <UserData>
                <AcceptEula>true</AcceptEula>
            </UserData>
        </component>
    </settings>
    <settings pass="specialize">
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <ComputerName></ComputerName>
            <ShowWindowsLive>false</ShowWindowsLive>
        </component>
	<component name="Microsoft-Windows-Shell-Setup" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <ComputerName></ComputerName>
            <ShowWindowsLive>false</ShowWindowsLive>
        </component>
        <component name="Microsoft-Windows-IE-InternetExplorer" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <FilterLevel>High</FilterLevel>
            <PlaySound>false</PlaySound>
            <ShowInformationBar>false</ShowInformationBar>
            <DisableAccelerators>true</DisableAccelerators>
            <DisableDevTools>true</DisableDevTools>
            <DisableFirstRunWizard>true</DisableFirstRunWizard>
            <DisableOOBAccelerators>true</DisableOOBAccelerators>
            <Home_Page>http://www.google.com</Home_Page>
 	    <SearchScopes>
 		<Scope wcm:action="add">
      		<ScopeDefault>true</ScopeDefault>
                <ScopeDisplayName>Google</ScopeDisplayName>
                <ScopeKey>SearchProvider1</ScopeKey>
                <ScopeUrl>http://www.google.com/search?q={searchTerms}</ScopeUrl>
                </Scope>
            </SearchScopes>
        </component>
        <component name="Microsoft-Windows-IE-InternetExplorer" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <FilterLevel>High</FilterLevel>
            <PlaySound>false</PlaySound>
            <ShowInformationBar>false</ShowInformationBar>
            <DisableAccelerators>true</DisableAccelerators>
            <DisableDevTools>true</DisableDevTools>
            <DisableFirstRunWizard>true</DisableFirstRunWizard>
            <DisableOOBAccelerators>true</DisableOOBAccelerators>
            <Home_Page>http://www.google.com</Home_Page>
 	    <SearchScopes>
 		<Scope wcm:action="add">
      		<ScopeDefault>true</ScopeDefault>
                <ScopeDisplayName>Google</ScopeDisplayName>
                <ScopeKey>SearchProvider1</ScopeKey>
                <ScopeUrl>http://www.google.com/search?q={searchTerms}</ScopeUrl>
                </Scope>
            </SearchScopes>
         </component>
        <component name="Microsoft-Windows-ErrorReportingCore" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DisableWER>1</DisableWER>
        </component>
        <component name="Microsoft-Windows-ErrorReportingCore" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DisableWER>1</DisableWER>
        </component>
    </settings>
    <settings pass="oobeSystem">
         <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <OOBE>
                 <ProtectYourPC>1</ProtectYourPC>
                <NetworkLocation>Home</NetworkLocation>
            </OOBE>
      <AutoLogon>
         <Password>
            <Value>Pa55w0rd</Value> 
            <PlainText>true</PlainText> 
         </Password>
         <Username>Administrator</Username> 
         <Enabled>true</Enabled> 
         <LogonCount>1</LogonCount> 
      </AutoLogon>
            <UserAccounts>
		<AdministratorPassword>
			<Value>Pa55w0rd</Value> 
			<PlainText>true</PlainText> 
		</AdministratorPassword>
                <LocalAccounts>
                    <LocalAccount wcm:action="add">
                        <Password>
                            <Value>Pa55w0rd</Value>
                            <PlainText>true</PlainText>
                        </Password>
                        <Name>localadmin1</Name>
                        <Group>Administrators</Group>
                    </LocalAccount>
                    <LocalAccount wcm:action="add">
                        <Password>
                            <Value>Pa55w0rd</Value>
                            <PlainText>true</PlainText>
                        </Password>
                        <Name>localuser</Name>
                        <Group>Users</Group>
                    </LocalAccount>
                </LocalAccounts>
            </UserAccounts>
            <TimeZone>GMT Standard Time</TimeZone>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <OOBE>
                 <ProtectYourPC>1</ProtectYourPC>
                <NetworkLocation>Home</NetworkLocation>
            </OOBE>
      <AutoLogon>
         <Password>
            <Value>Pa55w0rd</Value> 
            <PlainText>true</PlainText> 
         </Password>
         <Username>Administrator</Username> 
         <Enabled>true</Enabled> 
         <LogonCount>1</LogonCount> 
      </AutoLogon>
            <UserAccounts>
		<AdministratorPassword>
			<Value>Pa55w0rd</Value> 
			<PlainText>true</PlainText> 
		</AdministratorPassword>
                <LocalAccounts>
                    <LocalAccount wcm:action="add">
                        <Password>
                            <Value>Pa55w0rd</Value>
                            <PlainText>true</PlainText>
                        </Password>
                        <Name>localadmin1</Name>
                        <Group>Administrators</Group>
                    </LocalAccount>
                    <LocalAccount wcm:action="add">
                        <Password>
                            <Value>Pa55w0rd</Value>
                            <PlainText>true</PlainText>
                        </Password>
                        <Name>localuser</Name>
                        <Group>Users</Group>
                    </LocalAccount>
                </LocalAccounts>
            </UserAccounts>
            <TimeZone>GMT Standard Time</TimeZone>
        </component>
    </settings>
    <cpi:offlineImage cpi:source="wim:/sources/install.wim#Windows 7 ULTIMATE" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>


Sysprep image

When you are happy it is setup how you need for your situation, run sys.bat from the floppy image (W7.IMG), which simply automates copying unattend.xml and running sysprep. 

copy /y unattend.xml c:\Windows\System32\sysprep\
c:
cd c:\Windows\System32\sysprep\
sysprep /generalize /oobe /shutdown /unattend:unattend.xml


unattend.xml , (it will get modified later, this is just enough for testing so it will get through an install automatically.) 

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
   <settings pass="oobeSystem">
	<component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <OOBE>
   		<HideEULAPage>true</HideEULAPage>
  		<SkipMachineOOBE>true</SkipMachineOOBE>
  		<SkipUserOOBE>true</SkipUserOOBE>
            </OOBE>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <OOBE>
   		<HideEULAPage>true</HideEULAPage>
  		<SkipMachineOOBE>true</SkipMachineOOBE>
  		<SkipUserOOBE>true</SkipUserOOBE>
            </OOBE>
        </component>
    </settings>
</unattend>

Win XP

Sysprep image

TODO

Creating the Image

Once the image is finished, shutdown the machine if it isn't already.


Clone/Merge the snapshot out

You can either user the GUI Snapshots > Clone > Full Clone > Current Machine State (current versions doesn't seem to let you select a path though so it will be slow writing to the same drive)

Or Goto the Snapshots sub-folder of the VM and look at the newest .vdi, if it were {eade675a-8362-473d-91aa-f5c1c248e801}.vdi then the UUID is eade675a-8362-473d-91aa-f5c1c248e801 so this is the command to clone it (dont worry it does snapshots merge!) 

$ vboxmanage clonehd eade675a-8362-473d-91aa-f5c1c248e801 /media/7200.12/test.vdi

Mount the .vdi snapshot as block device

Now mount the image as a block device with qemu-nbd (first need to add the device driver module with modprobe). qemu-nbd which has the added bonus it will mount each partition in the image, without you needing to work any offset out.

Despite being in the disk group you still need sudo to mount it, if it worked you should see nbd0pX representing each partition (depending on how many partitions it has). In the below example it happens to be Windows 7 so has two by default.

  • Note If you dont use sudo it will fail but be stuck in a state where you then cannot mount it with sudo unless you unmount first with sudo.
$ sudo modprobe nbd
$ sudo qemu-nbd -c /dev/nbd0 "/media/7200.12/test.vdi"
$ ls -al /dev/nb*
brw-rw---- 1 root disk 43,   0 2012-01-07 22:54 /dev/nbd0
brw-rw---- 1 root disk 43,   1 2012-01-07 22:54 /dev/nbd0p1
brw-rw---- 1 root disk 43,   2 2012-01-07 22:54 /dev/nbd0p2
brw-rw---- 1 root disk 43,  16 2012-01-07 20:14 /dev/nbd1
<snip>

Mount the block device to cleanup

Mount the operating system partition to remove uneeded files, this will probably be the second one with Windows 7 and the only one with XP. Probably only worth deleting the main windows fluff. The unmount.

Windows 7 removal list

ntfs.log hiberfil.sys pagefile.sys "System Volume Information" RECYCLER Windows\CSC

Windows XP removal list

TODO

Example removal

$ udisks --mount /dev/nbd0p2
Mounted /org/freedesktop/UDisks/devices/nbd0p2 at /media/disk
$ rm -rf '/media/disk/pagefile.sys'
$ rm -rf '/media/disk/System Volume Information'
$ rm -rf '/media/disk/$Recycle.bin'
$ udisks --unmount /dev/nbd0p2

Save the NTFS partitions out

Save the NTFS image of partition 1 using ntfsclone, using its special image format, pipe it to std out, compress it through gzip and output that on stdout directed to the image archive file test.p1.gz.

Ideally this wants to be on a different hdd to speed it up.

Gzip although not offering the best compression it is a lot faster and more importantly faster to decompress. It's choice is really a balancing act of your limiting factors. With the assumption most of your imaging will be over gigabit, the limiting factor is the write speed of the hdd or the image decompression time. Gzip will work multithreaded with pigz, which on most machines is faster than the hdd write speed. If you are imaging across the internet you may well be better using bzip2. Bear in mind it decompresses as piped output as it goes so it needs to be a streamed format, so 7z or rar isn't possible as they need to see the start and end of the file (seeking) before they can do anything (they are archivers, not compressors). Note, Could modify to use other compression programs , lzop would be fastest but larget image, although if hdd limiting then of no consequence, xz (LZMA2) better compression but slower to decompress. 

$ ntfsclone --save-image -o - /dev/nbd0p1 | gzip -9 -c > /media/7200.11/test.p1.gz

You could have the output on an NFS drive share to your server so it writes it directly too it (or std out to an FTP client that writes it directly to the FTP server, on your server)

However you do it you eventually need this outputted file on the server in the /cyclone/img/ folder, then update the database with the new image.

Both partitions in this case will need cloning, the second perhaps as test.p2.gz with /dev/nbd0p2 as the source.

Complete Example: Mount, remove cruft, image

Remember you need to be in the disks group.

Assume first partition is Windows Boot partition, second partition contains Windows proper. Pigz is used to speedup the full partition ntfsclone operation, it makes no difference for the boot. 

vboxmanage clonehd eade675a-8362-473d-91aa-f5c1c248e801 /media/7200.12/test.vdi
sudo modprobe nbd
sudo qemu-nbd -c /dev/nbd0 /media/7200.12/test.vdi
udisks --mount /dev/nbd0p2
rm -rf '/media/disk/pagefile.sys'
rm -rf '/media/disk/hiberfil.sys'
rm -rf '/media/disk/System Volume Information'
rm -rf '/media/disk/$Recycle.bin'
rm -rf '/media/disk/Windows/CSC'
udisks --unmount /dev/nbd0p2
ntfsclone --save-image -o - /dev/nbd0p1 | gzip -9 -c > /media/7200.11/test.p1.gz
ntfsclone --save-image -o - /dev/nbd0p2 | pigz -9 -c > /media/7200.11/test.p2.gz
sudo qemu-nbd -d /dev/nbd0


The partition images can then be put in the cyclone/images folder.

The cyclone/db.sqlite database needs changing at various points to fill in the new image.

Update Ubuntu 15.10

udisks no longer exists, it is now udisksctl, which doesnt appear to work. Also now using KVM/qemu instead of virtualbox. 

$ udisksctl mount -b /dev/nbd0p2
Object /org/freedesktop/UDisks2/block_devices/nbd0p2 is not a mountable filesystem.

Have to use mount with root instead, thus in full 

sudo modprobe nbd
sudo qemu-nbd -c /dev/nbd0 /media/3/VMM/win7_x64.qcow2
sudo mkdir -p /tmp/ntfs
sudo mount /dev/nbd0p2 /tmp/ntfs
rm -rf '/tmp/ntfs/pagefile.sys'
rm -rf '/tmp/ntfs/hiberfil.sys'
rm -rf '/tmp/ntfs/System Volume Information'
rm -rf '/tmp/ntfs/$Recycle.Bin'
rm -rf '/tmp/ntfs/Windows/CSC'
rm -rf '/tmp/ntfs/Windows/SoftwareDistribution'
sudo umount /tmp/ntfs
sudo ntfsclone --save-image -o - /dev/nbd0p1 | pigz -9 -c > /nfs/xxx.xxx.xxx.xxx/cyclone/img/w7-x64-standalone-20151214-p1.gz
sudo ntfsclone --save-image -o - /dev/nbd0p2 | pigz -9 -c > /nfs/xxx.xxx.xxx.xxx/cyclone/img/w7-x64-standalone-20151214-p2.gz
sudo qemu-nbd -d /dev/nbd0

Update Ubuntu 16.04 kernel > 4.4

Need to add max_part=16 otherwise you cant mount the partitions! 

sudo modprobe nbd max_part=16

Update Ubuntu 18.10

Win10 image with Tinycore UEFI x86_64 firmware (/usr/share/OVMF/OVMF_CODE.ms.fd), needs external snapshots as UEFI not supported with internal ones. 

#create snapshot, boot snapshot
virsh snapshot-create-as --domain Windows10_1803 "w10_office_installed" --diskspec vda,file=/media/1/VMM/Win10_1803/snapshot/office2,snapshot=external --disk-only --atomic

#add WIN10.IMG as floppy in VirtManager, open admin cmd prompt and run a:\sys.bat - [note no A: will appear in Explorer]. This will sysprep it at the end of the batch file.

#mount image
sudo modprobe nbd
sudo qemu-nbd -c /dev/nbd0 /media/1/VMM/Win10_1803/snapshot/office2

#cleanup
sudo mkdir /tmp/ntfs
sudo mount /dev/nbd0p4 /tmp/ntfs
rm -rf '/tmp/ntfs/System Volume Information'
rm -rf '/tmp/ntfs/$Recycle.Bin'
rm -rf '/tmp/ntfs/pagefile.sys'
rm -rf '/tmp/ntfs/swapfile.sys' 
rm -rf '/tmp/ntfs/$RECYCLE.BIN'
sudo umount /tmp/ntfs

#create images

#P1 Windows recovery Environment (NTFS), , P4 Microsoft Basic Data (NTFS, is resized on restore to fit disk)
sudo ntfsclone --save-image -o - /dev/nbd0p1 | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p1.gz
sudo ntfsclone --save-image -o - /dev/nbd0p4 | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p4.gz

#P2 EFI System (raw), P3 Microsoft Reserved (raw)
sudo dd if=/dev/nbd0p2  | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p2.gz
sudo dd if=/dev/nbd0p3  | pigz -9 -c > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-p3.gz

sudo fdisk -l /dev/nbd0 > /nfs/143.210.24.235/opt/cyclone/img/w10-1803-x64-standalone-20181214-original_partition_table.txt

sudo qemu-nbd -d /dev/nbd0

Update Ubuntu 19.10

change --diskspec vda to --diskspec sda

Retrieved from "https://www.richud.com/w/index.php?title=Cyclone_Create_OS_Image_File&oldid=5140"