Cyclone Client Build

=Building the client= This will create the client that gets booted on the machine you are imaging.

You will need the following to get everything working. sudo apt-get install netpbm

Design goals
I am using buildroot in order to use uclibc, which makes the final files smaller at the expense of a bit more complication compiling them. This is optional.
 * Work on all network and storage chipsets (kernel supported).
 * Fast as possible - aim to work at the max wire speed
 * Small as possible - running on a slow link want as little data transfer as possible.
 * Flexible - can alter anything you like, can image anything from anywhere across the internet.
 * Reliable - it should just work

Getting the sources
These are the core programs I suggest including.

Linux Kernel
The core of the system if you will, (all the drivers in Cyclone are compiled into it rather than modular loading.)

http://www.kernel.org/

http://www.kernel.org/pub/linux/kernel/v3.0/linux-3.2.1.tar.bz2

Bash
Bourne again shell, building block of the core system. Needed as Bourne shell (sh) in Busybox lacks some abilities of bash. This isn't vital but makes things easier.

http://www.gnu.org/software/bash/

http://ftp.gnu.org/gnu/bash/bash-4.1.tar.gz

(Had problems with bash 4.2 cross compiled with new 'lastpipe' code, stick with 4.1 as it is only needed for a few utils.)

BIND
This is purely for the ability of nslookup in ISC BIND to return TXT entries from DNS records. This is probably quite custom to my system and you may well not need it.

http://www.isc.org/downloads

ftp://ftp.isc.org/isc/bind9/9.8.1-P1/bind-9.8.1-P1.tar.gz

BusyBox
Core component, provides init amongst other things and a myriad of little programs to tie the scripts together. Busybox's sh shell doesn't quite do enough to replace Bash shell. http://busybox.net/

http://busybox.net/downloads/busybox-1.19.3.tar.bz2

Had trouble with latest source git which had udhcpd segfault.

git clone git://git.busybox.net/busybox.git

chntpw
For editing Windows target registry, but certain things don't work and in the previous version to this caused registry corruption. Currently not used.

http://pogostick.net/~pnh/ntpasswd/

http://pogostick.net/~pnh/ntpasswd/chntpw-source-110511.zip

coreutils
This is purely to produce a working 'join' binary as the Busybox one doesnt do everything it needs to.

http://www.gnu.org/software/coreutils/

http://ftp.gnu.org/gnu/coreutils/coreutils-8.15.tar.xz

dmidecode
For pulling out serial number and board information etc. from the BIOS

http://download.savannah.gnu.org/releases/dmidecode/

http://download.savannah.gnu.org/releases/dmidecode/dmidecode-2.11.tar.gz

hexedit
non vital - for debugging and developing

http://rigaux.org/hexedit.html http://rigaux.org/hexedit-1.2.12.src.tgz

nano
non vital - for debugging and developing

ftp://ftp.gnu.org/pub/gnu/nano/ ftp://ftp.gnu.org/pub/gnu/nano/nano-2.3.1.tar.gz

ntfs-3g
For creating/accessing/modifying the NTFS filesystem on the target drive. This now contains Ntfsprogs and Fuser, rather than needing them separately. (Fuser support is automatically enabled in kernel via Makefile)

The patch included modifies ntfsresize.c (automatically in Makefile) which by default sets a windows chkdisk to occur by setting the volume dirty. If your image is correctly made and you have tested it works without corrupting anything then it appears to be fine to stop this happening. (Personally it doesn't bother me but it makes it look like something went wrong and it wastes time. I have imaged several hundreds of machines with this check removed and not had a problem.) If you dont want this to happen just remove the line form the Makefile that applies the patch.

http://www.tuxera.com/community/ntfs-3g-download/

http://tuxera.com/opensource/ntfs-3g_ntfsprogs-2011.4.12.tgz

http://tuxera.com/opensource/ntfs-3g_ntfsprogs-2011.10.9-RC.tgz

ntfsfixboot
"Set geometry and location parameters in NTFS filesystem, so it can boot." This fixes the NTFS boot sector on the target machine (start of the partition). E.g. if you image was created on a VM that had 16 heads, you would need to change it to 255 on a normal hdd format style.

http://sourceforge.net/projects/ntfsfixboot/files/ntfsfixboot/

http://sourceforge.net/projects/ntfsfixboot/files/latest/download?source=files

pigz
Speeds up unzipping of images but using other threads for other processing tasks. Benchmark: On test system with ~3.4Gb image / gigabit, gunzip takes 1:41s and pigz 1:12s (When a multithreaded xz decompressor comes along hopefully this can be replaced as the images are so much smaller.)

http://zlib.net/pigz/

http://zlib.net/pigz/pigz-2.2.3.tar.gz

strace
non vital - for debugging and developing

http://sourceforge.net/projects/strace/files/ http://sourceforge.net/projects/strace/files/latest/download?source=files

wget
This is used (along with lighttpd on server) for fetching all images/files etc. from the server, it is almost as fast (within seconds) of using NFS (for files) and FTP (for images). However it is far more flexible, very easy to set up and will work over the internet. It is also secure as doesn't need login accounts and can also be used for writing to the DB via PHP. Traffic over port 80 is easy to deal with! cURL may be a suitable alternative but untested. wget in Busybox is exactly the same speed xferring images, but lacks features needed for pulling down files/folders.

http://ftp.gnu.org/gnu/wget/

http://ftp.gnu.org/gnu/wget/wget-1.13.tar.xz

xz
Currently unused but is used for extracting lzma and xz images which are far smaller - but slow. If you have 100Mbit or slower you may well be better off using this than gzipped images as the slower decompression is outweighed by the network speed. (xz/lzma decoding using busyboxes built in tools is significantly slower than using the ones in xz)

http://tukaani.org/xz

http://tukaani.org/xz/xz-5.1.1alpha.tar.gz

Extract Sources
Extract all the source archives from cyclone/archive to cyclone/source

A quick way to do that is $ cd cyclone/archive $ for i in $(ls) ; do tar -C ../source/ -axf $i;done

Any that tar cant handle you will need to manually extract (like any .zip)

Buildroot
Buildroot will create the environment in which to cross compile with uclibc.

This will add complication but create smaller files and hopefully make these docs work for anyone rather than being distro specific.

In the below example git will automatically create sub folder buildroot. I have a hdd specially for this mounted in /media/7200.12/, with a folder cyclone/ to keep everything in.

$ cd cyclone $ git clone git://git.buildroot.net/buildroot Cloning into buildroot... remote: Counting objects: 76365, done. remote: Compressing objects: 100% (32468/32468), done. remote: Total 76365 (delta 51817), reused 66040 (delta 43530) Receiving objects: 100% (76365/76365), 26.62 MiB | 127 KiB/s, done. Resolving deltas: 100% (51817/51817), done. $ cd buildroot

Update it (if you haven't just clone'd it), make sure you are in the buildroot folder now cyclone/buildroot $ git pull

Setup configuring, make sure you are in the buildroot folder now

$ make defconfig $ make menuconfig

Change the following

Build options > gcc optimization level > select 3 Package selection > Libraries > Compression and decompression > select zlib,lzo,libarchive Package selection > Libraries > Crypto > openssl Package selection > Libraries > Text and terminal handline > select libiconv,ncurses,readline Toolchain > Kernel Headers > Linux 3.1.x kernel headers [select whatever matches the kernel you are using] Toolchain > GCC Compiler Version > gcc 4.6.x [select newest] Toolchain > Enable large file (files > 2 GB) support Toolchain > Enable IPV6 Toolchain > Enable RPC Toolchain > Enable WHCAR Toolchain > Enable C++ support

Build it, this takes ages (hours) $ make -j4

Configuring the kernel
Currently you don't need to manually do anything, Cyclone is using the default kernel made with defconfig and the Makefile is taking care of the rest. The makefile will automatically add new network and hard drive controllers support, for;
 * 100 Mbit and 1000 MBit netcards.
 * PATA & SATA hdd controllers

Look at Makefile on how to adjust things to include automatically.

Note, Intel HDA type audio is needed (enabled by default) to be able to detect the sub-audio devices for driverpack matching. Without this it wont see it and miss out audio drivers. This is a special case, it also needs ALSA installed (by default).

(You may well know how to tweak this section better than I can tell you, I have not devoted a lot of time to it. I imagine if you are intending to clone over the internet it may be worth investing some time on reducing the kernel size by chopping out uneeded bits)

Get the latest (stable or mainline kernel) from kernel.org, make sure buildroot is set with same version kernel headers (not sure how important this is)

Makefile
This should hopefully build the whole thing in one go, once you have done the above. If any source versions are altered they are very likely to break either because of bugs or new features, hence I have left links in to versions I used which defintely work (at the time of writing). This is especially true when cross compiling as bugs are less reported/found, hence the use of git as bugs are often only fixed after people have tried cross compiling the release version. If its a new feature that is causing a problem you can often turn it off by altering the ./configure line and disabling it, if its a bug you will have to try using git/beta/alpha version. A new feature or change of software may require additions to buildroot in order to support it.

Custom
These are the files in \custom

fbsplash.conf
BAR_LEFT=400 BAR_TOP=25 BAR_WIDTH=380 BAR_HEIGHT=25 BAR_R=50 BAR_G=50 BAR_B=150

inittab

 * sysinit:/etc/init.d/rcS
 * ctrlaltdel:/sbin/reboot
 * shutdown:/bin/umount -a -r

tty1::respawn:/bin/sh tty2::respawn:/bin/sh

rcS

 * 1) !/bin/sh
 * 2) Cyclone Linux #
 * 3)  RMDC 2008-12 #
 * 1)  RMDC 2008-12 #

mount -t proc proc /proc mount -t sysfs none /sys
 * 1) mount system virtual file systems

mount -t devpts devpts /dev/pts
 * 1) UNIX98 pseudoterminal support

ln -s /proc/mounts /etc/mtab
 * 1) link mounted filesystems


 * 1) stop kernel messages from popping onto the screen
 * 2) echo 0 > /proc/sys/kernel/printk

mdev -s echo /sbin/mdev > /proc/sys/kernel/hotplug
 * 1) hotplug support with bb mdev

udhcpc -s /etc/udhcpc.script &>/dev/null
 * 1) configure network via DHCP

dropbear -r /etc/dropbear_rsa_host_key
 * 1) start dropbear SSH server

eval $(grep -oE "[^ ]+=[^ ]+" /proc/cmdline)
 * 1) get kernel cmdline args as vars, cyclone server name/IP & scripts path & run script

wget -P "/tmp/" -R index.html -nH -q -r -np http://$serveraddr/$spath/ chmod 755 /tmp/$spath/*.sh && "/tmp/$spath/$runinit"
 * 1) pull and execute startup


 * 1) setsid sh -c 'exec "/tmp/$spath/$runinit" /dev/tty1 2>&1'

profile
TERM=linux PATH=/sbin:/usr/bin:/bin:/usr/bin

Building
If everything is ok, then the following should work, ('make all' also works but I wouldn't tempt fate.)
 * compile kernel (make linux)
 * bzImage produced > \output\bzImage
 * compile all other programs (make progs)
 * \sources compiled (any patching done automatically) > \install
 * put system together (make system)
 * \install parsed, stripped and tweaked, combined with \custom > \initrd
 * \initrd compressed > \output\initramfs.cp.lzma (not using .gz 2 x larger, xz (at time of writing) oddly slow to decompress)

$ cd cyclone $ make linux $ make progs $ make system

If you have a problem, which you will surely do to start with, you can build each program individually, you should be able to see from the Makefile what they are, e.g. this should just build busybox and you should see stuff end up in cyclone/target $ cd cyclone $ make busybox

When everything compiles correctly, dont forget to redo 'make system' which will produce a shiny new \output\initramfs.cp.lzma image.

Compatability

 * 1) Drivers are reliant on the kernel, which is fairly quick to get updated. (It has always been miles ahead of the kit my work gets)
 * 2) fakeRAID support is included, but you will need to modify the run script (cyclone.sh) to work with it as it uses different block device names.

Reliability
The only instances I have ever had this crash under (in years) is when there is faulty hardware on the target, often bad RAM.