Advanced Topics

Image Format

snf-image supports 3 types of image formats:

  • extdump: a raw dump of an ext{2,3,4} file system
  • ntfsdump: a raw dump of an NTFS file system
  • diskdump (recommended): a raw dump of a disk

extdump and ntfsdump image formats

Those two formats are dumps (raw copies using dd) of partitions hosting Linux systems on ext{2,3,4} and Windows systems on NTFS file systems respectively. Partitions hosting a Windows or Linux system that are suitable for dumping should have the following properties:

  • Be the first partition in the file system
  • The OS they host should not depend on any other partitions
  • Start at sector 2048
  • Have a boot loader installed in the boot sector of the partition (not MBR)
  • Have the root device in /etc/fstab specified in a persistent way, using UUID or LABEL (for extdump only)

Known Issues

  • For Linux systems, having GRUB installed on a partition is fragile and things can go wrong if you shrink the partition.
  • More complicated partition schemes are not supported.

Windows Deployment

snf-image performs Windows customization by installing an Answer File for Unattended Installation (typically named Unattend.xml) into the VM’s hard disk and customizing the file accordingly. The VM will auto-configure itself the first time it boots. For this to work, the used Windows image must have previously been generalized [2] with a command like this:

Sysprep /generalize /shutdown /oobe

The pre-included Unattend.xml file that snf-image will by default install on the VM’s hard disk is this one:

<?xml version="1.0" encoding="utf-8"?>
<unattend xmlns="urn:schemas-microsoft-com:unattend">
    <settings pass="specialize">
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DisableAutoDaylightTimeSet>false</DisableAutoDaylightTimeSet>
            <TimeZone>@TIMEZONE@</TimeZone>
            <ComputerName>*</ComputerName>
            <RegisteredOrganization>Microsoft</RegisteredOrganization>
            <RegisteredOwner>AutoBVT</RegisteredOwner>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <DisableAutoDaylightTimeSet>false</DisableAutoDaylightTimeSet>
            <TimeZone>@TIMEZONE@</TimeZone>
            <ComputerName>*</ComputerName>
            <RegisteredOrganization>Microsoft</RegisteredOrganization>
            <RegisteredOwner>AutoBVT</RegisteredOwner>
        </component>
        <component name="Microsoft-Windows-Deployment" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <RunSynchronous>
                <RunSynchronousCommand wcm:action="add">
                    <Description>Enable Automatic Updates</Description>
                    <Order>1</Order>
                    <Path>cmd /C reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v AUOptions /t REG_DWORD /d 4 /f</Path>
                </RunSynchronousCommand>
                <RunSynchronousCommand wcm:action="add">
                    <Description>Supress local user account setup</Description>
                    <Order>2</Order>
                    <Path>cmd /C reg add HKLM\Software\Microsoft\Windows\CurrentVersion\Setup\OOBE /v UnattendCreatedUser /t REG_DWORD /d 1 /f</Path>
                </RunSynchronousCommand>
            </RunSynchronous>
        </component>
        <component name="Microsoft-Windows-Deployment" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <RunSynchronous>
                <RunSynchronousCommand wcm:action="add">
                    <Description>Enable Automatic Updates</Description>
                    <Order>1</Order>
                    <Path>cmd /C reg add "HKEY_LOCAL_MACHINE\SOFTWARE\Microsoft\Windows\CurrentVersion\WindowsUpdate\Auto Update" /v AUOptions /t REG_DWORD /d 4 /f</Path>
                </RunSynchronousCommand>
                <RunSynchronousCommand wcm:action="add">
                    <Description>Supress local user account setup</Description>
                    <Order>2</Order>
                    <Path>cmd /C reg add HKLM\Software\Microsoft\Windows\CurrentVersion\Setup\OOBE /v UnattendCreatedUser /t REG_DWORD /d 1 /f</Path>
                </RunSynchronousCommand>
            </RunSynchronous>
        </component>
    </settings>
    <settings pass="oobeSystem">
        <component name="Microsoft-Windows-International-Core" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <InputLocale>en-US</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
        <component name="Microsoft-Windows-International-Core" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <InputLocale>en-US</InputLocale>
            <SystemLocale>en-US</SystemLocale>
            <UILanguage>en-US</UILanguage>
            <UILanguageFallback>en-US</UILanguageFallback>
            <UserLocale>en-US</UserLocale>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="amd64" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <Display>
                <ColorDepth>24</ColorDepth>
                <HorizontalResolution>1024</HorizontalResolution>
                <VerticalResolution>768</VerticalResolution>
            </Display>
            <OOBE>
                <HideEULAPage>true</HideEULAPage>
                <ProtectYourPC>1</ProtectYourPC>
                <NetworkLocation>Other</NetworkLocation>
            </OOBE>
        </component>
        <component name="Microsoft-Windows-Shell-Setup" processorArchitecture="x86" publicKeyToken="31bf3856ad364e35" language="neutral" versionScope="nonSxS" xmlns:wcm="http://schemas.microsoft.com/WMIConfig/2002/State" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
            <Display>
                <ColorDepth>24</ColorDepth>
                <HorizontalResolution>1024</HorizontalResolution>
                <VerticalResolution>768</VerticalResolution>
            </Display>
            <OOBE>
                <HideEULAPage>true</HideEULAPage>
                <ProtectYourPC>1</ProtectYourPC>
                <NetworkLocation>Other</NetworkLocation>
            </OOBE>
        </component>
    </settings>
    <cpi:offlineImage cpi:source="catalog:d:/sources/install_windows server 2008 r2 serverstandard.clg" xmlns:cpi="urn:schemas-microsoft-com:cpi" />
</unattend>

The file above is expected to work with all releases (Server or Desktop) of Microsoft Windows starting from version 6.1. The @TIMEZONE@ containers you see are replaced by snf-image with the value of the WINDOWS_TIMEZONE configuration variable during the deployment. Also, as you may have already noticed, all component elements of the pre-included Unattend.xml are in 2 copies. One for processorArchitecture amd64 and one for x86. This is done in order to support both 32-bit and 64-bit versions of Windows using only one file.

The table below lists the releases the developers have confirmed it to work with:

Version Marketing name Editions
6.1 Windows 7 Professional, Ultimate
Windows Server 2008 R2 Datacenter
6.2 Windows 8 Professional
Windows Server 2012 Datacenter
6.3 Windows 8.1 Professional
Windows Server 2012 R2 Datacenter

Nevertheless, the user may want to use a custom Unattend.xml file that better fits his needs. To do so, he can either use the os_answer_file OS parameter during the deployment to specify one or put his copy of the file in the root directory of the image’s %SystemDrive%. snf-image will not install an Unattend.xml file if it is already present in the image, unless IGNORE_UNATTEND image property is defined.

Windows Legacy Deployment

For Windows OSes prior to Windows Vista, the System Preparation Tool (Sysprep) utility is completely different than the one used on later versions. It is not installed with the OS itself. It’s located in the Windows product CD in the /Sypport/Tools/Deploy.cab file and needs to be extracted under the %SYSTEMDRIVE%\Sysprep folder by the user. The commands that needs to be executed in order to put the system in a state ready for cloning looks like this one:

Sysprep -mini -quiet

The answer file which is typically called Sysprep.inf has an INI format. The one snf-image hosts and will install under C:\Sysprep\Sysprep.inf is this one:

;SetupMgrTag
[Unattended]
    OemSkipEula=Yes
    InstallFilesPath=C:\sysprep\i386

[GuiUnattended]
    OEMSkipRegional=1
    TimeZone=@TIMEZONE_INDEX@
    OemSkipWelcome=1

[UserData]
    FullName=User
    OrgName=GRNET
    ComputerName=*

[SetupMgr]
    DistFolder=C:\windist
    DistShare=windist

[Identification]
    JoinWorkgroup=WORKGROUP

[Networking]
    InstallDefaultComponents=Yes

[sysprepcleanup]

The @TIMEZONE_INDEX@ container will be replaced during the installation of the file by the index value of the time zone which is specified through the WINDOWS_TIMEZONE configuration variable. For more info check here. This file is known to work with Windows XP and in order for the deployment to be completely unattended, the user must provide a Windows Product key using the os_product_key OS parameter snf-image offers. As with the newer Windows versions, the user is free to use his custom copy of sysprep.inf either by putting it inside the image (under C:\Sysprep\Sysprep.inf) or by using the os_answer_file OS parameter.

Progress Monitoring Interface

snf-image has an embedded mechanism for transmitting progress messages during an image deployment. A user may specify an external executable by overwriting the PROGRESS_MONITOR variable under /etc/default/snf-image and snf-image will redirect the progress messages to the standard input of this program. In this section we will describe the format and the fields of the progress messages.

The progress messages are JSON strings with standardized fields. All messages have a type field whose value is a string and a timestamp field whose value is a floating point number referring to a time encoded as the number of seconds elapsed since the epoch. The rest of the field depend on the specific type.

image-info

This message type is used to display arbitrary progress information. It has an extra messages field whose value is a list of strings. A valid image-info message looks like this:

{"messages": ["Starting image copy..."], "type": "image-info", "timestamp": 1378914866.209169}

image-error

This message type is used to display a fatal error that occurred during image deployment. It may either have an extra messages field to display the error message or an stderr field to display the last lines of the standard error output stream of the OS creation script. Valid image-error messages look like this:

{"messages": ["Image customization failed."], "type": "image-error", "timestamp": 1379507045.924449}

image-copy-progress

One of the tasks snf-image has to accomplish is to copy the image file into the VM’s hard disk before configuring it. Messages of type image-copy-progress are used to display the progress of this task. The extra fields this message type has is position, total and progress. The position field is used to display the number of bytes written to the hard disk. The total field indicates the overall size (in bytes) of the image, and finally the progress field indicates the percent of the accomplished work. Messages of this type look like this:

{"position": 335547996, "total": 474398720, "type": "image-copy-progress", "timestamp": 1378914869.312985, "progress": 70.73}

image-helper

This is a family of messages that are created when snf-image-helper runs. Each message of this type has a subtype field.

task-start

Messages with subtype task-start indicate that snf-image-helper started running a configuration task on the image. Messages of this type have an extra task field whose value is the name of the task snf-image-helper started, and look like this:

{"subtype": "task-start", "task": "FixPartitionTable", "type": "image-helper", "timestamp": 1379507040.456931}

task-stop

Messages with subtype task-stop are produced every time a configuration task successfully exits. As with the task-start messages, the task field is present:

{"subtype": "task-end", "task": "FixPartitionTable", "type": "image-helper", "timestamp": 1379507041.357184}

warning

This messages are produced to display a warning. The actual warning message itself is present in the messages field:

{"subtype": "warning", "type": "image-helper", "messages": ["No swap partition defined"], "timestamp": 1379075807.71704}

error

The last image-helper message that may occur is the error message. As with the image-error messages, either a messages field that hosts the actual error message or a stderr field that hosts the last 10 lines of the standard error output stream of snf-image-helper. Valid error messages look like this:

{"subtype": "error", "type": "image-helper", "messages": ["The image contains a(n) MSDOS partition table.  For FreeBSD images only GUID Partition Tables are supported."], "timestamp": 1379507910.799365}

Configuration Tasks Enviroment

When an snf-image-helper configuration task runs, it expects to find the required information in its environment. In the table below we describe the environment variables that are present when the configuration tasks run.

Name [1] Details
DEV_COUNT The number of the instance’s disks.
DEV_%N The device file of the Nth disk.
DEV The device file of the first disk (we keep this for backward compatibility)
PERSONALITY The value of the img_personality OS parameter.
HOSTNAME The instance’s name.
PASSWD The value of the img_passwd OS parameter.
PASSWD_HASH The value of the img_passwd_hash OS parameter.
OS_PRODUCT_KEY The value of the os_product_key parameter.
PROPERTY_* The value of a specific image property that was specified in json through the img_properties OS parameter.
RESIZE_PART The number of the partition that will be enlarged.
TARGET The directory the instance’s file systems are mounted under.
NIC_COUNT The number of network interface controllers of the instance.
NIC_%N_* The Ganeti provided environment variable for the Nth network interface controller. Check here
DHCP_TAGS The value of the DHCP_TAGS configuration parameter (see Configuration Parameters)
STATEFUL_DHCPV6_TAGS The value of the STATEFUL_DHCPV6_TAGS configuration parameter (see Configuration Parameters)
STATELESS_DHCPV6_TAGS The value of the STATELESS_DHCPV6_TAGS configuration parameter (see Configuration Parameters)
UNATTEND The path to the Windows setup answer file (Unattend.xml)
SYSPREPINF The path to the Windows legacy setup answer file (sysprep.inf)
[1]all environment variable names are prefixed with SNF_IMAGE_

Overwriting Configuration Tasks

One way to extend snf-image to work on images that are not directly supported by it, is the overwriting mechanism for the configuration tasks. If the ALLOW_MOUNTED_TASK_OVERWRITING image property is defined with yes, then the presence of an executable under the path /root/snf-image/helper/overwrite_task_<TASK> inside the image’s file system will make snf-image-helper interrupt the execution of the actual configuration task and the image’s file will be executed instead. This executable should make use of the Configuration Tasks Environment. Also keep in mind that this only works for configuration tasks that run while the image is mounted (have a priority between 31 and 79).

Return Code and post execution

If the return code of the execution of the image-defined executable is non-zero, snf-image will translate this as a failure of the configuration task. The only exception to this is the return code 101, which will make snf-image execute the original configuration task and then the image’s executable once again. The first time the image’s executable will be called with pre-exec as its first argument and the second with post-exec. This way the executable itself can decide if it is going to be executed in conjunction with the original configuration task and in which order (before, after or in both cases).

Footnotes

[2]http://technet.microsoft.com/en-us/library/hh824938.aspx