Fedora Linux Migration Part 3: GParted Magic and Encrypted Home

Posted in Technology at 22:55:42 by Jeff Hardy

GParted is among the most well-known and respected open-source partition editors. From creation to deletion, resizing, copying, and moving, it is a tool built to manage the full gamut of disk partitionining operations.

In part one of our migration process, we cloned the installation to a new drive in new hardware. In part two, we moved to a LUKS-encrypted root. Now we have a few adjustments we wish to make to our partitioning scheme, regaining space so that we may move the rest of the system beneath the umbrella of encryption.

The Plan

We have the following arrangement on the system drive:

/dev/sda (150GB)
     sda1: ext3 /boot (300MB)
     sda2: ext3 oldroot (38GB)
     sda3: lvm Volume00 (112GB)
          Volume00-home: ext3 /home (70GB)
          Volume00-swap: swap (2GB)
          Volume00-root: LUKS (40GB)
               luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34: / ext3 (40GB)

Despite our efforts thus far, the issue remains that our home volume is not yet protected by encryption. We can tackle this two ways, either by building a new home crypto-volume and copying data as we did for root, or by simply reclaiming space enough to abandon a separate home volume and copy that data beneath the already encrypted root. I elect for the latter, mainly because the use of a single crypto-volume avoids the need for multiple pass-phrase entry at boot, and also because there is no compelling reason for a separate home on this particular system.

In order to gain the space necessary, we will remove the old unused root, add that space for use by LVM, grow our new crypto-root, and then copy the data from home. This is no big deal, and the partition is easily deleted with fdisk or gparted. But there is one issue. Because this is not the last partition on the disk, resizing for our needs is not as simple as an fdisk delete and recreate, moving the end boundary to the end of the drive. Now we need to move the start of the partition. Enter GParted.


While GParted can be run at any time, some functionality for a given storage device may not be available, nor entirely safe, if it is in-use (mounted) by the operating system. For this reason, we elect to boot the system with the GParted Live CD.

NOTE: Be aware that there is a perfectly good alternative to partition resizing of any kind. We could instead elect to create an LVM partition using the unallocated space and do standard LVM operations to incorporate it into our root logical volume (pvcreate, vgextend, lvextend). But we continue with full confidence in GParted, the main benefit being mostly cosmetic in the end: a tidy partition table.

Booted into the live environment, we start GParted. After deleting the old root partition at /dev/sda2, we are left with a good illustration of the situation at hand. We have a region of unallocated space before the LVM partition:

We need to resize/move the sda3 partition, which contains our priceless crypto-LVM/root within it:

The resize dialog gets right to the point. Adjusting the amount of space before and after the partition moves its start and end positions on disk:

Drag the start boundary all the way to the left (or enter 0) to effect a move/resize, absorbing every bit of the unallocated space before the partition:

Operation pending. We can still back out without affecting anything. Click apply, and be prepared for a lengthy wait.

When all is said and done, we have moved and resized our LVM partition to utilize the full disk:

Note that used space has not changed. We address that next.

Resize LVM, LUKS, Ext3

The work done so far has resized the outermost container, the partition. Like Russian nesting dolls, we need to grow each in turn before we can address the filesystem. A testament to the tools, we can actually do all of the follwoing booted into the live system.

First LVM:

  PV         VG       Fmt  Attr PSize   PFree 
  /dev/sda3  Volume00 lvm2 a--  148.75g 36.98g

  VG       #PV #LV #SN Attr   VSize   VFree 
  Volume00   1   3   0 wz--n- 148.75g 36.98g

  LV   VG       Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  home Volume00 -wn-ao---- 69.00g                                             
  root Volume00 -wi-ao---- 40.76g                                             
  swap Volume00 -wi-ao----  2.00g

lvresize Volume00/root -l+100%FREE
  Extending logical volume root to 77.75 GiB
  Logical volume root successfully resized

  LV   VG       Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  home Volume00 -wn-ao---- 69.00g                                             
  root Volume00 -wi-ao---- 77.75g                                             
  swap Volume00 -wi-ao----  2.00g

NOTE: I would normally expect to first do a pvresize to grow the PV, but the PV already knew about the extra space, likely a gift from GParted.

Then LUKS:

cryptsetup resize luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34  
echo $?

Finally the filesystem:

df -h /
Filesystem                                             Size  Used Avail Use% Mounted on
/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34   40G   29G  9.2G  76% /

resize2fs /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is mounted on /; on-line resizing required
old_desc_blocks = 3, new_desc_blocks = 5
The filesystem on /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is now 20380160 blocks long.

df -h /
Filesystem                                             Size  Used Avail Use% Mounted on
/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34   77G   29G   45G  40% /

We are left with a nice large root to accommodate our /home data.

Home Copy

With 45GB free in our root, we should have plenty of space after cleaning out some garbage in home:

df -h /home
Filesystem                 Size  Used Avail Use% Mounted on
/dev/mapper/Volume00-home   64G  5.0G   56G   9% /home

A good old rsync will suffice to copy that whopping 5GB of data from the existing home volume to the home within the crypto-root. First things first, bug out to runlevel 3 (or single/rescue mode for overkill) to ensure home is not in use so we can mount it somewhere else:

init 3
umount /home
mkdir /oldhome
mount /dev/mapper/Volume00-home /oldhome

Finally rsync the data:

rsync -a /oldhome/ /home

When done, remember to pound out the /home mount from /etc/fstab lest we accidentally mount our old unencrypted home volume on next boot:

#/dev/Volume00/home	/home			ext3	defaults	1 2

A reboot confirms that everything is working as it should.

LVM Cleanup

Almost half the disk is tied up in our abandoned home. Once happy with the new arrangement, we can easily reclaim that space starting at the LVM layer, and then working our way up. This is just a second run at container resizing as above, LVM, LUKS, Ext3, all live.

  LV   VG       Attr       LSize  Pool Origin Data%  Move Log Cpy%Sync Convert
  home Volume00 -wn-a----- 69.00g                                             
  root Volume00 -wi-ao---- 77.75g                                             
  swap Volume00 -wi-ao----  2.00g

lvremove Volume00/home
Do you really want to remove active logical volume home? [y/n]: y
  Logical volume "home" successfully removed

  VG       #PV #LV #SN Attr   VSize   VFree 
  Volume00   1   2   0 wz--n- 148.75g 69.00g

lvextend -L+50G Volume00/root
  Extending logical volume root to 127.75 GiB
  Logical volume root successfully resized

cryptsetup resize luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34  
echo $?

resize2fs /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is mounted on /; on-line resizing required
old_desc_blocks = 5, new_desc_blocks = 8
The filesystem on /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is now 33487360 blocks long.

df -h /
Filesystem                                             Size  Used Avail Use% Mounted on
/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34  126G   36G   85G  30% /

NOTE: With ample disk available, we left some space unallocated in the VG. This is just a hedge for flexibility. If we need more space in the future, we can always add it live just as we did here.

Partition Re-Numbering

Our partition table consists of the following, as revealed by fdisk:

Device    Boot     Start       End    Blocks  Id System
/dev/sda1 *           63    619919    309928+ 83 Linux
/dev/sda3         620544 312580095 155979776  8e Linux LVM

While it is no problem whatsoever to be “missing” a partition in sequence, we continue the intrepid pursuit of the trivial and wish to renumber sda3 to sda2. The easiest way to rectify this is to simply delete and re-create the partition using fdisk. Since it is the last partition, fdisk will normally default to the same start and end sectors when creating the new one (something to double-check). While this can be done live followed by an immediate reboot, it is safest to do this from rescue mode with the disk not in use.

fdisk /dev/sda
ddelete, partition number 3
new, primary, partition number 2, first sector default, last sector default
type, partition number 2, 8e

Reboot. Just a reminder, this is completely unnecessary. :-)

Everything Encrypted

We leave with this on the system disk:

/dev/sda (150GB)
     sda1: ext3 /boot (300MB)
     sda2: lvm Volume00 (150GB)
          Volume00-swap: swap (2GB)
          Volume00-root: LUKS (127GB)
               luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 ext3 (127GB)
          unallocated: (19GB)

A tidy partition and volume layout with everything encrypted (almost).


Working Dracut

Posted in Technology at 23:23:55 by Jeff Hardy

Dracut is an initramfs infrastructure, as well as the name of tool needed to create initramfs images. It is also the name of the prompt/environment made available at boot time by choice or by error, allowing for troubleshooting of boot-time problems.

Building Initramfs Images

It always pays to make backups:

cd /boot
cp -a initramfs-3.12.8-300.fc20.i686.img initramfs-3.12.8-300.fc20.i686.img.backup

The dracut tool is used to build initramfs images:

dracut -f /boot/initramfs-3.12.8-300.fc20.i686.img 3.12.8-300.fc20.i686

Images are best built from within the intended root of the Linux system, and the tool can be run on a live system with no issues. If the system is in a state where this is unattainable, it is possible to build a clean environment by assembling the necessary mounts in rescue mode. Do not select “find drives automatically,” we will do so manually:

mkdir /mnt/sysimage
vgchange -ay
mount /dev/mapper/Volume00-root
mount /dev/sda1 /mnt/sysimage/boot
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
chroot /mnt/sysimage

NOTE: Devices holding boot and root will vary from system to system.

Now run dracut.

The Dracut Shell

If the boot process fails due to an issue with the initial ram disk, we will likely be dropped to the dracut shell (this may require adding rd.shell as a kernel argument).

For example, failure due to a missing volume:

Warning: Could not boot.
Warning: /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 does not exist

The beauty of the dracut prompt is that we have nearly the full flexibility of Linux at our disposal, with whatever tools exist in the initramfs image.

This particular error is something I have commonly experienced when switching root, and is probably due to the systemd setup within the initramfs specifically looking for a device that did not exist when we created it (for instance, if we ran dracut while not within the final resting place of root).

We can see what is actually contained within the initrams image:

lsinitrd /boot/initramfs-3.12.8-300.fc20.i686.img

If a volume cannot be found, it is likely there is no entry for it in etc/systemd/system/initrd.target.wants. We can attempt to urge things along manually:

lvm vgchange -ay
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
Enter passphrase for /dev/mapper/Volume00-root:

If this now boots the system, the path forward is to re-generate the initramfs image correctly, which usually entails doing so from within the appropriate root (as above).

Finally, here is a tidbit showing what kinds of things can be done for gathering intelligence for later followup:

debug=$(lvm pvdisplay; lvm vgdisplay; lvm lvdisplay; blkid; dmesg)
lvm vgchange -a y
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
Enter passphrase for /dev/mapper/Volume00-root:
mount /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /mnt
echo "$debug" > /mnt/dracut.debug

NOTE: All this does is set an environment variable “debug” to the output of a bunch of commands, then activate storage somewhere to store that output, in this case showing how it is even possible to use a crypto-volume. Perhaps it is USB or simple volume instead.


  • http://wiki.centos.org/TipsAndTricks/CreateNewInitrd
  • http://lists.centos.org/pipermail/centos/2013-March/133604.html

Working GRUB

Posted in Technology at 23:22:23 by Jeff Hardy

GNU GRUB is the bootloader of choice for most Linux distributions, offering massive flexibility to boot a multitude of operating systems. With GRUB version 2, configuration remains straightforward, but can be largely auto-generated with modern tools. It also retains the full flexibility to modify the boot configuration at run-time.


It always pays to make backups:

cd /boot/grub2
cp -a grub.cfg grub.cfg.backup

In order to generate grub2 configuration:

grub2-mkconfig -o /boot/grub2/grub.cfg

The tool will probe the system looking for partitions containing vmlinuz and initramfs images, and build configuration appropriately. Administrator tunables that affect the process are stored in /etc/default/grub. Example config:

GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
# GRUB_TERMINAL="serial console"
# GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600"

Configuration is best built from within the intended root of the Linux system, and the tool can be run on a live system with no issues. If the system is in a state where this is unattainable, it is possible to build a clean environment by assembling the necessary mounts in rescue mode. Do not select “find drives automatically,” we will do so manually:

mkdir /mnt/sysimage
vgchange -ay
mount /dev/mapper/Volume00-root
mount /dev/sda1 /mnt/sysimage/boot
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
chroot /mnt/sysimage

NOTE: Devices holding boot and root will vary from system to system.

Now run grub2-mkconfig.

Run-Time Changes

Hitting “e” at the grub menu allows one to peruse and change configuration. This is probably most commonly used to alter kernel command-line arguments, such as adding “single” to boot to single-mode, but any aspect of the configuration can be changed before boot.

For example, we may wish to attempt to boot an alternate root volume. While in this mode, we need simply find the kernel we wish to boot and alter its command-line arguments, one being the selection of the intended root:

linux /vmlinuz-3.12.8-300.fc20.i686 ro root=/dev/sda2

NOTE: Add/remove other arguments such as quiet, rhgb, or rd.* as necessary.

We may also have a need to load an alternate initramfs image:

initrd  /initramfs-3.12.8-300.fc20.i686.img.backup

When satisfied with the changes, hit F10 to boot.

Grub Prompt

It is actually entirely possible to attempt boot with no grub configuration at all, whether dropping voluntarily or involuntarily to the grub prompt. Given all the same assumptions:

grub> ls
(hd0) (hd0,1) (hd0,2) (hd0,3)
grub> set root=(hd0,1)
grub> linux /vmlinuz-3.12.8-300.fc20.i686 ro root=/dev/sda2
grub> initrd /initramfs-3.12.8-300.fc20.i686.img.20140608
grub> boot

NOTE: The partition nomenclature is different in grub, but we are specifying the partition containing vmlinuz and initrd images (eventually known as sda1 and mounted at /boot by linux). Also note that tab-completion thankfully works.


  • http://tuxers.com/main/instigating-a-manual-boot-from-the-grub-prompt

Fedora Linux Migration Part 2: Switch to Encrypted Root

Posted in Technology at 23:21:07 by Jeff Hardy

Full-disk encryption is essentially a requirement for a mobile OS. While there are a few choices for full block device encryption on Linux (such as Truecrypt), at this time the clear choice is Linux Unified Key Setup. LUKS integrates with dm-crypt as the encryption backend, producing a device mapper entry for the decrypted block device that can be used like any other.

In part one of our Fedora migration odyssey, we cloned the installation to a laptop. Now in part two, we will shift things around to move to a LUKS-encrypted root. What follows assumes that we are using the new laptop for all operations, and that our cloned system drive is the only one in place (sda). While everything can be done in rescue mode, some operations require it and will be noted.

The Plan

We have the following partition and volume layout:

/dev/sda (150GB)
     sda1: ext3 /boot (300MB)
     sda2: ext3 / (38GB)
     sda3: lvm Volume00 (112GB)
          Volume00-home: ext3 /home (70GB)
          Volume00-swap: swap (2GB)
          unallocated: (40GB)

The official documentation indicates that there is no way to encrypt-in-place and thereby preserve data (although tools such as luksipc suggest otherwise). Regardless, we will use that unallocated space, perfectly sized, to create a new root filesystem on top of a crypto-volume using LVM/LUKS, and then copy data. A move provides a secondary benefit here: it will fold root back into LVM, its excursion to a bare partition being only ever a mere workaround.

Last but not least, a copy leaves our existing root undisturbed as a fallback in the event of problems.

Create a New Crypto Volume

First we create an LVM volume using all of that free space:

  VG #PV #LV #SN Attr VSize VFree
  Volume00 1 2 0 wz--n- 111.76g 40.76g

lvcreate -n root -l+100%FREE Volume00

  VG #PV #LV #SN Attr VSize VFree
  Volume00 1 3 0 wz--n- 111.76g 0

  LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
  home Volume00 -wn-ao---- 69.00g
  root Volume00 -wi-ao---- 40.76g
  swap Volume00 -wi-ao---- 2.00

Rather than put a filesystem directly on the new volume, we will set up for encryption. For the best encryption results, it is recommended to write random data to the device:

dd if=/dev/urandom of=/dev/mapper/Volume00-root
dd: writing to ‘/dev/mapper/Volume00-root’: No space left on device
85483521+0 records in
85483520+0 records out
43767562240 bytes (44 GB) copied, 11330.7 s, 3.9 MB/s

NOTE: This will take some time.

Encrypt the volume:

cryptsetup --verbose --verify-passphrase luksFormat /dev/mapper/Volume00-root

This will overwrite data on /dev/mapper/Volume00-root irrevocably.

Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
Command successful.

NOTE: Remember that passphrase. If lost, data will be irrecoverable.

Review the encryption summary for the device, noting the UUID:

cryptsetup luksDump /dev/mapper/Volume00-root
cryptsetup luksDump /dev/mapper/Volume00-root | grep UUID
UUID:          	cc7e78ee-87a9-4ad0-9c82-31de01389b34

Open the crypto-volume. The last argument specifies the device-mapper name by which this volume will be opened:

cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34

NOTE: Naming by UUID is how Fedora sets these volumes up when creating them at install time, practice we will adhere to later when setting up the new root. For consistency’s sake, we do so now, but we could just as easily call it “luks-root” or some such for convenience at this stage.

Create a filesystem:

mkfs -t ext3 /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34

Copy Data

At this point, we should boot to rescue mode if not already. Because we are copying the existing root volume, we cannot afford to have it mounted at / where it could change. A read-only remount could suffice, but is likely to cause issues on a live system. Additionally, though we tell rsync not to cross filesystem boundaries anyway (-x), it is best reduce to zero the risk of copying special fileystems or other partitions that may be mounted beneath root.

We mount the source and destination, and rsync the contents:

mkdir /oldroot
mount /dev/sda2 /oldroot
mkdir /newroot
vgchange -ay
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
mount /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /newroot
rsync -avx /oldroot/ /newroot

Requirements for Switching to a New Root

There are three elements necessary for successful switch to a new root, be it partition, lvm, crypto-volume, floppy, or who knows what else. In the order they are encountered in the boot process:

  1. GRUB: The bootloader needs to be configured properly to load the appropriate kernel executable with necessary arguments indicating where to find the new root.
  2. Initial RAM Disk: The initramfs image (or older initrd) needs to be built with enough utility so that the kernel can make use of it to find the real root filesystem. It may or may not already be capable, depending on suitable kernel arguments specified in grub.
  3. fstab: Configuration in /etc/fstab must be set to mount the new root at / with suitable mount options.

It is certainly possible to attempt to fight through each stage of the boot process, making run-time manual changes to grub and the dracut environments in order to find one’s way into the new root, confirming it works before making configuration changes. This entails having a decent amount of intelligence about how various rd.* kernel arguments will relate to one’s system. This is definitely a worthy endeavor and is certainly the approach to take when troubleshooting. But in the end, we will proceed directly to making the system changes for the new root, first making backups of the involved files, thereby allowing us to use the original root as a fallback without too much trouble.

Configure the System for a New Root

The modern tools necessary to build the bootloader config (grub2-mkconfig) and initramfs image (dracut), do a wonderful job of auto-configuring themselves. This is both a blessing and a curse, as we need to carefully establish an environment from which they can reliably draw their bits. This means running them from within the new root for best results, necessitating a boot to rescue mode.

Boot to rescue mode and make sure not to select “find drives automatically.” Instead, we will assemble what we need manually. The following is the surest path I have found to setup an environment for building clean grub configuration and initramfs image:

mkdir /mnt/sysimage
vgchange -ay
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
Enter passphrase for /dev/mapper/Volume00-root:
mount /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /mnt/sysimage
mount /dev/sda1 /mnt/sysimage/boot
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
chroot /mnt/sysimage

NOTE: If this was not a crypto-volume, the procedure still stands, mounting the LVM volume directly.

We have assembled the system from new root up, and are now prepared to re-configure and rebuild.

Make backups of the initramfs image and grub configuration:

cd /boot
cp -a initramfs-3.12.8-300.fc20.i686.img initramfs-3.12.8-300.fc20.i686.img.20140608
cd /boot/grub2
cp -a grub.cfg grub.cfg.20140608

NOTE: Grub configuration problems are relatively easy to work around at boot time, especially with a backup of the initramfs. Initramfs-specific problems less so. The initramfs may also be difficult to restore to original state depending on the nature of the problem. In other words, make sure to back up the initramfs.

Build the new initramfs image:

dracut -f /boot/initramfs-3.12.8-300.fc20.i686.img 3.12.8-300.fc20.i686

Alter grub configuration at /etc/default/grub, adding an argument informing our kernel that it should open our new crypto-volume:

GRUB_CMDLINE_LINUX="quiet rhgb rd.luks.uuid=cc7e78ee-87a9-4ad0-9c82-31de01389b34"

NOTE: Multiple crypto-volumes can be listed here, each resulting in a password prompt at boot time.

Build the new grub config:

grub2-mkconfig -o /boot/grub2/grub.cfg

NOTE: We should see entries for our old root as well, from the os-prober stage of grub2-mkconfig, though they may not work without some urging at the dracut prompt during boot.

This should produce a grub.cfg entry containing a kernel line reflecting our changes:

linux   /vmlinuz-3.12.8-300.fc20.i686 root=/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 ro quiet rd.luks.uuid=cc7e78ee-87a9-4ad0-9c82-31de01389b34

Finally, alter fstab to mount our new root volume:

/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34	/	ext3	defaults,user_xattr 1 1

NOTE: Over the years, distros have moved from mounting by device to mounting by label or UUID, or at least device-mapper name, since devices can change. We could do this here as well, for example mounting by the UUID of the filesystem within our crypto volume (UUID=551227f2-7ea4-4c5d-8bed-a5bff1d7354f). Fedora seems to mount crypto volumes by device-mapper name and so shall we.

Crypttab? It is possible to create an /etc/crypttab file containing an entry like this:

luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /dev/mapper/Volume00-root none

A stock Fedora install on a crypto-root does not create this file, but its existence may (or may not) eliminate the need to specify rd.luks.uuid on the kernel command-line, among other things. I did not create this file on my system. Your mileage may vary.

Boot the New Root

At this point, we can boot into our new crypto-root. Following the above procedure should drastically reduce the chances of problems, but if they do occur, we are well-prepared to troubleshoot and recover with the knowledge we have gained and the backups we have made (see Working GRUB and Working Dracut).

We are left with this on the system drive:

/dev/sda (150GB)
     sda1: ext3 /boot (300MB)
     sda2: ext3 / (38GB)
     sda3: lvm Volume00 (112GB)
          Volume00-home: ext3 /home (70GB)
          Volume00-swap: swap (2GB)
          Volume00-root: LUKS (40GB)
               luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 ext3 (40GB)

Ultimately, a successful move to a new crypto-root. We still have other unencrypted volumes which which we may soon contend.


Fedora Linux Migration Part 1: Hardware Migration

Posted in Technology at 23:15:04 by Jeff Hardy

My ten year Fedora upgrade streak, now at 20, is still running on hardware that is nearly as old and I thought it time to move the install to a (slightly) newer laptop that I am using more frequently. This will involve a number of storage operations and system re-configuration, new concerns such as encryption, and more to come. Cloning an install and re-fashioning it in such a way will indeed be more work than a fresh install and copy, but imposes some interesting obstacles to work around, and remains within the spirit of maintaining a legacy install.

Here in part one, we concentrate on merely getting the system onto a new drive, and booting the new hardware. This will merge the data from two drives in its current desktop environment, to a single new laptop drive. What follows assumes that cloning, partitioning, and LVM operations are conducted with the old and new storage accessed from a Linux system in an offline manner. Single or rescue mode are not required as long as we can access the block devices, perhaps connected as secondary PATA/SATA or USB drives. In no case should they be mounted anywhere.

Dump the Drive Images

There are certainly more exotic tools for disk cloning, but let us stick to the good old UNIX standby ‘dd.’ We can be tricky and dd over a network pipe directly to a destination drive, but with storage available (in this case external USB mounted at /storage), we get a safe offline copy/backup of the original drives:

dd if=/dev/sda of=/storage/fritzdesk_sda
dd if=/dev/sdb of=/storage/fritzdesk_sdb

The partition info on sda, revealing our /boot and / partitions:

fdisk -l fritzdesk_sda

Disk fritzdesk_sda: 37.3 GiB, 40037760000 bytes, 78198750 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa5cba5cb

Device Boot Start End Blocks Id System
fritzdesk_sda1 * 63 619919 309928+ 83 Linux
fritzdesk_sda2 619920 78185519 38782800 83 Linux

The partition info on sdb, revealing our LVM partition containing logical volumes for home and swap:

fdisk -l fritzdesk_sdb

Disk fritzdesk_sdb: 74.5 GiB, 80026361856 bytes, 156301488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x59d50c69

Device Boot Start End Blocks Id System
fritzdesk_sdb1 * 63 156296384 78148161 8e Linux LVM

Ultimately, we have this from our source system:

/dev/sda (40GB)
     sda1: ext3 /boot (300MB)
     sda2: ext3 / (38GB)
/dev/sdb (72GB)
     sdb1: lvm Volume00 (72GB)
          Volume00-home: ext3 /home (70GB)
          Volume00-swap: swap (2GB)

Clone to the New Drive

We will combine the partitions from these two source drives into the new 150GB laptop drive. No matter how the new drive is connected (direct PATA/SATA or USB), we will assume the new drive is /dev/sdc for clarity.

First a clone of sda, as that contains our MBR, boot, and root:

dd if=/storage/fritzdesk_sda of=/dev/sdc

Check partitions on the new drive:

fdisk -l /dev/sdc
Disk /dev/sdc: 149.1 GiB, 160041885696 bytes, 312581808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa5cba5cb

Device Boot Start End Blocks Id System
/dev/sdc1 * 63 619919 309928+ 83 Linux
/dev/sdc2 619920 78185519 38782800 83 Linux

We now need to create an LVM partition to hold the contents of sdb:

fdisk /dev/sdc
new, primary, partition number 3, first sector default, last sector default
type, partition number 3, 8e

Check partitions on the new drive:

fdisk -l /dev/sdc

Disk /dev/sdc: 149.1 GiB, 160041885696 bytes, 312581808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa5cba5cb

Device Boot Start End Blocks Id System
/dev/sdc1 * 63 619919 309928+ 83 Linux
/dev/sdc2 619920 78185519 38782800 83 Linux
/dev/sdc3 78186496 312581807 117197656 8e Linux LVM

We are interested in the one LVM partition inside the fritzdesk_sdb image rather than the entire drive. We must expose it with kpartx:

kpartx -av /storage/fritzdesk_sdb
add map loop0p1 (253:3): 0 156296322 linear /dev/loop0 63

Scan for the new LVM physical volume just exposed to ensure everything was found:

  PV /dev/mapper/loop0p1 VG Volume00 lvm2 [74.52 GiB / 3.52 GiB free]

NOTE: It is good practice to ensure unique VG names when doing system installs, lest a generic name like Volume00 collide with another, such as when used in this way on another system. Luck is on our side.

Clone the exposed source LVM partition to the new LVM partition, physical volume to physical volume:

dd if=/dev/mapper/loop0p1 of=/dev/sdc3

Claim New LVM PV Space

This last clone was of a smaller LVM partition image into a much larger LVM partition on the new drive. We will want to claim that space for future LVM use. The pvresize command will claim all space up to the partition or block device boundary (now further out) unless otherwise specified:

pvresize /dev/sdc3
  PV         VG       Fmt  Attr PSize   PFree
  /dev/sdc3  Volume00 lvm2 a--  111.76g 40.76g
  VG       #PV #LV #SN Attr   VSize   VFree
  Volume00   1   2   0 wz--n- 111.76g 40.76g

NOTE: This may need to be prefaced with a pvscan to find the physical volume.

Now we have an additional 40.76GB in the LVM volume group to grow or create new logical volumes.

Boot the New Machine

At this point, we have directly cloned the system to a new drive with the fewest changes possible. With that drive physically installed as the sole drive in the new system, we boot to ensure all pieces are present, troubleshooting any startup issues and checking things before introducing any more complexity. For example, there is trouble starting X11, and a boot to single mode and check of the X11 logs indicates that there is a need to install xorg-x11-drv-intel. Other minor issues involve installation of xorg-x11-drv-synaptics for full touchpad functionality. Things of this nature are not surprising given a move to new hardware.

We are left with this on the destination drive (now /dev/sda in-use in the new system):

/dev/sda (150GB)
     sda1: ext3 /boot (300MB)
     sda2: ext3 / (38GB)
     sda3: lvm Volume00 (112GB)
          Volume00-home: ext3 /home (70GB)
          Volume00-swap: swap (2GB)
          unallocated: (40GB)

Ultimately, a simple, successful clone of fritzdesk to fritzlap. We have left some room to maneuver, which we will soon need. In part 2, we will worry about encryption.


Fedora Upgrade: F19 -> F20

Posted in Technology at 10:03:36 by Jeff Hardy

First ensuring I was running fedup-0.8 to avoid the fedup-0.7 upgrade bug, this Fedora upgrade was off to the races with fedup --network 20. Unfortunately, after rebooting to the upgrade, my machine ended up wedged at the following:

systemd[1]: Failed to initialize SELinux context: No such file or directory

This fedup selinux bug is easily worked around with selinux=0 enforcing=0 on the kernel command-line. Afterwards, the upgrade completed without issue. I am also pleased to see that one particular Pidgin bug that plagued me throughout Fedora 19 seems to be fixed.

Meanwhile, an unrelated fedup upgrade I conducted on a laptop required that I work around some bugs related to finding my encrypted root partition, by specifying rd.luks.uuid=<uuid> in grub. I find myself wondering if we as an industry are making adequate progress reconciling and mitigating the increasing problems that come with increasing technical complexity.


SUNY Federation with SimpleSAMLphp in HA

Posted in Technology at 22:16:49 by Jeff Hardy

SimpleSAMLphp is the ideal choice as a SAML 2.0 IdP for SUNY Federation. As it will be the core authentication mechanism for SUNY services, high-availability and load-balancing are an important consideration. Here is one approach to achieving this with HAProxy and Keepalived.

We intend to have a pair (at least) of Apache servers running identical SimpleSAMLphp installations. These will sit behind a pair of load-balancer machines running HAProxy and Keepalived, the first to balance HTTP traffic to the SAML servers, the second to provide virtual IP failover between the balancers. This forward-facing virtual IP will be stunnel-encryped, and advertised as our SAML Identity Provider (IdP).


Senior sysadmin Greg Kuchyt pioneered our use of these technologies for load-balancing and failover of our 389-DS LDAP services in a manner much like this, an approach that has served us well for many years, and which we have used for other services.

Our SAML setup is based on the document collection in the SUNY IDM Wiki, particularly the excellent Installing SimpleSAMLphp for SUNY Federation guide. Suffice it to say that I will not go into any detail on our IdP setup, instead covering only what SimpleSAMLphp changes are necessary to share session data for load-balancing and failover.

Shared Session Data with Memcache

On the SAML servers, in order to pave the way for balancing load across multiple machines, we need to ensure that SimpleSAMLphp session handling is switched over to memcache:

yum install memcached php-pecl-memcache
service memcached start
chkconfig memcached on

Note that SimpleSAMLphp uses PHP memcache and not the newer PHP memcached, but both use the system memcached daemon.

Firewalls will need to be opened to allow incoming memcache connections on port 11211.

In the SimpleSAMLphp configuration, set the store.type, and then adjust the memcache_store.servers array to taste (see docs). The following will cause all session data to be saved to the memcache servers on both boxes:


'store.type' => 'memcache',
'memcache_store.servers' => array(
                array('hostname' => 'saml1.potsdam.edu'),
                array('hostname' => 'saml2.potsdam.edu'),

Load-Balancing with HAProxy

On the balance servers we first install HAProxy. The following configuration uses the least connection algorithm to balance HTTP sessions between the two SAML servers behind them.


    mode                    http
    log                     global
    option                  httplog
    option                  dontlognull
    option http-server-close
    option forwardfor       except
    option                  redispatch
    retries                 3
    timeout http-request    10s
    timeout queue           1m
    timeout connect         10s
    timeout client          1m
    timeout server          1m
    timeout http-keep-alive 10s
    timeout check           10s
    maxconn                 3000

listen https
    balance leastconn
    option  httpclose
    option  forwardfor
    option  httplog
    option  httpchk         GET /
    server  saml1 saml1.potsdam.edu:80 check inter 5000 downinter 500
    server  saml2 saml2.potsdam.edu:80 check inter 5000 downinter 500

Note that hostnames are used here for clarity, but it is probably preferable to use IP addresses for server config lines. Also note that we bind our instance to localhost:4430, since we do not intend for public connections to hit this port directly. Instead they will hit SSL on port 443 provided by stunnel (later).

Start haproxy:

service haproxy start

We now have two balancers that will each balance HTTP connections to the SAML servers.

Failover with Keepalived

On the balance servers we next install Keepalived. On balance1, the following configuration establishes a virtual IP address to be maintained between the balance servers.


global_defs {
        notification_email {
        notification_email_from devnull@potsdam.edu
        smtp_connect_timeout 30

vrrp_instance VI_1 {
        virtual_router_id 1
        state MASTER
        priority 100
        interface eth2


        authentication {
                auth_type AH
                auth_pass SomeKindofPasswordHere!
        virtual_ipaddress {
       brd dev eth2

On balance2, we install the same configuration with two key differences:

        state BACKUP
        priority 50

Start keepalived:

service keepalived start

We now have a front-facing virtual IP address providing failover between the balancers. You can view the status of the IP with: ip addr show.

SSL with Stunnel

On the balance servers we next install stunnel to encrypt communication on the virtual IP port 443.


#CAfile = /etc/pki/tls/certs/entrust-chain.crt
cert = /etc/pki/tls/certs/saml.potsdam.edu.crt
key = /etc/pki/tls/private/saml.potsdam.edu.key
        # public virtual ip address
        accept =
        connect =
	verify = 0

An example self-signed cert:

openssl genrsa -aes256 -out pass.key 2048
openssl rsa -in pass.key -out server.key
openssl req -new -key server.key -x509 -out server.crt -days 999

Note that key and cert will need to be copied into the locations referenced above.

Start stunnel with the following, probably in an init script:

stunnel /etc/stunnel/https.conf

With stunnel running, we now have encrypted communication on our virtual IP.

Hacking SimpleSAMLphp for Offloaded SSL

SimpleSAMLphp makes use of internal functions to determine port number, HTTP versus HTTPS, and to build links based on those determinations. Because we offloaded SSL to our balancer, it throws off these functions, with the result that links and such are improperly constructed.

Discussion at https://groups.google.com/forum/#!topic/simplesamlphp/m0aqJoURl7I suggests that there may be simpler ways to do this, eventually, but I settled on the changes represented in the following patch to lib/SimpleSAML/Utilities.php:

--- Utilities.dist.php	2013-04-08 04:44:05.000000000 -0400
+++ Utilities.php	2013-12-14 15:35:04.000000000 -0500
@@ -87,6 +87,8 @@
 	public static function isHTTPS() {
+return TRUE;
 		$url = self::getBaseURL();
 		$end = strpos($url,'://');
@@ -105,6 +107,8 @@
 	private static function getServerHTTPS() {
+return TRUE;
 		if(!array_key_exists('HTTPS', $_SERVER)) {
 			/* Not an https-request. */
 			return FALSE;
@@ -128,6 +132,8 @@
 	private static function getServerPort() {
+return '';
 		if (isset($_SERVER["SERVER_PORT"])) {
 			$portnumber = $_SERVER["SERVER_PORT"];
 		} else {

After applying the patch, we should find that all clicks everywhere remain based on https://saml.potsdam.edu.


SUNY Printing with LPSKIM

Posted in Technology at 21:48:01 by Jeff Hardy

SUNY Potsdam (and probably all SUNY schools) need to accommodate administrative print jobs from a number of legacy (COBOL) sources from SUNY Central in Albany. These jobs arrive bearing many of the marks of the era of mainframes and dot matrix printers: text-only, and replete with inconvenient line feeds, odd characters, and alignment issues. Historically this has meant ruler measurements, physical printer adjustment, and later, perhaps some arcane PCL magic. Unfortunately, these jobs seemingly cannot be modified at the source to make them more palatable for modern PostScript printers.

Printing at SUNY Potsdam has been backed by CUPS for over ten years, and because it is open-source, with well-documented APIs for extension, we have been able to accommodate a wide variety of printing scenarios by writing our own applications. These have included accounting filters, PDF writers, hardware backends, and a bevy of specialized solutions to handle unique printing needs.

The latest offering is lpskim, so named because it is a text manipulation filter designed to be used as a System V interface script. A SYSV interface script differs from the now traditional PPD-driven document-type filtering inherent to CUPS, instead assuming all filtering itself as in the old LPR days. So, rather than define a series of isolated scripts to handle the needs for each queue, many repeated, some unique, all needing maintenance, lpskim arose as a generic universal filter that is driven by a configuration file. It has grown to handle everything we need to print these jobs to modern printers and beyond.

In our case, we created several SUNY queues (SUNY1-SUNYE) on our local CUPS server, each corresponding to an Albany queue for Potsdam. Each will receive these Albany-sourced jobs and merely re-spool to the queues with which our offices are familiar. This gives us a useful, consistent abstraction layer so that local queue and printer changes need no coordination with remote hands in Albany. It is also where we attach lpskim. The following sets up a queue for use this way:

lpadmin -p SUNYD -v file:/dev/null -i /path/to/lpskim.pl

Note that the path is wherever lpskim was downloaded, as lpadmin simply copies it to /etc/cups/interfaces/SUNYD. Also note that we define the printer URI as file-backed /dev/null since we handle re-spooling to the final queue in lpskim itself. There we have necessary flexibility to change the destination of print jobs based on patterns in the jobs themselves (more below). This is also a way to create mischief by spooling a job to all queues on the print server, but that is left as an exercise for the reader.

We now need to define configuration for SUNYD in /etc/cups/lpskim.conf, otherwise the job will simply hit /dev/null. The following is assumed to be all one line:

SUNYD | debug : save_job : respool=RAY412-5 : lpoption=cpi=12 : \ 
lpoption=lpi=6.6 : lpoption=page-left=54 : lpoption=page-top=54

Note that we are re-spooling the job to an office queue, and that when doing so, we specify several lp options to affect formatting as the job is converted to the output appropriate for the final destination printer as specified in CUPS. Here, these are all margin adjustments, our simplest setup.

I provide our full SUNY lpskim configuration for all queues across a range of HP and Dell PostScript laser printers for reference. This file should be saved at /etc/cups/lpskim.conf. Notes follow.


  • SUNY1 (checks): Fairly straightforward text manipulation and formatted re-spooling, except for a cut/replace to move the date for new check stock.
  • SUNY4 (batch control): Formatted re-spooling with conf_switch options to re-base the configuration on either the SUNY4_CHECKREG or SUNY4_QUICKPAY configs based on patterns in the job. With all options the same, these proved unnecessary in the end, but were left in case new printers introduced new wrinkles.
  • SUNY9 (refund): Minute adjustments to cpi and lpi so output would fit a rough photocopy of a form, and line deletions near the end to eliminate a few extraneous lines that unavoidably shift to a second page as a result.
  • SUNYE (vouchers): A pair of filters to change line feeds and the like. Also note the commented-out alternate configuration that prepended a PCL string for a previous printer.

The rest of the queues are straightforward formatted re-spooling, a few involving a change to landscape, some margin adjustments, etc.

See http://fritz.potsdam.edu/projects/cupsapps/lpskim to download lpskim and for full documentation of all options. Happy printing.


Fedora Upgrade: F18 -> F19

Posted in Technology at 03:46:41 by Jeff Hardy

Another fedup upgrade brought me up to Fedora 19 a couple of weeks ago. Pre-upgrade, my outstanding oom-killer bug seemed to get worse before it got better. Since I consider it a last resort to change likely unrelated variables in the face of a problem (entire OS), I stuck it out on Fedora 18 until, luckily, a kernel update arrived that seemed to stabilize the issue (inadvertently or otherwise, 3.10.7). With a known good kernel available in both new and old, it paved the way for upgrade.

Stability has persisted, despite the fact that I am still on my legacy hardware, three graphics cards, four monitors setup. I am experiencing a problem with pidgin, for which I have opened a new bug. Such is technology.


Fedora Upgrade: F17 -> F18

Posted in Technology at 01:50:54 by Jeff Hardy

The new fedup process completed my upgrade to Fedora 18 with no issues a couple of months ago. After the normal minor wrangling to get four monitors working again, I immediately began experiencing a more serious issue: persistent oom-killer, typically a few hours after boot and despite plenty of memory.

Oom-killer continued to target various aspects of X, so trying to rule things out, I tried different desktop environments. Unfortunately, it has persisted across XFCE, LXDE, and KDE. All attempts at isolating a single problematic application have come up empty. A decent resource on tracing memory issues like this is http://bl0rg.krunch.be/oom-frag.html, which analysis points to fragmentation, but in my case I seem to be exhausting DMA memory. I created a bug tracking the issue.

vm.panic_on_oom = 1
kernel.panic = 10

I had to resort to the above sysctl.conf, with further mitigation being to reboot at the beginning and end of the day, leaving the system in runlevel 3 when not in use. Barely tolerable. My least successful upgrade.

« Previous entries Next Page » Next Page »