07.10.14
Posted in Technology at 22:55:42 by Jeff Hardy
GParted is among the most well-known and respected open-source partition editors. From creation to deletion, resizing, copying, and moving, it is a tool built to manage the full gamut of disk partitionining operations.
In part one of our migration process, we cloned the installation to a new drive in new hardware. In part two, we moved to a LUKS-encrypted root. Now we have a few adjustments we wish to make to our partitioning scheme, regaining space so that we may move the rest of the system beneath the umbrella of encryption.
The Plan
We have the following arrangement on the system drive:
/dev/sda (150GB)
sda1: ext3 /boot (300MB)
sda2: ext3 oldroot (38GB)
sda3: lvm Volume00 (112GB)
Volume00-home: ext3 /home (70GB)
Volume00-swap: swap (2GB)
Volume00-root: LUKS (40GB)
luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34: / ext3 (40GB)
Despite our efforts thus far, the issue remains that our home volume is not yet protected by encryption. We can tackle this two ways, either by building a new home crypto-volume and copying data as we did for root, or by simply reclaiming space enough to abandon a separate home volume and copy that data beneath the already encrypted root. I elect for the latter, mainly because the use of a single crypto-volume avoids the need for multiple pass-phrase entry at boot, and also because there is no compelling reason for a separate home on this particular system.
In order to gain the space necessary, we will remove the old unused root, add that space for use by LVM, grow our new crypto-root, and then copy the data from home. This is no big deal, and the partition is easily deleted with fdisk or gparted. But there is one issue. Because this is not the last partition on the disk, resizing for our needs is not as simple as an fdisk delete and recreate, moving the end boundary to the end of the drive. Now we need to move the start of the partition. Enter GParted.
GParted
While GParted can be run at any time, some functionality for a given storage device may not be available, nor entirely safe, if it is in-use (mounted) by the operating system. For this reason, we elect to boot the system with the GParted Live CD.
NOTE: Be aware that there is a perfectly good alternative to partition resizing of any kind. We could instead elect to create an LVM partition using the unallocated space and do standard LVM operations to incorporate it into our root logical volume (pvcreate, vgextend, lvextend). But we continue with full confidence in GParted, the main benefit being mostly cosmetic in the end: a tidy partition table.
Booted into the live environment, we start GParted. After deleting the old root partition at /dev/sda2, we are left with a good illustration of the situation at hand. We have a region of unallocated space before the LVM partition:
We need to resize/move the sda3 partition, which contains our priceless crypto-LVM/root within it:
The resize dialog gets right to the point. Adjusting the amount of space before and after the partition moves its start and end positions on disk:
Drag the start boundary all the way to the left (or enter 0) to effect a move/resize, absorbing every bit of the unallocated space before the partition:
Operation pending. We can still back out without affecting anything. Click apply, and be prepared for a lengthy wait.
When all is said and done, we have moved and resized our LVM partition to utilize the full disk:
Note that used space has not changed. We address that next.
Resize LVM, LUKS, Ext3
The work done so far has resized the outermost container, the partition. Like Russian nesting dolls, we need to grow each in turn before we can address the filesystem. A testament to the tools, we can actually do all of the follwoing booted into the live system.
First LVM:
pvs
PV VG Fmt Attr PSize PFree
/dev/sda3 Volume00 lvm2 a-- 148.75g 36.98g
vgs
VG #PV #LV #SN Attr VSize VFree
Volume00 1 3 0 wz--n- 148.75g 36.98g
lvs
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
home Volume00 -wn-ao---- 69.00g
root Volume00 -wi-ao---- 40.76g
swap Volume00 -wi-ao---- 2.00g
lvresize Volume00/root -l+100%FREE
Extending logical volume root to 77.75 GiB
Logical volume root successfully resized
lvs
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
home Volume00 -wn-ao---- 69.00g
root Volume00 -wi-ao---- 77.75g
swap Volume00 -wi-ao---- 2.00g
NOTE: I would normally expect to first do a pvresize to grow the PV, but the PV already knew about the extra space, likely a gift from GParted.
Then LUKS:
cryptsetup resize luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
echo $?
0
Finally the filesystem:
df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 40G 29G 9.2G 76% /
resize2fs /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is mounted on /; on-line resizing required
old_desc_blocks = 3, new_desc_blocks = 5
The filesystem on /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is now 20380160 blocks long.
df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 77G 29G 45G 40% /
We are left with a nice large root to accommodate our /home data.
Home Copy
With 45GB free in our root, we should have plenty of space after cleaning out some garbage in home:
df -h /home
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/Volume00-home 64G 5.0G 56G 9% /home
A good old rsync will suffice to copy that whopping 5GB of data from the existing home volume to the home within the crypto-root. First things first, bug out to runlevel 3 (or single/rescue mode for overkill) to ensure home is not in use so we can mount it somewhere else:
init 3
umount /home
mkdir /oldhome
mount /dev/mapper/Volume00-home /oldhome
Finally rsync the data:
rsync -a /oldhome/ /home
When done, remember to pound out the /home mount from /etc/fstab lest we accidentally mount our old unencrypted home volume on next boot:
#/dev/Volume00/home /home ext3 defaults 1 2
A reboot confirms that everything is working as it should.
LVM Cleanup
Almost half the disk is tied up in our abandoned home. Once happy with the new arrangement, we can easily reclaim that space starting at the LVM layer, and then working our way up. This is just a second run at container resizing as above, LVM, LUKS, Ext3, all live.
lvs
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
home Volume00 -wn-a----- 69.00g
root Volume00 -wi-ao---- 77.75g
swap Volume00 -wi-ao---- 2.00g
lvremove Volume00/home
Do you really want to remove active logical volume home? [y/n]: y
Logical volume "home" successfully removed
vgs
VG #PV #LV #SN Attr VSize VFree
Volume00 1 2 0 wz--n- 148.75g 69.00g
lvextend -L+50G Volume00/root
Extending logical volume root to 127.75 GiB
Logical volume root successfully resized
cryptsetup resize luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
echo $?
0
resize2fs /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
resize2fs 1.42.8 (20-Jun-2013)
Filesystem at /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is mounted on /; on-line resizing required
old_desc_blocks = 5, new_desc_blocks = 8
The filesystem on /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 is now 33487360 blocks long.
df -h /
Filesystem Size Used Avail Use% Mounted on
/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 126G 36G 85G 30% /
NOTE: With ample disk available, we left some space unallocated in the VG. This is just a hedge for flexibility. If we need more space in the future, we can always add it live just as we did here.
Partition Re-Numbering
Our partition table consists of the following, as revealed by fdisk:
Device Boot Start End Blocks Id System
/dev/sda1 * 63 619919 309928+ 83 Linux
/dev/sda3 620544 312580095 155979776 8e Linux LVM
While it is no problem whatsoever to be “missing” a partition in sequence, we continue the intrepid pursuit of the trivial and wish to renumber sda3 to sda2. The easiest way to rectify this is to simply delete and re-create the partition using fdisk. Since it is the last partition, fdisk will normally default to the same start and end sectors when creating the new one (something to double-check). While this can be done live followed by an immediate reboot, it is safest to do this from rescue mode with the disk not in use.
fdisk /dev/sda
ddelete, partition number 3
new, primary, partition number 2, first sector default, last sector default
type, partition number 2, 8e
write
Reboot. Just a reminder, this is completely unnecessary. 
Everything Encrypted
We leave with this on the system disk:
/dev/sda (150GB)
sda1: ext3 /boot (300MB)
sda2: lvm Volume00 (150GB)
Volume00-swap: swap (2GB)
Volume00-root: LUKS (127GB)
luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 ext3 (127GB)
unallocated: (19GB)
A tidy partition and volume layout with everything encrypted (almost).
Permalink
06.21.14
Posted in Technology at 23:23:55 by Jeff Hardy
Dracut is an initramfs infrastructure, as well as the name of tool needed to create initramfs images. It is also the name of the prompt/environment made available at boot time by choice or by error, allowing for troubleshooting of boot-time problems.
Building Initramfs Images
It always pays to make backups:
cd /boot
cp -a initramfs-3.12.8-300.fc20.i686.img initramfs-3.12.8-300.fc20.i686.img.backup
The dracut tool is used to build initramfs images:
dracut -f /boot/initramfs-3.12.8-300.fc20.i686.img 3.12.8-300.fc20.i686
Images are best built from within the intended root of the Linux system, and the tool can be run on a live system with no issues. If the system is in a state where this is unattainable, it is possible to build a clean environment by assembling the necessary mounts in rescue mode. Do not select “find drives automatically,” we will do so manually:
mkdir /mnt/sysimage
vgchange -ay
mount /dev/mapper/Volume00-root
mount /dev/sda1 /mnt/sysimage/boot
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
chroot /mnt/sysimage
NOTE: Devices holding boot and root will vary from system to system.
Now run dracut.
The Dracut Shell
If the boot process fails due to an issue with the initial ram disk, we will likely be dropped to the dracut shell (this may require adding rd.shell as a kernel argument).
For example, failure due to a missing volume:
Warning: Could not boot.
Warning: /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 does not exist
The beauty of the dracut prompt is that we have nearly the full flexibility of Linux at our disposal, with whatever tools exist in the initramfs image.
This particular error is something I have commonly experienced when switching root, and is probably due to the systemd setup within the initramfs specifically looking for a device that did not exist when we created it (for instance, if we ran dracut while not within the final resting place of root).
We can see what is actually contained within the initrams image:
lsinitrd /boot/initramfs-3.12.8-300.fc20.i686.img
If a volume cannot be found, it is likely there is no entry for it in etc/systemd/system/initrd.target.wants. We can attempt to urge things along manually:
lvm vgchange -ay
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
Enter passphrase for /dev/mapper/Volume00-root:
exit
If this now boots the system, the path forward is to re-generate the initramfs image correctly, which usually entails doing so from within the appropriate root (as above).
Finally, here is a tidbit showing what kinds of things can be done for gathering intelligence for later followup:
debug=$(lvm pvdisplay; lvm vgdisplay; lvm lvdisplay; blkid; dmesg)
lvm vgchange -a y
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
Enter passphrase for /dev/mapper/Volume00-root:
mount /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /mnt
echo "$debug" > /mnt/dracut.debug
NOTE: All this does is set an environment variable “debug” to the output of a bunch of commands, then activate storage somewhere to store that output, in this case showing how it is even possible to use a crypto-volume. Perhaps it is USB or simple volume instead.
Resources
- http://wiki.centos.org/TipsAndTricks/CreateNewInitrd
- http://lists.centos.org/pipermail/centos/2013-March/133604.html
Permalink
Posted in Technology at 23:22:23 by Jeff Hardy
GNU GRUB is the bootloader of choice for most Linux distributions, offering massive flexibility to boot a multitude of operating systems. With GRUB version 2, configuration remains straightforward, but can be largely auto-generated with modern tools. It also retains the full flexibility to modify the boot configuration at run-time.
Configuration
It always pays to make backups:
cd /boot/grub2
cp -a grub.cfg grub.cfg.backup
In order to generate grub2 configuration:
grub2-mkconfig -o /boot/grub2/grub.cfg
The tool will probe the system looking for partitions containing vmlinuz and initramfs images, and build configuration appropriately. Administrator tunables that affect the process are stored in /etc/default/grub. Example config:
GRUB_TIMEOUT=5
GRUB_DISTRIBUTOR="$(sed 's, release .*$,,g' /etc/system-release)"
GRUB_DEFAULT=saved
# GRUB_TERMINAL="serial console"
# GRUB_SERIAL_COMMAND="serial --unit=0 --speed=9600"
GRUB_CMDLINE_LINUX="quiet rhgb"
GRUB_DISABLE_RECOVERY="true"
Configuration is best built from within the intended root of the Linux system, and the tool can be run on a live system with no issues. If the system is in a state where this is unattainable, it is possible to build a clean environment by assembling the necessary mounts in rescue mode. Do not select “find drives automatically,” we will do so manually:
mkdir /mnt/sysimage
vgchange -ay
mount /dev/mapper/Volume00-root
mount /dev/sda1 /mnt/sysimage/boot
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
chroot /mnt/sysimage
NOTE: Devices holding boot and root will vary from system to system.
Now run grub2-mkconfig.
Run-Time Changes
Hitting “e” at the grub menu allows one to peruse and change configuration. This is probably most commonly used to alter kernel command-line arguments, such as adding “single” to boot to single-mode, but any aspect of the configuration can be changed before boot.
For example, we may wish to attempt to boot an alternate root volume. While in this mode, we need simply find the kernel we wish to boot and alter its command-line arguments, one being the selection of the intended root:
linux /vmlinuz-3.12.8-300.fc20.i686 ro root=/dev/sda2
NOTE: Add/remove other arguments such as quiet, rhgb, or rd.* as necessary.
We may also have a need to load an alternate initramfs image:
initrd /initramfs-3.12.8-300.fc20.i686.img.backup
When satisfied with the changes, hit F10 to boot.
Grub Prompt
It is actually entirely possible to attempt boot with no grub configuration at all, whether dropping voluntarily or involuntarily to the grub prompt. Given all the same assumptions:
grub> ls
(hd0) (hd0,1) (hd0,2) (hd0,3)
grub> set root=(hd0,1)
grub> linux /vmlinuz-3.12.8-300.fc20.i686 ro root=/dev/sda2
grub> initrd /initramfs-3.12.8-300.fc20.i686.img.20140608
grub> boot
NOTE: The partition nomenclature is different in grub, but we are specifying the partition containing vmlinuz and initrd images (eventually known as sda1 and mounted at /boot by linux). Also note that tab-completion thankfully works.
Resources
- http://tuxers.com/main/instigating-a-manual-boot-from-the-grub-prompt
Permalink
Posted in Technology at 23:21:07 by Jeff Hardy
Full-disk encryption is essentially a requirement for a mobile OS. While there are a few choices for full block device encryption on Linux (such as Truecrypt), at this time the clear choice is Linux Unified Key Setup. LUKS integrates with dm-crypt as the encryption backend, producing a device mapper entry for the decrypted block device that can be used like any other.
In part one of our Fedora migration odyssey, we cloned the installation to a laptop. Now in part two, we will shift things around to move to a LUKS-encrypted root. What follows assumes that we are using the new laptop for all operations, and that our cloned system drive is the only one in place (sda). While everything can be done in rescue mode, some operations require it and will be noted.
The Plan
We have the following partition and volume layout:
/dev/sda (150GB)
sda1: ext3 /boot (300MB)
sda2: ext3 / (38GB)
sda3: lvm Volume00 (112GB)
Volume00-home: ext3 /home (70GB)
Volume00-swap: swap (2GB)
unallocated: (40GB)
The official documentation indicates that there is no way to encrypt-in-place and thereby preserve data (although tools such as luksipc suggest otherwise). Regardless, we will use that unallocated space, perfectly sized, to create a new root filesystem on top of a crypto-volume using LVM/LUKS, and then copy data. A move provides a secondary benefit here: it will fold root back into LVM, its excursion to a bare partition being only ever a mere workaround.
Last but not least, a copy leaves our existing root undisturbed as a fallback in the event of problems.
Create a New Crypto Volume
First we create an LVM volume using all of that free space:
vgs
VG #PV #LV #SN Attr VSize VFree
Volume00 1 2 0 wz--n- 111.76g 40.76g
lvcreate -n root -l+100%FREE Volume00
vgs
VG #PV #LV #SN Attr VSize VFree
Volume00 1 3 0 wz--n- 111.76g 0
lvs
LV VG Attr LSize Pool Origin Data% Move Log Cpy%Sync Convert
home Volume00 -wn-ao---- 69.00g
root Volume00 -wi-ao---- 40.76g
swap Volume00 -wi-ao---- 2.00
Rather than put a filesystem directly on the new volume, we will set up for encryption. For the best encryption results, it is recommended to write random data to the device:
dd if=/dev/urandom of=/dev/mapper/Volume00-root
dd: writing to ‘/dev/mapper/Volume00-root’: No space left on device
85483521+0 records in
85483520+0 records out
43767562240 bytes (44 GB) copied, 11330.7 s, 3.9 MB/s
NOTE: This will take some time.
Encrypt the volume:
cryptsetup --verbose --verify-passphrase luksFormat /dev/mapper/Volume00-root
WARNING!
========
This will overwrite data on /dev/mapper/Volume00-root irrevocably.
Are you sure? (Type uppercase yes): YES
Enter passphrase:
Verify passphrase:
Command successful.
NOTE: Remember that passphrase. If lost, data will be irrecoverable.
Review the encryption summary for the device, noting the UUID:
cryptsetup luksDump /dev/mapper/Volume00-root
...
cryptsetup luksDump /dev/mapper/Volume00-root | grep UUID
UUID: cc7e78ee-87a9-4ad0-9c82-31de01389b34
Open the crypto-volume. The last argument specifies the device-mapper name by which this volume will be opened:
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
NOTE: Naming by UUID is how Fedora sets these volumes up when creating them at install time, practice we will adhere to later when setting up the new root. For consistency’s sake, we do so now, but we could just as easily call it “luks-root” or some such for convenience at this stage.
Create a filesystem:
mkfs -t ext3 /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
Copy Data
At this point, we should boot to rescue mode if not already. Because we are copying the existing root volume, we cannot afford to have it mounted at / where it could change. A read-only remount could suffice, but is likely to cause issues on a live system. Additionally, though we tell rsync not to cross filesystem boundaries anyway (-x), it is best reduce to zero the risk of copying special fileystems or other partitions that may be mounted beneath root.
We mount the source and destination, and rsync the contents:
mkdir /oldroot
mount /dev/sda2 /oldroot
mkdir /newroot
vgchange -ay
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
mount /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /newroot
rsync -avx /oldroot/ /newroot
Requirements for Switching to a New Root
There are three elements necessary for successful switch to a new root, be it partition, lvm, crypto-volume, floppy, or who knows what else. In the order they are encountered in the boot process:
-
GRUB: The bootloader needs to be configured properly to load the appropriate kernel executable with necessary arguments indicating where to find the new root.
-
Initial RAM Disk: The initramfs image (or older initrd) needs to be built with enough utility so that the kernel can make use of it to find the real root filesystem. It may or may not already be capable, depending on suitable kernel arguments specified in grub.
-
fstab: Configuration in /etc/fstab must be set to mount the new root at / with suitable mount options.
It is certainly possible to attempt to fight through each stage of the boot process, making run-time manual changes to grub and the dracut environments in order to find one’s way into the new root, confirming it works before making configuration changes. This entails having a decent amount of intelligence about how various rd.* kernel arguments will relate to one’s system. This is definitely a worthy endeavor and is certainly the approach to take when troubleshooting. But in the end, we will proceed directly to making the system changes for the new root, first making backups of the involved files, thereby allowing us to use the original root as a fallback without too much trouble.
Configure the System for a New Root
The modern tools necessary to build the bootloader config (grub2-mkconfig) and initramfs image (dracut), do a wonderful job of auto-configuring themselves. This is both a blessing and a curse, as we need to carefully establish an environment from which they can reliably draw their bits. This means running them from within the new root for best results, necessitating a boot to rescue mode.
Boot to rescue mode and make sure not to select “find drives automatically.” Instead, we will assemble what we need manually. The following is the surest path I have found to setup an environment for building clean grub configuration and initramfs image:
mkdir /mnt/sysimage
vgchange -ay
cryptsetup luksOpen /dev/mapper/Volume00-root luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34
Enter passphrase for /dev/mapper/Volume00-root:
mount /dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /mnt/sysimage
mount /dev/sda1 /mnt/sysimage/boot
mount --bind /proc /mnt/sysimage/proc
mount --bind /dev /mnt/sysimage/dev
mount --bind /sys /mnt/sysimage/sys
chroot /mnt/sysimage
NOTE: If this was not a crypto-volume, the procedure still stands, mounting the LVM volume directly.
We have assembled the system from new root up, and are now prepared to re-configure and rebuild.
Make backups of the initramfs image and grub configuration:
cd /boot
cp -a initramfs-3.12.8-300.fc20.i686.img initramfs-3.12.8-300.fc20.i686.img.20140608
cd /boot/grub2
cp -a grub.cfg grub.cfg.20140608
NOTE: Grub configuration problems are relatively easy to work around at boot time, especially with a backup of the initramfs. Initramfs-specific problems less so. The initramfs may also be difficult to restore to original state depending on the nature of the problem. In other words, make sure to back up the initramfs.
Build the new initramfs image:
dracut -f /boot/initramfs-3.12.8-300.fc20.i686.img 3.12.8-300.fc20.i686
Alter grub configuration at /etc/default/grub, adding an argument informing our kernel that it should open our new crypto-volume:
GRUB_CMDLINE_LINUX="quiet rhgb rd.luks.uuid=cc7e78ee-87a9-4ad0-9c82-31de01389b34"
NOTE: Multiple crypto-volumes can be listed here, each resulting in a password prompt at boot time.
Build the new grub config:
grub2-mkconfig -o /boot/grub2/grub.cfg
NOTE: We should see entries for our old root as well, from the os-prober stage of grub2-mkconfig, though they may not work without some urging at the dracut prompt during boot.
This should produce a grub.cfg entry containing a kernel line reflecting our changes:
linux /vmlinuz-3.12.8-300.fc20.i686 root=/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 ro quiet rd.luks.uuid=cc7e78ee-87a9-4ad0-9c82-31de01389b34
Finally, alter fstab to mount our new root volume:
/dev/mapper/luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 / ext3 defaults,user_xattr 1 1
NOTE: Over the years, distros have moved from mounting by device to mounting by label or UUID, or at least device-mapper name, since devices can change. We could do this here as well, for example mounting by the UUID of the filesystem within our crypto volume (UUID=551227f2-7ea4-4c5d-8bed-a5bff1d7354f). Fedora seems to mount crypto volumes by device-mapper name and so shall we.
Crypttab? It is possible to create an /etc/crypttab file containing an entry like this:
luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 /dev/mapper/Volume00-root none
A stock Fedora install on a crypto-root does not create this file, but its existence may (or may not) eliminate the need to specify rd.luks.uuid on the kernel command-line, among other things. I did not create this file on my system. Your mileage may vary.
Boot the New Root
At this point, we can boot into our new crypto-root. Following the above procedure should drastically reduce the chances of problems, but if they do occur, we are well-prepared to troubleshoot and recover with the knowledge we have gained and the backups we have made (see Working GRUB and Working Dracut).
We are left with this on the system drive:
/dev/sda (150GB)
sda1: ext3 /boot (300MB)
sda2: ext3 / (38GB)
sda3: lvm Volume00 (112GB)
Volume00-home: ext3 /home (70GB)
Volume00-swap: swap (2GB)
Volume00-root: LUKS (40GB)
luks-cc7e78ee-87a9-4ad0-9c82-31de01389b34 ext3 (40GB)
Ultimately, a successful move to a new crypto-root. We still have other unencrypted volumes which which we may soon contend.
Permalink
06.16.14
Posted in Technology at 23:15:04 by Jeff Hardy
My ten year Fedora upgrade streak, now at 20, is still running on hardware that is nearly as old and I thought it time to move the install to a (slightly) newer laptop that I am using more frequently. This will involve a number of storage operations and system re-configuration, new concerns such as encryption, and more to come. Cloning an install and re-fashioning it in such a way will indeed be more work than a fresh install and copy, but imposes some interesting obstacles to work around, and remains within the spirit of maintaining a legacy install.
Here in part one, we concentrate on merely getting the system onto a new drive, and booting the new hardware. This will merge the data from two drives in its current desktop environment, to a single new laptop drive. What follows assumes that cloning, partitioning, and LVM operations are conducted with the old and new storage accessed from a Linux system in an offline manner. Single or rescue mode are not required as long as we can access the block devices, perhaps connected as secondary PATA/SATA or USB drives. In no case should they be mounted anywhere.
Dump the Drive Images
There are certainly more exotic tools for disk cloning, but let us stick to the good old UNIX standby ‘dd.’ We can be tricky and dd over a network pipe directly to a destination drive, but with storage available (in this case external USB mounted at /storage), we get a safe offline copy/backup of the original drives:
dd if=/dev/sda of=/storage/fritzdesk_sda
dd if=/dev/sdb of=/storage/fritzdesk_sdb
The partition info on sda, revealing our /boot and / partitions:
fdisk -l fritzdesk_sda
Disk fritzdesk_sda: 37.3 GiB, 40037760000 bytes, 78198750 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa5cba5cb
Device Boot Start End Blocks Id System
fritzdesk_sda1 * 63 619919 309928+ 83 Linux
fritzdesk_sda2 619920 78185519 38782800 83 Linux
The partition info on sdb, revealing our LVM partition containing logical volumes for home and swap:
fdisk -l fritzdesk_sdb
Disk fritzdesk_sdb: 74.5 GiB, 80026361856 bytes, 156301488 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0x59d50c69
Device Boot Start End Blocks Id System
fritzdesk_sdb1 * 63 156296384 78148161 8e Linux LVM
Ultimately, we have this from our source system:
/dev/sda (40GB)
sda1: ext3 /boot (300MB)
sda2: ext3 / (38GB)
/dev/sdb (72GB)
sdb1: lvm Volume00 (72GB)
Volume00-home: ext3 /home (70GB)
Volume00-swap: swap (2GB)
Clone to the New Drive
We will combine the partitions from these two source drives into the new 150GB laptop drive. No matter how the new drive is connected (direct PATA/SATA or USB), we will assume the new drive is /dev/sdc for clarity.
First a clone of sda, as that contains our MBR, boot, and root:
dd if=/storage/fritzdesk_sda of=/dev/sdc
Check partitions on the new drive:
fdisk -l /dev/sdc
Disk /dev/sdc: 149.1 GiB, 160041885696 bytes, 312581808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa5cba5cb
Device Boot Start End Blocks Id System
/dev/sdc1 * 63 619919 309928+ 83 Linux
/dev/sdc2 619920 78185519 38782800 83 Linux
We now need to create an LVM partition to hold the contents of sdb:
fdisk /dev/sdc
new, primary, partition number 3, first sector default, last sector default
type, partition number 3, 8e
write
Check partitions on the new drive:
fdisk -l /dev/sdc
Disk /dev/sdc: 149.1 GiB, 160041885696 bytes, 312581808 sectors
Units: sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disklabel type: dos
Disk identifier: 0xa5cba5cb
Device Boot Start End Blocks Id System
/dev/sdc1 * 63 619919 309928+ 83 Linux
/dev/sdc2 619920 78185519 38782800 83 Linux
/dev/sdc3 78186496 312581807 117197656 8e Linux LVM
We are interested in the one LVM partition inside the fritzdesk_sdb image rather than the entire drive. We must expose it with kpartx:
kpartx -av /storage/fritzdesk_sdb
add map loop0p1 (253:3): 0 156296322 linear /dev/loop0 63
Scan for the new LVM physical volume just exposed to ensure everything was found:
pvscan
PV /dev/mapper/loop0p1 VG Volume00 lvm2 [74.52 GiB / 3.52 GiB free]
NOTE: It is good practice to ensure unique VG names when doing system installs, lest a generic name like Volume00 collide with another, such as when used in this way on another system. Luck is on our side.
Clone the exposed source LVM partition to the new LVM partition, physical volume to physical volume:
dd if=/dev/mapper/loop0p1 of=/dev/sdc3
Claim New LVM PV Space
This last clone was of a smaller LVM partition image into a much larger LVM partition on the new drive. We will want to claim that space for future LVM use. The pvresize command will claim all space up to the partition or block device boundary (now further out) unless otherwise specified:
pvresize /dev/sdc3
pvs
PV VG Fmt Attr PSize PFree
/dev/sdc3 Volume00 lvm2 a-- 111.76g 40.76g
vgs
VG #PV #LV #SN Attr VSize VFree
Volume00 1 2 0 wz--n- 111.76g 40.76g
NOTE: This may need to be prefaced with a pvscan to find the physical volume.
Now we have an additional 40.76GB in the LVM volume group to grow or create new logical volumes.
Boot the New Machine
At this point, we have directly cloned the system to a new drive with the fewest changes possible. With that drive physically installed as the sole drive in the new system, we boot to ensure all pieces are present, troubleshooting any startup issues and checking things before introducing any more complexity. For example, there is trouble starting X11, and a boot to single mode and check of the X11 logs indicates that there is a need to install xorg-x11-drv-intel. Other minor issues involve installation of xorg-x11-drv-synaptics for full touchpad functionality. Things of this nature are not surprising given a move to new hardware.
We are left with this on the destination drive (now /dev/sda in-use in the new system):
/dev/sda (150GB)
sda1: ext3 /boot (300MB)
sda2: ext3 / (38GB)
sda3: lvm Volume00 (112GB)
Volume00-home: ext3 /home (70GB)
Volume00-swap: swap (2GB)
unallocated: (40GB)
Ultimately, a simple, successful clone of fritzdesk to fritzlap. We have left some room to maneuver, which we will soon need. In part 2, we will worry about encryption.
Permalink
01.30.14
Posted in Technology at 10:03:36 by Jeff Hardy
First ensuring I was running fedup-0.8 to avoid the fedup-0.7 upgrade bug, this Fedora upgrade was off to the races with fedup --network 20. Unfortunately, after rebooting to the upgrade, my machine ended up wedged at the following:
systemd[1]: Failed to initialize SELinux context: No such file or directory
This fedup selinux bug is easily worked around with selinux=0 enforcing=0 on the kernel command-line. Afterwards, the upgrade completed without issue. I am also pleased to see that one particular Pidgin bug that plagued me throughout Fedora 19 seems to be fixed.
Meanwhile, an unrelated fedup upgrade I conducted on a laptop required that I work around some bugs related to finding my encrypted root partition, by specifying rd.luks.uuid=<uuid> in grub. I find myself wondering if we as an industry are making adequate progress reconciling and mitigating the increasing problems that come with increasing technical complexity.
Permalink
12.19.13
Posted in Technology at 22:16:49 by Jeff Hardy
SimpleSAMLphp is the ideal choice as a SAML 2.0 IdP for SUNY Federation. As it will be the core authentication mechanism for SUNY services, high-availability and load-balancing are an important consideration. Here is one approach to achieving this with HAProxy and Keepalived.
We intend to have a pair (at least) of Apache servers running identical SimpleSAMLphp installations. These will sit behind a pair of load-balancer machines running HAProxy and Keepalived, the first to balance HTTP traffic to the SAML servers, the second to provide virtual IP failover between the balancers. This forward-facing virtual IP will be stunnel-encryped, and advertised as our SAML Identity Provider (IdP).
__________vip__________
______keepalived______
balance1____balance2
|____\__haproxy__/____|
saml1___________saml2
Senior sysadmin Greg Kuchyt pioneered our use of these technologies for load-balancing and failover of our 389-DS LDAP services in a manner much like this, an approach that has served us well for many years, and which we have used for other services.
Our SAML setup is based on the document collection in the SUNY IDM Wiki, particularly the excellent Installing SimpleSAMLphp for SUNY Federation guide. Suffice it to say that I will not go into any detail on our IdP setup, instead covering only what SimpleSAMLphp changes are necessary to share session data for load-balancing and failover.
Shared Session Data with Memcache
On the SAML servers, in order to pave the way for balancing load across multiple machines, we need to ensure that SimpleSAMLphp session handling is switched over to memcache:
yum install memcached php-pecl-memcache
service memcached start
chkconfig memcached on
Note that SimpleSAMLphp uses PHP memcache and not the newer PHP memcached, but both use the system memcached daemon.
Firewalls will need to be opened to allow incoming memcache connections on port 11211.
In the SimpleSAMLphp configuration, set the store.type, and then adjust the memcache_store.servers array to taste (see docs). The following will cause all session data to be saved to the memcache servers on both boxes:
config.php:
'store.type' => 'memcache',
'memcache_store.servers' => array(
array(
array('hostname' => 'saml1.potsdam.edu'),
),
array(
array('hostname' => 'saml2.potsdam.edu'),
),
),
Load-Balancing with HAProxy
On the balance servers we first install HAProxy. The following configuration uses the least connection algorithm to balance HTTP sessions between the two SAML servers behind them.
/etc/haproxy/haproxy.cfg:
defaults
mode http
log global
option httplog
option dontlognull
option http-server-close
option forwardfor except 127.0.0.0/8
option redispatch
retries 3
timeout http-request 10s
timeout queue 1m
timeout connect 10s
timeout client 1m
timeout server 1m
timeout http-keep-alive 10s
timeout check 10s
maxconn 3000
listen https
balance leastconn
bind 127.0.0.1:4430
option httpclose
option forwardfor
option httplog
option httpchk GET /
server saml1 saml1.potsdam.edu:80 check inter 5000 downinter 500
server saml2 saml2.potsdam.edu:80 check inter 5000 downinter 500
Note that hostnames are used here for clarity, but it is probably preferable to use IP addresses for server config lines. Also note that we bind our instance to localhost:4430, since we do not intend for public connections to hit this port directly. Instead they will hit SSL on port 443 provided by stunnel (later).
Start haproxy:
service haproxy start
We now have two balancers that will each balance HTTP connections to the SAML servers.
Failover with Keepalived
On the balance servers we next install Keepalived. On balance1, the following configuration establishes a virtual IP address to be maintained between the balance servers.
/etc/keepalived/keepalived.conf:
global_defs {
notification_email {
devnull@potsdam.edu
}
notification_email_from devnull@potsdam.edu
smtp_server 10.137.110.104
smtp_connect_timeout 30
}
vrrp_instance VI_1 {
virtual_router_id 1
state MASTER
priority 100
interface eth2
smtp_alert
authentication {
auth_type AH
auth_pass SomeKindofPasswordHere!
}
virtual_ipaddress {
10.137.100.101/24 brd 10.137.100.255 dev eth2
}
}
On balance2, we install the same configuration with two key differences:
state BACKUP
priority 50
Start keepalived:
service keepalived start
We now have a front-facing virtual IP address 10.137.100.101 providing failover between the balancers. You can view the status of the IP with: ip addr show.
SSL with Stunnel
On the balance servers we next install stunnel to encrypt communication on the virtual IP port 443.
/etc/stunnel/https.conf:
#CAfile = /etc/pki/tls/certs/entrust-chain.crt
cert = /etc/pki/tls/certs/saml.potsdam.edu.crt
key = /etc/pki/tls/private/saml.potsdam.edu.key
[https]
# public virtual ip address
accept = 10.137.100.101:443
connect = 127.0.0.1:4430
verify = 0
An example self-signed cert:
openssl genrsa -aes256 -out pass.key 2048
openssl rsa -in pass.key -out server.key
openssl req -new -key server.key -x509 -out server.crt -days 999
Note that key and cert will need to be copied into the locations referenced above.
Start stunnel with the following, probably in an init script:
stunnel /etc/stunnel/https.conf
With stunnel running, we now have encrypted communication on our virtual IP.
Hacking SimpleSAMLphp for Offloaded SSL
SimpleSAMLphp makes use of internal functions to determine port number, HTTP versus HTTPS, and to build links based on those determinations. Because we offloaded SSL to our balancer, it throws off these functions, with the result that links and such are improperly constructed.
Discussion at https://groups.google.com/forum/#!topic/simplesamlphp/m0aqJoURl7I suggests that there may be simpler ways to do this, eventually, but I settled on the changes represented in the following patch to lib/SimpleSAML/Utilities.php:
--- Utilities.dist.php 2013-04-08 04:44:05.000000000 -0400
+++ Utilities.php 2013-12-14 15:35:04.000000000 -0500
@@ -87,6 +87,8 @@
*/
public static function isHTTPS() {
+return TRUE;
+
$url = self::getBaseURL();
$end = strpos($url,'://');
@@ -105,6 +107,8 @@
*/
private static function getServerHTTPS() {
+return TRUE;
+
if(!array_key_exists('HTTPS', $_SERVER)) {
/* Not an https-request. */
return FALSE;
@@ -128,6 +132,8 @@
*/
private static function getServerPort() {
+return '';
+
if (isset($_SERVER["SERVER_PORT"])) {
$portnumber = $_SERVER["SERVER_PORT"];
} else {
After applying the patch, we should find that all clicks everywhere remain based on https://saml.potsdam.edu.
Permalink
12.04.13
Posted in Technology at 21:48:01 by Jeff Hardy
SUNY Potsdam (and probably all SUNY schools) need to accommodate administrative print jobs from a number of legacy (COBOL) sources from SUNY Central in Albany. These jobs arrive bearing many of the marks of the era of mainframes and dot matrix printers: text-only, and replete with inconvenient line feeds, odd characters, and alignment issues. Historically this has meant ruler measurements, physical printer adjustment, and later, perhaps some arcane PCL magic. Unfortunately, these jobs seemingly cannot be modified at the source to make them more palatable for modern PostScript printers.
Printing at SUNY Potsdam has been backed by CUPS for over ten years, and because it is open-source, with well-documented APIs for extension, we have been able to accommodate a wide variety of printing scenarios by writing our own applications. These have included accounting filters, PDF writers, hardware backends, and a bevy of specialized solutions to handle unique printing needs.
The latest offering is lpskim, so named because it is a text manipulation filter designed to be used as a System V interface script. A SYSV interface script differs from the now traditional PPD-driven document-type filtering inherent to CUPS, instead assuming all filtering itself as in the old LPR days. So, rather than define a series of isolated scripts to handle the needs for each queue, many repeated, some unique, all needing maintenance, lpskim arose as a generic universal filter that is driven by a configuration file. It has grown to handle everything we need to print these jobs to modern printers and beyond.
In our case, we created several SUNY queues (SUNY1-SUNYE) on our local CUPS server, each corresponding to an Albany queue for Potsdam. Each will receive these Albany-sourced jobs and merely re-spool to the queues with which our offices are familiar. This gives us a useful, consistent abstraction layer so that local queue and printer changes need no coordination with remote hands in Albany. It is also where we attach lpskim. The following sets up a queue for use this way:
lpadmin -p SUNYD -v file:/dev/null -i /path/to/lpskim.pl
Note that the path is wherever lpskim was downloaded, as lpadmin simply copies it to /etc/cups/interfaces/SUNYD. Also note that we define the printer URI as file-backed /dev/null since we handle re-spooling to the final queue in lpskim itself. There we have necessary flexibility to change the destination of print jobs based on patterns in the jobs themselves (more below). This is also a way to create mischief by spooling a job to all queues on the print server, but that is left as an exercise for the reader.
We now need to define configuration for SUNYD in /etc/cups/lpskim.conf, otherwise the job will simply hit /dev/null. The following is assumed to be all one line:
SUNYD | debug : save_job : respool=RAY412-5 : lpoption=cpi=12 : \
lpoption=lpi=6.6 : lpoption=page-left=54 : lpoption=page-top=54
Note that we are re-spooling the job to an office queue, and that when doing so, we specify several lp options to affect formatting as the job is converted to the output appropriate for the final destination printer as specified in CUPS. Here, these are all margin adjustments, our simplest setup.
I provide our full SUNY lpskim configuration for all queues across a range of HP and Dell PostScript laser printers for reference. This file should be saved at /etc/cups/lpskim.conf. Notes follow.
lpskim.txt
- SUNY1 (checks): Fairly straightforward text manipulation and formatted re-spooling, except for a cut/replace to move the date for new check stock.
- SUNY4 (batch control): Formatted re-spooling with conf_switch options to re-base the configuration on either the SUNY4_CHECKREG or SUNY4_QUICKPAY configs based on patterns in the job. With all options the same, these proved unnecessary in the end, but were left in case new printers introduced new wrinkles.
- SUNY9 (refund): Minute adjustments to cpi and lpi so output would fit a rough photocopy of a form, and line deletions near the end to eliminate a few extraneous lines that unavoidably shift to a second page as a result.
- SUNYE (vouchers): A pair of filters to change line feeds and the like. Also note the commented-out alternate configuration that prepended a PCL string for a previous printer.
The rest of the queues are straightforward formatted re-spooling, a few involving a change to landscape, some margin adjustments, etc.
See http://fritz.potsdam.edu/projects/cupsapps/lpskim to download lpskim and for full documentation of all options. Happy printing.
Permalink
09.25.13
Posted in Technology at 03:46:41 by Jeff Hardy
Another fedup upgrade brought me up to Fedora 19 a couple of weeks ago. Pre-upgrade, my outstanding oom-killer bug seemed to get worse before it got better. Since I consider it a last resort to change likely unrelated variables in the face of a problem (entire OS), I stuck it out on Fedora 18 until, luckily, a kernel update arrived that seemed to stabilize the issue (inadvertently or otherwise, 3.10.7). With a known good kernel available in both new and old, it paved the way for upgrade.
Stability has persisted, despite the fact that I am still on my legacy hardware, three graphics cards, four monitors setup. I am experiencing a problem with pidgin, for which I have opened a new bug. Such is technology.
Permalink
04.08.13
Posted in Technology at 01:50:54 by Jeff Hardy
The new fedup process completed my upgrade to Fedora 18 with no issues a couple of months ago. After the normal minor wrangling to get four monitors working again, I immediately began experiencing a more serious issue: persistent oom-killer, typically a few hours after boot and despite plenty of memory.
Oom-killer continued to target various aspects of X, so trying to rule things out, I tried different desktop environments. Unfortunately, it has persisted across XFCE, LXDE, and KDE. All attempts at isolating a single problematic application have come up empty. A decent resource on tracing memory issues like this is http://bl0rg.krunch.be/oom-frag.html, which analysis points to fragmentation, but in my case I seem to be exhausting DMA memory. I created a bug tracking the issue.
vm.panic_on_oom = 1
kernel.panic = 10
I had to resort to the above sysctl.conf, with further mitigation being to reboot at the beginning and end of the day, leaving the system in runlevel 3 when not in use. Barely tolerable. My least successful upgrade.
Permalink
« Previous entries Next Page » Next Page »