Browsed by
Month: March 2011

Mounting Individual Parititons in RAW Disk Images

Mounting Individual Parititons in RAW Disk Images

Let us say, for the sake of simplicity, that you backup whole disks by using dd in Linux. Now let us assume that you need just a single file out of a backup and do not want to (or can not realistically) restore the whole backup to disk for the single file. You can not mount -o loop because the backup does not contain a single partition. “What to do” you ask? Well I will tell you, wary traveler.

  1. Mount your raw disk image as a loopback device: losetup /dev/loop0 [path to image]
  2. Run kpartx and: kpartx -va /dev/loop0
    This will add your partitions to /dev/mapper/loop0pX where each X is a different partition
  3. You can now mount each partition: mount /dev/mapper/loop0pX /media/partition
    If you want to make sure you do not write anything to your backup simply mount it as read-only: mount -o ro /dev/mapper/loop0pX /media/partition

Once you are finished you will need to do your clean up.

  1. Unmount: umount /media/partition
  2. Remove the mapper devices: kpartx -d /dev/loop0
  3. Remove the loopback device: losetup -d /dev/loop0

These commands may all need to be run as root. If loop0 reports that it is busy just pick another loop device (/dev/loop1, ect).

Crazy RAID Fun

Crazy RAID Fun

I have been experimenting with a number of things in Linux as of late. Breaking out my mad scientist cap– did I ever put it away?– one such thing I started screwing with was software-based RAID arrays in Linux. I wanted to see if I could create a number of encrypted, compressed files, spread them all over the world, and mount them in an RAID array (there is no way to say that any simpler without taking away from it). I put together a little plan in my head and I was off.

I stuck a bunch of Ubuntu installations all over different geographical locations and connected them via OpenVPN and NFS (CIFS would also have worked although probably with a noticeable performance hit). Using dd I created a 1GB file on each of them. Using losetup and its built-in encryption I mounted them all as loopback devices on my local machine. Using mdadm I turned the loopback devices into a RAID array and stuck ext4 on it. Using fusecompress I mounted said ext4 volume to compress everything on the fly.

What possible use could this serve? Well, there is the reason I did it: it is jawesome sauce and I wanted to see if I could. Those reasons, of course, are not a purpose. I suppose, if one really wanted to hide their data, they could use this in a RAID0– or a RAID5/6, even– to spread the data around using a very small cluster size. Theoretically, no two "neighbouring" clusters on the filesystem would be at the same geographical location. This means that if one– or more– sites were compromised not only would the attacker have so little of the data as to be useless but due to the way encryption tends to work (it is nearly impossible to unencrypted anything without the "neighbouring" data) it would be extra useless to them.

Like so many of my experiments this was never meant to be a practical anything. Yeah, I got it working with little difficulty– the real problem was driving around and convincing my friends to let me use their houses and bandwidth– but I can think of a number of issue that would arise in real-world use. First off, the whole concept is predicated on the use of files mounted on network exports. Files on exports potentially far, far away from you. If a network link goes down– or hiccups– you would probably have a bit of an issue. Sure it is all in a RAID but your NFS– or CIFS– mounts are going to wait a bit before they decide to let you know they have gone down. I imagine this would manifest itself as a temporary “freeze” of the filesystem but I have not tested it. A second issue is if you are using a RAID0 for maximum security (as mentioned above) losing anything at all would kill the whole setup. Consider that if your backup is not as secure as your primary then what is the point? Thirdly, depending on which RAID level you choose, you may quickly realise that not all link bandwidth or latency is created equal. I did run into a common mdadm issue where it did not release the devices but did not put any effort into fixing it.

All-in-all I am pretty excited for no good reason; I suppose I just thought it was neat. I do not recommend relying on this unless you can solve the above problems. iSCSI was designed for such things so if you are hell-bent on implementing this idea I suggest you use that instead of the loopback devices. You might have to find a new way to encrypt everything (I bet TrueCrypt would work).

I did write a few short scripts to make my lifer easier but they have no error checking so I decided not to post them. If there is enough interest I could finish them (IE add the error checking) and post them.

Clone, Wipe, or Image Media Using dd to Create RAW Disk Images

Clone, Wipe, or Image Media Using dd to Create RAW Disk Images

I recently came across five Windows machines I needed to get working. These were all to be used for the same purpose and thus I would have had to do the same work five times over. I decided to do the work once and then just clone the original disk using dd on Linux.

What follows are a number of example usages of the dd utility:

  • Make an ISO from a CD/DVD:

    dd if=[device] of=disk.iso bs=2048 conv=sync

  • Overwrite a disk with random data:

    dd if=/dev/urandom of=[device]

    One could use /dev/zero instead of /dev/urandom if they just wanted to zero out a disk.
    Darik’s Boot And Nuke might be a better tool for this.

  • Clone a partition:

    dd if=[device partition source] of=[device partition destination] bs=4K conv=noerror,sync

    Create unformatted partitions first.

  • Clone a disk:

    dd if=[device source] of=[device destination] bs=4K conv=noerror,sync

  • Create an image from a disk:

    dd if=[device source] of=[image destination] bs=4K conv=noerror,sync

  • Restore an image to a disk:

    dd if=[image source] of=[device destination] bs=4K

    .

  • Create a compressed image from a disk:

    dd if=[device source] bs=4K conv=noerror,sync | gzip -c > [image destination]

    It would be a good idea to have your image name end with .gz so you later remember it is a compressed image and also clear out temporary directories.

    Adding --fast to gzip will not stress your CPU as badly (good for netbooks and the like). Adding --best to gzip will give you the best compression but drastically slow you down.

    On a well-traveled filesystem zeroing out the free space can lead to huge space savings. In Linux you can use cd [mounted filesystem] && sudo dd if=/dev/zero of=delme-0 bs=1M &> /dev/null ; sudo dd if=/dev/zero of=delme-1 bs=1 &> /dev/null ; sync && sudo rm delme-1 delme-0 on the mounted filesystem (if you have multiple partitions you should do it for each) while Windows offers sdelete -z [drive letter]:. Depending on the amount of free space this could take a good long while.

  • Restore a compressed image to a disk:

    gzip -d -c [image source] | dd of=[image destination]

If you do not have a working Linux installation you can always use a Live-CD. Use fdisk -l or GParted to figure out what a device is named.