trying to solve IT problems

For this project the back-end was written in Java and stored in a mercurial (hg) repository, and the front-end was written in Ruby and stored in a git repository. To allow having branches across the project, we wanted to merge the two repositories into one. The target SCM was git.

In the result repository, there needed two be two subdirectories, back-end and front-end which contain the original projects.

Let’s start by converting the mercurial repository to a git:

git init back-end
cd back-end
hg-fast-export -r /path/to_hg/repo
git checkout 

Now move the contents into a subdirectory, rewriting history to preserve it:

git filter-branch --index-filter \
    'git ls-files -s | sed "s-\t\"*-&back-end/-" |
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
            git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD

This repository is now ready for merging.

cd ..

Now get the front-end repo and put its content in a subdirectory preserving history:

git clone <front-end-git-url> front-end
cd front-end
it filter-branch --index-filter \
    'git ls-files -s | sed "s-\t\"*-&front-end/-" |
        GIT_INDEX_FILE=$GIT_INDEX_FILE.new \
            git update-index --index-info &&
    mv $GIT_INDEX_FILE.new $GIT_INDEX_FILE' HEAD
cd ..

Now create a new repository for the combined project:

git init merged
cd merged
git pull ../back-end
git pull ../front-end

Done. Add the remote origin and push to finish.

Thanks to my colleague @huugsy and Henrik Nuy. With some additions to include the same information for Mercurial.

Include the snippet below in your /etc/bash.bashrc file:

# http://henrik.nyh.se/2008/12/git-dirty-prompt
# http://www.simplisticcomplexity.com/2008/03/13/show-your-git-branch-name-in-your-prompt/
# Mercurial additions see http://blog.progs.be/?p=351
#
# joachim likes it like this:
# joachim:~/dev/dir[master]$ # clean working directory 
# joachim:~/dev/dir[master⚡]$ # dirty working directory
 
function parse_git_dirty {
  [[ $(git status 2> /dev/null | tail -n1) != "nothing to commit, working directory clean" ]] && echo "⚡"
}
function parse_git_branch {
  git branch --no-color 2> /dev/null | sed -e '/^[^*]/d' -e "s/* \(.*\)/[\1$(parse_git_dirty)]/"
}
PS1='${debian_chroot:+($debian_chroot)}\u@\h:\w\$ '
 
function parse_hg_dirty {
  [[ $( hg status 2> /dev/null ) != "" ]] && echo "⚡"
}
function parse_hg_branch {
  hg branch 2> /dev/null | sed -e "s/\(.*\)/[\1$(parse_hg_dirty)]/"
}
 
 
export PS1='\u:\[\033[1;33m\]\w\[\033[0m\]$(parse_git_branch)$(parse_hg_branch)$ '

Do make sure you .gitignore or .hgignore file is complete or you will always see the dirty thunderbolt.

Once there was a server which contained two Xen virtual machines. All Ubuntu.
Then, suddenly (what else), disk failures. The backup was still on the todo list, so…

Many hours later trying to recover everything, I learned the following

Rescuing the domU images

(GNU) ddrescue to the rescue. This is a command which is similar to dd (which does low-level byte-by-byte disk copying), but with some added features for accessing failing disks. I used the command
ddrescue -v --no-split /dev/sdc sdc.img sdc.log
You could add additional steps to retry the failed parts hoping to recover more. This can only work if your new disk is bigger than the failing disk. Making the copy takes a lot of time. Copying files several GB in size takes a lot of time between healthy disks, reading a failing disk only makes it slower.

The disk can then be accessed using the loopback device. For starts, you can have a look at the partition table (giving valuable information for later, included in the output

$ fdisk -l -u -C 9039 sdc.img 

Disk sdc.img: 0 MB, 0 bytes
255 heads, 63 sectors/track, 9039 cylinders, total 0 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x000c6c37

  Device Boot      Start         End      Blocks   Id  System
sdc.img1   *   119684250   139219289     9767520   83  Linux
sdc.img2       139219290   145211534     2996122+   5  Extended
sdc.img3              63   119684249    59842093+  83  Linux
sdc.img5       139219353   145211534     2996091   82  Linux swap / Solaris

You need to know the number of cylinders of your original disk to view this info (the “-C 9039” parameter), which you can read from a fdisk on the original disk. So this command serves more as a sanity check that the copying was at least partially ok.

You can now access the partitions using the loopback device (as root, the mout point “/tst” in this example should already exist)

mount -o loop,offset=32256 -t ext3 sdc.img /test/
mount -o loop,offset=61278336000 -t ext3 sdc.img /test/

Once mounted, you can go through the fisk and retrieve the xen disk images. These images are also images just like the ones created using dd (or ddrescue) and can be mounted in the same way (as these are partition images, you don’t need the offset).

The rescueing may have broken the xen disk image. To fix (please do take another copy of the disk image first), unmount the disk and check/fix:

losetup /dev/loop0 disk.img
e2fsck -f /dev/loop0
losetup -d /dev/loop0

Fresh dom0 installation

While I love Ubuntu, recent versions don’t have Xen support anymore (dom0 was 8.04, the latest version where it was still included). The guides to get it installed did not work for me (including recompile of xen, kernel etc). Finally I ended up installing Debian Squeeze. See the guide at http://wiki.debian.org/Xen.

One point where the guide is not clear, you have to add the line
(network-script 'network-bridge antispoof=yes')
in /etc/xen/xend-config.sxp to make the network bridge work (the file seems to indicate using a network bridge is the default, but it made a difference for me).

Getting the domU system up and running again

This was (in hindsight, there was a lot of trial-and-error and googling to get to this point) quite easy.

In the domU configuration file, change the kernel and ramdisk to point to the booted version, in my case this was

kernel = '/boot/vmlinuz-2.6.32-5-xen-amd64'
ramdisk = '/boot/initrd.img-2.6.32-5-xen-amd64'

I could notw create the virtual machines, but they could not yet access the disk. Apparently Debian does not include the blocktap device for licensing reasons, so accessing the disk using the (supposedly faster) “tap:iao:” protocol fails. This should be replaced by “file:”. Secondly, my original configuration created the disk as “sda1”. For some reason this failed and needs to be replaced by “xvda1” (also in the “root” configuration).

Backups?

This time around, I did fix the backups. As I want these to be remote, I used dropbox http://wiki.dropbox.com/TipsAndTricks/TextBasedLinuxInstall. This works nicely but (as I don’t see how you can restrict what data is coming in without a GUI, it is best to create a specific dropbox account for the machine – the data can always be shared in other dropboxes).