Workaround: reiserfsck –rebuild tree “out of space aborted”

This is an article about why not to use reiserfs with mythtv and what to do if it goes horribly wrong. So I came home to my media box about a month ago and it was locked up. I shell’d in and poked around and it said the /home partition was full which seemed unlikely since its 550 gigs so I thought I would give the box a reboot and it would be ok. Reiser has just made some free space mistake I thought. Just a small problem I thought, be quickly fixed.

The reiserfsck said the disk had some corruption and needed the tree rebuilt. No problem I said and got that started. Turns out that takes a long time – for me in the vicinity of four days. After four days of patiently waiting I was greeted by the nice message “out of space aborted”.

Well it turns out that you are not meant to use mythtv with ReiserFS v3, and there is some debate about whose fault it is. Seems there is a bug in mythtv which triggers a bug in reiser causing this corruption.

This is how I fixed it based on the very thoughtful advice of one of the reiser developers. I created a linear raid and added an extra gig on to the end. This is genius because other types of raid would be impossible to back out of, and I must admit up until this point I never understood the point of linear raid. But by using this technique you can add a small amount of space to filesystem and then remove it at a later date.

Of course we are not using a superblock because these are not raid formatted disks. Using this technique you can completely concatenate two disks in a rather dumb fashion, I had no idea mdadm could do this.

Then I simply ran the reiserfsck again

I’m not sure why at this point you say no to using a resizer. I guess because this would resize the partition and you would run into the same problem again?

Finally five days later it finished. I then resized the partition down again

Dear god that made me nervous, I did not want to sit for another five days waiting for it to repair. Luckily it finished fine. I then destroyed /dev/md3 and went back to using good old trusty md2.

General Linux Filesystem Ramblings

The folks over at reiser are a bit upset about this because it has given their filesystem a bad name and made it look bad and they blame mythtv for their bug. I don’t fully understand what mythtv is supposed to have done wrong but really the way I see it you can’t trust every app that you put on a box but you have to trust the filesystem. Especially since its in the kernel and things go horribly wrong if it doesn’t do its job. For this reason I think its very important that a filesystem is able to handle any bad malformed use of its API.

Although this was a huge waste of time and kept me from my media I’m not bitter or anti-reiser now. The problem I always face is that ext3 while rock solid is not very space efficient nor speedy, especially when deleting large trees of data or for directories that have hundreds of entries. Yet reiserfs and xfs with their logical journals seem prone to corruption in the face of powerfailures or crashes. Added to that I’ve seen xfs kernel dump production systems and when you are the one who has to fix these things you tend to be once bitten twice shy. At the moment I’m running JFS on my media box and we’ll see how that goes, I like the fact that it is backed by IBM.