Yesterday I spent half the day lamenting my own stupidity. Remember that large RAID array I mentioned previously? Well I did a stupid thing. I'm getting ahead of myself though.
When I put together the new server I decided to add a couple more of the Seagate 2TB Barracuda XTs to the array. I had wanted a hot-spare on the array (an empty drive set aside so that if a drive from the array fails the RAID card will automatically rebuild the array on the hot-spare) and the extra one was for a little more space to "future-proof" me just a bit. Adding the hot-spare was no problem. After installing the Adaptec Storage Manager on the server (one of the few graphical tools worth jumping to level 5 for) I simply added the first new drive as a hot-spare and no extra work was necessary. Expanding the array with the other drive was also easy, but a little time-consuming (it took three days) since the array was still accessible during that time, albeit at a severe loss to speed.
So now I have a larger array but a partition that doesn't cover the whole space. To fix this, I got into the yast partition program. I figured it would be as simple as resizing the partition to encompass the extra space. I'd done that before on other drives and figured it would be no problem. What I didn't realize was that when I first created a partition on my array Linux had defaulted to making it a GPT (GUID Partition Table), which means that when a partition is created, the backup GPT files were put at the end of the array. Because I'd extended the array, the backup file was now somewhere in the middle of it. Yast didn't bother to tell me that, it just refused to re-size the array. If you're reading this and you are working as a developer on the openSUSE project, this is something you might think about fixing. Anyway, I thought that maybe the partition was still somehow being mounted to the system (despite making sure to umount the drive), so I decided I'd better boot up PartedMagic Live from a USB stick.
PartedMagic is a small but powerful distribution of Linux that takes up just over a hundred MBs of space. It is generally loaded directly into the RAM and thus does not have to mount any hard disc in order to run. This makes it the absolutely perfect tool to do exactly what it was designed to do, namely: partition drives. I popped in my Live USB and booted to RAM. Next I opened up the Partition Editor (GParted) and attempted to grow the partition to fill the array. For about a minute it scanned the drive and all looked like it was going well, until it stopped completely and threw out the exact error I described above. It told me that the backup GPT was in the middle of the logical drive and that it needed to be moved to the end. It did not offer any tips on how to do this. If you are reading this and you are a developer for PartedMagic or GParted, you might want to think about including the ability to do this from that program, or at least point people in the right direction. As it was, I did the absolute wrong thing.
First, I got on the openSUSE forum and asked for advice on how to move the GPT backup. This was the right thing. The wrong thing was getting tired of waiting for a response and trying to follow a man (manual) page for sgdisk. On top of which I totally f'd up the command. I typed "# sgdisk -e /dev/sdb1". Now, anyone who has a firm understanding of the Linux file system can immediately tell what I did wrong with this. sgdisk and gdisk are utilities designed to be to GPT disks what fdisk is to MBR (Master Boot Record) disks. The -e option moves the GPT backup to the end of the disk, or at least it's supposed to, and if I had typed "# sgdisk -e /dev/sdb" that's what it would have done. sdb is the entire array. If it were actually a physical disk, sdb would be the whole disk. Since it's technically a RAID array it's actually a
logical disk, but you get the idea. sdb1 on the other hand, is the partition I'd originally set up. By telling sgdisk I wanted the backup moved to the end of sdb1, my best guess is that it overwrote important file system information (like maybe a superblock or two). Not yet seeing my mistake, I rebooted the system.
When I got back to PartedMagic and ran Partition Manager, I found the array completely unreadable. I kind of panicked. I immediately recalled my error and started flipping out, assuming the worst. Well, not exactly flipping out. All important info was backed up on two separate external drives, but it would have been a pain to re-rip my dvd's to xvid. Now, at this juncture I'm going to point out some flaws in my logic. The first is, that if sgdisk is worth it's weight in dung it should realize that no user wants their backup GPT put in the middle of a disk, and should automatically correct you and put it in the right spot. The second is that since the first partition spanned the entire size of the logical volume when it was created (which it did) then moving it to the end of that partition would have put it right on top of itself and shouldn't have had any effect. Regardless, I tried to unf#*# my mistake by running sgdisk again with the correct sdb instead of sdb1. Rebooted... same problem.
Again, I'm going to make myself look quite stupid by telling you that at the time I was running under the assumption that it was a partition problem, when I should have seen quite clearly that the Partition Manager could still see the partition, just couldn't recognize the file system. After running gdisk -v I found that it was recognizing an error as a GPT MBR mismatch, making me further believe that it was a partition problem. I almost deleted the partition so that I could attempt to recreate it using testdisk (great utility I wish I had time to talk about now). Luckily, I finally realized that it had to be a file system error and searched about a bit until I found fsck. fsck (File System ChecK) is a utility that used to be commonly run to fix file system errors, mostly before journaling file systems came to prominence. I found it on a page talking about replacing lost superblocks and ran the utility on the partition just to see if maybe this was the problem. It immediately recognized that the file system was broken and asked if I wanted to fix it. I figured what the hell, I didn't really have much of a choice. My only regret in this regard is that I wish I would have appended -a to the command because I spent the next hour and a half hitting the "y" key (fsck asks pretty often if you want it to do stuff, usually very small, miniscule tasks - a lot of them). At the end of which BAM, there was my beautiful array again, working like a charm. Back in PartedMagic I tried again to grow the partition to it's physical limits and sure enough it worked this time.
I still don't know what happened. If you do, by all means leave me a comment and tell me. Obviously I screwed up the file system somehow, I'm just not sure how. Regardless, in some ways I'm glad it happened. First off, it reinforced my desire to do the one most important thing you can do with your data: back it up, back it up, back it up (yes, just like all the rappers tell you to). Second, it gave me a great reason to do a whole lot of research about file systems, partitioning structures and Linux in general. Every time I make a mistake I end up learning so much more than if my operation had turned out as planned. I have SO MUCH to learn!
