Yesterday I spent half the day lamenting my own stupidity. Remember that large RAID array I mentioned previously? Well I did a stupid thing. I'm getting ahead of myself though.
When I put together the new server I decided to add a couple more of the Seagate 2TB Barracuda XTs to the array. I had wanted a hot-spare on the array (an empty drive set aside so that if a drive from the array fails the RAID card will automatically rebuild the array on the hot-spare) and the extra one was for a little more space to "future-proof" me just a bit. Adding the hot-spare was no problem. After installing the Adaptec Storage Manager on the server (one of the few graphical tools worth jumping to level 5 for) I simply added the first new drive as a hot-spare and no extra work was necessary. Expanding the array with the other drive was also easy, but a little time-consuming (it took three days) since the array was still accessible during that time, albeit at a severe loss to speed.
So now I have a larger array but a partition that doesn't cover the whole space. To fix this, I got into the yast partition program. I figured it would be as simple as resizing the partition to encompass the extra space. I'd done that before on other drives and figured it would be no problem. What I didn't realize was that when I first created a partition on my array Linux had defaulted to making it a GPT (GUID Partition Table), which means that when a partition is created, the backup GPT files were put at the end of the array. Because I'd extended the array, the backup file was now somewhere in the middle of it. Yast didn't bother to tell me that, it just refused to re-size the array. If you're reading this and you are working as a developer on the openSUSE project, this is something you might think about fixing. Anyway, I thought that maybe the partition was still somehow being mounted to the system (despite making sure to umount the drive), so I decided I'd better boot up PartedMagic Live from a USB stick.
PartedMagic is a small but powerful distribution of Linux that takes up just over a hundred MBs of space. It is generally loaded directly into the RAM and thus does not have to mount any hard disc in order to run. This makes it the absolutely perfect tool to do exactly what it was designed to do, namely: partition drives. I popped in my Live USB and booted to RAM. Next I opened up the Partition Editor (GParted) and attempted to grow the partition to fill the array. For about a minute it scanned the drive and all looked like it was going well, until it stopped completely and threw out the exact error I described above. It told me that the backup GPT was in the middle of the logical drive and that it needed to be moved to the end. It did not offer any tips on how to do this. If you are reading this and you are a developer for PartedMagic or GParted, you might want to think about including the ability to do this from that program, or at least point people in the right direction. As it was, I did the absolute wrong thing.
First, I got on the openSUSE forum and asked for advice on how to move the GPT backup. This was the right thing. The wrong thing was getting tired of waiting for a response and trying to follow a man (manual) page for sgdisk. On top of which I totally f'd up the command. I typed "# sgdisk -e /dev/sdb1". Now, anyone who has a firm understanding of the Linux file system can immediately tell what I did wrong with this. sgdisk and gdisk are utilities designed to be to GPT disks what fdisk is to MBR (Master Boot Record) disks. The -e option moves the GPT backup to the end of the disk, or at least it's supposed to, and if I had typed "# sgdisk -e /dev/sdb" that's what it would have done. sdb is the entire array. If it were actually a physical disk, sdb would be the whole disk. Since it's technically a RAID array it's actually a logical disk, but you get the idea. sdb1 on the other hand, is the partition I'd originally set up. By telling sgdisk I wanted the backup moved to the end of sdb1, my best guess is that it overwrote important file system information (like maybe a superblock or two). Not yet seeing my mistake, I rebooted the system.
When I got back to PartedMagic and ran Partition Manager, I found the array completely unreadable. I kind of panicked. I immediately recalled my error and started flipping out, assuming the worst. Well, not exactly flipping out. All important info was backed up on two separate external drives, but it would have been a pain to re-rip my dvd's to xvid. Now, at this juncture I'm going to point out some flaws in my logic. The first is, that if sgdisk is worth it's weight in dung it should realize that no user wants their backup GPT put in the middle of a disk, and should automatically correct you and put it in the right spot. The second is that since the first partition spanned the entire size of the logical volume when it was created (which it did) then moving it to the end of that partition would have put it right on top of itself and shouldn't have had any effect. Regardless, I tried to unf#*# my mistake by running sgdisk again with the correct sdb instead of sdb1. Rebooted... same problem.
Again, I'm going to make myself look quite stupid by telling you that at the time I was running under the assumption that it was a partition problem, when I should have seen quite clearly that the Partition Manager could still see the partition, just couldn't recognize the file system. After running gdisk -v I found that it was recognizing an error as a GPT MBR mismatch, making me further believe that it was a partition problem. I almost deleted the partition so that I could attempt to recreate it using testdisk (great utility I wish I had time to talk about now). Luckily, I finally realized that it had to be a file system error and searched about a bit until I found fsck. fsck (File System ChecK) is a utility that used to be commonly run to fix file system errors, mostly before journaling file systems came to prominence. I found it on a page talking about replacing lost superblocks and ran the utility on the partition just to see if maybe this was the problem. It immediately recognized that the file system was broken and asked if I wanted to fix it. I figured what the hell, I didn't really have much of a choice. My only regret in this regard is that I wish I would have appended -a to the command because I spent the next hour and a half hitting the "y" key (fsck asks pretty often if you want it to do stuff, usually very small, miniscule tasks - a lot of them). At the end of which BAM, there was my beautiful array again, working like a charm. Back in PartedMagic I tried again to grow the partition to it's physical limits and sure enough it worked this time.
I still don't know what happened. If you do, by all means leave me a comment and tell me. Obviously I screwed up the file system somehow, I'm just not sure how. Regardless, in some ways I'm glad it happened. First off, it reinforced my desire to do the one most important thing you can do with your data: back it up, back it up, back it up (yes, just like all the rappers tell you to). Second, it gave me a great reason to do a whole lot of research about file systems, partitioning structures and Linux in general. Every time I make a mistake I end up learning so much more than if my operation had turned out as planned. I have SO MUCH to learn!
Sunday, August 21, 2011
Friday, August 19, 2011
Samba Sharing
Before recently putting together the openSUSE server I showed in the last post, I had the desktop set up as a slightly more humble home server. The RAID card and array were set up in it and it was running openSUSE 11.3, just one step down from what the server's running now. One of the basic functions of my server is to share the RAID array with the other computers in my house. I have terabytes of movies and television ripped from my discs into Xvids that take up a lot of space. I used to have a WDTV Live that streamed everything off my server but while I was at my last duty station in Maryland I decided to start watching more Hulu which I couldn't do from my WDTV Live, plus I thought it would be cool to browse the net from the couch. I purchased a Logitech Revue but found out that most sites block Google TV devices, and that it was just all around not very powerful for surfing the web and such, so I purchased the components for the HTPC and put it together, first trying Ubuntu and XBMC, and then eventually switching to Windows.
My server has to be able to share the RAID array with the HTPC, as well as my wife's laptop and mine too, as well as both the desktops. It houses not just my movie and TV collection but our family photos, videos and documents as well, not to mention my vast library of RAW photos from my Nikon D700. Now before you start preaching about how a RAID array does not guarantee data safety I'll tell you that all important documents, videos and photos are backed up onto two separate external HDDs, one attached to the server and one attached to desktop 2. I use RSync for that function and I'll talk more about that in a later post.
My point is that the server has to be able to share all that information securely and efficiently over our wireless network. Those of you who have some computer experience will know that Linux and Windows do not automatically get along when it comes to sharing files. Linux's open source nature is awesome when it comes to reading other file systems and handshaking with other OS's, but Microsoft likes to make things difficult by being proprietary with their code. The answer is Samba. Samba is a daemon (background program) that works in virtually all Linux distros. It makes it possible for a Linux machine to become part of a workgroup and share files and directories with Windows machines. On my last server I had Samba up and running with a guest only share of my RAID array, meaning no username or password were necessary to log on. I wanted this server to require both a username and password, but I thought that with only minor tweaking my /etc/samba/smb.conf file would run samba just the same on my new system. I was dead wrong.
Samba uses a configuration file called smb.conf to know how to set up shares in Linux. I thought by simply transferring it to the new system and tweaking it a little to include security features it would be up and running in no time flat. I initially configured Samba using Yast (openSUSE's configuration tool, stands for "Yet Another Setup Tool"), mostly to configure openSUSE to start Samba during bootup, then I copied the smb.conf file into /etc/samba, taking care to backup the original. I edited out the guest only and guest options from the share and reset the source to it's new home at /srv, hopped on my desktop and opened up the Network. Nothing. Couldn't even see the system at all.
Then I thought to myself "DOH, I haven't set up any users for Samba." I'm not sure why I thought that would help not being able to see the system but it was a necessary step. First I created a new group of users within openSUSE using "groupadd Samba". I then created two new users in group Samba using "useradd -G Samba 'username'". We'll call them Jack and Jill. Jack was my logon and I changed the owner of the the RAID drive using "chown -R Jack /srv", then I changed the permissions of /srv using "chmod -R 750 /srv" which makes it so that Jack can read, write and execute files from /srv, Jill (and anyone else in group Samba) can read and execute files on /srv, and anyone else (not in group Samba) can't do jack, pun intended. Then I went ahead and added each of the users to the Samba user list by using "smbpasswd -a Jack" and ditto for Jill. With my users all ready to go I tried accessing again and was once more stymied, no Server showed up.
After literally beating my brain against the wall for what seemed like forever but was in reality about three hours, I stumbled onto a site that mentioned something interesting in regard to Samba and openSUSE 11.4. Apparently as of 11.4, Novell's AppArmor defaults to quashing Samba unless told to do otherwise. Here's how you do it: Start up yast, select "AppArmor Control Panel" then "Configure Profile Modes" and the "Set profile modes [Configure]" In the Profile Mode Configuration screen, set usr.sbin.nmbd and usr.sbin.smbd for "complain" instead of "enforce". BAM, problem solved. Checked my desktop and sure enough there was the Server. Logged into both users just to check and sure enough had no problems.
Hope somebody followed along and found this useful. If you know how I could have done this better or have suggestions then be sure to leave a comment! Tomorrow I have quite the story to tell about RAID array extension, partition growth and file system errors. Stay tuned!
My server has to be able to share the RAID array with the HTPC, as well as my wife's laptop and mine too, as well as both the desktops. It houses not just my movie and TV collection but our family photos, videos and documents as well, not to mention my vast library of RAW photos from my Nikon D700. Now before you start preaching about how a RAID array does not guarantee data safety I'll tell you that all important documents, videos and photos are backed up onto two separate external HDDs, one attached to the server and one attached to desktop 2. I use RSync for that function and I'll talk more about that in a later post.
My point is that the server has to be able to share all that information securely and efficiently over our wireless network. Those of you who have some computer experience will know that Linux and Windows do not automatically get along when it comes to sharing files. Linux's open source nature is awesome when it comes to reading other file systems and handshaking with other OS's, but Microsoft likes to make things difficult by being proprietary with their code. The answer is Samba. Samba is a daemon (background program) that works in virtually all Linux distros. It makes it possible for a Linux machine to become part of a workgroup and share files and directories with Windows machines. On my last server I had Samba up and running with a guest only share of my RAID array, meaning no username or password were necessary to log on. I wanted this server to require both a username and password, but I thought that with only minor tweaking my /etc/samba/smb.conf file would run samba just the same on my new system. I was dead wrong.
Samba uses a configuration file called smb.conf to know how to set up shares in Linux. I thought by simply transferring it to the new system and tweaking it a little to include security features it would be up and running in no time flat. I initially configured Samba using Yast (openSUSE's configuration tool, stands for "Yet Another Setup Tool"), mostly to configure openSUSE to start Samba during bootup, then I copied the smb.conf file into /etc/samba, taking care to backup the original. I edited out the guest only and guest options from the share and reset the source to it's new home at /srv, hopped on my desktop and opened up the Network. Nothing. Couldn't even see the system at all.
Then I thought to myself "DOH, I haven't set up any users for Samba." I'm not sure why I thought that would help not being able to see the system but it was a necessary step. First I created a new group of users within openSUSE using "groupadd Samba". I then created two new users in group Samba using "useradd -G Samba 'username'". We'll call them Jack and Jill. Jack was my logon and I changed the owner of the the RAID drive using "chown -R Jack /srv", then I changed the permissions of /srv using "chmod -R 750 /srv" which makes it so that Jack can read, write and execute files from /srv, Jill (and anyone else in group Samba) can read and execute files on /srv, and anyone else (not in group Samba) can't do jack, pun intended. Then I went ahead and added each of the users to the Samba user list by using "smbpasswd -a Jack" and ditto for Jill. With my users all ready to go I tried accessing again and was once more stymied, no Server showed up.
After literally beating my brain against the wall for what seemed like forever but was in reality about three hours, I stumbled onto a site that mentioned something interesting in regard to Samba and openSUSE 11.4. Apparently as of 11.4, Novell's AppArmor defaults to quashing Samba unless told to do otherwise. Here's how you do it: Start up yast, select "AppArmor Control Panel" then "Configure Profile Modes" and the "Set profile modes [Configure]" In the Profile Mode Configuration screen, set usr.sbin.nmbd and usr.sbin.smbd for "complain" instead of "enforce". BAM, problem solved. Checked my desktop and sure enough there was the Server. Logged into both users just to check and sure enough had no problems.
Hope somebody followed along and found this useful. If you know how I could have done this better or have suggestions then be sure to leave a comment! Tomorrow I have quite the story to tell about RAID array extension, partition growth and file system errors. Stay tuned!
Equipment
I'm going to start off by detailing my equipment. This serves two purposes. First, I put a lot of time and effort into putting my systems together so I want to show them off a little. Second, and far more importantly, is to inform others of what I'm working with so if I have a problem that is hardware related others might be able to spot it. Here's my work space, you'll probably need to click it and make it larger to see the text.
And here's a photo of the server with the LCD console out.
I don't normally run the GUI on the server but my RAID card's storage manager program is much easier to deal with in GUI form and I'm expanding my RAID 5 array. That brings me to the stats.
Server:
-openSUSE 11.4 defaulting to runlevel 3 (it boots to a command line)
-Norco RPC-4020 4U Server Case w/ 20 Hot-Swappable SATA/SAS Drive Bays
-Corsair Professional Series Gold High Performance 750-Watt Power Supply CMPSU-750AX
-12x 4GB DDR3 @1333 MHz RDIMM
-ASUS Motherboard KGPE-D16 with ASMB4-IKVM card
-2x AMD Opteron 6128 (16 physical cores @ 2.0GHz)
-OCZ 30GB Vertex Turbo Series SSD OCZSSD2-1VTXT30G (for the OS)
-Adaptec Raid 5805Z running 6x Seagate 2TB Barracuda XT in RAID 5 ext4 array (five in array & one hotspare making 7.266 usable TBs) mounted at /srv
-Lite ON 8X DVDRW Slim Internal
-ATEN CL1000M 17" SingleRail LCD Integrated Console (on rack)
You might be wondering why I payed for the LCD console when I have a several computer monitors mere feet away. Well, for one, the server rack will probably change locations. Two, I intend to add on to the rack with a dedicated firewall system shortly which will share a KVM switch with the server. Now to the desktop.
Desktop 1:
-Windows 7
-NZXT TEMPEST Enthusiast Steel ATX Midtower
-Corsair CMPSU-750TX 750-Watt TX Series 80 Plus Certified Power Supply
-CORSAIR XMS3 DHX 8GB ( 4 X 2GB ) PC3-10666 1333MHz DDR3
-ASUS Crosshair IV Formula - AM3 - AMD 890FX - DDR3 - Republic of Gamers - ATX Motherboard
-AMD Phenom II X6 1055T Thuban 2.8 GHz Six-Core Processor (Overclocked to 3.5 GHz)
-XFX Radeon HD 5770 1GB DDR5
-Corsair Hydro Series H80 High Performance Liquid CPU Cooler (CWCH80)
-Intel 510 Series 120GB SATA SSD (for OS and programs)
-Western Digital Scorpio 2.5-Inch 500 GB SATA (for storage)
Desktop 2 is a cannibalized system that used to be my main desktop but now runs openSUSE 11.4. It's my network slave for use in testing. It's stats aren't really worth mentioning. The netbook is running openSUSE 11.4 and if you want to see it's stats you can just look up the HP-Mini 210.
Oh, I almost forgot the Home Theater PC.
HTPC:
-Windows 7 (tried running Ubuntu but I just couldn't get all my movie files to play well)
- Apex MI-100 4BAY Desktop Mitx
-Diablotek 380-Watt Micro ATX Power Supply PHD380M
-Corsair 4 GB Memory Kit 1333mhz DDR3
-ZOTAC LGA 1156 Mini ITX Intel Motherboard H55ITX-A-E
-Intel Core I3-540 3.06GHz 4MB LGA1156 CPU
-Sapphire Radeon HD4550 512 MB DDR3 Video Card
-Seagate Barracuda 500 GB 7200RPM HDD
Not sure why I bothered with 500GB since it only streams off the network anyway, but oh well. The blue antenna thing thing hanging out in front there is a wireless antenna. Between that and the wireless extender coming off my router I get five solid bars of wireless N signal. Not an easy feat here in Italy due to the fact that their walls are cinder block, concrete and steel. Although my computer room and the living room the HTPC resides in are no more than 50ft from each other, I couldn't get more than three bars of signal until I added the extender and the antenna. This might seem trivial but with just three bars I was getting a lot of stuttering when streaming HD video content over the network. In my last place I just ran Cat5 to it but due to Italian construction that just wasn't feasible here.
That's it, I promise. At least for now... I'll be adding another system to the rack when I get a chance that will act as a dedicated firewall. I need to become more proficient at IPtables before I do that, but more about that later. Like my systems? Think they're poorly designed and terrible? Either way, comment and let me know!
And here's a photo of the server with the LCD console out.
I don't normally run the GUI on the server but my RAID card's storage manager program is much easier to deal with in GUI form and I'm expanding my RAID 5 array. That brings me to the stats.
Server:
-openSUSE 11.4 defaulting to runlevel 3 (it boots to a command line)
-Norco RPC-4020 4U Server Case w/ 20 Hot-Swappable SATA/SAS Drive Bays
-Corsair Professional Series Gold High Performance 750-Watt Power Supply CMPSU-750AX
-12x 4GB DDR3 @1333 MHz RDIMM
-ASUS Motherboard KGPE-D16 with ASMB4-IKVM card
-2x AMD Opteron 6128 (16 physical cores @ 2.0GHz)
-OCZ 30GB Vertex Turbo Series SSD OCZSSD2-1VTXT30G (for the OS)
-Adaptec Raid 5805Z running 6x Seagate 2TB Barracuda XT in RAID 5 ext4 array (five in array & one hotspare making 7.266 usable TBs) mounted at /srv
-Lite ON 8X DVDRW Slim Internal
-ATEN CL1000M 17" SingleRail LCD Integrated Console (on rack)
You might be wondering why I payed for the LCD console when I have a several computer monitors mere feet away. Well, for one, the server rack will probably change locations. Two, I intend to add on to the rack with a dedicated firewall system shortly which will share a KVM switch with the server. Now to the desktop.
Desktop 1:
-Windows 7
-NZXT TEMPEST Enthusiast Steel ATX Midtower
-Corsair CMPSU-750TX 750-Watt TX Series 80 Plus Certified Power Supply
-CORSAIR XMS3 DHX 8GB ( 4 X 2GB ) PC3-10666 1333MHz DDR3
-ASUS Crosshair IV Formula - AM3 - AMD 890FX - DDR3 - Republic of Gamers - ATX Motherboard
-AMD Phenom II X6 1055T Thuban 2.8 GHz Six-Core Processor (Overclocked to 3.5 GHz)
-XFX Radeon HD 5770 1GB DDR5
-Corsair Hydro Series H80 High Performance Liquid CPU Cooler (CWCH80)
-Intel 510 Series 120GB SATA SSD (for OS and programs)
-Western Digital Scorpio 2.5-Inch 500 GB SATA (for storage)
Desktop 2 is a cannibalized system that used to be my main desktop but now runs openSUSE 11.4. It's my network slave for use in testing. It's stats aren't really worth mentioning. The netbook is running openSUSE 11.4 and if you want to see it's stats you can just look up the HP-Mini 210.
Oh, I almost forgot the Home Theater PC.
HTPC:
-Windows 7 (tried running Ubuntu but I just couldn't get all my movie files to play well)
- Apex MI-100 4BAY Desktop Mitx
-Diablotek 380-Watt Micro ATX Power Supply PHD380M
-Corsair 4 GB Memory Kit 1333mhz DDR3
-ZOTAC LGA 1156 Mini ITX Intel Motherboard H55ITX-A-E
-Intel Core I3-540 3.06GHz 4MB LGA1156 CPU
-Sapphire Radeon HD4550 512 MB DDR3 Video Card
-Seagate Barracuda 500 GB 7200RPM HDD
Not sure why I bothered with 500GB since it only streams off the network anyway, but oh well. The blue antenna thing thing hanging out in front there is a wireless antenna. Between that and the wireless extender coming off my router I get five solid bars of wireless N signal. Not an easy feat here in Italy due to the fact that their walls are cinder block, concrete and steel. Although my computer room and the living room the HTPC resides in are no more than 50ft from each other, I couldn't get more than three bars of signal until I added the extender and the antenna. This might seem trivial but with just three bars I was getting a lot of stuttering when streaming HD video content over the network. In my last place I just ran Cat5 to it but due to Italian construction that just wasn't feasible here.
That's it, I promise. At least for now... I'll be adding another system to the rack when I get a chance that will act as a dedicated firewall. I need to become more proficient at IPtables before I do that, but more about that later. Like my systems? Think they're poorly designed and terrible? Either way, comment and let me know!
1st Post
I'm Nick. Full time US Army soldier and part-time-nerd. By this I mean that in my spare time I work with computers, not just because I have an abiding interest in technology, but also because my long-term goal is to leave the military at the end of my contract (still two and a half years away) and become a program developer / Linux network administrator. Before I joined the military I was an English major with three years under my belt. When I get out of the military I'm going to be switching gears completely and getting a Computer Science degree using my GI Bill money. In the meantime, I want to learn as much as a I can about computer systems in an effort to become Linux and Oracle certified so that when I go back to school I can work part-time in the industry and thus have a stronger resume going into the job market. I have a long way to go and I'm WAY behind the curve. I'm 26 years old and have only an enthusiasts grasp of computer basics. This blog is going to be an ongoing public record of my struggle to learn the in's and out's of Linux (specifically openSUSE as it's my chosen distribution), networking, and software development. I hope that through documenting this process I can be informative to others and also gather feedback and advice from enthusiasts and those in the industry. My current job in the military is as a Combat Camerman, which means taking photos and video for the military (the photo above is me in my photo studio). I'm stationed in Vicenza, Italy.
Subscribe to:
Posts (Atom)