Recovering data from a HP Media Vault

I had a customer bring in a drive from a HP Media Vault that had failed (or flaked out or something, we're really not sure yet). It was kind of interesting trying to find data on the drive, so I thought I'd share...

First, the drive obviously didn't have a standard PC partition table.

# fdisk -l /dev/hdi

Disk /dev/hdi: 300.0 GB, 300069052416 bytes
255 heads, 63 sectors/track, 36481 cylinders
Units = cylinders of 16065 * 512 = 8225280 bytes

Disk /dev/hdi doesn't contain a valid partition table

I should note that I tried testdisk at this point too, but it didn't find anything. Next, I checked for anything recognizable at the beginning of the disk.

# dd if=/dev/hdi count=8 | strings
Broadcom NAS Version 1.1 MBR Tag
SYSTEM
8+0 records in
8+0 records out
4096 bytes (4.1 kB) copied, 0.00013262 s, 30.9 MB/s

After a little Googling for "broadcom nas", "hp media vault", and a few other things, I figured out there was a reiserfs filesystem on the thing somewhere. (Note: As pointed out in the first comment to this post, there is some great technical documentation on this thing online. Rather than follow it, I chose to cheat and do this the (relatively) easy way.) Google found me this nice document describing the on-disk structure of reiserfs. That's how I figured out that I was looking for a magic string "ReIsEr2Fs" (or "ReIsErFs" for version 1, or "ReIsEr3Fs" for version 3, according to some other search results). I used hexedit to find the offset of the magic string by doing hexedit /dev/hdi, hitting tab, hitting /, then typing in ReIsEr.

2685A020   84 03 00 00  1E 00 00 00  00 00 00 00  00 10 CC 03  06 00 01 00  52 65 49 73  ....................ReIs
2685A038   45 72 32 46  73 00 00 00  03 00 00 00  05 00 B7 08  02 00 00 00  CC A3 00 00  Er2Fs...................

In this case, the magic string was at hex location 2685A034, which means the beginning of the superblock was at 2685A000, or (decimal) 646291456 bytes. The beginning of the superblock is 64k bytes before that, so I set up a loop device there:

# losetup -o $[ 646291456 - 65536 ] /dev/loop0 /dev/hdi
# mkdir /mnt/tmp
# mount -r -t reiserfs /dev/loop0 /mnt/tmp

The files my customer needed were in /mnt/tmp/FileShare/.

Why I suck at business

So anyone who knows me probably knows about the store, but I try not to make a big deal out of it here or in my normal away-from-the-store interaction with people. The fact is, I just don't like mentioning it, for fear that people will think I'm advertising, and I'm only in it for the money. This post is already making me incredibly uncomfortable...

That said, I need some help figuring out some way to advertise some classes I'll be teaching starting in February. Kara and I have invested a lot in the training room in our new store (which makes the local Linux users group meetings much more comfortable), so we have to start making some money on it soon. As it happens, I'm a darn good sysadmin (IMHO), and I like to talk, so teaching is a natural thing for me. ;-)

The classes I'll be teaching are all Linux system administration-related. Half of them are LPI Linux certification classes (on account of the O'Reilly book on the subject I helped write). There's a Google Calendar with dates, but no class descriptions yet. (Details available on request, but I obviously need to get them posted somewhere...)

So I need to figure out a way to get the word out, preferably to the St. Louis area (and the midwest in general, I suppose) that a) doesn't cost a ton of money (because that's all tied up in the new shop), b) doesn't make me feel like a total weasel, and c) doesn't make the intended audience think I'm a total weasel. Suggestions (like, what advertising do you actually pay any attention to and maybe trust a little) would be most appreciated.

Oh, since I'm talking about work, and in case my lovely and talented wife reads this, I have to point out for the record that she owns and runs the business. (In addition to being extremely averse to "sales", I also happen to be horrible with money. On the other hand, she happens to be good at that and more. :-)

Hard drives suck

In my day job, we go through a lot of hard drives. It seems now that no matter how good a drive manufacturer is, drives still show up DOA, die early, die not-so-early, etc. This is the reason I like Seagate so much. While I fully expect their drives to start dying naturally at 18 months or so (they don't always, but that's when I really start keeping an eye on them), at least with their 5-year warranty I know I can get replacement drives without spending a lot of money constantly.

Anyway, a while back I got rather tired of replacing drives that were DOA or after less than 30 days of service, so I figured out that burning them in with a combination of SMART self tests and badblocks seemed to weed out the early failures rather well. I wrote this script to run those tests for me. All it needs is smartctl and badblocks (which are probably available on most live Linux CDs, but definitely on Fedora and Knoppix). It will work on any IDE or SATA drive. With a little work, it could probably handle SCSI drives, ATA drives on 3ware controllers, etc.

cpanspec 1.74

I released cpanspec 1.74 this morning. There's some information about it on the Fedora wiki, but the short version is that cpanspec generates spec files (or, optionally, rpms) for Perl packages on CPAN. As an example, you could build a source rpm for String::Random like this:

$ cpanspec -v -s String::Random
Updating /home/steve/.cpan/sources/modules/02packages.details.txt.gz...
Fetching /home/steve/.cpan/sources/modules/02packages.details.txt.gz from http://www.cpan.org/modules/02packages.details.txt.gz...
Fetching String-Random-0.22.tar.gz from http://www.cpan.org/authors/id/S/ST/STEVE/String-Random-0.22.tar.gz...
Trying to fetch description from String-Random-0.22/lib/String/Random.pm...
Writing perl-String-Random.spec...
Building source rpm from perl-String-Random.spec
Wrote: /home/steve/src/fedora/rpms/cpanspec/devel/tmp/perl-String-Random-0.22-1.fc7.src.rpm

Unlike most other automated (or semi-automated) solutions for building spec files for Perl modules, cpanspec actually gets the Summary, License, URL, description, and dependencies (if available) from the CPAN metadata and other sources. In many cases, packages generated with cpanspec are completely done and ready to use (or submit for package review).

Obviously there are a few things cpanspec doesn't handle well. See BUGS and TODO. Patches always welcome. :-)

Overdue update

I'm leaving for the Ohio LinuxFest in a few hours, so I thought this would be a good time to give a quick update.

• The store is coming together, slowly. My KVM head units finally showed up, and now that I've seen them, I've reconsidered how I want to cable everything a bit. (One of these years, I'll have to write up what we're doing, maybe with some pictures...) My switches in the front of the store are too loud, so I have to find a way to quiet them down before I can use them. The lighting in the front of the store still isn't right. Our training room furniture still needs to be assembled. We still have a lot of stuff that needs to be unpacked. There's probably more that I'm forgetting, but at least we are making progress...
• We've had some drama with our employees. We had one person quit after Kara and I went over some complaints. (At least we won't have to worry about those issues any more.) One employee is moving away to go back to school. Another gave us notice so he could take another job, but I think we convinced him to stay. And then we have a new guy starting next week, which means we get to train him for a while...
  
• I'm very slowly easing back into doing Fedora work, mostly just to take a break from all of the above. Hopefully I'll get a chance to try out 7.91 over the weekend. (I'm burning a copy to take with me now.)

With all this stuff going on, I'm not thrilled about leaving for a few days, but I probably need the break.

Moving sucks

Construction on our new store is done (finally), so we're moving in now, just a little later than originally intended. We're going to shut down this weekend so we can get everything in the new store set up. Hopefully we'll be mostly functional on Tuesday (2007-09-04, after the Labor Day holiday), but I have a feeling it will be a week or two before things settle down to something like normal...

On buying laptops

I posted the following in the LinkedIn Answers section earlier today while I was taking a little break. Maybe it will be useful to someone...

Since we work on laptops that are both in and out of warranty at my job, my first concern is usually how easy a laptop is to work on.

With that in mind, whatever you do, don't buy Dell. Dell's primary business goals appear to a) lock people in, and b) make sure systems self-destruct in a relatively short time so that the customer buys another computer.

Avoid any laptop that has a power cord that doesn't have a normal round plug with the opening for a center pin. That rules out Dell, half the Toshiba laptops, and various others. One of the most common repairs on a laptop is a broken power jack, and the round ones tend to be easier to fix. Plus replacement power adapters tend to be easier to find (and cheaper).

For most people, Lenovo laptops (especially the Thinkpads) would be my first choice... They're reasonably priced, solid, and fairly easy to work on. (I thought that even before our shop was set up as an authorized service center, although that certainly helps with our customers.) For some of the higher-end models, especially gaming laptops, we usually recommend MSI and Asus. If you want really small and light, Averatec usually has some really nice models, although be sure to shop around a bit if you want to go that route. Often you can get an identical laptop from MSI or another company either with better specs or a bit cheaper.

The one exception to my normal rules is the MacBooks. They have a weird power adapter, they aren't terribly easy to repair, and it looks like most of the engineering effort on them went to making them pretty, not providing good airflow, etc. Still, they are nice to look at, they are light, and they're pleasant to use. Just be prepared for them to be a bit more disposable than your average laptop.

A lot of the bigger name-brand laptops (Lenovo, Acer, etc.) will let you buy an extended warranty (Acer will go to 3 years, but I think Lenovo will go up to 5) that covers accidental damage. If you travel a lot, or you are at all worried about accidents, it is well worth the extra money. Not to mention the odds of needing a warranty in 3-5 years is much higher than in the first year.

Oh, and one last thing... Make sure you have good backups of anything on your laptop. Laptop hard drives get much more abuse than desktop drives, so they tend to fail faster than desktop drives. Unfortunately, they tend to be a lot harder to recover data from than desktop drives too.

Store update

It's been hard to find time to get much work done lately, much less post updates here. The new store is coming along nicely. We should be able to start moving in next weekend, and we should be completely moved and operating from the new store on September 1. I've been spending most of my time working on all the little details related to the move (keeping an eye on the construction, turning on utilities, designing a counter, etc., etc., etc.).

I set up a Google calendar to at least start to schedule classes in the new training room the other day.

If all goes well (and I don't lose my mind first), I'll get back to somewhat normal work in a couple of weeks after we're done moving.

More Perl 6 modules

I know it isn't what I'm supposed to be working on, but I happened to be looking through the list of Perl 6-on-Perl 5 modules on CPAN. I ran the list through cpanspec --follow and came up with another 51 packages.

As I have time, I'll go through the pile, clean them up, and start submitting them for review.

Week in review

Yikes. I seem to have spaced off making any updates this week. Oops.

Earlier this week I updated my pugs package to a svn checkout. It builds fine now, but I still have a ton of rpmlint warnings to address.

In Fedora news, all but one of the packages I submitted last week has been reviewed and approved. I also spent some time working on building my packages for EPEL. That's going to get a lot easier once the Module::Build package's dependencies get worked out.

There's been a lot more progress on the new store. We have permits now, and all the electrical and plumbing in the floor is done. We should have a floor on Tuesday, so either Wednesday or Thursday they should be building walls.

This past Thursday was Emma's 6th birthday. I took her to see the Cardinals/Cubs game. That turned out to be a lot of fun... The Cardinals won 11 to 1, with home runs from Rolen and Pujols and a grand slam from Chris Duncan. Emma seemed to especially enjoy all the fireworks. :-)

Parrot build failure

Building parrot on x86_64 without any of the tweaks my spec file does still fails "make test", but it doesn't fail as many tests...

Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/compilers/pge/06-grammar.t 0 11 16 26 162.50% 4-16
6 tests and 581 subtests skipped.
Failed 1/295 test scripts, 99.66% okay. 13/7356 subtests failed, 99.82% okay.

Perl 6 microgrant

Apparently I forgot to mention here that I applied for, and was accepted for, one of the Perl 6 Microgrants. I'll be working as much as possible (which, given what's going on with the store, might not be as much as I'd like) on finishing the various Perl 6-related packages for Fedora. I'll be giving regular status updates here, and I've gone back and tagged a couple of old updates appropriately.

By "finish", I mean resolving a few issues (like fixing pugs so it knows about vendor_perl and allows for a arch/noarch split), cleaning up the packages to match the Fedora packaging guidelines and fix anything rpmlint complains about, and actually get the packages accepted in Fedora. Ideally Fedora 8 will support everything Perl 6-related out of the box.

On that note, I tried updating my parrot package tonight at the St. Louis Perl Mongers meeting. The build fails, at least on x86_64.

Failed Test Stat Wstat Total Fail Failed List of Failed
-------------------------------------------------------------------------------
t/compilers/pge/06-grammar.t 0 11 16 26 162.50% 4-16
t/examples/shootout.t 1 256 20 1 5.00% 16
6 tests and 587 subtests skipped.
Failed 2/295 test scripts, 99.32% okay. 14/7356 subtests failed, 99.81% okay.

Pugs hates me

My new notebook is finally running somewhat stably (more on that later), so I took it and got out of the store for a while today to work on my pugs and parrot packages. I lost a bit of time because I didn't have all the necessary development packages installed, and it's a lot slower to install a bunch of packages over the Internet than over a gigabit LAN. Still, eventually I got everything installed on the laptop, and I tried to build pugs. The build failed. To make a long story short, I eventually tried building on my desktop, which built pugs fine a few months ago. It failed there too.

So apparently something in Fedora 7 (likely a newer version of ghc, if I had to guess) breaks pugs. Whee.

I think I may just jump to a svn checkout next to see if that builds.

Package flood

It isn't what I was supposed to be working on today, but I just went through my queue of packages that needed to be submitted for review and did that for most of them. The majority of these are dependencies for Bricolage.

• 248393 perl-Text-WordDiff - Track changes between documents
• 248394 perl-Text-LevenshteinXS - XS implementation of the Levenshtein edit distance
• 248396 perl-Text-Diff-HTML - XHTML format for Text::Diff::Unified
• 248400 perl-Test-File-Contents - Test routines for examining the contents of files
• 248403 perl-Test-Class - Easily create test classes in an xUnit/JUnit style
• 248407 perl-Term-ReadPassword - Asking the user for a password
• 248410 perl-Params-CallbackRequest - Functional and object-oriented callback architecture
• 248414 perl-Net-SFTP - Secure File Transfer Protocol client
• 248425 perl-File-Sync - Perl access to fsync() and sync() function calls
• 248427 perl-Authen-PAM - Authen::PAM Perl module
• 248431 perl-Net-FTPServer - Secure, extensible and configurable Perl FTP server
• 248434 perl-HTTP-DAV - WebDAV client library for Perl5
• 248451 perl-MasonX-Interp-WithCallbacks - Mason callback support via Params::CallbackRequest
• 248457 perl-Devel-Profiler - Perl profiler compatible with dprofpp

All this gets me down from 25 packages waiting to only 11, and I may never submit a few of those for various reasons.

Stupid hardware tricks

My backup server (one of the oldest boxes I still have in service, a 900MHz P3 Celeron) finally got on my nerves one too many times this week, so I replaced the motherboard/CPU/RAM with a Asus M2NPV-VM, a Athlon64 X2 3800+, and 2GB of DDR2/667. Since it was down anyway, I upgraded it to Fedora 7 (x86_64). That all went fine, but now the thing won't boot from the 3ware controller (7500-8), and for the life of me I can't figure out why... All I get is a blank screen and a blinking cursor. I've reinstalled grub, but the behavior hasn't changed. I have to wonder if there's some kind of incompatibility between the board and the 3ware card.

Monday I became the proud owner of an ancient Power Computing Mac clone. I thought it would be nice to have something PPC around that I could test Fedora on. After poking at it a bit and asking around on IRC, I found out it is an "Old World" Mac, which means you have to jump through some hoops to get it to boot Linux. Whee.

Oh, and I finally decided that my 2-year-old (give or take) LG DVD burner is dead. It spins, it will eject media, but it won't detect media in the tray. Next time I reboot, I think I'm going to replace it with one of the Lite-Ons we carry at the store. (Don't get me wrong... The LG burner was great right up until the point that it died. I'd probably get another LG, except these Lite-On burners seem good, they're cheap, and I have them on-hand.)

Weekend update

I've been working off and on for a month getting my new laptop set up. It's an Asus Z92T that is equipped with a Turion64 X2 TL-64 (2.2GHz dual core), 2GB DDR2/667 RAM, a 160GB IDE hard drive, and a Intel Pro Wireless 2915. It's an odd laptop, since it isn't an old model, but it comes with IDE and Cardbus instead of SATA and ExpressCard. Still, it's reasonably Linux-friendly, and with the evil proprietary NVIDIA drivers installed, it plays Quake 4 at 1280x800 at a completely respectable frame rate.

The laptop runs Fedora 7 fine, except it occasionally seems to lock up. After messing with it a bit, I think I figured out that the network flakes out, and X (possibly as a result of the network going away, but probably not) stops responding. Alt-SysRq-K gets me back to a console though. Now that I've figured all that out, maybe I'll be able to debug what's happening. Then again, I seem to have a ton of updates to install, so maybe everything will just work after that...

On a related note, I was running my linkdups script (functionally the same as the hardlink program that comes with Fedora, but I wrote my script a long time ago, so it's what I use), and I noticed it was chewing through files a little slower than I thought it should on the new laptop. After some debugging, I noticed the script was running /bin/pwd a lot. Obviously that didn't come from my code, so I traced it down to File::Spec::Unix::_cwd(), which seems to get called for every file opened by IO::File (and by extension FileHandle, which is what I was actually using). File::Spec::Unix::_cwd() looks like this:

sub _cwd {
require Cwd;
Cwd::cwd();
}

Cwd::cwd() then runs /bin/pwd for some reason I really don't understand. In any case, changing that to Cwd::getcwd() made a huge improvement in the speed of my linkdups script. The original code would md5sum around 600 files per second. With _cwd() redefined, it was checking almost 3000 files/sec.

I reported this as a bug upstream and with Fedora.

While working with an image of a customer's hard drive, I finally figured out how to handle partitions in a disk image:

losetup /dev/loop0 file
kpartx -a /dev/loop0

After that, the partitions will show up as /dev/mapper/loop0pn. When you are done:

kpartx -d /dev/loop0
losetup -d /dev/loop0

Just make sure the device-mapper-multipath package is installed if you want to try this out.

On the Fedora front, I updated perl-Pugs-Compiler-Rule, amavisd-new, tuxpaint, tuxpaint-stamps, and perl-Test-Base in rawhide over the weekend.

There was probably more, but that's all I can think of at the moment...

Movin' on up...

We've officially signed off on all the construction plans for the new store location. At this rate, there is still some (slim) chance we'll be moved in before our current lease runs out at the end of August.

Fedora miscellany

While looking through the big mess of pending reviews, I decided I really should try to do at least a couple of merge reviews. Those two were enough of a pain that I've decided not to do any more today. :-) I'll try to squeeze in one or two a day for a while though. It's the least I can do, especially since I've historically been really bad about doing package reviews...

I also talked to bpepple a bit about doing a semi-official Fedora event at the Ohio LinuxFest in September. Let's hope something comes of that...

Perl 6 update

I've been working for around a year now on packaging everything Perl 6-related for Fedora. For a while now, everyone has been able to install perl-Perl6-Bible to get the available Perl 6 documentation. I also have been working on packages for Parrot and Pugs. Those packages aren't quite done, but I've submitted them for review so everyone knows I'm working on them.

• parrot review
• Perl-Perl6-Pugs review

cpanspec 1.71

I just released cpanspec 1.71. The new version handles bzip2-compressed tar archives, filters out a few more obviously bogus files from %doc, and adds some options to cpanget. From the changelog:

* Add option processing to cpanget.  It now accepts the following options:

 -h          Print usage message
 -n          Don't download, only print URLs
 -q          Work quietly

The other day I had a reason (and honestly I don't remember what it was :-) to get the URL for something on CPAN, and, since the code was there in cpanget anyway... Now you can do this:

$ cpanget -q -n String::Random
String::Random: http://www.cpan.org/authors/id/S/ST/STEVE/String-Random-0.22.tar.gz

For the next version, I want to do the right thing with the new split perl package in F7 and later. I posted a note to fedora-perl-devel-list to get some opinions on how best to handle that.

Catching up...

Thanks to back-to-back bouts of cold/flu that left me unable to sleep (which left me a zombie when I was awake) a few months ago, I got really behind on Fedora work. Then we found a new location for the store, so I got sucked in to the insane amount of planning that requires. With all that, my contributions to Fedora have suffered to the point that a couple of my packages don't even run on the recently-released Fedora 7.

Slowly but surely I'm catching up though. I have an update to amavisd-new in updates-testing now. I've updated all my Perl module packages in the development tree. I've closed out or commented on several pending package reviews. I've touched base with the SoC project I'm mentoring.

Next, the bugzilla queue...

Is this thing on?

I'm normally violently opposed to using hosted services for things. When I decided to set up a blog for updates on the million-and-one projects I'm working on, I looked around for a good Perl-based solution for blogging. (Why Perl? I'm good enough with Perl that I feel completely comfortable that there won't be a problem I can't at least understand, if not fix, with Perl-based projects. I can't say the same for other languages.)

After looking around and asking my local Perl Mongers group, I came to the conclusion that blogging is one of those things that Perl works great for, and in fact it is so easy to set up a blog with Perl that the standard suggestion is to just roll your own. There are dozens of ad-hoc blog implementations, but no clear leader that does everything the average user would expect from a blog, maybe because blogging is such a personal thing that rolling your own is appealing. Or maybe because the various solutions that are currently available are just good enough that people consider this a solved problem. Or maybe everybody else gave up and started using hosted services too.

So, since I don't have the time to roll my own right now, and the idea of running the solutions that are out there right now on my own servers scares me a little (small open-source projects don't necessarily have enough eyes on them to make me feel comfortable that they are secure), I'm setting aside my own bias against hosted services to try this thing out for a while.