RAID - Data Recovery with Open-Source Tools (part 7)

This is part 7 of a multi-part series. See part 1 for the beginning of the series.

Software RAID

It's becoming increasingly common (in 2009) on desktop PCs to use some form of BIOS-based software RAID. In most cases dealing with a single-drive failure in a software RAID isn't terribly difficult. For example, with NVIDIA's software RAID, even when one drive out of a stripe (RAID 0) set fails, if the drive is recoverable, you can simply clone it to a new identically-sized drive and the RAID will just work. Unfortunately, this isn't so simple with Intel's software RAID, which appears to store the serial numbers of the drives in the RAID metadata, meaning an exact clone won't work. While it would most likely be possible to simply edit the RAID metadata using hexedit to update the drive information, a somewhat simpler solution is to make a backup clone of the drives in the array, then re-create the RAID exactly as it was before in the RAID BIOS, then boot into Linux and run testdisk on the RAID device. More on that in part 8.

Most often the RAID metadata for drives in a software RAID volume is stored toward the end of the drive. In some cases, if you are forced to clone a failing RAID drive to a larger drive, you can make Linux (and maybe the BIOS and Windows) see the drive as a RAID device by copying the last few blocks from the failing drive to the last few blocks of the replacement drive.

old_end=$(( $( blockdev --getsz /dev/sda ) / 2 ))
end=$(( $( blockdev --getsz /dev/sdb ) / 2 ))
dd_rescue -d -D -b 4k -B 4k -s $(( $old_end - 1024 ))k -S $(( $end - 1024 ))k /dev/sdb /dev/sdb

Hardware RAID

Unfortunately, the ways that hardware RAID controllers store metadata don't tend to be quite as predictable as software RAID. If you attach a hardware RAID member drive to a non-RAID controller, some of the tricks mentioned above might work, but there are by no means any guarantees.

Also be aware that hardware RAID controllers are very likely to take a drive offline at the first sign of an error rather than report back the error and continue as most non-RAID controllers would. While this makes hardware RAID controllers largely unusable for data recovery, it does mean that a failing RAID member drive is quite likely to be recoverable.

To be continued in part 8.

Wiping Drives - Data Recovery with Open-Source Tools (part 6)

This is part 6 of a multi-part series.  See part 1 for the beginning of the series.

Wiping drives

To properly wipe a drive so it is effectively unrecoverable, the best solution is to use DBAN. It can be downloaded from https://sourceforge.net/projects/dban/.

Note from 2024: The DBAN project is mostly dead. Currently I would recommend nwipe, which is available in the standard package repositories for a number of Linux distributions, from source at https://github.com/martijnvanbrummelen/nwipe, or on bootable media like SystemRescue.  In fact, SystemRescue has a page in their documentation on this very topic.

In many cases, it is sufficient to simply zero out the entire drive. This can be done using dd_rescue.

To zero out /dev/sda, you can use the following command:

dd_rescue -D -b 1M -B 4k -m $(( $( blockdev --getsz /dev/sda ) / 2 ))k /dev/zero /dev/sda

This uses a bit of a shell scripting trick to avoid multiple commands and copy & paste, but it is still fairly simple. The output of blockdev --getsz gives us the size of the device in 512-byte blocks, so we divide that number by 2 to get the size in 1kB blocks, which we pass to the -m option (with a trailing k) to denote kB) to specify the maximum amount of data to transfer. Using a default block size of 1MB (-b) with a fallback of 4kB (-B, to match the host page size, which is required for direct I/O) should give us decent throughput.

Note that we're using -D to turn on direct I/O to the destination drive (/dev/sda), but we're not using direct I/O (-d) to read /dev/zero since /dev/zero is a character device that does not support direct I/O.

To just clear the MS-DOS partition table (and boot sector) on /dev/sda, you could do the following:

dd if=/dev/zero of=/dev/sda count=1

To be continued in part 7.

Cloning Drives - Data Recovery with Open-Source Tools (part 5)

This is part 5 of a multi-part series. See part 1 for the beginning of the series.

Cloning hard drives with dd_rescue

In cases where a hard drive is failing, often simply cloning the drive is all that is required to recover data. There are many other situations where cloning a drive is important though, such as when attempting to recover from a broken partition table or major filesystem corruption.

The primary tool for cloning drives is called dd_rescue. Running dd_rescue -h or simply dd_rescue with no options will give you a summary of the various command-line options:

dd_rescue Version 1.14, garloff@suse.de, GNU GPL
 ($Id: dd_rescue.c,v 1.59 2007/08/26 13:42:44 garloff Exp $)
dd_rescue copies data from one file (or block device) to another.
USAGE: dd_rescue [options] infile outfile
Options: -s ipos start position in input file (default=0),
	     -S opos start position in output file (def=ipos),
	     -b softbs block size for copy operation (def=65536),
	     -B hardbs fallback block size in case of errs (def=512),
	     -e maxerr exit after maxerr errors (def=0=infinite),
	     -m maxxfer maximum amount of data to be transfered (def=0=inf),
	     -y syncfrq frequency of fsync calls on outfile (def=512*softbs),
	     -l logfile name of a file to log errors and summary to (def=""),
	     -o bbfile name of a file to log bad blocks numbers (def=""),
	     -r reverse direction copy (def=forward),
	     -t truncate output file (def=no),
	     -d/D use O_DIRECT for input/output (def=no),
	     -w abort on Write errors (def=no),
	     -a spArse file writing (def=no),
	     -A Always write blocks, zeroed if err (def=no),
	     -i interactive: ask before overwriting data (def=no),
	     -f force: skip some sanity checks (def=no),
	     -p preserve: preserve ownership / perms (def=no),
	     -q quiet operation,
	     -v verbose operation,
	     -V display version and exit,
	     -h display this help and exit.
Note: Sizes may be given in units b(=512), k(=1024), M(=1024^2) or G(1024^3) bytes
This program is useful to rescue data in case of I/O errors, because
 it does not necessarily abort or truncate the output.

Note that there is also a GNU ddrescue with a similar feature set, but with entirely incompatible command-line arguments.

In the simplest of cases, dd_rescue can be used to copy infile (let's say, for example, /dev/sda) to outfile (again, for example, /dev/sdb).

dd_rescue /dev/sda /dev/sdb

In most cases, you'll want a little more control over how dd_rescue behaves though. For example, to clone failing /dev/sda to /dev/sdb:

dd_rescue -d -D -B 4k /dev/sda /dev/sdb

(to use the default 64k block size) or, for really bad drives, to force only one read attempt:

dd_rescue -d -D -B 4k -b 4k /dev/sda /dev/sdb

Adding the -r option to read backwards also helps sometimes.

Changing block sizes

By default, dd_rescue uses a block size of 64k (overridden with -b). In the event of a read error, it tries to read again in 512-byte chunks (overridden with -B). If a drive is good (or only beginning to fail), a larger block size (usually in the 512kB-1MB range) will give you significantly better performance.

If a drive is failing, forcing the default block size to the same value as the fall-back size will keep dd_rescue from re-reading (and therefore possibly damaging) failed blocks.

Direct I/O

The -d and -D options turn on direct I/O for the input and output files respectively. Direct I/O turns off all OS caching, both read-ahead and write-behind. This is much more efficient (and safer) when reading from and writing to hard drives, but should generally be avoided when using regular files.

Other useful options

-r        Read backwards. Sometimes works more reliably. (Very handy trick...)

-s num    Start position in input file.

-S num    Start position in output file. (Defaults to the same as -s.)

-e num    Stop after num errors.

-m num    Maximum amount of data to read.

-l file   Write a log to file.

Copying partitions

Let's say you have a drive with a MS-DOS partition table.  The drive has two partitions.  The first is a NTFS partition that seems to be intact.  The second partition is an unknown type.  Rather than copying every block using dd_rescue, you want to copy only the blocks that are in use to a drive that is the same size.

To do this, first copy the boot sector and partition table from /dev/sda to /dev/sdb using dd:

dd if=/dev/sda of=/dev/sdb count=1

The default block size of dd is 512 bytes, which, conveniently, is the size of boot sector + partition table at the beginning of the drive.

Note: This trick doesn't quite work on MS-DOS partition tables with extended partitions! In that case, use sfdisk to copy the partition table (after running the above command to pick up the boot sector):

sfdisk -d /dev/sda | sfdisk /dev/sdb

Next, re-read the partition table on /dev/sdb using hdparm:

hdparm -z /dev/sdb

Next we can clone the NTFS filesystem on /dev/sda1 to /dev/sdb1 using the ntfsclone command from ntfsprogs:

ntfsclone --rescue -O /dev/sdb1 /dev/sda1

Finally we clone /dev/sda2 to /dev/sdb2 using dd_rescue using a 1MB block size (for speed):

dd_rescue -d -D -B 4k -b 1M /dev/sda2 /dev/sdb2

To be continued in part 6.

Burn-in Testing for Spinning Disks - Data Recovery with Open-Source Tools (part 4)

This is part 4 of a multi-part series.  See part 1 for the beginning of the series.

Note that this was written long before solid state drives were common (or possibly before they existed), so when I say "drive", I mean traditional spinning hard drives.  Burn-in testing like this on SSDs makes a lot less sense and will likely only reduce their useful lifespan.

Burn-in testing

A good way to do a burn-in test on a new drive is to use a combination of SMART self-tests and the badblocks utility.  An example of how to do this can be found at https://github.com/silug/drivetest.

This script does the following:

1. Enables SMART on the drive
2. Checks for existing SMART health problems
3. Runs a SMART conveyance or short test if the drive advertises that capability
4. Uses badblocks to do a non-destructive read/write test of the whole drive
5. Checks for resulting SMART errors
6. Runs an extended SMART test

Depending on the size of the drive, this can take many hours, but the result will be a drive that should be past any early failures.

To be continued in part 5.

SMART - Data Recovery with Open-Source Tools (part 3)

This is part 3 of a multi-part series.  See part 1 for the beginning of the series.

SMART

SMART (Self-Monitoring, Analysis, and Reporting Technology) can, in many cases, be used to detect drive failures. The utility smartctl (from the smartmontools package, see https://www.smartmontools.org/) can be used to view SMART data, initiate self-tests, etc.

Specifying device types

Historically, smartctl has guessed that devices named /dev/hdn are ATA (IDE) drives, and devices named /dev/sdn are SCSI drives. Since SATA drives and IDE drives using the libata driver show up as /dev/sdn, recent versions of smartctl have been modified to generally detect ATA drives named /dev/sdn, but to be sure, or in cases where smartctl needs to be told what type of device you're accessing, use the -t option. To test how you are accessing the drive, use the -i (AKA --info) option.

• ATA (SATA and IDE drives)

smartctl -d ata -i /dev/sdn

• SCSI

smartctl -d scsi -i /dev/sdn

• 3ware controller, port n

smartctl -d 3ware,n -i /dev/twe0 (8000-series and earlier controllers)

smartctl -d 3ware,n -i /dev/twa0 (9000-series controllers)

smartctl supports various other device types (other RAID controllers, some USB-to-ATA bridges, etc.). See the man page or the smartmontools web site for more information.

Enabling SMART

If SMART is not enabled on the device (like when it is disabled in the BIOS), it can be enabled with smartctl -s on device. There is also a -S option that turns on autosave of vendor-specific attributes. In most cases, it shouldn't be necessary to turn this on, but it can't hurt to turn it on.

Displaying SMART data

If you only remember one option for smartctl, make sure it is -a. That will show you everything smartctl knows about a drive. It is equivalent to -H -i -c -A -l error -l selftest -l selective for ATA drives and -H -i -A -l error -l selftest for SCSI drives.

Health

Drives use a combination of factors to determine their overall health. The drive's determination can be displayed with smartctl -H. For a failing drive, the output might look like this:

# smartctl -d ata -H /dev/sdb
smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: FAILED!
Drive failure expected in less than 24 hours. SAVE ALL DATA.
Failed Attributes:
ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE
5 Reallocated_Sector_Ct 0x0033 033 033 036 Pre-fail Always FAILING_NOW 2747

For a drive that isn't failing (or, more accurately, that SMART on the drive doesn't think is failing), the output will look like this:

# smartctl -d ata -H /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

Please note that a failing health self-assessment should always be taken as a clear indication of a failure, but passing this test should not be used as an indication that a drive is fine. Most actively failing drives do not trip this test.

Information

As previously mentioned, the -i option for smartctl will report drive information, such as model number, serial number, capacity, etc. The output of smartctl -i will look something like this:

# smartctl -d ata -i /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF INFORMATION SECTION ===
Model Family: Seagate Barracuda 7200.12 family
Device Model: ST31000528AS
Serial Number: X4JZDJRF
Firmware Version: CC38
User Capacity: 1,000,204,886,016 bytes
Device is: In smartctl database [for details use: -P show]
ATA Version is: 8
ATA Standard is: ATA-8-ACS revision 4
Local Time is: Wed Jul 7 21:01:41 2010 CDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

In some cases, drives that are known to have firmware bugs will also give output like this:

==> WARNING: There are known problems with these drives,
see the following Seagate web pages:
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207931
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207951
http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207957

Capabilities

The -c option for smartctl displays drive capabilities. The most interesting bit of information displayed with this option is the suggested amount of time required for various self-tests. The full output will look like this:

# smartctl -d ata -c /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
General SMART Values:
Offline data collection status: (0x82) Offline data collection activity
                                       was completed without error.
                                       Auto Offline Data Collection: Enabled.
Self-test execution status:     (   0) The previous self-test routine completed
                                       without error or no self-test has ever
                                       been run.
Total time to complete Offline
data collection:                ( 600) seconds.
Offline data collection
capabilities:                   (0x7b) SMART execute Offline immediate.
                                       Auto Offline data collection on/off support.
                                       Suspend Offline collection upon new
                                       command.
                                       Offline surface scan supported.
                                       Self-test supported.
                                       Conveyance Self-test supported.
                                       Selective Self-test supported.
SMART capabilities:           (0x0003) Saves SMART data before entering
                                       power-saving mode.
                                       Supports SMART auto save timer.
Error logging capability:       (0x01) Error logging supported.
                                       General Purpose Logging supported.
Short self-test routine
recommended polling time:       (   1) minutes.
Extended self-test routine
recommended polling time:       ( 175) minutes.
Conveyance self-test routine
recommended polling time:       (   2) minutes.
SCT capabilities:             (0x103f) SCT Status supported.
                                       SCT Feature Control supported.
                                       SCT Data Table supported.

SMART attributes

The -A option for smartctl displays vendor-specific device attributes that are stored by the device.

# smartctl -d ata -A /dev/sdb

smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)

Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===

SMART Attributes Data Structure revision number: 10

Vendor Specific SMART Attributes with Thresholds:

ID# ATTRIBUTE_NAME          FLAG   VALUE WORST THRESH TYPE     UPDATED WHEN_FAILED RAW_VALUE

  1 Raw_Read_Error_Rate     0x000f 099   087   006    Pre-fail Always  -           134820080

  3 Spin_Up_Time            0x0003 095   095   000    Pre-fail Always  -           0

  4 Start_Stop_Count        0x0032 100   100   020    Old_age  Always  -           16

  5 Reallocated_Sector_Ct   0x0033 033   033   036    Pre-fail Always  FAILING_NOW 2748

  7 Seek_Error_Rate         0x000f 072   062   030    Pre-fail Always  -           16103679

  9 Power_On_Hours          0x0032 097   097   000    Old_age  Always  -           3165

 10 Spin_Retry_Count        0x0013 100   100   097    Pre-fail Always  -           0

 12 Power_Cycle_Count       0x0032 100   100   020    Old_age  Always  -           8

183 Runtime_Bad_Block       0x0032 100   100   000    Old_age  Always  -           0

184 End-to-End_Error        0x0032 100   100   099    Old_age  Always  -           0

187 Reported_Uncorrect      0x0032 100   100   000    Old_age  Always  -           0

188 Command_Timeout         0x0032 100   099   000    Old_age  Always  -           8590065676

189 High_Fly_Writes         0x003a 100   100   000    Old_age  Always  -           0

190 Airflow_Temperature_Cel 0x0022 071   065   045    Old_age  Always  -           29 (Lifetime Min/Max 27/30)

194 Temperature_Celsius     0x0022 029   040   000    Old_age  Always  -           29 (0 9 0 0)

195 Hardware_ECC_Recovered  0x001a 044   020   000    Old_age  Always  -           134820080

197 Current_Pending_Sector  0x0012 100   100   000    Old_age  Always  -           0

198 Offline_Uncorrectable   0x0010 100   100   000    Old_age  Offline -           0

199 UDMA_CRC_Error_Count    0x003e 200   200   000    Old_age  Always  -           0

240 Head_Flying_Hours       0x0000 100   253   000    Old_age  Offline -           257186936654939

241 Total_LBAs_Written      0x0000 100   253   000    Old_age  Offline -           2601921204

242 Total_LBAs_Read         0x0000 100   253   000    Old_age  Offline -           551656776

Generally speaking, these attributes should be mostly self-explanatory. For example, attribute #9, Power_On_Hours, stores the number of hours that the drive has been powered on. In this example, the drive has been on 3165 hours (seen in the RAW_VALUE column), which is a bit over 4 months.

Drives store thresholds for what value indicates a failure. In this example, note that attribute 5, Reallocated_Sector_Ct, which has a value of 2748, is considered FAILING_NOW.

SMART logs

The -l name option for smartctl displays the SMART log name stored on the device. There are several such logs that any given device might support, but the most interesting are the error and selftest logs.

The error log is, like the name suggests, a log of events that are seen as errors by the drive. A device that supports (and stores) a SMART error log, but currently has nothing logged, will look like this:

# smartctl -d ata -l error /dev/sda
smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===
SMART Error Log Version: 1
No Errors Logged

And here's an example of a device with one error logged:

# smartctl -d ata -l error /dev/sda

smartctl 5.39.1 2010-01-28 r3054 [i386-redhat-linux-gnu] (local build)

Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===

SMART Error Log Version: 1

ATA Error Count: 1

        CR = Command Register [HEX]

        FR = Features Register [HEX]

        SC = Sector Count Register [HEX]

        SN = Sector Number Register [HEX]

        CL = Cylinder Low Register [HEX]

        CH = Cylinder High Register [HEX]

        DH = Device/Head Register [HEX]

        DC = Device Command Register [HEX]

        ER = Error register [HEX]

        ST = Status register [HEX]

Powered_Up_Time is measured from power on, and printed as

DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,

SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 1 occurred at disk power-on lifetime: 4775 hours (198 days + 23 hours)

  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:

  ER ST SC SN CL CH DH

  -- -- -- -- -- -- --

  40 51 00 aa b9 2f 04

  Commands leading to the command that caused the error were:

  CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name

  -- -- -- -- -- -- -- -- ---------------- --------------------

  60 00 08 a7 b9 2f 44 00 5d+18:49:37.312 READ FPDMA QUEUED

  61 00 10 87 2e de 44 00 5d+18:49:37.296 WRITE FPDMA QUEUED

  61 00 01 9a 7b 56 40 00 5d+18:49:37.272 WRITE FPDMA QUEUED

  61 00 20 ff ff ff 4f 00 5d+18:49:37.235 WRITE FPDMA QUEUED

  60 00 10 f7 98 59 40 00 5d+18:49:37.212 READ FPDMA QUEUED

The error log will only show the five most recent entries, but that is usually enough context to get an idea what is wrong.

SMART self-tests

The -t type option tells smartctl to run a self-test of type type on the drive. type can be one of several options, although the most common are short, long, and conveyance. smartctl -t short runs a SMART Short Self Test, which usually finishes in just a couple of minutes. smartctl -t long runs a SMART Extended Self Test, which often will take an hour or more to run. smartctl -t conveyance runs a SMART Conveyance Self Test, which checks for damage sustained during transport (drops and such).

The output will look like this:

# smartctl -t short /dev/sdb
smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)
Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Mon Sep 6 20:22:49 2010

Use smartctl -X to abort test.

After waiting the appropriate amount of time (2 minutes, in the previous case, as seen in the smartctl -t short output, but which can also be found with smartctl -c), you can use smartctl -l selftest to view the self-test results.

# smartctl -l selftest /dev/sdb

smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)

Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net

=== START OF READ SMART DATA SECTION ===

SMART Self-test log structure revision number 1

Num Test_Description     Status                  Remaining LifeTime(hours) LBA_of_first_error

# 1 Short offline        Completed: read failure       90%    8835         17135

# 2 Short offline        Completed without error       00%    0            -

In the example above, a short test completed successfully at a lifetime of 0 hours, but another short test failed with a read failure with 90% remaining at a lifetime of 8835 hours. (Test results are listed in order of most recent to oldest.)

More information

Google has done some excellent work in determining how SMART and various other data relates to drive failure. See https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf.

To be continued in part 4.