tag:blogger.com,1999:blog-81082778099375547922024-03-15T13:46:26.048-05:00SteveCoSteven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.comBlogger69125tag:blogger.com,1999:blog-8108277809937554792.post-26988255573033766952024-03-15T13:44:00.003-05:002024-03-15T13:44:39.352-05:00Cloning Drives - Data Recovery with Open-Source Tools (part 5)<p><i>This is part 5 of a multi-part series. See <a href="https://blog.stevecoinc.com/2024/02/data-recovery-with-open-source-tools.html">part 1</a>
for the beginning of the series.</i></p>
<h3>Cloning hard drives with <span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue</span></h3>
<p>In cases where a hard drive is failing, often simply cloning the drive is all that is required to recover data.
There are many other situations where cloning a drive is important though, such as when attempting to
recover from a broken partition table or major filesystem corruption.</p>
<p>The primary tool for cloning drives is called <span style="font-family: Inconsolata,'Courier New',monospace;"><a href="https://www.garloff.de/kurt/linux/ddrescue/" target="_blank">dd_rescue</a></span>.
Running <span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue -h</span>
or simply <span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue</span>
with no options will give you a summary of the various command-line options:</p>
<pre style="font-size: smaller;">dd_rescue Version 1.14, garloff@suse.de, GNU GPL
($Id: dd_rescue.c,v 1.59 2007/08/26 13:42:44 garloff Exp $)
dd_rescue copies data from one file (or block device) to another.
USAGE: dd_rescue [options] infile outfile
Options: -s ipos start position in input file (default=0),
-S opos start position in output file (def=ipos),
-b softbs block size for copy operation (def=65536),
-B hardbs fallback block size in case of errs (def=512),
-e maxerr exit after maxerr errors (def=0=infinite),
-m maxxfer maximum amount of data to be transfered (def=0=inf),
-y syncfrq frequency of fsync calls on outfile (def=512*softbs),
-l logfile name of a file to log errors and summary to (def=""),
-o bbfile name of a file to log bad blocks numbers (def=""),
-r reverse direction copy (def=forward),
-t truncate output file (def=no),
-d/D use O_DIRECT for input/output (def=no),
-w abort on Write errors (def=no),
-a spArse file writing (def=no),
-A Always write blocks, zeroed if err (def=no),
-i interactive: ask before overwriting data (def=no),
-f force: skip some sanity checks (def=no),
-p preserve: preserve ownership / perms (def=no),
-q quiet operation,
-v verbose operation,
-V display version and exit,
-h display this help and exit.
Note: Sizes may be given in units b(=512), k(=1024), M(=1024^2) or G(1024^3) bytes
This program is useful to rescue data in case of I/O errors, because
it does not necessarily abort or truncate the output.
</pre>
<p><i><b>Note that there is also a <a href="https://www.gnu.org/software/ddrescue/" target="_blank">GNU ddrescue</a> with a similar feature set, but with entirely incompatible command-line arguments.</b></i></p><p>In
the simplest of cases, <span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue</span>
can be used to copy <i>infile</i>
(let's say, for example, <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sda</span>) to <i>outfile</i>
(again, for example, <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sdb</span>).</p>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue /dev/sda /dev/sdb</span></p></blockquote>
<p>In most cases, you'll want a little more control over how <span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue</span> behaves though.
For example, to clone failing <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sda</span> to <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sdb</span>:</p>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue -d -D -B 4k /dev/sda /dev/sdb</span></p></blockquote>
<p>(to use the default 64k block size) or, for really bad drives, to force only one read attempt:</p>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue -d -D -B 4k -b 4k /dev/sda /dev/sdb</span></p></blockquote>
<p><i>Adding the </i><span style="font-family: Inconsolata,'Courier New',monospace;">-r</span><i> option to read backwards also helps sometimes.</i></p>
<h4>Changing block sizes</h4>
<p>By default, <span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue</span>
uses a block size of 64k (overridden with <span style="font-family: Inconsolata,'Courier New',monospace;">-b</span>). In the event of a read error, it tries to read
again in 512-byte chunks (overridden with <span style="font-family: Inconsolata,'Courier New',monospace;">-B</span>). If a drive is good (or only beginning to fail), a
larger block size (usually in the 512kB-1MB range) will give you significantly better performance.</p>
<p>If a drive is failing, forcing the default block size to the same value as the fall-back size will keep
<span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue</span>
from re-reading (and therefore possibly damaging) failed blocks.</p>
<h4>Direct I/O</h4>
<p>The <span style="font-family: Inconsolata,'Courier New',monospace;">-d</span> and <span style="font-family: Inconsolata,'Courier New',monospace;">-D</span> options turn on direct I/O for the input and output files respectively. Direct I/O turns
off all OS caching, both read-ahead and write-behind. This is <b>much</b>
more efficient (and safer) when reading from and writing to hard drives, but should generally be avoided
when using regular files.</p>
<h4>Other useful options</h4>
<p><span style="font-family: Inconsolata,'Courier New',monospace;">-r </span>Read backwards. Sometimes works more reliably. (Very handy trick...)</p>
<p><span style="font-family: Inconsolata,'Courier New',monospace;">-s num </span>Start position in input file.</p>
<p><span style="font-family: Inconsolata,'Courier New',monospace;">-S num </span>Start position in output file. (Defaults to the same as -s.)</p>
<p><span style="font-family: Inconsolata,'Courier New',monospace;">-e num </span>Stop after <i>num</i> errors.</p>
<p><span style="font-family: Inconsolata,'Courier New',monospace;">-m num </span>Maximum amount of data to read.</p>
<p><span style="font-family: Inconsolata,'Courier New',monospace;">-l file </span>Write a log to <i>file</i>.</p>
<h3>Copying partitions</h3><div>Let's say you have a drive with a MS-DOS partition table. The drive has two partitions. The first is a NTFS partition that seems to be intact. The second partition is an unknown type. Rather than copying every block using <span style="font-family: Inconsolata;">dd_rescue</span>, you want to copy only the blocks that are in use to a drive that is the same size.</div><div><br /></div><div>To do this, first copy the boot sector and partition table from <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sda</span>
to <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sdb</span> using <span style="font-family: Inconsolata;">dd</span>:</div>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">dd if=/dev/sda of=/dev/sdb count=1</span></p></blockquote>
<p><i>The default block size of </i><span style="font-family: Inconsolata,'Courier New',monospace;">dd</span><i> is
512 bytes, which, conveniently, is the size of boot sector + partition table at the beginning of the
drive.</i></p>
<p><i><b>Note: This trick doesn't quite work on MS-DOS partition tables with extended partitions!</b>
In that case, use </i><span style="font-family: Inconsolata;">sfdisk</span><i> to copy the partition table (after running the above command to pick up the boot sector):</i></p>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">sfdisk
-d /dev/sda | sfdisk /dev/sdb</span></p></blockquote>Next, re-read the partition table on <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sdb</span> using <span style="font-family: Inconsolata;">hdparm</span>:<br /><ul>
</ul>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">hdparm
-z /dev/sdb</span></p></blockquote>
Next we can clone the NTFS filesystem on <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sda1</span> to <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sdb1</span> using the <span style="font-family: Inconsolata;">ntfsclone</span> command from <a href="https://github.com/tuxera/ntfs-3g" target="_blank">ntfsprogs</a>:<br /><ul>
</ul>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">ntfsclone
--rescue -O /dev/sdb1 /dev/sda1</span></p></blockquote>
<p style="text-align: left;">Finally we clone <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sda2</span> to <span style="font-family: Inconsolata,'Courier New',monospace;">/dev/sdb2</span> using dd_rescue using a 1MB block size (for speed):</p><ul>
</ul>
<blockquote style="border: none; margin: 0 0 0 40px; padding: 0px;"><p style="text-align: left;"><span style="font-family: Inconsolata,'Courier New',monospace;">dd_rescue -d -D -B 4k -b 1M /dev/sda2 /dev/sdb2</span></p></blockquote>
<p><i>To be continued in part 6.</i></p>
Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-163020007364133102024-03-08T13:19:00.003-06:002024-03-15T13:45:54.762-05:00Burn-in Testing for Spinning Disks - Data Recovery with Open-Source Tools (part 4)<p><i>This is part 4 of a multi-part series. See <a href="https://blog.stevecoinc.com/2024/02/data-recovery-with-open-source-tools.html">part 1</a> for the beginning of the series.</i></p><p><i><b>Note that this was written long before solid state drives were common (or possibly before they existed), so when I say "drive", I mean traditional spinning hard drives. Burn-in testing like this on SSDs makes a lot less sense and will likely only reduce their useful lifespan.</b></i></p><h1 style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt; text-align: left;">Burn-in testing</h1><p dir="ltr" style="line-height: 1.2; margin-bottom: 12pt; margin-top: 0pt;">A good way to do a burn-in test on a new drive is to use a combination of SMART self-tests and the <span style="font-family: 'Courier New',monospace; white-space: pre;">badblocks</span> utility. An example of how to do this can be found at <a href="https://github.com/silug/drivetest">https://github.com/silug/drivetest</a>.</p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12pt; margin-top: 0pt;">This script does the following:</p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12pt; margin-top: 0pt;"></p><ol style="text-align: left;"><li>Enables SMART on the drive</li><li>Checks for existing SMART health problems</li><li>Runs a SMART conveyance or short test if the drive advertises that capability</li><li>Uses badblocks to do a non-destructive read/write test of the whole drive</li><li>Checks for resulting SMART errors</li><li>Runs an extended SMART test</li></ol>Depending on the size of the drive, this can take many hours, but the result will be a drive that should be past any early failures.<p></p><p><i>To be continued in <a href="https://blog.stevecoinc.com/2024/03/cloning-drives-data-recovery-with-open.html">part 5</a>.</i></p>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-45263355789037674542024-03-01T14:38:00.001-06:002024-03-08T13:20:27.110-06:00SMART - Data Recovery with Open-Source Tools (part 3)<p><i>This is part 3 of a multi-part series. See <a href="https://blog.stevecoinc.com/2024/02/data-recovery-with-open-source-tools.html">part 1</a> for the beginning of the series.</i><br /></p><h1 style="text-align: left;">SMART</h1><p></p>SMART (Self-Monitoring, Analysis, and Reporting Technology) can, in many cases, be used to detect drive failures. The utility <span style="font-family: Inconsolata;">smartctl</span> (from the smartmontools package, see <a href="https://www.smartmontools.org/">https://www.smartmontools.org/</a>) can be used to view SMART data, initiate self-tests, etc.<br /><h4 style="text-align: left;">Specifying device types</h4>Historically, <span style="font-family: Inconsolata;">smartctl</span> has guessed that devices named <span style="font-family: Inconsolata;">/dev/hd<i>n</i></span> are ATA (IDE) drives, and devices named <span style="font-family: Inconsolata;">/dev/sd<i>n</i></span> are SCSI drives. Since SATA drives and IDE drives using the libata driver show up as <span style="font-family: Inconsolata;">/dev/sd<i>n</i></span>, recent versions of <span style="font-family: Inconsolata;">smartctl</span> have been modified to generally detect ATA drives named <span style="font-family: Inconsolata;">/dev/sd<i>n</i></span>, but to be sure, or in cases where <span style="font-family: Inconsolata;">smartctl</span> needs to be told what type of device you're accessing, use the <span style="font-family: Inconsolata;">-t</span> option. To test how you are accessing the drive, use the <span style="font-family: Inconsolata;">-i</span> (AKA <span style="font-family: Inconsolata;">--info</span>) option.<br /><ul style="text-align: left;"><li>ATA (SATA and IDE drives)</li></ul><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="font-family: Inconsolata;">smartctl -d ata -i /dev/sd<i>n</i></span></blockquote><ul style="text-align: left;"><li>SCSI</li></ul><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="font-family: Inconsolata;">smartctl -d scsi -i /dev/sd<i>n</i></span></blockquote><ul style="text-align: left;"><li>3ware controller, port <i>n</i></li></ul><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="font-family: Inconsolata;">smartctl -d 3ware,<i>n</i> -i /dev/twe0</span> (8000-series and earlier controllers)</blockquote><blockquote style="border: none; margin: 0px 0px 0px 40px; padding: 0px; text-align: left;"><span style="font-family: Inconsolata;">smartctl -d 3ware,<i>n</i> -i /dev/twa0</span> (9000-series controllers)</blockquote><br /><span style="font-family: Inconsolata;">smartctl</span> supports various other device types (other RAID controllers, some USB-to-ATA bridges, etc.). See the man page or the smartmontools web site for more information.<br /><h4 style="text-align: left;">Enabling SMART</h4>If SMART is not enabled on the device (like when it is disabled in the BIOS), it can be enabled with <span style="font-family: Inconsolata;">smartctl -s on <i>device</i></span>. There is also a <span style="font-family: Inconsolata;">-S</span> option that turns on autosave of vendor-specific attributes. In most cases, it shouldn't be necessary to turn this on, but it can't hurt to turn it on.<br /><h4 style="text-align: left;">Displaying SMART data</h4>If you only remember one option for <span style="font-family: Inconsolata;">smartctl</span>, make sure it is <span style="font-family: Inconsolata;">-a</span>. That will show you everything <span style="font-family: Inconsolata;">smartctl</span> knows about a drive. It is equivalent to <span style="font-family: Inconsolata;">-H -i -c -A -l error -l selftest -l selective</span> for ATA drives and <span style="font-family: Inconsolata;">-H -i -A -l error -l selftest</span> for SCSI drives.<br /><h4>Health</h4>Drives use a combination of factors to determine their overall health. The drive's determination can be displayed with <span style="font-family: Inconsolata;">smartctl -H</span>. For a failing drive, the output might look like this:<br /><br /><span style="font-family: Inconsolata;"># smartctl -d ata -H /dev/sdb<br />smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)<br />Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net<br /><br />=== START OF READ SMART DATA SECTION ===<br />SMART overall-health self-assessment test result: FAILED!<br />Drive failure expected in less than 24 hours. SAVE ALL DATA.<br />Failed Attributes:<br />ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE<br /> 5 Reallocated_Sector_Ct 0x0033 033 033 036 Pre-fail Always FAILING_NOW 2747</span><br /><br />For a drive that isn't failing (or, more accurately, that SMART on the drive doesn't think is failing), the output will look like this:<br /><br /><span style="font-family: Inconsolata;"># smartctl -d ata -H /dev/sda<br />smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)<br />Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net<br /><br />=== START OF READ SMART DATA SECTION ===<br />SMART overall-health self-assessment test result: PASSED</span><br /><br />Please note that a failing health self-assessment should always be taken as a clear indication of a failure, but passing this test should not be used as an indication that a drive is fine. Most actively failing drives do not trip this test.<br /><h4 style="text-align: left;">Information</h4>As previously mentioned, the <span style="font-family: Inconsolata;">-i</span> option for <span style="font-family: Inconsolata;">smartctl</span> will report drive information, such as model number, serial number, capacity, etc. The output of <span style="font-family: Inconsolata;">smartctl -i</span> will look something like this:<br /><br /><span style="font-family: Inconsolata;"># smartctl -d ata -i /dev/sda<br />smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)<br />Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net<br /><br />=== START OF INFORMATION SECTION ===<br />Model Family: Seagate Barracuda 7200.12 family<br />Device Model: ST31000528AS<br />Serial Number: X4JZDJRF<br />Firmware Version: CC38<br />User Capacity: 1,000,204,886,016 bytes<br />Device is: In smartctl database [for details use: -P show]<br />ATA Version is: 8<br />ATA Standard is: ATA-8-ACS revision 4<br />Local Time is: Wed Jul 7 21:01:41 2010 CDT<br />SMART support is: Available - device has SMART capability.<br />SMART support is: Enabled<br /></span><br />In some cases, drives that are known to have firmware bugs will also give output like this:<br /><br /><span style="font-family: Inconsolata;">==> WARNING: There are known problems with these drives,<br />see the following Seagate web pages:<br />http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207931<br />http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207951<br />http://seagate.custkb.com/seagate/crm/selfservice/search.jsp?DocId=207957<br /></span><h4 style="text-align: left;">Capabilities</h4>The <span style="font-family: Inconsolata;">-c</span> option for <span style="font-family: Inconsolata;">smartctl</span> displays drive capabilities. The most interesting bit of information displayed with this option is the suggested amount of time required for various self-tests. The full output will look like this:<br /><br /><span style="font-family: Inconsolata;"># smartctl -d ata -c /dev/sda<br />smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)<br />Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net<br /><br />=== START OF READ SMART DATA SECTION ===<br />General SMART Values:<br />Offline data collection status: (0x82) Offline data collection activity<br /> was completed without error.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Auto Offline Data Collection: Enabled.<br />Self-test execution status: ( 0) The previous self-test routine completed<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">without error or no self-test has ever<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">been run.<br />Total time to complete Offline<br />data collection: ( 600) seconds.<br />Offline data collection<br />capabilities: (0x7b) SMART execute Offline immediate.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Auto Offline data collection on/off support.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Suspend Offline collection upon new<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">command.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Offline surface scan supported.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Self-test supported.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Conveyance Self-test supported.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Selective Self-test supported.<br />SMART capabilities: (0x0003) Saves SMART data before entering<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">power-saving mode.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">Supports SMART auto save timer.<br />Error logging capability: (0x01) Error logging supported.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">General Purpose Logging supported.<br />Short self-test routine<br />recommended polling time: ( 1) minutes.<br />Extended self-test routine<br />recommended polling time: ( 175) minutes.<br />Conveyance self-test routine<br />recommended polling time: ( 2) minutes.<br />SCT capabilities: (0x103f) SCT Status supported.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">SCT Feature Control supported.<br /></span><span style="font-family: Inconsolata;"> </span><span style="font-family: Inconsolata;">SCT Data Table supported.</span><br /><h4 style="text-align: left;">SMART attributes</h4>The <span style="font-family: Inconsolata;">-A</span> option for <span style="font-family: Inconsolata;">smartctl</span> displays vendor-specific device attributes that are stored by the device.<br /><br /><div><span style="font-family: Inconsolata;"># smartctl -d ata -A /dev/sdb</span></div><div><span style="font-family: Inconsolata;">smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)</span></div><div><span style="font-family: Inconsolata;">Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net</span></div><div><span style="font-family: Inconsolata;"><br /></span></div><div><span style="font-family: Inconsolata;">=== START OF READ SMART DATA SECTION ===</span></div><div><span style="font-family: Inconsolata;">SMART Attributes Data Structure revision number: 10</span></div><div><span style="font-family: Inconsolata;">Vendor Specific SMART Attributes with Thresholds:</span></div><div><span style="font-family: Inconsolata;">ID# ATTRIBUTE_NAME FLAG VALUE WORST THRESH TYPE UPDATED WHEN_FAILED RAW_VALUE</span></div><div><span style="font-family: Inconsolata;"> 1 Raw_Read_Error_Rate 0x000f 099 087 006 Pre-fail Always - 134820080</span></div><div><span style="font-family: Inconsolata;"> 3 Spin_Up_Time 0x0003 095 095 000 Pre-fail Always - 0</span></div><div><span style="font-family: Inconsolata;"> 4 Start_Stop_Count 0x0032 100 100 020 Old_age Always - 16</span></div><div><span style="font-family: Inconsolata;"> 5 Reallocated_Sector_Ct 0x0033 033 033 036 Pre-fail Always FAILING_NOW 2748</span></div><div><span style="font-family: Inconsolata;"> 7 Seek_Error_Rate 0x000f 072 062 030 Pre-fail Always - 16103679</span></div><div><span style="font-family: Inconsolata;"> 9 Power_On_Hours 0x0032 097 097 000 Old_age Always - 3165</span></div><div><span style="font-family: Inconsolata;"> 10 Spin_Retry_Count 0x0013 100 100 097 Pre-fail Always - 0</span></div><div><span style="font-family: Inconsolata;"> 12 Power_Cycle_Count 0x0032 100 100 020 Old_age Always - 8</span></div><div><span style="font-family: Inconsolata;">183 Runtime_Bad_Block 0x0032 100 100 000 Old_age Always - 0</span></div><div><span style="font-family: Inconsolata;">184 End-to-End_Error 0x0032 100 100 099 Old_age Always - 0</span></div><div><span style="font-family: Inconsolata;">187 Reported_Uncorrect 0x0032 100 100 000 Old_age Always - 0</span></div><div><span style="font-family: Inconsolata;">188 Command_Timeout 0x0032 100 099 000 Old_age Always - 8590065676</span></div><div><span style="font-family: Inconsolata;">189 High_Fly_Writes 0x003a 100 100 000 Old_age Always - 0</span></div><div><span style="font-family: Inconsolata;">190 Airflow_Temperature_Cel 0x0022 071 065 045 Old_age Always - 29 (Lifetime Min/Max 27/30)</span></div><div><span style="font-family: Inconsolata;">194 Temperature_Celsius 0x0022 029 040 000 Old_age Always - 29 (0 9 0 0)</span></div><div><span style="font-family: Inconsolata;">195 Hardware_ECC_Recovered 0x001a 044 020 000 Old_age Always - 134820080</span></div><div><span style="font-family: Inconsolata;">197 Current_Pending_Sector 0x0012 100 100 000 Old_age Always - 0</span></div><div><span style="font-family: Inconsolata;">198 Offline_Uncorrectable 0x0010 100 100 000 Old_age Offline - 0</span></div><div><span style="font-family: Inconsolata;">199 UDMA_CRC_Error_Count 0x003e 200 200 000 Old_age Always - 0</span></div><div><span style="font-family: Inconsolata;">240 Head_Flying_Hours 0x0000 100 253 000 Old_age Offline - 257186936654939</span></div><div><span style="font-family: Inconsolata;">241 Total_LBAs_Written 0x0000 100 253 000 Old_age Offline - 2601921204</span></div><div><span style="font-family: Inconsolata;">242 Total_LBAs_Read 0x0000 100 253 000 Old_age Offline - 551656776</span></div><br />Generally speaking, these attributes should be mostly self-explanatory. For example, attribute #9, <span style="font-family: Inconsolata;">Power_On_Hours</span>, stores the number of hours that the drive has been powered on. In this example, the drive has been on 3165 hours (seen in the <span style="font-family: Inconsolata;">RAW_VALUE</span> column), which is a bit over 4 months.<br /><br />Drives store thresholds for what value indicates a failure. In this example, note that attribute 5, <span style="font-family: Inconsolata;">Reallocated_Sector_Ct</span>, which has a value of 2748, is considered <span style="font-family: Inconsolata;">FAILING_NOW</span>.<br /><h4 style="text-align: left;">SMART logs</h4>The <span style="font-family: Inconsolata;">-l <i>name</i></span> option for <span style="font-family: Inconsolata;">smartctl</span> displays the SMART log name stored on the device. There are several such logs that any given device might support, but the most interesting are the <span style="font-family: Inconsolata;">error</span> and <span style="font-family: Inconsolata;">selftest</span> logs.<br /><br />The error log is, like the name suggests, a log of events that are seen as errors by the drive. A device that supports (and stores) a SMART error log, but currently has nothing logged, will look like this:<br /><br /><span style="font-family: Inconsolata;"># smartctl -d ata -l error /dev/sda<br />smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)<br />Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net<br /><br />=== START OF READ SMART DATA SECTION ===<br />SMART Error Log Version: 1<br />No Errors Logged</span><br /><br />And here's an example of a device with one error logged:<br /><br /><div><span style="font-family: Inconsolata;"># smartctl -d ata -l error /dev/sda</span></div><div><span style="font-family: Inconsolata;">smartctl 5.39.1 2010-01-28 r3054 [i386-redhat-linux-gnu] (local build)</span></div><div><span style="font-family: Inconsolata;">Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net</span></div><div><span style="font-family: Inconsolata;"><br /></span></div><div><span style="font-family: Inconsolata;">=== START OF READ SMART DATA SECTION ===</span></div><div><span style="font-family: Inconsolata;">SMART Error Log Version: 1</span></div><div><span style="font-family: Inconsolata;">ATA Error Count: 1</span></div><div><span style="font-family: Inconsolata;"> CR = Command Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> FR = Features Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> SC = Sector Count Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> SN = Sector Number Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> CL = Cylinder Low Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> CH = Cylinder High Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> DH = Device/Head Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> DC = Device Command Register [HEX]</span></div><div><span style="font-family: Inconsolata;"> ER = Error register [HEX]</span></div><div><span style="font-family: Inconsolata;"> ST = Status register [HEX]</span></div><div><span style="font-family: Inconsolata;">Powered_Up_Time is measured from power on, and printed as</span></div><div><span style="font-family: Inconsolata;">DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,</span></div><div><span style="font-family: Inconsolata;">SS=sec, and sss=millisec. It "wraps" after 49.710 days.</span></div><div><span style="font-family: Inconsolata;"><br /></span></div><div><span style="font-family: Inconsolata;">Error 1 occurred at disk power-on lifetime: 4775 hours (198 days + 23 hours)</span></div><div><span style="font-family: Inconsolata;"> When the command that caused the error occurred, the device was active or idle.</span></div><div><span style="font-family: Inconsolata;"><br /></span></div><div><span style="font-family: Inconsolata;"> After command completion occurred, registers were:</span></div><div><span style="font-family: Inconsolata;"> ER ST SC SN CL CH DH</span></div><div><span style="font-family: Inconsolata;"> -- -- -- -- -- -- --</span></div><div><span style="font-family: Inconsolata;"> 40 51 00 aa b9 2f 04</span></div><div><span style="font-family: Inconsolata;"><br /></span></div><div><span style="font-family: Inconsolata;"> Commands leading to the command that caused the error were:</span></div><div><span style="font-family: Inconsolata;"> CR FR SC SN CL CH DH DC Powered_Up_Time Command/Feature_Name</span></div><div><span style="font-family: Inconsolata;"> -- -- -- -- -- -- -- -- ---------------- --------------------</span></div><div><span style="font-family: Inconsolata;"> 60 00 08 a7 b9 2f 44 00 5d+18:49:37.312 READ FPDMA QUEUED</span></div><div><span style="font-family: Inconsolata;"> 61 00 10 87 2e de 44 00 5d+18:49:37.296 WRITE FPDMA QUEUED</span></div><div><span style="font-family: Inconsolata;"> 61 00 01 9a 7b 56 40 00 5d+18:49:37.272 WRITE FPDMA QUEUED</span></div><div><span style="font-family: Inconsolata;"> 61 00 20 ff ff ff 4f 00 5d+18:49:37.235 WRITE FPDMA QUEUED</span></div><div><span style="font-family: Inconsolata;"> 60 00 10 f7 98 59 40 00 5d+18:49:37.212 READ FPDMA QUEUED</span></div><div><br /></div>The error log will only show the five most recent entries, but that is usually enough context to get an idea what is wrong.<br /><h4 style="text-align: left;">SMART self-tests</h4>The <span style="font-family: Inconsolata;">-t <i>type</i></span> option tells <span style="font-family: Inconsolata;">smartctl</span> to run a self-test of type <span style="font-family: Inconsolata;"><i>type</i></span> on the drive. <i style="font-family: Inconsolata;">type</i> can be one of several options, although the most common are <span style="font-family: Inconsolata;">short</span>, <span style="font-family: Inconsolata;">long</span>, and <span style="font-family: Inconsolata;">conveyance</span>. <span style="font-family: Inconsolata;">smartctl -t short</span> runs a SMART Short Self Test, which usually finishes in just a couple of minutes. <span style="font-family: Inconsolata;">smartctl -t long</span> runs a SMART Extended Self Test, which often will take an hour or more to run. <span style="font-family: Inconsolata;">smartctl -t conveyance</span> runs a SMART Conveyance Self Test, which checks for damage sustained during transport (drops and such).<br /><br />The output will look like this:<br /><br /><span style="font-family: Inconsolata;"># smartctl -t short /dev/sdb<br />smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)<br />Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net<br /><br />=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===<br />Sending command: "Execute SMART Short self-test routine immediately in off-line mode".<br />Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.<br />Testing has begun.<br />Please wait 2 minutes for test to complete.<br />Test will complete after Mon Sep 6 20:22:49 2010<br /><br />Use smartctl -X to abort test.</span><br /><br />After waiting the appropriate amount of time (2 minutes, in the previous case, as seen in the <span style="font-family: Inconsolata;">smartctl -t short</span> output, but which can also be found with <span style="font-family: Inconsolata;">smartctl -c</span>), you can use <span style="font-family: Inconsolata;">smartctl -l selftest</span> to view the self-test results.<br /><br /><div><span style="font-family: Inconsolata;"># smartctl -l selftest /dev/sdb</span></div><div><span style="font-family: Inconsolata;">smartctl 5.39.1 2010-01-28 r3054 [x86_64-redhat-linux-gnu] (local build)</span></div><div><span style="font-family: Inconsolata;">Copyright (C) 2002-10 by Bruce Allen, http://smartmontools.sourceforge.net</span></div><div><span style="font-family: Inconsolata;"><br /></span></div><div><span style="font-family: Inconsolata;">=== START OF READ SMART DATA SECTION ===</span></div><div><span style="font-family: Inconsolata;">SMART Self-test log structure revision number 1</span></div><div><span style="font-family: Inconsolata;">Num Test_Description Status Remaining LifeTime(hours) LBA_of_first_error</span></div><div><span style="font-family: Inconsolata;"># 1 Short offline Completed: read failure 90% 8835 17135</span></div><div><span style="font-family: Inconsolata;"># 2 Short offline Completed without error 00% 0 -</span></div><br />In the example above, a short test completed successfully at a lifetime of 0 hours, but another short test failed with a read failure with 90% remaining at a lifetime of 8835 hours. (Test results are listed in order of most recent to oldest.)<br /><h4 style="text-align: left;">More information</h4>Google has done some excellent work in determining how SMART and various other data relates to drive failure. See <a href="https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf">https://static.googleusercontent.com/media/research.google.com/en//archive/disk_failures.pdf</a>.<br /><p><i>To be continued in <a href="https://blog.stevecoinc.com/2024/03/burn-in-testing-for-spinning-disks-data.html">part 4</a>.</i></p>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-86403677713002662262024-02-24T12:21:00.001-06:002024-03-01T14:39:37.263-06:00Drive Failures - Data Recovery with Open-Source Tools (part 2)<p><i>This is part 2 of a multi-part series. See <a href="https://blog.stevecoinc.com/2024/02/data-recovery-with-open-source-tools.html">part 1</a> for the beginning of the series.</i></p><p><i>Note that this is material from 2010 and earlier that pre-dates the common availability of solid state drives.</i></p><h3 dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="text-decoration: none; vertical-align: baseline;">Detecting failures</span></h3><h4 dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline;">Mechanical failures</span></h4><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Mechanical drive failure is nearly always accompanied by some sort of audible noise. One common sound heard from failing hard drives is the so-called "Click of Death", a sound similar to a watch ticking (but much louder). This can have various causes, but it is commonly caused by the read/write head inside a drive being stuck or possibly trying to repeatedly read a failing block.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Another common noise is a very high-pitched whine. This is caused by bearings in a drive failing (most likely rubbing metal-on-metal), usually as a result of old age. Anything that moves inside a computer (fans, for example) can make a noise like this, so always check a suspect drive away from other sources of noise to verify that the sound is indeed coming from the drive.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Drive motors failing and head crashes can cause other distinctive noises. As a rule, any noise coming from a hard drive that does not seem normal is probably an indicator of imminent failure.</span></p><h4 dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline;">Electronic failures</span></h4><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Failing electronics can cause a drive to act flaky, not detect, and occasionally catch fire.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Hard drives have electronics on the inside of the drive which are inaccessible without destroying the drive (unless you happen to have a clean room). Unfortunately, if those fail, there isn't much you can do.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">The external electronics on a hard drive are usually a small circuit board that contains the interface connector and is held onto the drive with a few screws. In many cases, multiple versions of a drive (IDE, SATA, SCSI, SAS, etc.) exist with different controller interface boards. Generally speaking, it is possible to transplant the external electronics from a good drive onto a drive with failing electronics in order to get data off the failing drive. Usually the controller board will need to be off an identical drive with similar manufacturing dates.</span></p><h4 dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none; vertical-align: baseline;">Dealing with physical failures</span></h4><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">In addition to drive electronics transplanting, just about any trick you've heard of (freezing, spinning, smacking, etc.) has probably worked for someone, sometime. Whether any of these tricks work for you is a matter of trial and error. Just be careful.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Freezing drives seem to be especially effective. Unfortunately, as soon as a drive is operating, it will tend to heat up quickly, so some care needs to be taken to keep drives cool without letting them get wet from condensation.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;">Swapping electronics often works when faced with electronic failure, but only when the donor drive <b>exactly</b> matches the failed drive.</p>Freezing drives often helps in cases of crashed heads and electronic problems. Sometimes they will need help to stay cold (ice packs, freeze spray, etc.), but often once they start spinning, they'll stay spinning. Turning a drive on its side sometimes helps with physical problems as well.<p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;"><br /></span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none; vertical-align: baseline;">Unfortunately, we do have to get a drive to spin for any software data recovery techniques to work.</span></p><p><i>To be continued in <a href="https://blog.stevecoinc.com/2024/03/smart-data-recovery-with-open-source.html">part 3</a>.</i></p>
Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-35928917529221989552024-02-18T11:57:00.001-06:002024-02-24T12:25:04.812-06:00Data Recovery with Open-Source Tools (part 1)<p>This is material from a class I taught a long time ago. Some of it may still be useful. 🙂</p><p>The original copyright notice:</p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 8pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Copyright © 2009-2010 Steven Pritchard / K&S Pritchard Enterprises, Inc.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 8pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">This work is licensed under the Creative Commons Attribution-Noncommercial-No Derivative Works 3.0 United States License. To view a copy of this license, visit http://creativecommons.org/licenses/by-nc-nd/3.0/us/ or send a letter to Creative Commons, 171 Second Street, Suite 300, San Francisco, California, 94105, USA.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 8pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"><span style="font-family: "Times New Roman"; font-size: medium; white-space-collapse: collapse;"><br /></span></span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 8pt; font-variant: normal; font-weight: 400; text-decoration: none;"><span style="font-family: "Times New Roman"; font-size: medium; white-space-collapse: collapse;"><i>This is part 1 of a multi-part series.</i></span></span></p><h3 style="line-height: 1.2; margin-bottom: 12pt; margin-top: 12pt; text-align: left;"><span face="Verdana,sans-serif" style="font-size: 12pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none;">Identifying drives</span></h3><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">An easy way to get a list of drives attached to a system is to run </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">fdisk -l</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">. The output will look something like this:</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b id="docs-internal-guid-8511f08d-7fff-57d5-6747-d5137620ff7d" style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"># fdisk -l</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Disk /dev/sda: 80.0 GB, 80026361856 bytes</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">255 heads, 63 sectors/track, 9729 cylinders</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Units = cylinders of 16065 * 512 = 8225280 bytes</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Disk identifier: 0xcab10bee</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> Device Boot Start End Blocks Id System</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/dev/sda1 * 1 8673 69665841 7 HPFS/NTFS</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/dev/sda2 8675 9729 8474287+ c W95 FAT32 (LBA)</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">In many cases, you'll see a lot of (generally) uninteresting devices that are named /dev/dm-</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">n</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">. These are devices created by device mapper for everything from software RAID to LVM logical volumes. If you are primarily interested in the physical drives attached to a system, you can suppress the extra output of </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">fdisk -l</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> with a little bit of </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">sed</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">. Try the following:</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">fdisk -l 2>&1 | sed '/\/dev\/dm-/,/^$/d' | uniq</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Whole devices generally show up as /dev/sd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> (/dev/sda, /dev/sdb, etc.) or /dev/hd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> (/dev/hda, /dev/hdb, etc.). Partitions on the individual devices show up as /dev/sd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">xn</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> (/dev/sda1, /dev/sda2, etc.), or, in the case of longer device names, the name of the device with p</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">n</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> appended (an example might be /dev/mapper/loop0p1).</span></p><h3 style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt; text-align: left;"><span face="Verdana, sans-serif" style="font-size: 12pt; font-weight: 700; white-space-collapse: preserve;">Hardware</span></h3><h4 dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none;">PATA/SATA</span></h4><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">The vast majority of hard drives currently in use connect to a computer using either an IDE (or Parallel ATA) interface or a SATA (Serial ATA) interface. For the most part, SATA is just IDE with a different connector, but when SATA came out, the old Linux IDE driver had accumulated enough cruft that a new SATA driver (libata) was developed to support SATA controller chipsets. Later, the libata driver had support for most IDE controllers added, obsoleting the old IDE driver.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">There are some differences in the two drivers, and often those differences directly impact data recovery. One difference is device naming. The old IDE driver named devices /dev/hd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">, where </span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> is determined by the position of the drive.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/dev/hda </span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Master device, primary controller</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/dev/hdb </span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Slave device, primary controller</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/dev/hdc </span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Master device, secondary controller</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/dev/hdd </span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Slave device, secondary controller</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">And so on.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Unlike the IDE driver, the libata driver uses what was historically SCSI device naming, /dev/sd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">, where </span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> starts at "a" and increments upwards as devices are detected, which means that device names are more-or-less random, and won't be consistent across reboots.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">The other major difference between the old IDE driver and the libata driver that affects data recovery is how the drivers handle DMA (direct memory access). The ATA specification allows for various PIO (Programmed I/O) and DMA modes. Both the old IDE driver and the libata driver will determine the best mode, in most cases choosing a DMA mode initially, and falling back to a PIO mode in error conditions. The old IDE driver would also let you manually toggle DMA off and on for any device using the command </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">hdparm</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">hdparm -d /dev/hd</span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x <span> </span></span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Query DMA on/off state for /dev/hd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">hdparm -d0 /dev/hd</span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x<span> </span></span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Disable DMA on /dev/hd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">hdparm -d1 /dev/hd</span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x<span> </span></span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Enable DMA on /dev/hd</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">x</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">The libata driver currently lacks the ability to toggle DMA on a running system, but it can be turned off for all hard drives with the kernel command line option </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">libata.dma=6</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">, or for all devices (including optical drives) with </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">libata.dma=0</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">. On a running system, the value of libata.dma can be found in </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/sys/module/libata/parameters/dma</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">. (The full list of numeric values for this option can be found in </span><a href="http://www.kernel.org/doc/Documentation/kernel-parameters.txt" style="text-decoration: none;"><span face="Verdana,sans-serif" style="-webkit-text-decoration-skip: none; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration-skip-ink: none; text-decoration: underline;">http://www.kernel.org/doc/Documentation/kernel-parameters.txt</span></a><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">.) There does not appear to be a way to way to toggle DMA per device with the libata driver.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">There are several reasons why you might want to toggle DMA on or off for a drive. In some cases, failing drives simply won't work unless DMA is disabled, or even in some rare cases might not work unless DMA is enabled. In some cases the computer might have issues when reading from a failing drive with DMA enabled. (The libata driver usually handles these situations fairly well. The old IDE driver only began to handle these situations well in recent years.)</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">In addition to those reasons, PIO mode forces a drive to a maximum speed of 25MB/s (PIO Mode 6, others are even slower), while DMA modes can go up to 133MB/s. Some drives appear to work better at these lower speeds.</span></p><h4 dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 12.75pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none;">SCSI</span></h4><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">While SCSI drives and controllers are less common than they once were, all current hard drive controller interfaces now use the kernel SCSI device layers for device management and such. For example, all devices that use the SCSI layer will show up in </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/proc/scsi/scsi</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"># cat /proc/scsi/scsi</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Attached devices:</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Host: scsi0 Channel: 00 Id: 00 Lun: 00</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> Vendor: TSSTcorp Model: CD/DVDW TS-L632D Rev: AS05</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> Type: CD-ROM ANSI SCSI revision: 05</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Host: scsi1 Channel: 00 Id: 00 Lun: 00</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> Vendor: ATA Model: ST9160821A Rev: 3.AL</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> Type: Direct-Access ANSI SCSI revision: 05</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">Host: scsi3 Channel: 00 Id: 00 Lun: 00</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> Vendor: ATA Model: WDC WD10EACS-00Z Rev: 01.0</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> Type: Direct-Access ANSI SCSI revision: 05</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">In most cases, it is safe to remove a device that isn't currently mounted, but to be absolutely sure it is safe, you can also explicitly tell the kernel to disable a device by writing to </span><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">/proc/scsi/scsi</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">. For example, to remove the third device (the Western Digital drive in this example), you could do the following:</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-left: 30pt; margin-top: 0pt;"><span style="font-family: 'Courier New',monospace; font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">echo scsi remove-single-device 3 0 0 0 > /proc/scsi/scsi</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana, sans-serif" style="font-size: 10pt; white-space-collapse: preserve;">Note that the four numbers correspond to the controller, channel, ID, and LUN in the example.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><b style="font-weight: normal;"><br /></b></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">In cases where hot-added devices don't automatically show up, there is also a corresponding </span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: italic; font-variant: normal; font-weight: 400; text-decoration: none;">add-single-device</span><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;"> command.</span></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"><span face="Verdana, sans-serif" style="font-size: 10pt; white-space-collapse: preserve;">When recovering data from SCSI (and SCSI-like drives such as SAS), there are no special tricks like DMA.</span></p><h4 style="line-height: 1.2; margin-bottom: 12.75pt; margin-top: 0pt; text-align: left;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 700; text-decoration: none;">USB, etc.</span></h4><p dir="ltr" style="line-height: 1.2; margin-bottom: 0pt; margin-top: 0pt;"></p><p dir="ltr" style="line-height: 1.2; margin-bottom: 12pt; margin-top: 0pt;"><span face="Verdana,sans-serif" style="font-size: 10pt; font-style: normal; font-variant: normal; font-weight: 400; text-decoration: none;">The Linux USB drivers are rather resilient in the face of errors, so no special consideration needs to be given when recovering data from thumb drives and other flash memory (except that these devices tend to work or not, and, of course, dead shorts across USB ports are a Bad Thing). USB-to-ATA bridge devices are a different matter entirely though. They tend to lock up hard or otherwise behave badly when they hit errors on a failing drive. Generally speaking, they should be avoided for failing drives, but drives that are OK other than a trashed filesystem or partition table should be completely fine on a USB-to-ATA bridge device.</span></p><i>To be continued in <a href="https://blog.stevecoinc.com/2024/02/drive-failures-data-recovery-with-open.html">part 2</a>.</i>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-6203520808090889992023-05-17T12:26:00.000-05:002023-05-17T12:26:29.366-05:00libvirt surprise<p>I just noticed that <em>some</em> of my libvirt VMs had <tt>on_crash</tt> set to <tt>destroy</tt> instead of <tt>restart</tt>. It looks like there is an easy fix:</p><code>for vm in $( virsh list --name ) ; do virt-xml "$vm" --edit --events on_crash=restart ; done</code><p>I don't know if something changed in <tt>virt-manager</tt>/<tt>virt-install</tt> over the years, or if I ran into this a long time ago and forgot about it.</p><p>Now I just need to remember to add that <tt>--events</tt> option to <tt>virt-install</tt> in the future... 🙂</p>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-8554653799506842272023-05-09T18:17:00.001-05:002023-05-09T18:17:48.086-05:00Harvester HCI<p>I have been using libvirt on CentOS + <a href="https://zfsonlinux.org/" target="_blank">ZFS</a> for my home lab for somewhere around a decade now. For the last several years, I have been trying off and on to switch to some kind of hyperconverged infrastructure, usually <a href="https://ovirt.org/" target="_blank">oVirt</a> + a clustered storage solution (<a href="https://ceph.io/en/" target="_blank">Ceph</a>, <a href="https://www.gluster.org/" target="_blank">Gluster</a>). For various reasons, I've never quite managed to get all the pieces to work together correctly.</p><p>So, imagine how happy I was to hear about <a href="https://harvesterhci.io/" target="_blank">Harvester</a> a while back.</p><blockquote><p>Harvester is a modern Hyperconverged infrastructure (HCI) solution built for bare metal servers using enterprise-grade open source technologies including Kubernetes, Kubevirt and Longhorn.</p></blockquote><p>I love everything about this! It's a more modern take on hyperconverged infrastructure than what I was trying to assemble. Plus, all my problems magically disappear when all the pieces work together out of the box, right?</p><p>Well... Not quite. I installed version 0.3.0. It made for a cool demo, but, thanks to stability problems and a whole lot of missing features, it wasn't quite ready for anything resembling production use. (Granted, this is my home lab, but I still run virtualized firewalls and stuff like that on it, so I need it to <b>work</b>, and work reliably.)</p><p>I'll note here that I wrote all of the above over a year ago. I then closed with a list of reasons why Harvester wasn't good enough for me to actually use it at the time. It (<i>unintentionally!</i>) sounded negative enough that I decided not to publish the post.</p><p>So here we are a year or so later, and after a few more failed attempts I recently tried Harvester again, this time with version 1.1.1. Everything I <b>need</b> to work seems to, and I'm ready to start migrating some real workloads!</p><p>That's not to say that everything is perfect. There are a few useful features on the roadmap that I could benefit from (like anti-affinity rules, zero-downtime upgrades, ...), and I still have some challenges. Some examples:</p><p></p><ul style="text-align: left;"><li>Automating node installation is ... let's say difficult?</li><li>Networking is <b>almost</b> as functional as I want it, but I still haven't been able to figure out how to move storage replication to a network with jumbo frames.</li><li>I want real certs. I see how to manually manage the certs, but it's not immediately obvious how I could manage them automatically (for Let's Encrypt).</li></ul><div>Thankfully none of those things are keeping me from using Harvester. They're just things to look forward to in future upgrades. 😀</div>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-31158748028560361332022-09-10T15:31:00.000-05:002022-09-10T15:31:43.905-05:00Migrating libvirt VMs<p>I recently moved a bunch of libvirt VMs from a CentOS 7 host to a CentOS Stream 9 host. Normally moving virtual machines from one libvirt host to another is pretty easy. All you need to do is stop the VM on the original host, copy the disk image from host to host (with <tt>rsync</tt> or whatever is convenient), dump the VM config (with <tt>virsh dumpxml <i>guest</i></tt>), and import that config on the new host (with <tt>virsh define</tt>). It turns out a few things have changed that make that not quite work though...</p>
<p>The first thing (and the easiest to fix) was that a lot of old machine types that worked in CentOS 7's libvirt <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/considerations_in_adopting_rhel_9/assembly_virtualization_considerations-in-adopting-rhel-9#ref_changes-to-spice_assembly_virtualization:~:text=VM%20machine%20types%20based%20on%20RHEL%207.5%20and%20earlier%20are%20unsupported">no longer work</a>. The easy answer is to switch to the generic <tt>pc</tt> machine type.</p>
<p>The harder one to deal with was that <a href="https://access.redhat.com/documentation/en-us/red_hat_enterprise_linux/9/html/considerations_in_adopting_rhel_9/assembly_virtualization_considerations-in-adopting-rhel-9#ref_changes-to-spice_assembly_virtualization">Spice support was dropped</a>. This meant switching the graphics to VNC, using virtio for the virtual video hardware, and removing all the other Spice-related devices.</p>
<p>The libvirt VM configuration is all XML, so I wrote a script that uses <tt>xmlstarlet</tt> to make all the necessary changes.</p>
<br />
<pre style="height: 30em; overflow-x: auto; overflow-y: scroll; overflow: auto scroll;"><code class="language-bash">#!/bin/bash
set -e
remote="${REMOTE_HOST:?'Set REMOTE_HOST environment variable'}"
for guest in "$@" ; do
xml="$( mktemp XXXXXXXX.xml )"
trap "rm -fv '$xml'" EXIT
ssh root@"$remote" virsh dumpxml "$guest" | \
xmlstarlet ed \
-u '/domain/os/type[starts-with(@machine, "pc-i440fx")]/@machine' -v pc \
-u '/domain/os/type[starts-with(@machine, "rhel")]/@machine' -v pc \
-u '/domain/devices/video/model[@type="qxl"]/@type' -v virtio \
-d '/domain/devices/video/model[@type="virtio"]/@ram' \
-d '/domain/devices/video/model[@type="virtio"]/@vram' \
-d '/domain/devices/video/model[@type="virtio"]/@vgamem' \
-d '/domain/devices/graphics[@type="spice"]/@port' \
-i '/domain/devices/graphics[@type="spice"]' -t attr -n port -v -1 \
-u '/domain/devices/graphics[@type="spice"]/@type' -v vnc \
-d '/domain/devices/channel[@type="spicevmc"]' \
-d '/domain/devices/redirdev[@type="spicevmc"]' \
> "$xml"
virsh define "$xml"
rm -fv "$xml"
trap - EXIT
virsh autostart "$guest"
done
</code></pre>
<p>(The above script is also available at <a href="https://gist.github.com/silug/8c13cdfa0e50ca6e237333a79594be66">https://gist.github.com/silug/8c13cdfa0e50ca6e237333a79594be66</a>.)</p>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-10861121655991732142022-02-07T16:29:00.001-06:002022-02-07T16:29:27.585-06:00Recovering a ZFS array<p>
In August of 2016, I lost a (relatively) large (for me at the time) ZFS array. Rather than tell you how it happened to me, <a href="https://youtu.be/Npu7jkJk5nM" target="_blank">watch how it happened to Linus Tech Tips</a>.
</p>
<div>
<iframe allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen="" frameborder="0" height="315" src="https://www.youtube.com/embed/Npu7jkJk5nM" title="YouTube video player" width="560"></iframe>
</div>
<p>
My story is <em>almost</em> identical, except the array in question was <strong>much</strong> smaller, but to make matters worse it was mostly cobbled together with old hardware, including drives, so when the array died, there were a <strong>lot</strong> of bad drives.
</p>
<p>
My array started life as 15 750GB drives in a RAID-Z2. As the 750GB drives failed, they were replaced with 1TB drives. Unfortunately, I continued to use a mix of previously-used drives and some Seagate drives that apparently weren't Seagate's best work. The end result was that drives were failing rather often, and due to like of time, attention, and a ready supply of spare drives, I wasn't great at replacing them when they failed.
</p>
<p>
The biggest problem with RAID-5/6 and the RAID-Z equivalents are that rebuilds from a drive failure involve a lot of I/O to all of the drives in the array. RAID-Z2 allows you to lose two drives, but if you lose a third from the stress mid-rebuild, your whole array is toast. In my case, I didn't realize that I had a major problem until the third drive started to fail and ZFS took the array offline. A couple of the remaining drives had SMART errors and likely weren't going to survive a rebuild. I was going to have to clone all of the drives with errors before trying to rebuild. If I wanted a non-destructive fall-back plan, I needed to clone every drive, so if all else failed I could go back to the original array members to try again.
</p>
<p>
So... I didn't want to buy another 15 1TB drives. Where was I going to find enough disks (or raw space for disk images) to make a copy of the array?
</p>
<p>
My ultimate answer came almost 5.5 years later as I was working on rebuilding my <a href="https://harvesterhci.io/" target="_blank">Harvester</a> cluster (more on that some other time). I had several larger drives in the cluster, so while everything was down and disassembled, I put 5 of them in one system and built a ZFS pool. With around 18TB usable, I had more than enough space to store images of all of the drives in the array!
</p>
<p>
Enough time had passed that I wasn't sure which drives were which, so I <a href="https://gist.github.com/silug/570d8514710dab3e9c33c1e23d6e1cf1" target="_blank">wrote a script</a> to examine the metadata on each drive and then clone the drive to a name based on the pool, GUID, and last modified timestamp:
</p>
<pre style="height: 10em; overflow-x: auto; overflow-y: scroll; overflow: auto scroll;"><code class="language-bash">#!/bin/bash
set -e
dd_args=(
"-d"
"-b" "1M"
"-B" "4k"
"-w"
"-A"
)
warn() {
echo "$@" >&2
}
die() {
warn "$@"
exit 1
}
usage() {
warn "$( basename "$0" ) device [device [...]]"
}
get_first() {
key="$1"
text="$2"
value=$( echo "$text" | awk "(\$1 == \"$key:\") { print \$2; exit 0; }" )
echo "${value//\'/}"
}
get_pool() {
get_first "name" "$@"
}
get_guid() {
get_first "guid" "$@"
}
get_timestamp() {
text="$1"
timestamps=( $( echo "$text" | awk '($1 == "timestamp") { print $3 }' | sort -n ) )
echo "${timestamps[-1]}"
}
get_output_filename() {
base="$1"
n=1
while [ -f "${base}-${n}.img" ] ; do
warn "${base}-${n}.img exists."
(( n++ ))
done
echo "${base}-${n}.img"
}
if [ "$#" -lt 1 ] ; then
usage
exit 1
fi
cd /volumes/recovery/disks
for device in "$@" ; do
if [ ! -e "${device}1" ] ; then
die "Can't find first partition for device $device"
fi
zdb=$( zdb -l -u "${device}1" )
pool=$( get_pool "$zdb" )
guid=$( get_guid "$zdb" )
timestamp=$( get_timestamp "$zdb" )
echo "Recovering $guid from pool $pool last updated $( date --date="@$timestamp" )..."
mkdir -pv "${pool}/${guid}"
filename=$( get_output_filename "${pool}/${guid}/${timestamp}" )
logfile="${filename%.img}.log"
badfile="${filename%.img}.bad"
echo "Cloning $device to $filename (logging to $logfile)..."
dd_rescue "${dd_args[@]}" -l "$logfile" -o "$badfile" "$device" "$filename"
done</code></pre>
<p>
The script uses <a href="https://openzfs.github.io/openzfs-docs/man/8/zdb.8.html" target="_blank"><code>zdb</code></a> to get metadata from the drive, then uses <a href="http://www.garloff.de/kurt/linux/ddrescue/" target="_blank"><code>dd_rescue</code></a> to clone the drive to a file.
</p>
<p>
Once that finished, I made a snapshot of the entire filesystem (with <a href="https://openzfs.github.io/openzfs-docs/man/8/zfs-snapshot.8.html" target="_blank"><code>zfs snapshot</code></a>), mapped the files to block devices (with <code>losetup</code>), and activated partitions on the loopback devices (with <code>kpartx</code>). Then I was able to import the pool (with <a href="https://openzfs.github.io/openzfs-docs/man/8/zpool-import.8.html" target="_blank"><code>zpool import</code></a>) and find and fix all the errors (with <a href="https://openzfs.github.io/openzfs-docs/man/8/zpool-scrub.8.html" target="_blank"><code>zpool scrub</code></a>).
</p>
<p>
Very roughly, the commands I used went something like this:
</p>
<ul>
<li><code>zpool create -f -m /volumes/recovery -o ashift=12 recovery raidz /dev/disk/by-id/ata-TOSHIBA_!(*-part[0-9])</code></li>
<li><code>zfs create recovery/disks -o compress=zstd-fast</code></li>
<li><em>Insert the drives and run the script above against each one.</em></li>
<li><code>zfs snapshot recovery/disks@$( date +%Y%m%d%H%M%S )</code></li>
<li><code>for file in /volumes/recovery/disks/*/*/*.img ; do losetup -f -v $file; done</code></li>
<li><code>for loop in $( losetup -a | awk -F: '{print $1}' ) ; do kpartx -a $loop ; done</code></li>
<li><code>zpool import -d /dev/disk/by-id -f <em>pool_name</em></code></li>
<li><code>zpool scrub <em>pool_name</em></code></li>
</ul>
<p>
Now I just need to find enough space to <code>rsync</code> or <code>zfs send | zfs receive</code> all that data. 😀
</p>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-34499801332455033172022-02-01T21:25:00.001-06:002022-02-01T21:27:19.115-06:00Video from OLF 2021<p>I had two talks at <a href="https://olfconference.org/" target="_blank">OLF</a> in December. I just noticed that videos are up on YouTube for both of them.</p><h3 style="text-align: left;"><a href="https://youtu.be/BmeAJgCjHwg" target="_blank">I Like GitLab... and So Should You</a></h3><div><iframe width="560" height="315" src="https://www.youtube.com/embed/BmeAJgCjHwg" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div><h3 style="text-align: left;"><a href="https://youtu.be/0sbdTXV-zdc" target="_blank">Infrastructure Prototyping with Bolt and Vagrant</a></h3><div><iframe width="560" height="315" src="https://www.youtube.com/embed/0sbdTXV-zdc" title="YouTube video player" frameborder="0" allow="accelerometer; autoplay; clipboard-write; encrypted-media; gyroscope; picture-in-picture" allowfullscreen></iframe></div>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-25298266023440214942021-04-20T11:46:00.000-05:002021-04-20T11:46:41.777-05:00Dealing with old ssh implementations<p>Over the last several releases, Fedora has removed support for old, broken crypto algorithms. Unfortunately, this makes it harder to deal with old devices or servers that can't easily be upgraded. For example, I have a switch that I can't connect to with the <span style="font-family: Inconsolata;">ssh</span> on Fedora.</p><p>I can connect to it fine with the <span style="font-family: Inconsolata;">ssh</span> on CentOS 7 though... <span style="font-family: Inconsolata;">podman</span>/<span style="font-family: Inconsolata;">docker</span> to the rescue!</p><pre style="height: 10em; overflow-x: auto; overflow-y: scroll;"><code class="language-bash">#!/bin/bash
get_container_runtime() {
if [ -n "$CONTAINER_RUNTIME" ] ; then
container_runtime=$CONTAINER_RUNTIME
return
fi
podman=$( type -p podman )
if [ -n "$podman" ] ; then
container_runtime=$podman
return
fi
docker=$( type -p docker )
if [ -n "$docker" ] ; then
container_runtime=$docker
return
fi
echo 'No container runtime found.' >&2
exit 1
}
get_container_runtime
set -e
container=${CONTAINER:-"centos:7"}
ssh_cmd=$( mktemp /tmp/ssh.XXXXXX )
chmod 700 "$ssh_cmd"
trap "rm -fv $ssh_cmd" EXIT
cat > "$ssh_cmd" <<END
#!/bin/sh
set -e
yum -y install /usr/bin/ssh
ssh $@
END
run_args=(
-it
--rm
-v "$HOME/.ssh:/root/.ssh"
-v "$ssh_cmd:$ssh_cmd"
)
if [ -n "$SSH_AUTH_SOCK" ] ; then
run_args+=(
-e=SSH_AUTH_SOCK
-v "$SSH_AUTH_SOCK:$SSH_AUTH_SOCK"
)
fi
$container_runtime run ${run_args[@]} \
"$container" \
"$ssh_cmd"</code></pre><p>The script accepts all of the arguments that the container's <span style="font-family: Inconsolata;">ssh</span> accepts (because it blindly passes them along). It automatically maps your <span style="font-family: Inconsolata;">.ssh</span> directory and your <span style="font-family: Inconsolata;">ssh-agent</span> socket. YMMV, but I've tested it on Fedora with <span style="font-family: Inconsolata;">podman</span> and a Mac with <span style="font-family: Inconsolata;">docker</span>.</p>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-82381099585290469332020-08-24T10:57:00.001-05:002020-08-24T10:57:24.144-05:00Vagrant + libvirt on CentOS 7<p> I recently needed to set up vagrant-libvirt on a CentOS 7 VM. After finding a lot of outdated guides, I decided to write my own and post it on <a href="https://www.onyxpoint.com/blog/vagrant-libvirt-on-centos-7/">my work blog</a>.</p>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-84912379899017008862017-04-08T17:16:00.000-05:002017-04-08T17:16:08.870-05:00Delegating domain join privileges in Samba 4 from the command line (or not)I'm trying to solve a bit of a mystery. I'd like to set up Samba 4 without using Windows. Most things seem to be possible, but I can't figure out how to delegate domain join privileges. Unfortunately, even <a href="https://wiki.samba.org/index.php/Delegation/Joining_Machines_to_a_Domain">the official documentation</a> specifically references ADUC.<br />
<br />
So I did some digging into what it would take to delegate domain join privileges without a Windows system. After several dead ends, I ran across this page: <a href="https://www.sevecek.com/EnglishPages/Lists/Posts/Post.aspx?ID=48" target="_blank">https://www.sevecek.com/EnglishPages/Lists/Posts/Post.aspx?ID=48</a><br />
<br />
The important bit of that page is this script that uses the Windows command-line tool <span style="font-family: "courier new" , "courier" , monospace;">dsacls</span>:<br />
<br />
<pre style="background-color: whitesmoke; color: #4c4c4c; font-size: 10.6667px;">$user = 'gps\SCCM Client Computer Joiners'
$ou = 'OU=SCCM Test Clients,OU=SCCM,OU=Service,OU=Company,DC=gopas,DC=virtual'
DSACLS $ou /R $user
DSACLS $ou /I:S /G "$($user):GR;;computer"
DSACLS $ou /I:S /G "$($user):CA;Reset Password;computer"
DSACLS $ou /I:S /G "$($user):WP;pwdLastSet;computer"
DSACLS $ou /I:S /G "$($user):WP;Logon Information;computer"
DSACLS $ou /I:S /G "$($user):WP;description;computer"
DSACLS $ou /I:S /G "$($user):WP;displayName;computer"
DSACLS $ou /I:S /G "$($user):WP;sAMAccountName;computer"
DSACLS $ou /I:S /G "$($user):WP;DNS Host Name Attributes;computer"
DSACLS $ou /I:S /G "$($user):WP;Account Restrictions;computer"
DSACLS $ou /I:S /G "$($user):WP;servicePrincipalName;computer"
DSACLS $ou /I:S /G "$($user):CC;computer;organizationalUnit"</pre>
<br />
<span style="font-family: "courier new" , "courier" , monospace;">samba-tool</span> has a subcommand <span style="font-family: "courier new" , "courier" , monospace;">dsacl set</span> that I thought might be able to accomplish the same task. After a lot of work trying to get the arguments correct, I got to this point:
<br />
<blockquote>
<tt>[root@dc1 ~]# samba-tool dsacl set --action=allow --objectdn='cn=Computers,dc=samba4,dc=local' --trusteedn='cn=Domain Join,cn=Users,dc=samba4,dc=local' --sddl='GR;;computer' --realm=SAMBA4.LOCAL -U administrator --password="$( cat /root/.password )"<br />
new descriptor for cn=Computers,dc=samba4,dc=local:<br />
O:DAG:DAD:AI(A;;RPWPCRCCDCLCLORCWOWDSDDTSW;;;SY)(A;;RPWPCRCCDCLCLORCWOWDSW;;;DA)(OA;;CCDC;bf967a86-0de6-11d0-a285-00aa003049e2;;AO)(OA;;CCDC;bf967aba-0de6-11d0-a285-00aa003049e2;;AO)(OA;;CCDC;bf967a9c-0de6-11d0-a285-00aa003049e2;;AO)(OA;;CCDC;bf967aa8-0de6-11d0-a285-00aa003049e2;;PO)(A;;RPLCLORC;;;AU)(OA;;CCDC;4828cc14-1437-45bc-9b07-ad6f015e5f28;;AO)(OA;CIIOID;RP;4c164200-20c0-11d0-a768-00aa006e0529;4828cc14-1437-45bc-9b07-ad6f015e5f28;RU)(OA;CIIOID;RP;4c164200-20c0-11d0-a768-00aa006e0529;bf967aba-0de6-11d0-a285-00aa003049e2;RU)(OA;CIIOID;RP;5f202010-79a5-11d0-9020-00c04fc2d4cf;4828cc14-1437-45bc-9b07-ad6f015e5f28;RU)(OA;CIIOID;RP;5f202010-79a5-11d0-9020-00c04fc2d4cf;bf967aba-0de6-11d0-a285-00aa003049e2;RU)(OA;CIIOID;RP;bc0ac240-79a9-11d0-9020-00c04fc2d4cf;4828cc14-1437-45bc-9b07-ad6f015e5f28;RU)(OA;CIIOID;RP;bc0ac240-79a9-11d0-9020-00c04fc2d4cf;bf967aba-0de6-11d0-a285-00aa003049e2;RU)(OA;CIIOID;RP;59ba2f42-79a2-11d0-9020-00c04fc2d3cf;4828cc14-1437-45bc-9b07-ad6f015e5f28;RU)(OA;CIIOID;RP;59ba2f42-79a2-11d0-9020-00c04fc2d3cf;bf967aba-0de6-11d0-a285-00aa003049e2;RU)(OA;CIIOID;RP;037088f8-0ae1-11d2-b422-00a0c968f939;4828cc14-1437-45bc-9b07-ad6f015e5f28;RU)(OA;CIIOID;RP;037088f8-0ae1-11d2-b422-00a0c968f939;bf967aba-0de6-11d0-a285-00aa003049e2;RU)(OA;CIIOID;RP;b7c69e6d-2cc7-11d2-854e-00a0c983f608;bf967a86-0de6-11d0-a285-00aa003049e2;ED)(OA;CIIOID;RP;b7c69e6d-2cc7-11d2-854e-00a0c983f608;bf967a9c-0de6-11d0-a285-00aa003049e2;ED)(OA;CIIOID;RP;b7c69e6d-2cc7-11d2-854e-00a0c983f608;bf967aba-0de6-11d0-a285-00aa003049e2;ED)(OA;CIIOID;RPLCLORC;;4828cc14-1437-45bc-9b07-ad6f015e5f28;RU)(OA;CIIOID;RPLCLORC;;bf967a9c-0de6-11d0-a285-00aa003049e2;RU)(OA;CIIOID;RPLCLORC;;bf967aba-0de6-11d0-a285-00aa003049e2;RU)(OA;CIID;RPWPCR;91e647de-d96f-4b70-9557-d63ff4f3ccd8;;PS)(A;CIID;RPWPCRCCDCLCLORCWOWDSDDTSW;;;EA)(A;CIID;LC;;;RU)(A;CIID;RPWPCRCCLCLORCWOWDSDSW;;;BA)S:AI(OU;CIIOIDSA;WP;f30e3bbe-9ff0-11d1-b603-0000f80367c1;bf967aa5-0de6-11d0-a285-00aa003049e2;WD)(OU;CIIOIDSA;WP;f30e3bbf-9ff0-11d1-b603-0000f80367c1;bf967aa5-0de6-11d0-a285-00aa003049e2;WD)<br />
ERROR(<type 'exceptions.TypeError'>): uncaught exception - Unable to parse SDDL<br />
File "/usr/lib64/python2.7/site-packages/samba/netcmd/__init__.py", line 176, in _run<br />
return self.run(*args, **kwargs)<br />
File "/usr/lib64/python2.7/site-packages/samba/netcmd/dsacl.py", line 174, in run<br />
self.add_ace(samdb, objectdn, new_ace)<br />
File "/usr/lib64/python2.7/site-packages/samba/netcmd/dsacl.py", line 129, in add_ace<br />
desc = security.descriptor.from_sddl(desc_sddl, self.get_domain_sid(samdb))
</tt>
</blockquote>
So... I think the arguments to <span style="font-family: "courier new" , "courier" , monospace;">dsacls</span> are some kind of "friendly" names that resolve to UUIDs or SIDs or something on the back end, but I can't figure out how to do the mapping.<br />
<br />
Suggestions welcome.Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com4tag:blogger.com,1999:blog-8108277809937554792.post-68446489096008595772014-06-21T18:46:00.002-05:002014-06-21T18:46:42.619-05:00systemd 101By request, here's a presentation that I did for <a href="http://www.silug.org/">SILUG</a>: <a href="https://docs.google.com/presentation/d/10YwWZdBa3ffl7kVa2p21L9VqET2CRmVoWJpVBW6ujgg/edit?usp=sharing">https://docs.google.com/presentation/d/10YwWZdBa3ffl7kVa2p21L9VqET2CRmVoWJpVBW6ujgg/edit?usp=sharing</a>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-37731445841661553112012-10-27T22:48:00.000-05:002012-10-27T22:48:01.261-05:00Health updateI don't often post anything personal (or really anything at all, for that matter), but I'm going to make an exception today.<br />
<br />
Today marks 9 months since I started working on losing weight and generally improving my health. At the time, I weighed somewhere around twice what I should (maybe more), and I hadn't been at a healthy weight for nearly 20 years. The scary thing is that I really didn't see myself as <i>that</i> heavy, but obviously my self-image didn't match reality in the slightest... As a co-worker put it, you don't get to that size without a healthy dose of denial.<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-VeLCyhfIc88/UInKRySPnNI/AAAAAAAAAEo/wXMEnSDxmEs/s1600/60317_427118536965_1679827_n.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="132" src="http://2.bp.blogspot.com/-VeLCyhfIc88/UInKRySPnNI/AAAAAAAAAEo/wXMEnSDxmEs/s200/60317_427118536965_1679827_n.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">2 years ago</td></tr>
</tbody></table>
It took a weight loss competition organized by another co-worker to get me motivated, but I decided in January that I was going to lose weight and get back to some level of physical fitness. I've had a lot of success, as anyone who knows me can tell, so I get asked a lot how I lost all the weight. The short answer that I give is that I lost it the old-fashioned way - diet and exercise. There is a much longer answer though, so bear with me...<br />
<br />
Diet was definitely the biggest change for me. I've always eaten way too much food. It wasn't necessarily all bad food, although it often was, but the sheer quantity was what got me in trouble. I decided to do three things to address that. First of all (and most importantly), I finally started logging what I was eating (as my wife had been trying to get me to do for <i>years</i>). I found the <a href="http://www.loseit.com/">Lose It!</a> app, which made this painless. It was absolutely invaluable since it let me see what foods were OK to eat and which weren't (and to see just how bad those things were).<br />
<br />
Second, I tried to eat more of the right foods, like lean proteins and vegetables. I started to avoid sugar, starches (no pasta, bread, or rice), and high-fat foods (with a few exceptions like almonds, which became one of my favorite snacks). Conveniently, since I was watching my calorie intake, the things I was trying to eat are low-calorie, which meant I didn't have to starve myself at all.<br />
<br />
The third diet change that I made was to start snacking through the day, usually eating something every couple of hours. This was the weirdest part, focusing on eating regularly and often in order to lose weight, and it was odd never really being full, but at the same time I never really got hungry enough to have impulse control issues.<br />
<br />
Note that when I say "diet", I'm trying to avoid the connotation that the world normally holds. I never meant for any of this to be a short-term change in my eating, but rather I considered this to be a lifestyle change. I have no intention of going back to anything resembling my old diet, no matter what shape I'm in or how active I am.<br />
<br />
Speaking of activity, I struggled a bit to find exercise that I was physically capable of doing for any length of time, without hurting myself. My friend Artie (who had recently lost a large amount of weight himself, and who was my biggest inspiration for putting in all this effort) worked for a while to convince me to go out for short walks with him. With my bad knees, walking was extremely uncomfortable. Eventually I gave in though, and we started walking as often as possible. At first, a 15-minute walk would nearly kill me. I kept walking as often as I could though, either at lunch, in the afternoons just to clear my head, with my family in the evenings, you name it. By May, I walked a 5K with Artie (in just over 50 minutes). It was looking like I would be able to run a 5K this past month, but unfortunately an injury slowed me down just enough that I wasn't able to.<br />
<br />
Somewhere early on, I started riding our stationary recumbent bike (which had sat in our house, collecting dust for around 5 years). At first, I was lucky to do 5-10 minutes. After a few weeks, I recall doing an hour, non-stop, and feeling like I wasn't going to be able to walk afterwards. At some point around then, I started riding my real bike and found that I couldn't climb a hill. I kept working on it though, and eventually I was able to ride 10 miles, 15 miles, 25 miles, 33 miles, and ultimately 50 miles. (At some point in the near future, I'd like to try to ride 100 miles, but that's a pretty massive time commitment.)<br />
<br />
The most rewarding part of this entire experience has been the lifestyle change that my entire family has gone through. It's one thing for everyone to diet together, but that's not what we've done. We're all eating differently, cooking together, and finding ways to be active together. My wife Kara has been incredibly patient and understanding, even when I've been overly single-minded about trying to hit whatever goal I had on any given day. She has been on-board since the beginning, and has also managed to lose a significant amount of weight. (I'll leave it to her to give details.) All of the changes have been great for our daughter Emma too, who is in better shape now than she has been at any other point in her life. I know a lot of people who try to lose weight on their own, and I'm sure it can be done, but I certainly wouldn't recommend it.<br />
<br />
I'm fortunate to have a great support system. I mentioned Artie before (thanks, Artie!), but I also have to thank Mike for pushing me to do more, go a little faster, or go a little farther. There are many others (yes, I'm looking at you, Emma) who have helped, and I apologize for not naming every one of you, but I do appreciate all of the support.<br />
<br />
As of this morning, I have lost over 36% of the weight I was carrying at the end of January. I need to get to 50%, give or take, so I still have quite a bit to lose, but I have complete confidence that it will come off over the next few months. I have had to replace my wardrobe multiple times now (I'm already wearing shirts 4 sizes smaller than I was wearing when I started), so I'm perfectly OK with the loss leveling off for a while. :-)<br />
<br />
<table align="center" cellpadding="0" cellspacing="0" class="tr-caption-container" style="margin-left: auto; margin-right: auto; text-align: center;"><tbody>
<tr><td style="text-align: center;"><a href="http://2.bp.blogspot.com/-s_MDgCfq_-4/UInKTfKTlhI/AAAAAAAAAEw/n1A_s7cgTqw/s1600/617037_4287521264592_1194442838_o.jpg" imageanchor="1" style="margin-left: auto; margin-right: auto;"><img border="0" height="200" src="http://2.bp.blogspot.com/-s_MDgCfq_-4/UInKTfKTlhI/AAAAAAAAAEw/n1A_s7cgTqw/s200/617037_4287521264592_1194442838_o.jpg" width="200" /></a></td></tr>
<tr><td class="tr-caption" style="text-align: center;">Now</td></tr>
</tbody></table>
Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com9tag:blogger.com,1999:blog-8108277809937554792.post-58309540077761144902011-09-19T12:17:00.001-05:002011-09-19T12:17:35.862-05:00New cpanspec coming soonI haven't released a new version of <a href="https://github.com/silug/cpanspec">cpanspec</a> in quite a while, but I have been working on it off and on with the help of several other people. The big feature that I added was dependency extraction from tests, but I wasn't happy with the results of it. Luckily, other people made it better, plus knocked a bunch of stuff off my TODO list.<br />
<br />
The current list of changes looks like this:<br />
<br />
<br />
<ul>
<li>Extract dependencies from tests.</li>
<li>Add script detection (patch from Jeff Fearn).</li>
<li>Lots and lots of patches from Dennis Kaarsemaker and Gavin Carr:</li>
<ul>
<li>Drop cpanget and add the functionality to cpanspec.</li>
<li>Check the search path for rpm, rpmbuild, etc.</li>
<li>Add CC0 licence.</li>
<li>Change %{optimize} to %{optflags}.</li>
<li>Make tarball directory version component optional.</li>
<li>Add an option to print the generated specfile to stdout</li>
<li>Allow building rpms for slightly older perl versions</li>
<li>Check all build requirements against CPAN</li>
<li>Stop losing dependency version information for Module::Build, ExtUtils::MakeMaker, etc.</li>
<li>Strip any version comparison operator from the 'perl' build requirement</li>
<li>Add entries from configure_requres in META.yml as build dependencies</li>
<li>Detect scripts better</li>
<li>Don't let Module::AutoInstall run interactively</li>
<li>Add a simple blacklisting mechanism</li>
</ul>
</ul>
<br />
<div>
In my light testing, this version has been working beautifully, but I'd really like to hear some more positive feedback before I push this out into Fedora, so if you package Perl modules, give it a try and let me know what you think.</div>
Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-59891701696762192862011-08-28T22:23:00.002-05:002011-08-28T22:25:10.801-05:00Vim: From Essentials to Mastery at OLF 2011<br />
<div class="separator" style="clear: both; text-align: center;">
<a href="https://ohiolinux.org/" imageanchor="1" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="https://ohiolinux.org//sites/www.ohiolinux.org/files/images/promos/OLF2011.300x250_v1.png" /></a></div>
<a href="http://ohiolinux.org/">Ohio LinuxFest 2011</a> is coming up next week, September 9-11. As part of <a href="http://ohiolinux.org/olfi">OLF Institute</a>, Bill Odom and I will be teaching a full-day class on Vim on Friday, September 9. If you're like I was a year or so ago, and you think you know Vim just because you've been using <span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;">vi</span> forever, you really need to come to our class. Some of the things you can do with Vim will just blow your mind. (And if you don't even know <span class="Apple-style-span" style="font-family: 'Courier New',Courier,monospace;">vi</span>, the class will be life-changing. :-)<br />
<br />
The main conference is on Saturday, September 10. At 2PM, we'll have a special <a href="http://www.vimgeeks.org/">Vim Geeks</a> Columbus BoF session which, like the rest of the conference on Saturday, you can attend for free.<br />
<br />
For more information about the Vim class, see<br />
<br />
<a href="https://ohiolinux.org/olfi#VIM">https://ohiolinux.org/olfi#VIM</a><br />
<br />
For more information about OLF 2011, see the web site:<br />
<br />
<a href="http://ohiolinux.org/">http://ohiolinux.org/</a><br />
<br />
For more information about Vim Geeks, our local Vim users group, see<br />
<br />
<a href="http://www.vimgeeks.org/">http://www.vimgeeks.org/</a><br />
<br />
The full class description follows:<br />
<br />
<div style="text-align: center;">
<u>VIM: From Essentials to Mastery</u></div>
<br />
Instructors: Steven Pritchard and Bill Odom<br />
<br />
Vim deserves its reputation as one of the most powerful tools in an admin or developer's toolbox -- but it's not exactly friendly and approachable. Even long-time users rarely employ more than a fraction of its capabilities, and new users are often left wondering why so many apparently-sane people won't shut up about how awesome it is. The stark UI, the steep learning curve, the host of idiosyncrasies... mastering Vim is a challenge, and that's putting it politely.<br />
<br />
In this class, you'll learn why it's worth the effort.<br />
<br />
We'll start by covering the essentials of Vim, like modes, motions, operators, and commands, with an emphasis on <b>why</b> Vim works the way it does in addition to <b>how</b> it works. With the fundamentals firmly established, we'll work our way through real-world examples of using Vim to perform astounding feats that poor souls using lesser editors can only imagine. We'll cover ways to integrate Vim with your environment, tailor it to your work, and generally bend it to your every whim. We'll discuss important settings, advanced techniques, useful customizations, handy scripts, must-have plugins, crafty tips, and sneaky tricks.<br />
<br />
In short, we'll explore how to use Vim most effectively, so it lives up to the awesome reputation that you'll soon be telling all your disbelieving friends about.<br />
<br />
Bios: Steven Pritchard has nearly two decades of Linux and Unix experience. A dedicated Open Source advocate, he founded the Southern Illinois Linux Users Group in 1994 and has been a volunteer developer with Red Hat's Fedora Project since it began in 2003. He is also an author of the award winning LPI Linux Certification in a Nutshell, 2nd Ed (O'Reilly & Associates). Steven currently offers his technical services through the Computer Room, a retail technical sales and service company outside of St. Louis, Missouri.<br />
<br />
Bill Odom has over two decades of experience as a systems architect and software developer, working on everything from wiki software for Internet startups to global identity management solutions for Fortune 500 companies. He's also a long-time member of the Open Source community, an active member (and periodic leader) of several user groups in the St. Louis area, and served as president of the Perl Foundation from 2005 through 2007.<br />
<br />
Steven and Bill are both long-time users and advocates of Vim, and are the founders of the St. Louis Vim Geeks. They've given several well-received presentations, tutorials, and classes on Vim to many Midwest organizations.Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-82635605114819135332010-10-21T09:18:00.000-05:002010-10-21T09:18:04.478-05:00Regular ExpressionsSlides from my talk at last night's <a href="http://stlouis.pm.org/">St. Louis Perl Mongers</a> meeting:<br />
<br />
<iframe frameborder="0" height="342" src="https://docs.google.com/present/embed?id=dc4txvcb_54gd3qphg5" width="410"></iframe>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-29295304440224814602010-09-28T11:24:00.000-05:002011-10-10T12:55:53.685-05:00String::RandomI mentioned my one module on CPAN (String::Random) to a friend yesterday, and got the response "You wrote that?" Honestly, I was shocked that he'd heard of it. (There are so many modules on CPAN that I doubt most Perl programmers have heard of 99% of them.)<br />
<br />
I decided to Google for the module a bit to see if there were many mentions (fully expecting to find some "I looked at the code, and my eyes are still bleeding" comments), and I was pleasantly surprised to find this rather old tutorial on using the module: <a href="http://www.perlmonks.org/?node_id=88021">http://www.perlmonks.org/?node_id=88021</a><br />
<br />
I also found a Ruby port on GitHub: <a href="http://github.com/repeatedly/ruby-string-random">http://github.com/repeatedly/ruby-string-random</a><br />
<br />
So far, I haven't found anyone ripping into it, but I'm sure it's out there...Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com1tag:blogger.com,1999:blog-8108277809937554792.post-48368768193494598422010-08-19T21:01:00.000-05:002010-08-19T21:01:50.792-05:00Ohio Linux Fest<a href="http://www.ohiolinux.org/" style="clear: left; float: left; margin-bottom: 1em; margin-right: 1em;"><img border="0" src="http://www.ohiolinux.org/sites/www.ohiolinux.org/files/images/promos/125x125box-speaker.png" /></a>It's time again for the <a href="http://www.ohiolinux.org/">Ohio Linux Fest</a> (September 10-12, 2010 in Columbus, Ohio). I'm going. So should you. :-)<br />
<br />
I'm teaching an abbreviated version of my <a href="http://www.computerroom.us/2009/07/class-data-recovery-with-open-source.html">Data Recovery class</a> (for an abbreviated price, I might add) as part of the <a href="http://ohiolinux.org/olfu.html">OLFU program</a> on Friday, September 11. If you are responsible for any hard drives, I highly recommend the class.<br />
<br />
Speaking of classes, we also just announced a tag-team <a href="http://www.computerroom.us/2010/08/class-vi-basics.html">vi</a> and <a href="http://www.computerroom.us/2010/08/class-vim-fandango.html">vim</a> class lineup that Bill Odom and I will be doing late next month. Anyone in the St. Louis area who uses vi and thinks it is a burden (or avoids it like the plague) should take my vi basics class. <b>Everyone</b> should take Bill's vim class. (It's amazing stuff.)Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-80893632852599569972010-07-29T19:59:00.000-05:002010-07-29T19:59:17.490-05:00Generating ssh keys in PuTTY<span class="Apple-style-span" style="font-family: Arial;">I had to fire up Windows today to explain to someone how to generate ssh keys (for use with <a href="http://www.chiark.greenend.org.uk/~sgtatham/putty/">PuTTY</a>). I figured since I went to all that trouble, I should share...</span><div><span class="Apple-style-span" style="font-family: Arial;"><br />
</span><div><span class="Apple-style-span" style="-webkit-border-horizontal-spacing: 2px; -webkit-border-vertical-spacing: 2px;"><strong><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Note</span></strong><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">: The Linux version of </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">puttygen</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> is all command-line, so these instructions will only work with the Windows version.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">When you first run </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;"><a href="http://the.earth.li/~sgtatham/putty/latest/x86/puttygen.exe">puttygen</a></span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">, the default (along the bottom) should be to create a key type "SSH-2 RSA". If not, select that. 1024 bits is fine (box at the very bottom right), which should be the default.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Now, hit "Generate". It will ask you to move the mouse around a bit to generate some randomness. When that is done, it will generate the key. Put your email address in the "Key comment" field. Then select the key in the box at the top (under "Public key for pasting into OpenSSH authorized_keys file:"), copy it, and paste it into the </span><tt><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">.ssh/authorized_keys</span></tt><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> on the system you want to be able to login to with that key.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Fill in the box next to "Key passphrase" with something long that you'll remember. (It's fine to use a full sentence or something. Remember, it's a pass<b>phrase</b>, not a pass<b>word</b>.) Enter the passphrase again in the "Confirm passphrase" box.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">Next hit the "Save public key" button and save that half to a file with "public" in the file name. Then hit the "Save private key" button and save that half to a file with "private" in the file name.</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">In PuTTY, in the configuration dialog, expand "Connection" in the left pane (if it isn't already), then expand "SSH". Click on "Auth". Next to the box that says "Private key file for authentication", hit "Browse" and select the "private" file you just saved. Be sure to save your settings so you don't have to feed this in every time. (Click on "Session" at the top of the left pane, then under "Saved Sessions" click on "Default Settings" and hit "Save".)</span><br />
<span class="Apple-style-span" style="-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Arial, Helvetica, sans-serif;"><br />
</span><br />
<span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"></span><span class="Apple-style-span" style="-webkit-border-horizontal-spacing: 0px; -webkit-border-vertical-spacing: 0px; font-family: Arial, Helvetica, sans-serif;">Now when you try to login to the system you previously dropped your public key on, you should be prompted for the passphrase for your key rather than the password for your account on the system.</span></span></div></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"><br />
</span></div><div><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;">If I may add a little editorializing here, I do have to point out that this is all <b>much</b> easier with </span><span class="Apple-style-span" style="font-family: 'Courier New', Courier, monospace;">ssh-keygen</span><span class="Apple-style-span" style="font-family: Arial, Helvetica, sans-serif;"> on Linux. And, IMHO, if you're doing Linux administration from a Windows PC, you're doing it wrong. But that's just me. :-)</span></div>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-63340740626545591512010-02-26T14:36:00.002-06:002010-02-27T16:43:43.276-06:00Stupid git tricksI had a directory in CVS that I used for a catch-all for random scripts. (For example, this is where <a href="http://cpanspec.sourceforge.net/">cpanspec</a> lived before moving it to Sourceforge CVS.) Now that I'm using <a href="http://github.com/silug">github</a>, I'm trying to split these scripts up into separate git repos. This is the procedure I've come up with...<br /><br />First I use <b>git cvsimport</b> to pull in the whole CVS tree:<br /><blockquote><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">git cvsimport -d :ext:</span><i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">user</span></i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">@</span><i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">host</span></i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">:/</span><i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">cvsroot</span></i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;"> -C </span><i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">myscript</span></i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;"> </span><i><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">cvs_module</span></i></blockquote>This will create a directory named <i>myscript</i>. Next, go into that directory and use <b>git filter-branch</b> to remove everything but the file(s) we care about (in this case, myscript again).<br /><blockquote><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">git filter-branch --prune-empty --tree-filter 'find -maxdepth 1 -type f \! -name <i>myscript</i> -delete' HEAD</span></blockquote>This ends up leaving some stale objects that can be cleaned up by removing everything other than <span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">master</span> in <span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">.git/refs/heads/</span>, the entire directory <span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">.git/refs/original/</span>, and any unrelated tags in <span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">.git/refs/tags/</span><span class="Apple-style-span" style="font-family:inherit;"> (at least in my example with no branches and such), then cleaning up with a few git commands:</span><br /><blockquote><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">git gc --aggressive</span><br /><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">git prune</span><br /><span class="Apple-style-span" style="font-family:'Courier New', Courier, monospace;">git repack -a -d</span></blockquote><span class="Apple-style-span" style="font-family:inherit;"></span><br />The total number of objects listed by <b>git gc</b> and <b>git repack</b> should be much smaller than the original number <b>git cvsimport</b> reported. (I also confirmed that <b>git fsck --unreachable</b> doesn't find anything.)<div><br /></div><div><b>[Update]</b> Apparently I had found <a href="http://loupgaroublond.blogspot.com/2007/12/how-to-split-repository-in-git.html">this answer to my problem</a> a while back and forgot about it. Oops.</div>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com0tag:blogger.com,1999:blog-8108277809937554792.post-18990485828787752872010-02-22T17:24:00.000-06:002010-02-22T17:24:51.403-06:00Where do we go next with RAID?<span class="Apple-style-span" style="font-family: Arial; font-size: small;"><span class="Apple-style-span" style="font-size: 13px;">So a friend of mine sent me a link to </span></span><a href="http://blogs.zdnet.com/storage/?p=805&tag=nl.e539">this blog post</a>. A couple of things jumped out at me...<br />
<blockquote>When a drive fails in a 7 drive, 2 TB SATA disk RAID 5, you’ll have 6 remaining 2 TB drives. As the RAID controller is reconstructing the data it is very likely it will see an URE. At that point the RAID reconstruction stops.</blockquote>And later...<br />
<blockquote>RAID proponents assumed that disk failures are independent events, but long experience has shown this is not the case: 1 drive failure means another is much more likely.</blockquote>That sounds an <b>awful</b> lot like what I've been saying for 8 or 9 years now... (Well, not specifically about 2TB drives, but you know what I mean... :-)<br />
<div><br />
</div><div><div>So the myth that I've been hearing for the last 15 years or so is that you get speed and data security with RAID 5. The fact is that the speed of an intact array is terrible, and to use the word "speed" in regards</div><div>to a degraded array would be an oxymoron. Add that to the odds of a failure of one of your "good" drives during a rebuild, and you get one big pile of fail.</div><div><br />
</div><div>The advantage of RAID 5 is capacity. Period. Any other RAID solution costs more in terms of raw storage capacity. RAID 6 gives you one less drive of capacity in exchage for improving your odds of a successful</div><div>rebuild, but as you all know, I still don't trust it for anything that we don't have a mirror of somewhere.</div><div><br />
</div><div>We've been doing a lot of RAID 1 and RAID 1+0, which is fine, but ultimately you have the same problem there with likely failures while trying to rebuild an array, but you have the added bonus problem that</div><div>errors may go undetected. They may kill performance, but the checksums on RAID 5 and 6 do give you an added safety net since you can detect corrupted data.</div><div><br />
</div><div>For some of our largest arrays, we've been doing mirrored (or rsync'd) RAID 5 or 6, which, while extraordinarily wasteful in terms of storage space, gives us very good odds of recovery from catastrophic hardware failure.</div><div><br />
</div><div>I have to wonder if the real answer here might ultimately be to add parity to a stripe/mirror set, so that any combination of drive failures in an array of <i>n</i> drives that leaves you with at least <i>(n-2)/2</i> working</div><div>drives is easily recoverable... (Maybe doing RAID 6 over pairs of mirrored drives would be sufficient. I have to think on that a bit...)</div></div>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com3tag:blogger.com,1999:blog-8108277809937554792.post-83985402975912271282010-02-18T12:36:00.002-06:002020-04-23T09:08:12.667-05:00Open-Source Point of Sale?Dear Lazyweb,<br />
<div>
<br /></div>
<div>
I need an open-source POS solution for a client. They have a small cafeteria-type restaurant + gift shop. Currently they are using a craptastic closed-source commercial solution that offers no support despite requiring a huge service contract.</div>
<div>
<br /></div>
<div>
Lots of bonus points for something web-based, since the POS terminals they have are rather low-end.</div>
<div>
<br /></div>
<div>
FWIW, we've tried the following:</div>
<div>
<ul>
<li><a href="http://www.adempiere.com/index.php/ADempiere">Adempiere</a> + <a href="https://posterita.com/">Posterita</a></li>
<li><a href="http://www.openbravo.com/product/pos/">OpenBravo POS</a></li>
<li><a href="http://l-ane.sourceforge.net/">L’âne</a> (because the developer is local-ish)</li>
<li><a href="http://ofbiz.apache.org/">OFBiz</a> (both the web POS and the java client)</li>
</ul>
<div>
Posterita comes the closest to being what we want (web-based, AJAX-y, etc.), but development has gone closed-source apparently. OpenBravo POS is probably the most functional, but it's difficult to figure out how to do much of anything with it. OFBiz has a nice, simple POS app, but it's horribly buggy (and rather slow too).</div>
<div>
<br /></div>
<div>
Given an infinite amount of free time, I'd probably hack on the Adempiere + Posterita (last open-source release) combo, but, well, time is not on my side here...</div>
</div>
Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com31tag:blogger.com,1999:blog-8108277809937554792.post-11633438936005869482009-11-20T11:08:00.001-06:002009-11-21T10:07:40.018-06:00Why developers suck as adminsSo Fedora 12 <a href="https://www.redhat.com/archives/fedora-devel-list/2009-November/msg00926.html">allows regular users to install packages</a> as long as a) the user is logged in on the console, and b) they are signed and from a trusted repository.<br /><br />I think it's great that we can enable this functionality, and I'd even argue that it should be on by default on the Live spins (and installs from the Live spins), but in the general case this is a horrible idea.<br /><br />I've seen a few arguments why this doesn't matter. For example, anyone with local console access already owns the box, right? Well, console access != physical access. Think, well, anything in a server room (systems in locked cabinets attached to a KVM, or virtual machines). While this gives more ammo to us old-timers for not putting X on our servers (I'm looking at you, <a href="http://skvidal.wordpress.com/2009/11/18/polkit-and-package-kit-and-changing-settings/">skvidal</a>), the reality is that it's not realistic to expect all servers to run without a GUI.<br /><br />I don't want to re-hash the whole (long) thread linked to above, but I think it is important to point out some of the solid reasons why this change is a bad idea.<br /><br /><ul><li>The installation of one package shouldn't change the behavior of the system. (This one package changes the behavior of the system, plus allows for other packages to be installed that could do the same.) If you take into account that unintended dependencies tend to pull in random stuff during upgrades, this becomes especially important.</li><li>Can we really guarantee that there are <b>no</b> signed packages available that are exploitable, all the time?</li><li>This is a major change in behavior from Fedora 11 that did not go through the <a href="https://fedoraproject.org/wiki/Features/Policy">Feature Process</a>, <a href="https://fedoraproject.org/wiki/Releases/12/FeatureList">unless I'm missing something</a>.</li><li>Possibly even worse, if this "feature" makes it into RHEL 6, you run the risk of a <i>lot</i> of semi- and non-technical sysadmins having yet another security decision made for them, probably without their knowledge. (How many are aware of ctrl-alt-del, console users being able to shutdown/reboot, grub allowing kernel options ("single", for example) unless you set a password, etc.? Of course I've never understood the logic of all of that being open, but magic sysrq being off by default.)</li><li>At the very least, this is a DoS attack vector, although more likely due to somebody screwing up and installing a bunch of packages rather than somebody intentionally trying to fill /, /usr, or /var.</li></ul><br />Oh, and about the title... Developers of software get stuck into a mindset of "make my software work, no matter what", and, on a related note, tend to have tunnel vision about the use cases for their software. One of the things I love about Fedora is that we have a lot of sysadmins who happen to be coders, so we tend to find a good balance between "usability" (AKA letting the developers go nuts) and maintenance/security. This one slipped by us, but I hope the decision will be made to push an update with a more sane default. <b>[Update: </b><a href="https://www.redhat.com/archives/fedora-devel-list/2009-November/msg01445.html">It was.</a><b>]</b>Steven Pritchardhttp://www.blogger.com/profile/00716303018104544735noreply@blogger.com3