[wplug] Hardware RAID tuning

[Matthew Zwier]

Hi all,

Thanks, Nate, Nicholas, ans Martin for your help.  I appear to be
chasing Heisenbugs.  Today, all is much better.  I'm getting 250 MB/s
sustained write speeds on my RAID 5s and 330 MB/s on my RAID 50.  I'm
getting 250 MB/s sustained reads on my 7-disk RAID 5s and 320-400 MB/s
on my 15 disk RAID 50 and RAID 5 respectively, as measured with dd and
iotop.  Yesterday, the writes were topping out at 120 MB/s and the
reads at 40 MB/s by any test I chose to run.  I ran some filesystem
maintenence yesterday night, and the cluster is much more lightly
loaded, but that's all that's changed.

Today, xfsdump is fluctuating between 360 MB/s, 20 MB/s, and 875 kB/s,
with the fluctuations correlating well with read accesses.  The
filesystem I'm dumping is mounted but isn't in use, so it must be
reads from other arrays on the same controller.  Could this be a
poorly-configured disk scheduler?

Seeing as how sequential performance isn't really the 40 MB/s that I
feared, making this a problem with my backup idea instead of the RAID
array itself, I think I'll be trying something other than xfsdump.
rsync isn't the most palatable option (statting 37 million files takes
a long time), but just about anything is better than a 5-day-long
backup cycle.

Thanks again,
MZ