<br><br><div class="gmail_quote">On Tue, Jun 14, 2011 at 2:00 PM, Matthew Zwier <span dir="ltr"><<a href="mailto:mczwier@gmail.com">mczwier@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi all,<br>
<br>
I'm the systems administrator for a relatively small (~20 nodes, ~320<br>
cores) scientific computing cluster with relatively large (20 TB)<br>
storage needs. We have a couple of RAID5 arrays on a Dell PERC/5E<br>
(aka LSI MegaRAID) controller , both running XFS filesystems, and<br>
while performance is generally acceptable, it appears I can't get a<br>
backup in under five days for our 11 TB array. That leads me to a<br>
couple of questions:<br>
<br>
1) That translates to about a 40 MB/s sustained read, with frequent<br>
lengthy drops to around 2 MB/s (this using xfsdump into /dev/null, for<br>
the moment). For those of you with more experience than I...is that<br>
typical performance for a filesystem dump? Pleasantly fast?<br>
Unacceptably slow?<br>
<br>
2) Does anyone know of documentation about how to go about tuning an<br>
on-line hardware RAID array in Linux, specifically for file service?<br>
About all I can find are discussions about how to optimize MySQL<br>
performance, or tips on what parameters in /sys to tweak while piping<br>
zeros directly to /dev/sdb using dd, and the like. I can't find any<br>
documentation on how various hardware/kernel/filesystem parameters<br>
interact. The three-way optimization problem among RAID controller<br>
settings (i.e. read-ahead), disk- and controller-specific kernel<br>
settings (TCQ depth, read-ahead), and I/O-scheduler-specific settings<br>
(noop vs. deadline vs. cfq, queue size, etc) is just killing me.<br>
<br>
Matt Z.<br>
_______________________________________________<br>
wplug mailing list<br>
<a href="mailto:wplug@wplug.org">wplug@wplug.org</a><br>
<a href="http://www.wplug.org/mailman/listinfo/wplug" target="_blank">http://www.wplug.org/mailman/listinfo/wplug</a><br>
</blockquote></div><br>I have a dell system like this. The disks are smaller, 300G, 10K. <br>I see write speeds of 300M per second over 6 drives. <br><br>you will need 10G of free space to run this test.<br><br>run<br> $ dd if=/dev/zero of=/mount/yourraidfs/zero.txt bs=1M count=10000 <br>
<br>Do you see any error massages in the logs?<br><br>You must have a good backup of your data for the next test. The extra load from rebuilding can cause your array to fail. But it's better to have it fail when you have a good backup.<br>
"rebuild your array one disk at a time"<br>buy one new disk.<br>Pull the first disk and replace it.<br>The array will rebuild the disk. This will write data to every used sector. This is one way to prevent bitrot. <br>
check the logs after each rebuild.<br>repeat until you have pulled all disk.<br><br>