<div>Matt Z., </div><div><br></div>Could you be more specific about your backup mechanism? Are you doing a full backup? Over the network? Using NFS/SCP/RSYNC/RDIFF?<div><br></div><div>1. any indiciation why the drop from 40MB/s to 2MB/s</div>
<div>2. is the backup online? are jobs writing while you are reading?</div><div><br></div><div>Sequential read should get good performance on a raid5, if you are writing and reading at the same time, one will suffer (usually writing).</div>
<div><br></div><div>any benchmarking with IOZone?</div><div><br></div><div>what is the controller cache size?</div><div><br></div><div>Side notes:</div><div>why raid5? We've completely phased out raid5 in favor of raid6.</div>
<div><br></div><div>Martin Gehrke</div><div><br></div><div>--------------------------</div><div>Calling all Sysadmins. Be one of the first members of a new Syadmin group in Pittsburgh.</div><div><a href="http://snapgh.com">SNAPGH</a> System and Network Administrators of Pittsburgh<br>
<br><div class="gmail_quote">On Tue, Jun 14, 2011 at 2:00 PM, Matthew Zwier <span dir="ltr"><<a href="mailto:mczwier@gmail.com">mczwier@gmail.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex;">
Hi all,<br>
<br>
I'm the systems administrator for a relatively small (~20 nodes, ~320<br>
cores) scientific computing cluster with relatively large (20 TB)<br>
storage needs. We have a couple of RAID5 arrays on a Dell PERC/5E<br>
(aka LSI MegaRAID) controller , both running XFS filesystems, and<br>
while performance is generally acceptable, it appears I can't get a<br>
backup in under five days for our 11 TB array. That leads me to a<br>
couple of questions:<br>
<br>
1) That translates to about a 40 MB/s sustained read, with frequent<br>
lengthy drops to around 2 MB/s (this using xfsdump into /dev/null, for<br>
the moment). For those of you with more experience than I...is that<br>
typical performance for a filesystem dump? Pleasantly fast?<br>
Unacceptably slow?<br>
<br>
2) Does anyone know of documentation about how to go about tuning an<br>
on-line hardware RAID array in Linux, specifically for file service?<br>
About all I can find are discussions about how to optimize MySQL<br>
performance, or tips on what parameters in /sys to tweak while piping<br>
zeros directly to /dev/sdb using dd, and the like. I can't find any<br>
documentation on how various hardware/kernel/filesystem parameters<br>
interact. The three-way optimization problem among RAID controller<br>
settings (i.e. read-ahead), disk- and controller-specific kernel<br>
settings (TCQ depth, read-ahead), and I/O-scheduler-specific settings<br>
(noop vs. deadline vs. cfq, queue size, etc) is just killing me.<br>
<br>
Matt Z.<br>
_______________________________________________<br>
wplug mailing list<br>
<a href="mailto:wplug@wplug.org">wplug@wplug.org</a><br>
<a href="http://www.wplug.org/mailman/listinfo/wplug" target="_blank">http://www.wplug.org/mailman/listinfo/wplug</a><br>
</blockquote></div><br></div>