[wplug] Hardware RAID tuning

Wed Jun 15 10:48:20 EDT 2011

MZ,
	What tool have you been using to determine the spikes in IO latency? I
assume something like iostat?

	Also, have you considered using rsync? I've had great success in the
past moving around large filesystems with rsync. If SSH overhead isn't
your bottleneck, then you should have CPU to spare for the rsync
algorithm. With rsync and ssh with key based auth on both sides, you
should be able to easily do something like this:

rsync -avz -e "ssh -i $YOURKEY" $USER at 10.1.254.242:/backups/$DIR . 
	
	This, of course, will really only help if your filesystems don't change
more than 60% or so over the course of your backup interval.

HTH,
Nate	 


On Tue, 2011-06-14 at 15:08 -0400, Matthew Zwier wrote:
> Thanks for that link...looks very informative from the six paragraphs
> I've read so far.
> 
> As I mentioned previously, this is a full backup, not incremental, and
> I'm dumping either to a file over SSH or /dev/null locally, with no
> change in performance between the two, so reads by xfsdump must be
> forming the bottleneck.
> 
> MZ
> 
> On Tue, Jun 14, 2011 at 2:47 PM, Martin James Gehrke
> <martin at teamgehrke.com> wrote:
> > MZ,
> > Hard disks, Linux, RAID, server performance tuning
> > : http://www.fccps.cz/download/adv/frr/hdd/hdd.html
> > You can also email the question to lopsa-discus or jump onto IRC
> > Are you doing incremental backups using xfsdump?
> > what are you dumping to?
> > Martin
> >
> >
> >
> >
> > On Tue, Jun 14, 2011 at 2:34 PM, Matthew Zwier <mczwier at gmail.com> wrote:
> >>
> >> Hi Martin,
> >>
> >> Full backup using xfsdump.  I've tried it over the (gigabit) network
> >> (via SSH, into a file on a remote server) and also straight into
> >> /dev/null; numbers are identical.  The drop in throughput appears to
> >> be due to writes to another array on the same HBA channel, but that's
> >> been hard to isolate.
> >>
> >> Our controllers don't support RAID6, so RAID5 is the best I can do
> >> with what I have.  No IOZone benchmarks, since I haven't had any
> >> trouble with performance until now.  Controller cache size is 256 MB.
> >> Connection is 3Gbps over a 4X cable.
> >>
> >> MZ
> >>
> >> On Tue, Jun 14, 2011 at 2:22 PM, Martin James Gehrke
> >> <martin at teamgehrke.com> wrote:
> >> > Matt Z.,
> >> > Could you be more specific about your backup mechanism? Are you doing a
> >> > full
> >> > backup? Over the network? Using NFS/SCP/RSYNC/RDIFF?
> >> > 1. any indiciation why the drop from 40MB/s to 2MB/s
> >> > 2. is the backup online? are jobs writing while you are reading?
> >> > Sequential read should get good performance on a raid5, if you are
> >> > writing
> >> > and reading at the same time, one will suffer (usually writing).
> >> > any benchmarking with IOZone?
> >> > what is the controller cache size?
> >> > Side notes:
> >> > why raid5? We've completely phased out raid5 in favor of raid6.
> >> > Martin Gehrke
> >> > --------------------------
> >> > Calling all Sysadmins. Be one of the first members of a new Syadmin
> >> > group in
> >> > Pittsburgh.
> >> > SNAPGH System and Network Administrators of Pittsburgh
> >> >
> >> > On Tue, Jun 14, 2011 at 2:00 PM, Matthew Zwier <mczwier at gmail.com>
> >> > wrote:
> >> >>
> >> >> Hi all,
> >> >>
> >> >> I'm the systems administrator for a relatively small (~20 nodes, ~320
> >> >> cores) scientific computing cluster with relatively large (20 TB)
> >> >> storage needs.  We have a couple of RAID5 arrays on a Dell PERC/5E
> >> >> (aka LSI MegaRAID) controller , both running XFS filesystems, and
> >> >> while performance is generally acceptable, it appears I can't get a
> >> >> backup in under five days for our 11 TB array.  That leads me to a
> >> >> couple of questions:
> >> >>
> >> >> 1)  That translates to about a 40 MB/s sustained read, with frequent
> >> >> lengthy drops to around 2 MB/s (this using xfsdump into /dev/null, for
> >> >> the moment).  For those of you with more experience than I...is that
> >> >> typical performance for a filesystem dump?  Pleasantly fast?
> >> >> Unacceptably slow?
> >> >>
> >> >> 2)  Does anyone know of documentation about how to go about tuning an
> >> >> on-line hardware RAID array in Linux, specifically for file service?
> >> >> About all I can find are discussions about how to optimize MySQL
> >> >> performance, or tips on what parameters in /sys to tweak while piping
> >> >> zeros directly to /dev/sdb using dd, and the like.  I can't find any
> >> >> documentation on how various hardware/kernel/filesystem parameters
> >> >> interact.  The three-way optimization problem among RAID controller
> >> >> settings (i.e. read-ahead), disk- and controller-specific kernel
> >> >> settings (TCQ depth, read-ahead), and I/O-scheduler-specific settings
> >> >> (noop vs. deadline vs. cfq, queue size, etc) is just killing me.
> >> >>
> >> >> Matt Z.
> >> >> _______________________________________________
> >> >> wplug mailing list
> >> >> wplug at wplug.org
> >> >> http://www.wplug.org/mailman/listinfo/wplug
> >> >
> >> >
> >> > _______________________________________________
> >> > wplug mailing list
> >> > wplug at wplug.org
> >> > http://www.wplug.org/mailman/listinfo/wplug
> >> >
> >> >
> >> _______________________________________________
> >> wplug mailing list
> >> wplug at wplug.org
> >> http://www.wplug.org/mailman/listinfo/wplug
> >
> >
> > _______________________________________________
> > wplug mailing list
> > wplug at wplug.org
> > http://www.wplug.org/mailman/listinfo/wplug
> >
> >
> _______________________________________________
> wplug mailing list
> wplug at wplug.org
> http://www.wplug.org/mailman/listinfo/wplug