[wplug] Re: NFS Availability Issues -- why I should just help off-list ...

[Bryan J. Smith]

Jonathan Billings <billings at negate.org> wrote:
> Go right ahead.  Shooting yourself in the foot is a
> well-established side-effect in the IT industry!

True.  And I'm sorry.

And for every 3 people that know the issues I'm talking about some 97
people will think I'm "just being anal" and "it works for me." 
People think I "just have money" and I'm ignorant for using things
like NFS, the automounter, etc... -- let alone using a network
filesystem approach in general.  I mean, web and Internet services do
everything!  ;)

In this thread, I've also covered using local storage, rsync and
other options as well, minimizing the use of NFS.  There are other
options.  But you still need a home directory.  And I don't know many
apps that work without one.  ;)

Going thin client is another option.  But then you run into the
separate desktop v. all desktops on one server aspect.  And if you
use multiple servers to handle all those thin client desktops, then
you're back to square one on needing a distributed, network
filesystem.  ;)

So we go in circles while someone offers solutions that dance around
the problem, and the real details.

> To be honest, I've been following this thread with interest,

I know.  But 97% of Linux consultants are in the Internet and web
services arena.  They look at me like I don't know what I'm talking
about, or "you (your clients) have money."

I don't mean to be arrogant, but I have pulled a _lot_ of Internet
and web consultants out of the hole they've dug themselves with a
client.  I've literally had some clients turn to me and say, "well X
has been building high availability and clustered web solutions for Y
years" and my response is always, "then why am I here?"

Some of my biggest critics are the ones I've helped the most, largely
because I finally just stopped helping them.  The worst is a guy that
I helped regularly for over 3 years, and never charged him a dime
either (he had a wife and 3 kids to feed, so I used to look past his
boasting or even when he marginalized my help).  Now in all honesty,
I wouldn't even consider him a "good" Internet/web consultant, but
that's besides the point.

> because I run several high performance clusters, all of which
> share home and scratch disks through NFS. 
> We actually have a couple cluster file systems we use too - PVFS2 
> (http://www.pvfs.org), IBM's GPFS 
> (http://www-03.ibm.com/systems/clusters/software/gpfs.html) and
> I've been looking at Lustre (http://www.lustre.org) and when it
> finally exists, pNFS.
> Sadly, none of the open source cluster file systems we're using has
> very good failover.

Bingo!  ;)

There a various products and software out there.  They all offer
various features and limitations.  Many overlap, others have specific
foci, etc...  Heck, there are even the larger "volume management"
aspects that go with them too.  And breaking down all those
technologies and features -- many of which are either rather
eccentric, or take months of deployment to understand -- really make
you wonder if the complexity is worth it.

But web-centric Internet services?  They are largely stateless, or
the client maintains complete state utterly separate from the
protocol.  And they are "open read-only and close quickly" type
access.  They are point access.  They leave _nothing_ open.  They
care about _no_ state at the filesystem level.  So when I start to
point out such things -- Bam! -- "that's fine if you have the money."
 Huh?

> In fact, the most reliable remains to be the NFS volumes.

Bingo!

I remember when I first deployed IBM DFS.  It really tries.  It
requires "assumptions." And from there, if you don't know what you're
doing, it can make the assumption that goes against what you wanted
-- Bam!  ;)

Things like that really complicates it to the point that you wonder,
"okay, maybe I should step back and either pick:"
A.  Do I really need a network filesystem, or
B.  Isn't a single network filesystem server more reliable on its
own?

I understand the previous posters comments and why they do fit into
"A" -- do I really need a network filesystem.

But yes, Virginia, sometimes you do.  ;)

So that leaves me wondering if I'm really just at "B".

> Most of them are optimized for lots of storage, fast read
> or writes, for cheap.

There is a reason I love NetApp.  The whole combined volume
management with NVRAM and sync NFSv3 (plus SMB option), etc...  From
snapshots to hardware-based recovery -- "it just works."

Back in the kernel 2.2 days, Ext3 with a NVRAM board was an offering
from VA Research (aka VA Linux), that gave very good NFSv3 sync
operation and recovery time as well.  But it's still not as
integrated.

Not to tangent, but one of the biggest issues with XFS on Linux is
the lack of the complementary hardware.  That's a recurring theme. 
It's hard to get all those features working together, at to the
features you desire (let alone under differing assumptions), that "it
just works."

So many people in the web space can DNAT and redirect and thread and
do all sorts of things, because the protocols themselves are simple
and stateless.  It would be great if everything worked like that. 
But it doesn't.

Same deal in the X-terminal (aka thin client) / X-server (aka
Terminal Server) arena.  I can run a number of desktops and, guess
what?, they all local!  Problem solved, right?  Well, there comes a
point where the load is too much for a single, shared memory system. 
And that's when you're back to multiple nodes and ... again ...
network filesystems.  ;)

> It seems that the best way to keep the NFS volume available is to
> focus more on the underlying hardware than adapting the protocol.

Now you're talking my language.  ;)

> For example, using RAID, redundant power supplies,
> and effective monitoring.

And God forbid I start talking interconnect efficiency, in-line XOR
operations, TCP RX (receive) off-load with an HBA (host-based
adapter), etc...

That right there is where I get "beat up" by a lot of Linux web
consultants.  They talk about CPU utilization and how cheap,
software-based hardware works.  Well yeah, because you're
processor-bound, not I/O bound.

And then they'll say "I only see 10% CPU utilization" and the only
way I can show them that's because their entire interconnect is
utterly busy with I/O is if I have a PCI-X analysis (let alone PCIe
taps aren't really an option for PCIe ;).  I/O is everything to
network filesystems servers.

> Maybe I'm just missing something.

I've been known to be wrong.  And when I am, I fully admit it.  I
admit I am ignorant.  I admit I am not experienced in an area.  And I
learn.  And when it's where I'm not interested (e.g., the
"dime-a-dozen-consultant" Internet/web area), I keep my nose out.

Why it doesn't go the other way -- or worse yet -- why I'm with 2
other guys versus 97 who say "we're wrong" or "this works for me,"
well, I guess it's not well proliferated among others.

As I said, people pay me a lot of money -- and I'm not that smart, I
certainly barely got out of college with my EE w/computer option, and
I don't have experience in a lot of "common Linux" areas (like web). 
But know my limitations.  In fact, that's typically why I'm hired by
clients -- because I admit what I don't know (literally been told
that).

Anyhoo, end-of-rant.  I really need to shutup.

-- 
Bryan J. Smith   Professional, Technical Annoyance
b.j.smith at ieee.org    http://thebs413.blogspot.com
--------------------------------------------------
     Fission Power:  An Inconvenient Solution