[wplug] Interesting shell problem (possibly OT?)

[Nathan Embery]

Well, for those interested in the YTF portion of this little gem, I
think I've ran across the actual issue:

http://www.dovecot.org/list/dovecot/2004-October/005114.html 

the TL;DR comes down to dovecot is renaming lots of files inside of cur/
and in the process the readdir() from 'find' is missing those files.
Hence, my file counts are way off, and any files containing the text I'm
looking for will be missed. I have no idea why the number of files
missed would be on the order of several hundred thousand, but this is,
at least, a reasonable explanation.

Now, hopefully, I can move on to some other all consuming issue...

As an aside, in writing something to try and recreate this, I discovered
a neat way to make random data/passwords:

tr -dc [:graph:] </dev/urandom | head -c 100

-Nate 

On Fri, 2011-01-07 at 21:12 -0500, Mark S. wrote:
> Love it!
> 
> Can think of more On Topic....
> 
> 
> On Fri, Jan 7, 2011 at 3:34 AM, Vance Kochenderfer <vkochend at nyx.net>
> wrote:
>         Absolutely not off-topic, IMO.
>         
>         Nathan Embery wrote:
>         > Apparently, find or it's subshell is exiting early for some
>         reason, or
>         > at least not returning all the files correctly. The
>         directory contents
>         > are virtually guaranteed to change while this is processing,
>         but I
>         > wouldn't think that would be the issue since this is fairly
>         common (esp
>         > things like `find . -type f -print | xargs -n 1 rm` etc)
>         
>         
>         Hmm, I'm not too experienced with using process substitution
>         to
>         feed a while loop as you're doing, but in theory it ought to
>         work.
>         Bash should handle it by creating a named pipe or a file in
>         /dev/fd, so you shouldn't run into a limit on the amount of
>         data
>         that can be transferred to the loop.
>         
>         I would suggest instrumenting the script to provide additional
>         details as it's going through the directories.  To start, I'd
>         add
>         this after the while loop (make it the line before "else"):
>          echo Removed $ACOUNT of $MCOUNT files checked
>         
>         This would give you a cumulative directory-by-directory count
>         you
>         could compare to the expected values.  If this is coming up
>         short,
>         try changing the last line of the while loop to:
>          done < <( find $MAILDIR/$i/cur -type f | pv -l )
>         or if you don't have pv installed,
>          done < <( find $MAILDIR/$i/cur -type f | tee >(wc -l >&2) )
>         to see how many files the 'find' command is picking up.  If it
>         is
>         getting the right number, but $MCOUNT is wrong, then you've
>         run
>         into some limitation of bash, I think.  (If it is a process
>         substitution limit, the second form may be less useful in
>         picking
>         it up as it uses process substitution to determine the count.)
>         
>         A general cleanup is probably also in order - it's good
>         practice
>         to quote variables so that filenames with spaces get properly
>         handled (e.g., use "$fname" instead of $fname).  The script
>         will
>         also fail for filenames that contain a newline, but this is
>         admittedly a pathological case.
>         
>         The fastest and most reliable way would probably be to avoid
>         the
>         shell looping construct entirely and use a 'grep -R -l -Z'
>         command
>         in /var/mail/vmail to feed a list of names into 'xargs -0 rm'.
>         But that wouldn't give you the file counts, which I gather are
>         important.
>         
>         Vance Kochenderfer        |  "Get me out of these ropes and
>         into a
>         vkochend at nyx.net          |   good belt of Scotch"    -Nick
>         Danger
>         
>         _______________________________________________
>         wplug mailing list
>         wplug at wplug.org
>         http://www.wplug.org/mailman/listinfo/wplug
>         
> 
> _______________________________________________
> wplug mailing list
> wplug at wplug.org
> http://www.wplug.org/mailman/listinfo/wplug