[wplug] Download a large number of files
Kuzman Ganchev
kuzman at sccs.swarthmore.edu
Fri Apr 9 01:40:16 EDT 2004
On Fri, Apr 09, 2004 at 12:00:14AM -0400, Gentgeen wrote:
> The files are located in a number of subdirectories under the URL
> http://memory.loc.gov/afc/afcomaha/audio/, each sub-directory holds a
> number of different formats for the same file. Each sub-directory and
> file name is actually a number. IE - the subdirectory 0300 holds
> files 0376.mp3, 0376.ram, 0376.wav, 0377.mp3, 0377.ram, --- you get
> the idea.
wget = your friend.
> Would wget be the right thing, or is there a better option for what I
> need? After reading the man page, it looks like I could either list
> all the MP3 URLs, or download the entire directory. Would I be able
> to use some regex to limit the download?
wget -e "robots=off" -nc -r -l 1 -A mp3,MP3 <http://dir>
-e "robots=off": ignore robots file (because you're not going to be
crawling).
-nc: this is really just for restarts
-r: get the index and the files underneath it
-l 1: don't recurse farther
-A mp3,MP3: get only mp3 and MP3 files (see also -R).
Kuzman
More information about the wplug
mailing list