[wplug] wget with frames

[Poyner, Brandon]

Curl itself has no mirroring capability, you have to use an external
program that uses curl to do the actual retrieval.  This seems to be a
popular script.

http://curl.haxx.se/programs/curlmirror.txt

You might also have lftp installed.  Lftp can do http requests and it
can also mirror (at least for FTP, not certain about mirroring http).

I also saw searching google that some people modified the source code to
their wget so that it doesn't consider ~ to be an unsafe character.  It
looked very straight forward.

Brandon Poyner
Network Engineer III
CCAC - College Office
412-237-3086
 
 

> -----Original Message-----
> From: wplug-bounces+bpoyner=ccac.edu at wplug.org 
> [mailto:wplug-bounces+bpoyner=ccac.edu at wplug.org] On Behalf 
> Of John Harrold
> Sent: Tuesday, September 20, 2005 3:50 PM
> To: General user list
> Subject: Re: [wplug] wget with frames
> 
> Sometime in September Poyner, Brandon assaulted the keyboard 
> and produced:
> 
> | What you have is a stubborn battle of the wills.  Wget 
> considers ~ to be
> | an unsafe character and converts it to the hex %7e.  The server at
> | www.pitt.edu doesn't merely take that and convert it 
> internally to ~ and
> | return the content (Apache normally does this), but rather 
> it issues an
> | HTTP redirect to the URL with ~.  So wget attempts to follow the
> | provided link and converts ~ back to %7e.
> | 
> | Do you have something like curl handy?  I would consider using that
> | instead.
> 
> In fact I do, however I'm not really shure how to make it mirror or
> recursively fetch pages. I've looked through the man page for 
> curl, but I
> don't see that option anywhere. Any advice?
> 
> -- 
> ---------------------------------------------------------- 
>                             | /"\                         
>  john harrold               | \ / ASCII ribbon campaign   
>  john.m.harrold _at_ gmail  |  X  against HTML mail       
>  the most useful idiot      | / \                         
> ----------------------------------------------------------
>  What difference does it make to the dead, the orphans, 
>  and the homeless, whether the mad destruction is brought 
>  under the name of totalitarianism or the holy name of 
>  liberty and democracy?
>  --Gandhi
> ----------------------------------------------------------
>   gpg --keyserver pgp.mit.edu --recv-key B23241CB
> ----------------------------------------------------------
>