wget Usage: The Advance Guide

Well I'm back as promised. This time we are going to really geek it out with some advance features of wget,

You are able to create an exact copy or clone of a website with wget by using the -r and -l aruguments for example:

wget -r -l1 http://myhomesite.com

This command will create an exact mirror of the site you specify. Including the directory structure and everything, and it will save the information in a directory called /myhomesite.com. The -r command means do it recursively. The -l command tells wget how far do you want to dig into the site. If you don't specify a level it will clone the whole site which may take a long time depending on how complex the site is.

I mentioned previously that wget is useful for getting all the pictures or video files from a site well here is how to do it. It is a bit complicated but once you do it once or twice you will get it down.

wget -r -l 2 -np -A.gif http://www.myhomesite.com

The flags -np means no parent, which means reference to the parent directory is ignored. -A .gif means to accept only files with the .gif extension on it.

Suppose that you were in the middle of downloading some files and whoops the power goes out and your computer is dead what are you to do. YOU START OVER FROM FILE 1 OF 28, 344, 734 FILES. WRONG!!! wget has a solution, and this is why linux is so great. It is prepared for diesasters as long as you know the key.

wget -nc -r http://www.myhomesite.com

-nc stands for no clobber. In other words do not download files that already exists.

That is it for this post. Check back. I may post a super Ninja posting on get for those brave souls that can't get enough wget goodness.

See you next time,

The Linux Redneck

wget Usage: The Beginner's Guide

wget is a pretty nifty little program. wgs. et stands for web get, which does what the name says it will download an entire website if given the appropriate commands and argument. This is what this post and future posts will teach you about. Pretty soon you will be wgetting all sorts of things. Like my papa use to say you have to learn to crawl before you can spit. (I don't know what he meant by that but it is what he said, and sounds like good advice.) So on with the basics:

wget Basics
Say you wanted to get the homepage of a website you would use this syntax:

wget http://www.myhomesite.com

and what would be got from that command would be the index.html file. It would be saved into whatever directory you ran the wget command in. usually the /home directory in Linux.

Sometime you will run into of an error whenever wget times out. The default time is 25 tries to get a file, and for big files you may not be able to get the file such as a picture or video. So why not change the default? You can do that with the -t command:

wget -t 34 http://www.myhomesite.com/promo.mov

This downloads the promo.mov file, but since I know that it is a large file I want to give it a longer time period to try to download the file. So I extended the number of tries to 34 attempts.

Say you want to document the output of what you did instead of just having it appear on the screen. You can save the output into a file for review or documentation for later. Just add the -o command like this:

wget -o log http://www.myhomesite.com 


That is about it for basics of wget. Stay turned for some more hair tingling wget commands coming soon.


The Linux Redneck, signing out. :)