Wget

Wget tips n tricks

Wget is a command line tool available in most *nix systems, that is used to download the content from internet using URLs.

Download all files recursively

 wget -m <link>  

Turn Off robots.txt

 wget -e robots=off <link>  

Download with user name and password

 wget --user=<username> --password=<password> <link>  

I prefer

 wget -m -e robots=off --user=<username> --password=<password> <link>  

Refer this link for more info.