Wget – A Standard Command Line Download Utility For Linux

Wget (formerly known as Geturl) is a Free, open source, command line download tool which is retrieving files using HTTP, HTTPS and FTP, the most widely-used Internet protocols.

It is a non-interactive command line tool and Its name is derived from World Wide Web and get.

Wget handle download pretty much good compared with other tools, futures included working in background, recursive download, multiple file downloads, resume downloads, non-interactive downloads & large file downloads.

By default all the Linux Distribution included wget, so we can install easily from official repository, also we can install to windows and Mac OS too.

Wget has been designed for robustness over slow or unstable network connections, if a download fails due to a network problem, it will keep retrying until the whole file has been retrieved.

If the server supports regetting, it will instruct the server to continue the download from where it left off. Alternatively you can install other Download Utilities too.

Wget Features

  • Can resume aborted downloads, using REST and RANGE
  • Can use filename wild cards and recursively mirror directories
  • NLS-based message files for many different languages
  • Optionally converts absolute links in downloaded documents to relative, so that downloaded documents may link to each other locally
  • Runs on most UNIX-like operating systems as well as Microsoft Windows
  • Supports HTTP proxies
  • Supports HTTP cookies
  • Supports persistent HTTP connections
  • Unattended / background operation
  • Uses local file timestamps to determine whether documents need to be re-downloaded when mirroring

1) Install wget on Linux

We can easily install wget command line download tool to all the Linux Distribution such as Debian, Ubuntu, Mint, RHEL, CentOS, Fedora, suse, openSUSE, Arch Linux, Manjaro, Mageia, etc.. Just fire the below command to install.

For Fedora system, use DNF Command to install wget.

$ sudo dnf install wget

For Debian/Ubuntu systems, use APT-GET Command or APT Command to install wget.

$ sudo apt install wget

For Arch Linux based systems, use Pacman Command to install wget.

$ sudo pacman -S wget

For RHEL/CentOS systems, use YUM Command to install wget.

$ sudo yum install wget

For openSUSE Leap system, use Zypper Command to install wget.

$ sudo zypper install wget

2) Download Single File

The below command will download the file from given URL and stores in current directory, while downloading the file we can see the (date, time, download speed & download progress) of file.

# wget https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
--2016-03-22 07:27:20--  https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 2a01:4f8:192:50d6::3, 2a01:4f8:100:4349::3, 2a01:4f8:101:22a9::3, ...
Connecting to download.owncloud.org (download.owncloud.org)|2a01:4f8:192:50d6::3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22678208 (22M) [application/x-bzip2]
Saving to: âowncloud-9.0.0.tar.bz2â

100%[====================================================================================================>] 22,678,208  2.73MB/s   in 8.5s

2016-03-22 07:27:29 (2.55 MB/s) - âowncloud-9.0.0.tar.bz2â saved [22678208/22678208]

3) Save the file with different name

We can save the file with different name & format while initiate downloading, using -O (uppercase) option. Here we are going to save the filename with owncloud.zip.

# wget -O owncloud.zip https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
--2016-03-22 07:28:47--  https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 2a01:4f8:100:4349::3, 2a01:4f8:101:22a9::3, 2a01:4f8:130:806f::3, ...
Connecting to download.owncloud.org (download.owncloud.org)|2a01:4f8:100:4349::3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22678208 (22M) [application/x-bzip2]
Saving to: âowncloud.zipâ

100%[====================================================================================================>] 22,678,208  8.22MB/s   in 2.6s

2016-03-22 07:28:50 (8.22 MB/s) - âowncloud.zipâ saved [22678208/22678208]

4) Limit download speed

By default wget utilize full bandwidth for downloading file and we can’t use anything on server before download completion (Which will affect other service accessing bandwidth). So better use –limit-rate option to avoid further issue while downloading big size file.

# wget --limit-rate=500k https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
--2016-03-22 07:31:18--  https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 2a01:4f8:130:806f::3, 2a01:4f8:190:21ad::3, 2a01:4f8:192:50d6::3, ...
Connecting to download.owncloud.org (download.owncloud.org)|2a01:4f8:130:806f::3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22678208 (22M) [application/x-bzip2]
Saving to: âowncloud-9.0.0.tar.bz2â

100%[====================================================================================================>] 22,678,208   522KB/s   in 47s

2016-03-22 07:32:06 (471 KB/s) - âowncloud-9.0.0.tar.bz2â saved [22678208/22678208] 

5) Download Multiple Files

The below command will download more then on file from the location and stores in current directory, while downloading the file we can see the (date, time, download speed & download progress) of file.

# wget https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2 ftp://ftp.gnu.org/gnu/wget/wget-1.17.tar.gz
--2016-03-22 07:38:44--  https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 2a01:4f8:192:50d6::3, 2a01:4f8:100:4349::3, 2a01:4f8:101:22a9::3, ...
Connecting to download.owncloud.org (download.owncloud.org)|2a01:4f8:192:50d6::3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22678208 (22M) [application/x-bzip2]
Saving to: âowncloud-9.0.0.tar.bz2â

100%[====================================================================================================>] 22,678,208  2.50MB/s   in 8.9s

2016-03-22 07:38:53 (2.43 MB/s) - âowncloud-9.0.0.tar.bz2â saved [22678208/22678208]

--2016-03-22 07:38:53--  ftp://ftp.gnu.org/gnu/wget/wget-1.17.tar.gz
           => âwget-1.17.tar.gzâ
Resolving ftp.gnu.org (ftp.gnu.org)... 2001:4830:134:3::b, 208.118.235.20
Connecting to ftp.gnu.org (ftp.gnu.org)|2001:4830:134:3::b|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /gnu/wget ... done.
==> SIZE wget-1.17.tar.gz ... 3779445
==> EPSV ... done.    ==> RETR wget-1.17.tar.gz ... done.
Length: 3779445 (3.6M) (unauthoritative)

100%[====================================================================================================>] 3,779,445   4.28MB/s   in 0.8s

2016-03-22 07:38:54 (4.28 MB/s) - âwget-1.17.tar.gzâ saved [3779445]

FINISHED --2016-03-22 07:38:54--
Total wall clock time: 11s
Downloaded: 2 files, 25M in 9.7s (2.59 MB/s)

6) Resume Incomplete download

Make sure, whenever going to download big size of file (eg: ISO Images), i advise you to use -c option which will help us to resume the existing incomplete download from the state and complete as usual when we are facing any network connectivity issue or system problems. Otherwise when you are download again, it will initiate the fresh download and store to different file name (append .1 to the filename automatically).

# wget -c https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
--2016-03-22 07:40:23--  https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 2a01:4f8:101:22a9::3, 2a01:4f8:130:806f::3, 2a01:4f8:190:21ad::3, ...
Connecting to download.owncloud.org (download.owncloud.org)|2a01:4f8:101:22a9::3|:443... connected.
HTTP request sent, awaiting response... 206 Partial Content
Length: 22678208 (22M), 18942656 (18M) remaining [application/x-bzip2]
Saving to: âowncloud-9.0.0.tar.bz2â

100%[++++++++++++++++====================================================================================>] 22,678,208  6.75MB/s   in 2.7s

2016-03-22 07:40:26 (6.75 MB/s) - âowncloud-9.0.0.tar.bz2â saved [22678208/22678208]

7) Get the input from file

Alternatively wget can get the list of input URL’s from file and start downloading. We need to create a file and store each URL in separate line. Add -i option with wget command to perform this action.

# wget -i test-wget.txt
--2016-03-22 07:43:15--  https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
Resolving download.owncloud.org (download.owncloud.org)... 2a01:4f8:130:806f::3, 2a01:4f8:190:21ad::3, 2a01:4f8:192:50d6::3, ...
Connecting to download.owncloud.org (download.owncloud.org)|2a01:4f8:130:806f::3|:443... connected.
HTTP request sent, awaiting response... 200 OK
Length: 22678208 (22M) [application/x-bzip2]
Saving to: âowncloud-9.0.0.tar.bz2â

100%[====================================================================================================>] 22,678,208  3.02MB/s   in 8.0s

2016-03-22 07:43:23 (2.71 MB/s) - âowncloud-9.0.0.tar.bz2â saved [22678208/22678208]

--2016-03-22 07:43:23--  ftp://ftp.gnu.org/gnu/wget/wget-1.17.tar.gz
           => âwget-1.17.tar.gzâ
Resolving ftp.gnu.org (ftp.gnu.org)... 2001:4830:134:3::b, 208.118.235.20
Connecting to ftp.gnu.org (ftp.gnu.org)|2001:4830:134:3::b|:21... connected.
Logging in as anonymous ... Logged in!
==> SYST ... done.    ==> PWD ... done.
==> TYPE I ... done.  ==> CWD (1) /gnu/wget ... done.
==> SIZE wget-1.17.tar.gz ... 3779445
==> EPSV ... done.    ==> RETR wget-1.17.tar.gz ... done.
Length: 3779445 (3.6M) (unauthoritative)

100%[====================================================================================================>] 3,779,445   5.31MB/s   in 0.7s

2016-03-22 07:43:24 (5.31 MB/s) - âwget-1.17.tar.gzâ saved [3779445]

FINISHED --2016-03-22 07:43:24--
Total wall clock time: 9.6s
Downloaded: 2 files, 25M in 8.7s (2.91 MB/s)

8) Run the Download in Background

For big size of file download, we can initiate in Background to avoid further work interrupt using -b option. It will initiate the download and

# wget -b https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2
Continuing in background, pid 8114.
Output will be written to âwget-logâ.

# tail -f wget-log
 21800K .......... .......... .......... .......... .......... 98% 16.3M 0s
 21850K .......... .......... .......... .......... .......... 98% 15.5M 0s
 21900K .......... .......... .......... .......... .......... 99% 17.7M 0s
 21950K .......... .......... .......... .......... .......... 99% 15.8M 0s
 22000K .......... .......... .......... .......... .......... 99% 11.6M 0s
 22050K .......... .......... .......... .......... .......... 99% 2.44M 0s
 22100K .......... .......... .......... .......... ......    100% 44.5M=3.4s

2016-03-22 07:44:40 (6.40 MB/s) - âowncloud-9.0.0.tar.bz2â saved [22678208/22678208]

9) Download Group of Files

We can download group of files(Like, .pdf, .php, etc). The below command will download the group of files based on the extension such as .pdf, .php, .jpeg, .html, etc.. Add -r -A option with wget command to perform the download.

# wget -r -A png http://www.magesh.co.in
--2016-03-22 07:54:47--  http://www.magesh.co.in/
Resolving www.magesh.co.in (www.magesh.co.in)... 103.3.231.248
Connecting to www.magesh.co.in (www.magesh.co.in)|103.3.231.248|:80... connected.
HTTP request sent, awaiting response... 200 OK
Length: 54171 (53K) [text/html]
Saving to: âwww.magesh.co.in/index.htmlâ

100%[====================================================================================================>] 54,171      60.7KB/s   in 0.9s

2016-03-22 07:54:49 (60.7 KB/s) - âwww.magesh.co.in/index.htmlâ saved [54171/54171]

Loading robots.txt; please ignore errors.
--2016-03-22 07:54:49--  http://www.magesh.co.in/robots.txt
Reusing existing connection to www.magesh.co.in:80.
HTTP request sent, awaiting response... 404 Not Found
2016-03-22 07:54:49 ERROR 404: Not Found.

Removing www.magesh.co.in/index.html since it should be rejected.

--2016-03-22 07:54:49--  http://www.magesh.co.in/images/log.png
Reusing existing connection to www.magesh.co.in:80.
HTTP request sent, awaiting response... 200 OK
Length: 4605 (4.5K) [image/png]
Saving to: âwww.magesh.co.in/images/log.pngâ

100%[====================================================================================================>] 4,605       --.-K/s   in 0.01s

2016-03-22 07:54:49 (375 KB/s) - âwww.magesh.co.in/images/log.pngâ saved [4605/4605]

FINISHED --2016-03-22 07:54:49--
Total wall clock time: 2.3s
Downloaded: 2 files, 57K in 0.9s (65.0 KB/s)

10) Download a file from password protected site

Alternatively we can download a file from password protected site. The below command will download the file from password protected site.

# wget --http-user=xxx --http-password=xxx https://download.owncloud.org/community/owncloud-9.0.0.tar.bz2

# wget --ftp-user=xxx --ftp-password=xxx ftp://ftp.gnu.org/gnu/wget/wget-1.17.tar.gz

11) Read more about Wget

If you want to know more option which is available for wget, you can grep the details on your terminal itself by firing below commands..

# man wget 
or
# wget --help

Enjoy…)

Magesh Maruthamuthu

Love to play with all Linux distribution

You may also like...

%d bloggers like this: