Speed up file transfer

If you happen to be the customer of Malaysia’s leading ISP, sooner or later you will discover they wide disparity between the bandwidth that they advertised and sold to you and the real, day-to-day, “normal” speed that you can possibly ever get. Usually the variance is about 60%, where if you subscribe to the 1 Mbps ADSL package, you can normally get only around 200 kbps if you download files from any overseas site.

But I noticed that if you have multiple download in at the same time, you can actually reach the theoretical limit of your package (e.g. reaching around 900 kbps for 1 Mbps package). So somehow I suspect along the line this ISP does some traffic shaping probably, and they hired some fresh-graduate to configure the traffic shaper.

So how we can make the best use of this situation? Use a download accelerator of course! But before you continue, please be aware that most providers do not like the same user opening multiple connections to them, as it introduces unnecessary load on them. In other words, they might block or ban you. So don’t blame me later. Let’ move on.

Under Linux, there are numerous tools that can achieve this functionality. But two of my favorites are LFTP and Axel. Both LFTP and Axel are sophisticated ftp/http client.

1. To install LFTP and Axel under Ubuntu, make sure you have the extra repository enabled. To make the matter short, you need to modify your /etc/apt/sources.list file so it will look something like this:

deb http://archive.ubuntu.com/ubuntu/ gutsy main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ gutsy-updates main restricted universe multiverse
deb http://archive.ubuntu.com/ubuntu/ gutsy-backports main restricted universe multiverse
deb http://security.ubuntu.com/ubuntu gutsy-security main restricted universe multiverse

2. Install LFTP and Axel

sudo apt-get update
sudo apt-get install axel lftp

Axel

Axel can be used like a wget replacement. For example, to download latest stable kernel from kernel.org using 5 simultaneous connections, you can issue this command:

axel –n 5 http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2

The parameter “-n 5” is responsible to tell axel how many concurrent connections to open. To limit your download speed to certain value so that it will not consume all of your available bandwidth, the following command will do it:

axel -s 20000 -n 5 http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2

The directive “-s 20000” instruct Axel to download at a maximum of 20 Kbytes/sec.

LFTP

LFTP is my favorite downloader. I use it to download files via http, ftp, and scp. The interface is curses based, and it uses an almost similar command like an ftp client.

You can either open LFTP from terminal by typing lftp, and from lftp interface you issue the command, or you can straightaway enter the command you want lftp to execute as the program parameter.

For example, to repeat the above example, using these two methods:

$ lftp
lftp :~> pget -n 5 http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2
`/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2', got 267522 of 41863580 (0%) 76.8K/s eta:41m
..............................................................................................................................................

or

$ lftp -c "pget -n 5 http://kernel.org/pub/linux/kernel/v2.6/linux-2.6.18.tar.bz2"

Notice that unlike normal ftp client, LFTP use the command “pget” to initiate multiple download connections.
Apart from that, LFTP have other neat features such as directory mirroring. You can do both mirror and reverse mirror, where the former is mirroring the content of remote site to your system, and the later is vice versa (mirror local to remote system). For example, to mirror the whole Linux 2.6 kernel release directory, you can issue the following command:

$ lftp –c “mirror ftp://kernel.org/pub/linux/kernel/v2.6/”

Or, you can the multiple download connection for the mirror job itself, the command will do it.

$ lftp –c “mirror --parallel=5 ftp://kernel.org/pub/linux/kernel/v2.6/”

So, what are you waiting? Start saturating your pipe!