首页 > Linux系统 > system > Linux命令行多线程、断点续传下载工具axel和myget
2014
05-16

Linux命令行多线程、断点续传下载工具axel和myget

运维工作中常会在linux命令行下载外网文件或内网进行大文件传输,经常使用的文本下载工具wget、curl,今天给大家推荐支持Linux命令行多线程、断点续传下载工具axel和myget。

1、系统环境

# lsb_release -a
LSB Version:    :core-4.0-amd64:core-4.0-ia32:core-4.0-noarch:graphics-4.0-amd64:graphics-4.0-ia32:graphics-4.0-noarch:printing-4.0-amd64:printing-4.0-ia32:printing-4.0-noarch
Distributor ID: CentOS
Description:    CentOS release 5.8 (Final)
Release:        5.8
Codename:       Final

2、下载工具安装、使用方法介绍
2.1 wget
CentOS 默认已经安装,如需安装请运行

# yum install wget -y

wget版本信息
# wget -V
GNU Wget 1.11.4 Red Hat modified
此工具比较常用,使用方法、参数略
2.2 Axel 下载 安装
官方站点http://axel.alioth.debian.org/

源码包地址
http://alioth.debian.org/frs/?group_id=100070
最新版本axel-2.4源码包下载地址

http://alioth.debian.org/frs/download.php/3015/axel-2.4.tar.gz

rpm包地址
http://pkgs.repoforge.org/axel/
选择下载对应系统的rpm包
32位

wget http://file.xsdou.com/file/axel-2.4-1.el5.rf.i386.rpm

64位

wget http://file.xsdou.com/file/axel-2.4-1.el5.rf.x86_64.rpm
# wget http://pkgs.repoforge.org/axel/axel-2.4-1.el5.rf.x86_64.rpm

安装axel

# rpm -ivh axel-2.4-1.el5.rf.x86_64.rpm

axel版本

# axel -V
Axel version 2.4 (Linux)

axel命令使用方法:
axel [选项参数] url1 [url2] [url……]
axel 参数:
–max-speed=x #限速值最高速度
-s x
Specify maximum speed (bytes per second)
–num-connections=x
-n x #连接数
Specify maximum number of connections
–output=f #下载为本地文件
-o f
Specify local output file
–search[=x] #搜索镜像
-S [x]
Search for mirrors and download from x servers
–header=x
-H x #添加头文件字符串
Add header string
–user-agent=x #设置UA
-U x
Set user agent
–no-proxy #不使用代理服务器
-N
Just don’t use any proxy server –quiet
–quiet, -q
No output to stdout. #静默模式,不输出到标准输出
–verbose
-v
More status information #更多状态信息
–alternate
–help #帮助
-h
–version #版本
-V
2.3 myget 下载、安装
[root@cobbler-1014 ~]# wget http://myget.sourceforge.net/release/myget-0.1.2.tar.gz
[root@cobbler-1014 ~]# tar zxf myget-0.1.2.tar.gz
[root@cobbler-1014 ~]# cd myget-0.1.2
# ./configure && make && make install
mytget版本,注意myget命令为mytget
# mytget -v
myget 0.1.1
命令mytget用法
mytget [选项] [url]
参数
-b, –debug Show the debug message #看调试信息
-c, –count=num Set the retry count to [num], no limit when “0”, the default is “99” #设置重试次数,0为无限,默认是99次。
-d, –directory=dir Set the local direcotry to [dir], the default is “.” #指定下载本地目录,默认是当前目录
-f, –file=file Rename the file to [file] #重命名下载到本地的文件名
-h, –help A brief summary of all the options #简短的帮助摘要
-i, –interval=num Set the ftp retry interval to [num] seconds, the default is “5” #设置ftp重试间隔,单位s,默认5秒
-n, –number=num Use [num] connections instead of the default (4) #指定连接数,默认4
-r, –referer=URL Include `Referer: [URL]’ header in HTTP request. #包含请求头 Referer
-t, –timeout=num Set the connection timeout to [num] seconds, the default is “30” #设置连接超时时间,默认30秒
-v, –version Show the version of the myget and exit #查看版本信息
-x, –proxy=URL Set the proxy [URL] #设置代理
3、wget、axel、myget测试下载速度对比
3.1 下载速度对比
说明:同一台机器下载同一个数据源链接
3.1.1 wget下载
# wget http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
–2013-03-11 09:29:46– http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Connecting to 10.0.251.154:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 4353378304 (4.1G) [application/octet-stream]
Saving to: `CentOS-6.4-x86_64-bin-DVD1.iso’

0% [ ] 18,526,067 1.60M/s eta 46m 21s
3.2 axel下载
# axel http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Initializing download: http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
File size: 4353378304 bytes
Opening output file CentOS-6.4-x86_64-bin-DVD1.iso.0
Starting download

[ 0%] ………. ………. ………. ………. ………. [ 382.3KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 610.9KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 826.5KB/s]
[ 0%] ………. ………. ………. ………. ………. [1003.1KB/s]

[ 0%] ………. ………. ………. ………. ………. [2055.8KB/s]
[ 0%] ………. ………. ………. ………. ………. [2074.8KB/s]
[ 0%] ………. ………. ………. ………. ………. [2093.6KB/s]
[ 0%] ………. ………. ………. ………. ………. [2111.0KB/s]
[ 0%] ………. ………. ………. ………. …
Downloaded 2693.8 kilobytes in 1 second. (2113.05 KB/s)
# rm -f CentOS-6.4-x86_64-bin-DVD1.iso*
3.2.1 axel 使用10个线程下载
# axel -n 10 http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Initializing download: http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
File size: 4353378304 bytes
Opening output file CentOS-6.4-x86_64-bin-DVD1.iso
Starting download
[ 0%] ………. ………. ………. ………. ………. [ 440.9KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 707.9KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 935.1KB/s]
[ 0%] ………. ………. ………. ………. ………. [1138.8KB/s]
[ 0%] ………. ………. ………. ………. ………. [1310.5KB/s]

[ 0%] ………. ………. ………. ………. ………. [2180.3KB/s]
[ 0%] ………. ………. ………. ………. ………. [2196.9KB/s]
[ 0%] ………. ………. ………. ………. ………. [2189.7KB/s]
[ 0%] ………. ………. ………
Downloaded 4139.5 kilobytes in 1 second. (2176.65 KB/s)
3.3 myget下载
# mytget http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Begin to download: CentOS-6.4-x86_64-bin-DVD1.iso
Filesize: 4.1G
0% [ > > > ] [ 27M] [2.5M/s] [ETA:26:58]
3.3.1 myget指定10个线程下载
# mytget -n 10 http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Begin to download: CentOS-6.4-x86_64-bin-DVD1.iso
Filesize: 4.1G
0% [ > > > > > > > > > ] [ 39M] [2.6M/s] [ETA:26:27]
# ll CentOS-6.4-x86_64-bin-DVD1.iso.mg\!
-rw-r–r– 1 root root 4353378404 Mar 11 12:53 CentOS-6.4-x86_64-bin-DVD1.iso.mg!
下载速度对比结论:axel、myget支持多线程,且速度较快都在2M。

4、断点续传对比测试
4.1 axel
# axel -n 10 http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Initializing download: http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
File size: 4353378304 bytes
Opening output file CentOS-6.4-x86_64-bin-DVD1.iso
Starting download
[ 0%] ………. ………. ………. ………. ………. [ 491.8KB/s]
[ 0%] ………. ………. ………. ………. ………. [ 748.4KB/s]
[ 0%] ………. ………. ………. ………. ………. [1007.5KB/s]

[ 1%] ………. ……….. ………. ………. ………. [2509.3KB/s]
[ 1%] ………. ………. ………. ………. ………. [2509.5KB/s]
[ 1%] ………. ………. ………. ….
Downloaded 45.3 megabytes in 18 seconds. (2510.95 KB/s)
中止下载,进度已经到1%,下载了45.3mb

再次运行下载命令,继续从1%断点续传
# axel -n 10 http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Initializing download: http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
File size: 4353378304 bytes
Opening output file CentOS-6.4-x86_64-bin-DVD1.iso
State file found: 47549233 bytes downloaded, 4305829071 to go.
Starting download
,,,,,,,,,, ,,,,,,,,,, ,,,,,,,,,, ,,,,…… ………. [ 314.4KB/s]
[ 1%] ………. ………. ………. ………. ………. [ 603.0KB/s]
[ 1%] ………. ………. ………. ………. ………. [ 839.7KB/s]

[ 1%] ………. ………. ………. ………. ………. [1547.1KB/s]
[ 1%] ………. ………. ………. ………. ………. [1548.8KB/s]
# rm -f CentOS-6.4-x86_64-bin-DVD1.iso*
4.2 wget
# wget http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
–2013-03-11 09:51:46– http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Connecting to 10.0.251.154:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 4353378304 (4.1G) [application/octet-stream]
Saving to: `CentOS-6.4-x86_64-bin-DVD1.iso’
1% [ ] 47,760,647 1.49M/s eta 47m 42s

wget在进度1%中止已经下载47,760,647,再次运行命令,重新开始下载
# wget http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
–2013-03-11 09:53:24– http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Connecting to 10.0.251.154:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 4353378304 (4.1G) [application/octet-stream]
Saving to: `CentOS-6.4-x86_64-bin-DVD1.iso.1′
0% [ ] 1,032,445 1.21M/s
wget直接运行不支持断点续传
# rm -f CentOS-6.4-x86_64-bin-DVD1.iso*

wget重新测试使其支持断点续传
# wget http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
–2013-03-11 12:40:15– http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Connecting to 10.0.251.154:80… connected.
HTTP request sent, awaiting response… 200 OK
Length: 4353378304 (4.1G) [application/octet-stream]
Saving to: `CentOS-6.4-x86_64-bin-DVD1.iso’
1% [> ] 52,620,206 1.57M/s eta 44m 58s
在进度1%,已经下载52,620,206处停止
wget 加-c再次下载,支持断点续传
# wget -c http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
–2013-03-11 12:40:56– http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Connecting to 10.0.251.154:80… connected.
HTTP request sent, awaiting response… 206 Partial Content
Length: 4353378304 (4.1G), 4300661972 (4.0G) remaining [application/octet-stream]
Saving to: `CentOS-6.4-x86_64-bin-DVD1.iso’
1% [+ ] 59,663,080 1.59M/s eta 44m 14s

# rm -f CentOS-6.4-x86_64-bin-DVD1.iso.mg\!
4.3 myget
# mytget http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Begin to download: CentOS-6.4-x86_64-bin-DVD1.iso
Filesize: 4.1G
1% [ > > > ] [ 45M] [2.1M/s] [ETA:33:06]
下载进度到1% 文件大小45M
# ll CentOS-6.4-x86_64-bin-DVD1.iso.mg\!
-rw-r–r– 1 root root 4353378404 Mar 11 12:54 CentOS-6.4-x86_64-bin-DVD1.iso.mg!
再次执行命令,直接从进度1%开始下载
# mytget http://url/iso/Centos/x86_64/CentOS-6.4-x86_64-bin-DVD1.iso
Begin to download: CentOS-6.4-x86_64-bin-DVD1.iso
Filesize: 4.1G
1% [ > > > ] [ 51M] [2.9M/s] [ETA:23:57]
断点续传对比,三个工具都支持,但wget需要增加-c参数,axel、myget再次执行命令即可。
综上对比推荐大家工作中使用axel或myget提高效率,个人比较喜欢axel。
注意内网传输根据隧道带宽进行限速,别影响线上生产服务数据传输。

最后编辑:
作者:admin
捐 赠如果您觉得这篇文章有用处,请支持作者!鼓励作者写出更好更多的文章!

留下一个回复