Linux系列之学会使用CURL命令进行URL测试

curl命令是一个利用URL规则在shell终端命令行下工作的文件传输工具；curl命令作为一款强力工具，curl支持包括HTTP、HTTPS、ftp等众多协议，还支持POST、cookies、认证、从指定偏移处下载部分文件、用户代理字符串、限速、文件大小、进度条等特征；做网页处理流程和数据检索自动化。

语法：语法格式：curl [参数] [网址]

一. 查看帮助

1.1 查看命令参数

curl的参数很多，可以用命令curl --help查看：

[root@localhost ~]# curl --help
Usage: curl [options...] <url>
Options: (H) means HTTP/HTTPS only, (F) means FTP only --anyauth Pick "any" authentication method (H)
 -a, --append Append to target file when uploading (F/SFTP) --basic Use HTTP Basic Authentication (H) --cacert FILE   CA certificate to verify peer against (SSL) --capath DIR CA directory to verify peer against (SSL)
 -E, --cert CERT[:PASSWD] Client certificate file and password (SSL) --cert-type TYPE Certificate file type (DER/PEM/ENG) (SSL) --ciphers LIST  SSL ciphers to use (SSL) --compressed Request compressed response (using deflate or gzip)
 -K, --config FILE   Specify which config file to read --connect-timeout SECONDS  Maximum time allowed for connection
 -C, --continue-at OFFSET  Resumed transfer offset
 -b, --cookie STRING/FILE  String or file to read cookies from (H)
 -c, --cookie-jar FILE  Write cookies to this file after operation (H) --create-dirs   Create necessary local directory hierarchy --crlf Convert LF to CRLF in upload --crlfile FILE  Get a CRL list in PEM format from the given file
 -d, --data DATA HTTP POST data (H) --data-ascii DATA  HTTP POST ASCII data (H) --data-binary DATA  HTTP POST binary data (H) --data-urlencode DATA  HTTP POST data url encoded (H) --delegation STRING GSS-API delegation permission --digest Use HTTP Digest Authentication (H) --disable-eprt  Inhibit using EPRT or LPRT (F) --disable-epsv  Inhibit using EPSV (F)
 -D, --dump-header FILE  Write the headers to this file --egd-file FILE  EGD socket path for random data (SSL) --engine ENGINGE  Crypto engine (SSL). "--engine list" for list
 -f, --fail Fail silently (no output at all) on HTTP errors (H)
 -F, --form CONTENT  Specify HTTP multipart POST data (H) --form-string STRING  Specify HTTP multipart POST data (H) --ftp-account DATA  Account data string (F) --ftp-alternative-to-user COMMAND  String to replace "USER [name]" (F) --ftp-create-dirs  Create the remote dirs if not present (F) --ftp-method [MULTICWD/NOCWD/SINGLECWD] Control CWD usage (F) --ftp-pasv Use PASV/EPSV instead of PORT (F)
 -P, --ftp-port ADR  Use PORT with given address instead of PASV (F) --ftp-skip-pasv-ip Skip the IP address for PASV (F) --ftp-pret Send PRET before PASV (for drftpd) (F) --ftp-ssl-ccc   Send CCC after authenticating (F) --ftp-ssl-ccc-mode ACTIVE/PASSIVE  Set CCC mode (F) --ftp-ssl-control Require SSL/TLS for ftp login, clear for transfer (F)
 -G, --get Send the -d data with a HTTP GET (H)
 -g, --globoff Disable URL sequences and ranges using {} and []
 -H, --header LINE   Custom header to pass to server (H)
 -I, --head Show document info only
 -h, --help This help text --hostpubmd5 MD5  Hex encoded MD5 string of the host public key. (SSH)
 -0, --http1.0 Use HTTP 1.0 (H) --ignore-content-length  Ignore the HTTP Content-Length header
 -i, --include Include protocol headers in the output (H/F)
 -k, --insecure Allow connections to SSL sites without certs (H) --interface INTERFACE  Specify network interface/address to use
 -4, --ipv4 Resolve name to IPv4 address
 -6, --ipv6 Resolve name to IPv6 address
 -j, --junk-session-cookies Ignore session cookies read from file (H) --keepalive-time SECONDS  Interval between keepalive probes --key KEY Private key file name (SSL/SSH) --key-type TYPE Private key file type (DER/PEM/ENG) (SSL) --krb LEVEL Enable Kerberos with specified security level (F) --libcurl FILE  Dump libcurl equivalent code of this command line --limit-rate RATE  Limit transfer speed to this rate
 -l, --list-only List only names of an FTP directory (F) --local-port RANGE  Force use of these local port numbers
 -L, --location Follow redirects (H) --location-trusted like --location and send auth to other hosts (H)
 -M, --manual Display the full manual --mail-from FROM  Mail from this address --mail-rcpt TO  Mail to this receiver(s) --mail-auth AUTH  Originator address of the original email --max-filesize BYTES  Maximum file size to download (H/F) --max-redirs NUM  Maximum number of redirects allowed (H)
 -m, --max-time SECONDS  Maximum time allowed for the transfer --metalink Process given URLs as metalink XML file --negotiate Use HTTP Negotiate Authentication (H)
 -n, --netrc Must read .netrc for user name and password --netrc-optional Use either .netrc or URL; overrides -n --netrc-file FILE  Set up the netrc filename to use
 -N, --no-buffer Disable buffering of the output stream --no-keepalive  Disable keepalive use on the connection --no-sessionid  Disable SSL session-ID reusing (SSL) --noproxy List of hosts which do not use proxy --ntlm Use HTTP NTLM authentication (H)
 -o, --output FILE   Write output to <file> instead of stdout --pass PASS Pass phrase for the private key (SSL/SSH) --post301 Do not switch to GET after following a 301 redirect (H) --post302 Do not switch to GET after following a 302 redirect (H) --post303 Do not switch to GET after following a 303 redirect (H)
 -#, --progress-bar  Display transfer progress as a progress bar --proto PROTOCOLS  Enable/disable specified protocols --proto-redir PROTOCOLS  Enable/disable specified protocols on redirect
 -x, --proxy [PROTOCOL://]HOST[:PORT] Use proxy on given port --proxy-anyauth Pick "any" proxy authentication method (H) --proxy-basic   Use Basic authentication on the proxy (H) --proxy-digest  Use Digest authentication on the proxy (H) --proxy-negotiate Use Negotiate authentication on the proxy (H) --proxy-ntlm Use NTLM authentication on the proxy (H)
 -U, --proxy-user USER[:PASSWORD]  Proxy user and password --proxy1.0 HOST[:PORT]  Use HTTP/1.0 proxy on given port
 -p, --proxytunnel   Operate through a HTTP proxy tunnel (using CONNECT) --pubkey KEY Public key file name (SSH)
 -Q, --quote CMD Send command(s) to server before transfer (F/SFTP) --random-file FILE  File for reading random data from (SSL)
 -r, --range RANGE   Retrieve only the bytes within a range --raw Do HTTP "raw", without any transfer decoding (H)
 -e, --referer Referer URL (H)
 -J, --remote-header-name Use the header-provided filename (H)
 -O, --remote-name   Write output to a file named as the remote file --remote-name-all Use the remote file name for all URLs
 -R, --remote-time   Set the remote file's time on the local output
 -X, --request COMMAND  Specify request command to use --resolve HOST:PORT:ADDRESS  Force resolve of HOST:PORT to ADDRESS --retry NUM   Retry request NUM times if transient problems occur --retry-delay SECONDS When retrying, wait this many seconds between each --retry-max-time SECONDS  Retry only within this period
 -S, --show-error Show error. With -s, make curl show errors when they occur
 -s, --silent Silent mode. Don't output anything --socks4 HOST[:PORT]  SOCKS4 proxy on given host + port --socks4a HOST[:PORT]  SOCKS4a proxy on given host + port --socks5 HOST[:PORT]  SOCKS5 proxy on given host + port --socks5-hostname HOST[:PORT] SOCKS5 proxy, pass host name to proxy --socks5-gssapi-service NAME  SOCKS5 proxy service name for gssapi --socks5-gssapi-nec  Compatibility with NEC SOCKS5 server
 -Y, --speed-limit RATE  Stop transfers below speed-limit for 'speed-time' secs
 -y, --speed-time SECONDS  Time for trig speed-limit abort. Defaults to 30 --ssl Try SSL/TLS (FTP, IMAP, POP3, SMTP) --ssl-reqd Require SSL/TLS (FTP, IMAP, POP3, SMTP)
 -2, --sslv2 Use SSLv2 (SSL)
 -3, --sslv3 Use SSLv3 (SSL) --ssl-allow-beast Allow security flaw to improve interop (SSL) --stderr FILE   Where to redirect stderr. - means stdout --tcp-nodelay   Use the TCP_NODELAY option
 -t, --telnet-option OPT=VAL  Set telnet option --tftp-blksize VALUE  Set TFTP BLKSIZE option (must be >512)
 -z, --time-cond TIME  Transfer based on a time condition
 -1, --tlsv1 Use => TLSv1 (SSL) --tlsv1.0 Use TLSv1.0 (SSL) --tlsv1.1 Use TLSv1.1 (SSL) --tlsv1.2 Use TLSv1.2 (SSL) --trace FILE Write a debug trace to the given file --trace-ascii FILE  Like --trace but without the hex output --trace-time Add time stamps to trace/verbose output --tr-encoding   Request compressed transfer encoding (H)
 -T, --upload-file FILE  Transfer FILE to destination --url URL URL to work with
 -B, --use-ascii Use ASCII/text transfer
 -u, --user USER[:PASSWORD]  Server user and password --tlsuser USER  TLS username --tlspassword STRING TLS password --tlsauthtype STRING  TLS authentication type (default SRP)
 -A, --user-agent STRING  User-Agent to send to server (H)
 -v, --verbose Make the operation more talkative
 -V, --version Show version number and quit
 -w, --write-out FORMAT  What to output after completion --xattr Store metadata in extended file attributes
 -q If used as the first parameter disables .curlrc


  

 

  
1
  
2
  
3
  
4
  
5
  
6
  
7
  
8
  
9
  
10
  
11
  
12
  
13
  
14
  
15
  
16
  
17
  
18
  
19
  
20
  
21
  
22
  
23
  
24
  
25
  
26
  
27
  
28
  
29
  
30
  
31
  
32
  
33
  
34
  
35
  
36
  
37
  
38
  
39
  
40
  
41
  
42
  
43
  
44
  
45
  
46
  
47
  
48
  
49
  
50
  
51
  
52
  
53
  
54
  
55
  
56
  
57
  
58
  
59
  
60
  
61
  
62
  
63
  
64
  
65
  
66
  
67
  
68
  
69
  
70
  
71
  
72
  
73
  
74
  
75
  
76
  
77
  
78
  
79
  
80
  
81
  
82
  
83
  
84
  
85
  
86
  
87
  
88
  
89
  
90
  
91
  
92
  
93
  
94
  
95
  
96
  
97
  
98
  
99
  
100
  
101
  
102
  
103
  
104
  
105
  
106
  
107
  
108
  
109
  
110
  
111
  
112
  
113
  
114
  
115
  
116
  
117
  
118
  
119
  
120
  
121
  
122
  
123
  
124
  
125
  
126
  
127
  
128
  
129
  
130
  
131
  
132
  
133
  
134
  
135
  
136
  
137
  
138
  
139
  
140
  
141
  
142
  
143
  
144
  
145
  
146
  
147
  
148
  
149
  
150
  
151
  
152
  
153
  
154
  
155
  
156
  
157
  
158
  
159
  
160
 

国内翻译过来的curl文档：curl参数介绍

二. 查询使用手册

2.1 查看详细使用

也可以使用curl --manual查看基本用法例子：

三. 提取网页

3.1 查看网页源码

语法：curl uri

[root@localhost ~]# curl www.baidu.com
<!DOCTYPE html>
<!--STATUS OK--><html> <head><meta http-equiv=content-type content=text/html;charset=utf-8><meta http-equiv=X-UA-Compatible content=IE=Edge><meta content=always name=referrer><link rel=stylesheet type=text/css href=http://s1.bdstatic.com/r/www/cache/bdorz/baidu.min.css><title>百度一下，你就知道</title></head> <body link=#0000cc> <div id=wrapper> <div id=head> <div class=head_wrapper> <div class=s_form> <div class=s_form_wrapper> <div id=lg> <img hidefocus=true src=//www.baidu.com/img/bd_logo1.png width=270 height=129> </div> <form id=form name=f action=//www.baidu.com/s class=fm> <input type=hidden name=bdorz_come value=1> <input type=hidden name=ie value=utf-8> <input type=hidden name=f value=8> <input type=hidden name=rsv_bp value=1> <input type=hidden name=rsv_idx value=1> <input type=hidden name=tn value=baidu><span class="bg s_ipt_wr"><input id=kw name=wd class=s_ipt value maxlength=255 autocomplete=off autofocus></span><span class="bg s_btn_wr"><input type=submit id=su value=百度一下 class="bg s_btn"></span> </form> </div> </div> <div id=u1> <a href=http://news.baidu.com name=tj_trnews class=mnav>新闻</a> <a href=http://www.hao123.com name=tj_trhao123 class=mnav>hao123</a> <a href=http://map.baidu.com name=tj_trmap class=mnav>地图</a> <a href=http://v.baidu.com name=tj_trvideo class=mnav>视频</a> <a href=http://tieba.baidu.com name=tj_trtieba class=mnav>贴吧</a> <noscript> <a href=http://www.baidu.com/bdorz/login.gif?login&amp;tpl=mn&amp;u=http%3A%2F%2Fwww.baidu.com%2f%3fbdorz_come%3d1 name=tj_login class=lb>登录</a> </noscript> <script>document.write('<a href="http://www.baidu.com/bdorz/login.gif?login&tpl=mn&u='+ encodeURIComponent(window.location.href+ (window.location.search === "" ? "?" : "&")+ "bdorz_come=1")+ '" name="tj_login" class="lb">登录</a>');</script> <a href=//www.baidu.com/more/ name=tj_briicon class=bri style="display: block;">更多产品</a> </div> </div> </div> <div id=ftCon> <div id=ftConw> <p id=lh> <a href=http://home.baidu.com>关于百度</a> <a href=http://ir.baidu.com>About Baidu</a> </p> <p id=cp>&copy;2017&nbsp;Baidu&nbsp;<a href=http://www.baidu.com/duty/>使用百度前必读</a>&nbsp; <a href=http://jianyi.baidu.com/ class=cp-feedback>意见反馈</a>&nbsp;京ICP证030173号&nbsp; <img src=//www.baidu.com/img/gs.gif> </p> </div> </div> </div> </body> </html>


  

 

  
1
  
2
  
3
  
4
 

3.2 显示头信息

curl -i https://www.baidu.com

  

 

  
1
 

打印信息：

[root@localhost ~]# curl -i https://www.baiud.com
curl: (7) Failed connect to www.baiud.com:443; Connection refused
[root@localhost ~]# curl -i https://www.baidu.com
HTTP/1.1 200 OK
Accept-Ranges: bytes
Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
Connection: keep-alive
Content-Length: 2443
Content-Type: text/html
Date: Mon, 13 Jul 2020 07:22:02 GMT
Etag: "588603eb-98b"
Last-Modified: Mon, 23 Jan 2017 13:23:55 GMT
Pragma: no-cache
Server: bfe/1.0.8.18
Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/
...	

  

 

  
1
  
2
  
3
  
4
  
5
  
6
  
7
  
8
  
9
  
10
  
11
  
12
  
13
  
14
  
15
  
16
 

3.3 显示通信过程

curl -v www.baidu.com

  

 

  
1
 

curl -v www.baidu.com
* About to connect() to www.baidu.com port 80 (#0)
*   Trying 182.61.200.7...
* Connected to www.baidu.com (182.61.200.7) port 80 (#0)
> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: www.baidu.com
> Accept: */*
> 
< HTTP/1.1 200 OK
< Accept-Ranges: bytes
< Cache-Control: private, no-cache, no-store, proxy-revalidate, no-transform
< Connection: keep-alive
< Content-Length: 2381
< Content-Type: text/html
< Date: Mon, 13 Jul 2020 07:24:46 GMT
< Etag: "588604c8-94d"
< Last-Modified: Mon, 23 Jan 2017 13:27:36 GMT
< Pragma: no-cache
< Server: bfe/1.0.8.18
< Set-Cookie: BDORZ=27315; max-age=86400; domain=.baidu.com; path=/
< 
<!DOCTYPE html>
...

  

 

  
1
  
2
  
3
  
4
  
5
  
6
  
7
  
8
  
9
  
10
  
11
  
12
  
13
  
14
  
15
  
16
  
17
  
18
  
19
  
20
  
21
  
22
  
23
  
24
 

3.4 查看议定书

[root@localhost ~]# curl --trace-ascii output.txt https://www.baidu.com


  

 

  
1
  
2
 

3.5 查看时间

curl --trace-ascii output.txt --trace-time https://www.baidu.com

  

 

  
1
 

四. 用curl进行认证

4.1 referer设置

referer参数描述了一个网页的来源，可以按F12调试模式，查看：

访问百度时候，加上referer为csdn：

[root@localhost ~]# curl --referer https://blog.csdn.net https://www.baidu.com

  

 

  
1
 

4.2 用户代理设置

用户代理常被用于兼容ie，有时，您会看到获取带有curl的页面不会返回与使用浏览器获取页面时看到的页面相同的页面，这时候可以设置用户代理：

curl www.baidu.com --user-agent "Mozilla/5.0"
curl www.baidu.com -A "Mozilla/5.0"

  

 

  
1
  
2
 

4.3 用户信息校验

比如登录百度，可以用-u参数，接着会提示输入密码

[root@localhost ~]# curl -u admin https://www.baidu.net/
Enter host password for user 'admin':

  

 

  
1
  
2
 

五. 文件操作

5.1 保存文件

语法：curl -o filename uri

[root@localhost ~]# curl -o baidu.html www.baidu.com
  % Total % Received % Xferd  Average Speed   Time Time Time  Current Dload  Upload   Total   Spent Left  Speed
100  2381  100  2381 0 0  22761 0 --:--:-- --:--:-- --:--:-- 22894


  

 

  
1
  
2
  
3
  
4
  
5
 

5.2 文件下载

语法：curl uri filename --silent|progress

• –silent：不显示进度条
• –progress：显示进度条

[root@localhost ~]# curl https://avatar.csdnimg.cn/3/5/E/1_meituantech_1564971608.jpg -omeituan.jpg --progress
######################################################################## 100.0%


  

 

  
1
  
2
  
3
 

5.3 上传文件

语法：$ curl --form upload=@localfilename --form press=OK [URL]

[root@localhost ~]# curl --form upload=@baidu.html --form press=OK url


  

 

  
1
  
2
 

六. Cookie设置

6.1 设置cookie

设置cookie，可以用命令--cookie

6.2 cookie -b -c

-c cookie-file可以保存服务器返回的cookie到文件，-b cookie-file可以使用这个文件作为cookie信息，进行后续的请求。

$ curl -c cookies http://example.com
$ curl -b cookies http://example.com

  

 

  
1
  
2
 

[root@localhost ~]# curl -c "name=cookiename" www.baidu.com


  

 

  
1
  
2
 

6.3 指定cookie文件

也可以用–cookie-jar指定cookie文件

[root@localhost ~]# curl www.baidu.com --cookie-jar cookie_file


  

 

  
1
  
2
 

指定可下载的最大文件大小，可以用命令–limit-rate

[root@localhost ~]# curl www.baidu.com --limit-rate 50k


  

 

  
1
  
2
 

七. curl RESTFul

7.1 RESTFul API

curl支持RESTFul API格式的，所以先补充一些RESTFul API知识：

• GET 请求：获取服务器中的对象

  • 相当于SQL的Select命令
  • GET /blogs：列出所有博客信息

• POST 请求：在服务器上新增对象

  • 相当于SQL的Update命令
  • POST /blogs/ID：新增指定的博客信息

• PUT 请求：在服务器上更新对象

  • 相当于SQL的Create命令
  • PUT /blogs/ID：更新一个博客信息

• DELETE 请求：删除服务器中的对象

  • 相当于SQL的Delete命令
  • DELETE /blogs/ID：删除指定的博客

• HEAD 请求：仅仅用于获取对象的基础信息

7.2 POST请求

ps：-H指定请求header，-d指定参数

• CURL POST：

[root@localhost ~]# curl -H "Content-Type:application/json" -X POST localhost:9200/company/employees/1 -d '{"id" :1,"name":"admin","password":"123"}'


  

 

  
1
  
2
 

7.3 GET请求

• CURL GET：

[root@localhost ~]# curl -X GET localhost:9200/company/employees/1?pretty=true

  

 

  
1
 

7.4 DELETE请求

• CURL DELETE：

[root@localhost ~]# curl -X DELETE localhost:9200/company/employees/1

  

 

  
1
 

7.5 PUT请求

• CURL PUT：

curl -H "Content-Type:application/json" -X PUT localhost:9200/company/employees/1 -d '{"id" :1,"name":"admin","password":"111"}'

  

 

  
1
 

curl默认是GET方式请求url的，要调POST等接口，必须加上-X参数指定

八. CURL参考资料

CURL 官网教程：https://curl.haxx.se/docs/httpscripting.html
阮一峰老师教程：http://www.ruanyifeng.com/blog/2011/09/curl.html

文章来源: smilenicky.blog.csdn.net，作者：smileNicky，版权归原作者所有，如需转载，请联系作者。

原文链接：smilenicky.blog.csdn.net/article/details/107317880