I use cURL every single day to troubleshoot a variety of issues both at work and while working on projects. It is an excellent tool to get an idea of what is happening when a call is made to a website. While this tool is vast in its features including connecting via SFTP, I will not be covering those steps within this article. Let’s jump right into it!
Headers
One great feature of cURL is the ability to easily check the headers being returned from a website. In practical terms, this helps me identify the response code my request is returning, it has in the past helped me identify whether my request is hitting the right server and whether the right headers are being returned.
Returning headers
To have cURL return the headers, pass the -I (capital i) or --head flags:
➜ curl -I https://placeonthe.net
HTTP/2 200
date: Sun, 29 Nov 2020 03:58:17 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=db88f075e3e66663e55dde818cc9d8d171606622297; expires=Tue, 29-Dec-20 03:58:17 GMT; path=/; domain=.placeonthe.net; HttpOnly; SameSite=Lax
vary: Accept-Encoding
vary: Accept-Encoding
set-cookie: wp_visit_time_test=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0
link: https://placeonthe.net/wp-json/; rel="https://api.w.org/"
x-powered-by: WP Engine
x-cacheable: SHORT
vary: Accept-Encoding,Cookie
cache-control: max-age=600, must-revalidate
x-cache: HIT: 2
x-cache-group: normal
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin
content-security-policy-report-only: default-src 'none'; form-action 'none'; frame-ancestors 'none'; report-uri https://placeonthenet.report-uri.com/r/d/csp/wizard
x-donut: 🍩
strict-transport-security: max-age=63072000
cf-cache-status: DYNAMIC
cf-request-id: 06b3bfd48600009afaeaa71000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5f9968cdaa8e9afa-DFWThe above is the output we get from getting the headers for this site. A useful flag to include with --head is -L or --location, which tells cURL to follow 3XX redirects. This flag is useful when you want to ensure your request gets to the destination. Running curl -I https://www.placeonthe.net would stop at the first page, however the above actually returns a HTTP/2 301 header, as I have it redirecting to the apex domain, but without the --location flag we would not get to the next location (redirect). Adding the flag looks like this:
➜ curl -IL https://www.placeonthe.net
HTTP/2 301
date: Sun, 29 Nov 2020 04:04:05 GMT
content-type: text/html
content-length: 162
set-cookie: __cfduid=d7395a72e5eca04f4e44c94b23567256a1606622645; expires=Tue, 29-Dec-20 04:04:05 GMT; path=/; domain=.www.placeonthe.net; HttpOnly; SameSite=Lax
location: https://placeonthe.net/
strict-transport-security: max-age=63072000
cf-cache-status: DYNAMIC
cf-request-id: 06b3c524d400000e42a3a1d000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5f99714e1bcf0e42-DFW
HTTP/2 200
date: Sun, 29 Nov 2020 04:04:06 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d209a912320b2bfd76fbbd90ded628ce01606622645; expires=Tue, 29-Dec-20 04:04:05 GMT; path=/; domain=.placeonthe.net; HttpOnly; SameSite=Lax
vary: Accept-Encoding
vary: Accept-Encoding
set-cookie: wp_visit_time_test=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0
link: https://placeonthe.net/wp-json/; rel="https://api.w.org/"
x-powered-by: WP Engine
x-cacheable: SHORT
vary: Accept-Encoding,Cookie
cache-control: max-age=600, must-revalidate
x-cache: HIT: 3
x-cache-group: normal
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin
content-security-policy-report-only: default-src 'none'; form-action 'none'; frame-ancestors 'none'; report-uri https://placeonthenet.report-uri.com/r/d/csp/wizard
x-donut: 🍩
strict-transport-security: max-age=63072000
cf-cache-status: DYNAMIC
cf-request-id: 06b3c5266c0000eccf3fa6a000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5f997150a980eccf-DFWWithin this set, useful flag to keep in mind is -k (--insecure) which will allow you to proceed even if an SSL connection error is detected. The output on a certificate error looks like this:
curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html
curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.As the two examples above show, there is quite a lot of data there that may or may not be necessary for the task you are troubleshooting. To actually cover how I use the above here are some examples:
- Check for page redirects. I often check unexpected redirects while troubleshooting WordPress sites. Often plugins will set the
X-Redirect-By: WordPressheader as per the WordPress.org Developer code reference. Retrieving the headers can help identify the source of the loop - Confirm that the response code is as expected
- Often hosts will set their own custom headers to identify their platform. This can help determine whether the request is hitting the right server if a recent DNS change has taken place
- When setting custom headers such as
X-Frame-Options,Content-Security-Policyor other similar headers, you can confirm that these are present once set on the server - Checking whether a page is cached or not. This may vary on the host you are checking with, but often you will see cache headers displayed
Cookies
Cookies can be an important factor to test, whether the cookie was set or how a site reacts to using a specific cookie. It can be extremely useful to test cookie based cache exclusions with cURL by providing the request with the cookie that meets the specific rule. The two main cookie options that I use are:
-bor--cookiewhich tellscURLto use a specific cookie. An example would be:
curl -b 'utm_market=abc123' https://placeonthe.netThe above command sends the cookie named utm_market to the site with the content abc123. An example as mentioned would be to set a cookie that is expected to exclude a page from cache and using the -IL flag from the above section to see what cache headers are returned.
-cor--cookie-jarallows you to retrieve cookies either straight to standard output or to a file:
Standard output:
curl -c - https://placeonthe.net -IL
Save to a file:
curl -c my_cookie_file.txt https://placeonthe.net -ILThe output when writing to the standard output would be as follows:
curl -c - https://placeonthe.net -IL
HTTP/2 200
date: Sun, 29 Nov 2020 06:00:48 GMT
content-type: text/html; charset=UTF-8
set-cookie: __cfduid=d3d3323872f624290a71bdf44c5a2d8281606629648; expires=Tue, 29-Dec-20 06:00:48 GMT; path=/; domain=.placeonthe.net; HttpOnly; SameSite=Lax
vary: Accept-Encoding
vary: Accept-Encoding
set-cookie: wp_visit_time_test=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0
link: https://placeonthe.net/wp-json/; rel="https://api.w.org/"
x-powered-by: WP Engine
x-cacheable: SHORT
vary: Accept-Encoding,Cookie
cache-control: max-age=600, must-revalidate
x-cache: HIT: 1
x-cache-group: normal
x-xss-protection: 1; mode=block
x-content-type-options: nosniff
x-frame-options: SAMEORIGIN
referrer-policy: strict-origin
content-security-policy-report-only: default-src 'none'; form-action 'none'; frame-ancestors 'none'; report-uri https://placeonthenet.report-uri.com/r/d/csp/wizard
x-donut: 🍩
strict-transport-security: max-age=63072000
cf-cache-status: DYNAMIC
cf-request-id: 06b430016c00000935d4ad3000000001
expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
server: cloudflare
cf-ray: 5f9a1c48ab350935-SEA
# Netscape HTTP Cookie File
# https://curl.haxx.se/docs/http-cookies.html
# This file was generated by libcurl! Edit at your own risk.
HttpOnly_.placeonthe.net TRUE / FALSE 1609221648 __cfduid d3d3323872f624290a71bdf44c5a2d8281606629648
placeonthe.net FALSE / FALSE 1606629648 wp_visit_time_test deletedUser Agents, Referrers and IPs
Setting User Agents can sometimes help me track down my specific request within logs. Setting the other elements can also help test server side rules such as rules blocking User Agents, referrers or specific IP addresses.
Setting a custom User Agent is as simple as adding the -A or --user-agent such as:
cURL
curl -A "My test User Agent" https://placeonthe.net
Log entry
123.123.123.123 placeonthe.net - [29/Nov/2020:06:06:21 +0000] "GET / HTTP/1.0" 200 28406 "-" "My test User Agent"A referrer works much the same as the User Agent by setting it with the -e or --referer (please not that the spelling is with a single r):
cURL
curl -e "https://mysite.com" https://placeonthe.net
Log entry
123.123.123.123 placeonthe.net - [29/Nov/2020:06:10:05 +0000] "GET / HTTP/1.0" 200 28406 "https://mysite.com" "curl/7.58.0"You can even set the IP address your request is coming from, thus spoofing the origin of the request. Of note, for this to work you will need to ensure that the server you are requesting from has real_ip_header X-True-client-IP; set. By sending the header X-True-Client-IP: [your IP], the server will interpret and use the value provided in the request. The following headers are ideal for this:
NGINX Server headers:
set_real_ip_from 0.0.0.0/0;
real_ip_header X-True-Client-IP;
real_ip_recursive on;
cURL
curl -IL --header "X-True-Client-IP: 8.8.8.8" https://placeonthe.netThe above can be useful if you are trying to determine whether an IP address is blocked by a firewall or specific server side rules affecting specific IP addresses are working as expected.
Request method
Setting the request header is also a tool to have in your arsenal. By default, cURL will use the GET method. By providing the -X flag you can change the method to different HTTP Request methods. Sending a post request can significantly change how the server responds to your request, for example with WordPress sites, the xmlrpc.php file will require a POST method to respond correctly. Of note, when using the -d (--data) flag, cURL will automatically set the request method to POST:
Simple POST request:
curl -X POST https://placeonthe.net/contact
POST request using the data flag:
curl -d '<?xml version="1.0" encoding="utf-8"?><methodCall><methodName>system.listMethods</methodName><params></params></methodCall>' https://placeonthe.net/xmlrpc.php -A "cURL Tutorial"The data example will send a POST request with the methodCall to list all the available/supported methods via xmlrpc.php on a WordPress site.
Authentication
In a nutshell, cURL will allow a number of methods for you to set user credentials for basic authentication on a site. This can be done by directly supplying the username:password combo to the URL or using the -u (--user) option:
Set within the request:
curl https://testUSER:password123@placeonthe.net
Username and password with option:
curl -u 'testUSER:password123' https://placeonthe.net
Username with a blank password will prompt you for the password:
curl -u 'testUSER:' https://placeonthe.netThe last example may be the safest especially on a shared environment as you will be prompted for the password after making the request.
Conclusion
The above is definitely not an exhaustive list, however provides some of the commands I use on a daily basis along with examples and situations I use them in. Further options and use cases can be found on the cURL man pages.
.
