placeonthe.net

A site to share technological guides, news and scripts

weekly flashback no. 01

by

I am hoping to kick this post series off where I plan to post interesting articles, guides and other finds that I come across during the week. The number of links and content may vary on a week to week basis and as I move forward with this project idea things may change as we move along.

  • GIF production from asciinema – I have previously written about using asciinema in order to record the use of my scripts, however at times rather than linking to the video on the asciinema site, I’d rather bundle it in a to a gif. This is one of the main articles I used to achieve this.
  • Big-O notation explained by a self-taught programmer – I came across this post earlier today as I was reading about the Big-O notation which is a concept relating to algorithms and how they respond to different inputs. The article is well explained and is part of a series that provides an easy to understand explanation.
  • Stop using curl -u “user:password”! – This is a short but well written article that covers using the -n flag with curl and the .netrc file. This is a great way to prevent the credentials from showing up in the shell/terminal history files.
  • Shortened URL Security – One of the main points I’ve taken away from this technique is adding + to the end of a shortened URL in order to reveal the destination link.

I hope that you find the above links useful and stay tuned for the next installment!

troubleshooting with cURL

by

I use cURL every single day to troubleshoot a variety of issues both at work and while working on projects. It is an excellent tool to get an idea of what is happening when a call is made to a website. While this tool is vast in its features including connecting via SFTP, I will not be covering those steps within this article. Let’s jump right into it!

Headers

One great feature of cURL is the ability to easily check the headers being returned from a website. In practical terms, this helps me identify the response code my request is returning, it has in the past helped me identify whether my request is hitting the right server and whether the right headers are being returned.

Returning headers

To have cURL return the headers, pass the -I (capital i) or --head flags:

➜  curl -I https://placeonthe.net
 HTTP/2 200
 date: Sun, 29 Nov 2020 03:58:17 GMT
 content-type: text/html; charset=UTF-8
 set-cookie: __cfduid=db88f075e3e66663e55dde818cc9d8d171606622297; expires=Tue, 29-Dec-20 03:58:17 GMT; path=/; domain=.placeonthe.net; HttpOnly; SameSite=Lax
 vary: Accept-Encoding
 vary: Accept-Encoding
 set-cookie: wp_visit_time_test=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0
 link: https://placeonthe.net/wp-json/; rel="https://api.w.org/"
 x-powered-by: WP Engine
 x-cacheable: SHORT
 vary: Accept-Encoding,Cookie
 cache-control: max-age=600, must-revalidate
 x-cache: HIT: 2
 x-cache-group: normal
 x-xss-protection: 1; mode=block
 x-content-type-options: nosniff
 x-frame-options: SAMEORIGIN
 referrer-policy: strict-origin
 content-security-policy-report-only: default-src 'none'; form-action 'none'; frame-ancestors 'none'; report-uri https://placeonthenet.report-uri.com/r/d/csp/wizard
 x-donut: 🍩
 strict-transport-security: max-age=63072000
 cf-cache-status: DYNAMIC
 cf-request-id: 06b3bfd48600009afaeaa71000000001
 expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
 server: cloudflare
 cf-ray: 5f9968cdaa8e9afa-DFW

The above is the output we get from getting the headers for this site. A useful flag to include with --head is -L or --location, which tells cURL to follow 3XX redirects. This flag is useful when you want to ensure your request gets to the destination. Running curl -I https://www.placeonthe.net would stop at the first page, however the above actually returns a HTTP/2 301 header, as I have it redirecting to the apex domain, but without the --location flag we would not get to the next location (redirect). Adding the flag looks like this:

➜  curl -IL https://www.placeonthe.net
 HTTP/2 301
 date: Sun, 29 Nov 2020 04:04:05 GMT
 content-type: text/html
 content-length: 162
 set-cookie: __cfduid=d7395a72e5eca04f4e44c94b23567256a1606622645; expires=Tue, 29-Dec-20 04:04:05 GMT; path=/; domain=.www.placeonthe.net; HttpOnly; SameSite=Lax
 location: https://placeonthe.net/
 strict-transport-security: max-age=63072000
 cf-cache-status: DYNAMIC
 cf-request-id: 06b3c524d400000e42a3a1d000000001
 expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
 server: cloudflare
 cf-ray: 5f99714e1bcf0e42-DFW
 HTTP/2 200
 date: Sun, 29 Nov 2020 04:04:06 GMT
 content-type: text/html; charset=UTF-8
 set-cookie: __cfduid=d209a912320b2bfd76fbbd90ded628ce01606622645; expires=Tue, 29-Dec-20 04:04:05 GMT; path=/; domain=.placeonthe.net; HttpOnly; SameSite=Lax
 vary: Accept-Encoding
 vary: Accept-Encoding
 set-cookie: wp_visit_time_test=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0
 link: https://placeonthe.net/wp-json/; rel="https://api.w.org/"
 x-powered-by: WP Engine
 x-cacheable: SHORT
 vary: Accept-Encoding,Cookie
 cache-control: max-age=600, must-revalidate
 x-cache: HIT: 3
 x-cache-group: normal
 x-xss-protection: 1; mode=block
 x-content-type-options: nosniff
 x-frame-options: SAMEORIGIN
 referrer-policy: strict-origin
 content-security-policy-report-only: default-src 'none'; form-action 'none'; frame-ancestors 'none'; report-uri https://placeonthenet.report-uri.com/r/d/csp/wizard
 x-donut: 🍩
 strict-transport-security: max-age=63072000
 cf-cache-status: DYNAMIC
 cf-request-id: 06b3c5266c0000eccf3fa6a000000001
 expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
 server: cloudflare
 cf-ray: 5f997150a980eccf-DFW

Within this set, useful flag to keep in mind is -k (--insecure) which will allow you to proceed even if an SSL connection error is detected. The output on a certificate error looks like this:

curl: (60) SSL certificate problem: certificate has expired
More details here: https://curl.haxx.se/docs/sslcerts.html

curl failed to verify the legitimacy of the server and therefore could not
establish a secure connection to it. To learn more about this situation and
how to fix it, please visit the web page mentioned above.

As the two examples above show, there is quite a lot of data there that may or may not be necessary for the task you are troubleshooting. To actually cover how I use the above here are some examples:

  • Check for page redirects. I often check unexpected redirects while troubleshooting WordPress sites. Often plugins will set the X-Redirect-By: WordPress header as per the WordPress.org Developer code reference. Retrieving the headers can help identify the source of the loop
  • Confirm that the response code is as expected
  • Often hosts will set their own custom headers to identify their platform. This can help determine whether the request is hitting the right server if a recent DNS change has taken place
  • When setting custom headers such as X-Frame-Options, Content-Security-Policy or other similar headers, you can confirm that these are present once set on the server
  • Checking whether a page is cached or not. This may vary on the host you are checking with, but often you will see cache headers displayed

Cookies

Cookies can be an important factor to test, whether the cookie was set or how a site reacts to using a specific cookie. It can be extremely useful to test cookie based cache exclusions with cURL by providing the request with the cookie that meets the specific rule. The two main cookie options that I use are:

  • -b or --cookie which tells cURL to use a specific cookie. An example would be:
curl -b 'utm_market=abc123' https://placeonthe.net

The above command sends the cookie named utm_market to the site with the content abc123. An example as mentioned would be to set a cookie that is expected to exclude a page from cache and using the -IL flag from the above section to see what cache headers are returned.

  • -c or --cookie-jar allows you to retrieve cookies either straight to standard output or to a file:
Standard output:
curl -c - https://placeonthe.net -IL

Save to a file:
curl -c my_cookie_file.txt https://placeonthe.net -IL

The output when writing to the standard output would be as follows:

curl -c - https://placeonthe.net -IL
HTTP/2 200
 date: Sun, 29 Nov 2020 06:00:48 GMT
 content-type: text/html; charset=UTF-8
 set-cookie: __cfduid=d3d3323872f624290a71bdf44c5a2d8281606629648; expires=Tue, 29-Dec-20 06:00:48 GMT; path=/; domain=.placeonthe.net; HttpOnly; SameSite=Lax
 vary: Accept-Encoding
 vary: Accept-Encoding
 set-cookie: wp_visit_time_test=deleted; expires=Thu, 01-Jan-1970 00:00:01 GMT; Max-Age=0
 link: https://placeonthe.net/wp-json/; rel="https://api.w.org/"
 x-powered-by: WP Engine
 x-cacheable: SHORT
 vary: Accept-Encoding,Cookie
 cache-control: max-age=600, must-revalidate
 x-cache: HIT: 1
 x-cache-group: normal
 x-xss-protection: 1; mode=block
 x-content-type-options: nosniff
 x-frame-options: SAMEORIGIN
 referrer-policy: strict-origin
 content-security-policy-report-only: default-src 'none'; form-action 'none'; frame-ancestors 'none'; report-uri https://placeonthenet.report-uri.com/r/d/csp/wizard
 x-donut: 🍩
 strict-transport-security: max-age=63072000
 cf-cache-status: DYNAMIC
 cf-request-id: 06b430016c00000935d4ad3000000001
 expect-ct: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"
 server: cloudflare
 cf-ray: 5f9a1c48ab350935-SEA

 # Netscape HTTP Cookie File
 # https://curl.haxx.se/docs/http-cookies.html
 # This file was generated by libcurl! Edit at your own risk.
 
HttpOnly_.placeonthe.net       TRUE    /       FALSE   1609221648      __cfduid        d3d3323872f624290a71bdf44c5a2d8281606629648
placeonthe.net  FALSE   /       FALSE   1606629648      wp_visit_time_test      deleted

User Agents, Referrers and IPs

Setting User Agents can sometimes help me track down my specific request within logs. Setting the other elements can also help test server side rules such as rules blocking User Agents, referrers or specific IP addresses.

Setting a custom User Agent is as simple as adding the -A or --user-agent such as:

cURL
curl -A "My test User Agent" https://placeonthe.net

Log entry
123.123.123.123 placeonthe.net - [29/Nov/2020:06:06:21 +0000] "GET / HTTP/1.0" 200 28406 "-" "My test User Agent"

A referrer works much the same as the User Agent by setting it with the -e or --referer (please not that the spelling is with a single r):

cURL
curl -e "https://mysite.com" https://placeonthe.net

Log entry
123.123.123.123 placeonthe.net - [29/Nov/2020:06:10:05 +0000] "GET / HTTP/1.0" 200 28406 "https://mysite.com" "curl/7.58.0"

You can even set the IP address your request is coming from, thus spoofing the origin of the request. Of note, for this to work you will need to ensure that the server you are requesting from has real_ip_header X-True-client-IP; set. By sending the header X-True-Client-IP: [your IP], the server will interpret and use the value provided in the request. The following headers are ideal for this:

NGINX Server headers:

set_real_ip_from 0.0.0.0/0;
real_ip_header X-True-Client-IP;
real_ip_recursive on;

​cURL
curl -IL --header "X-True-Client-IP: 8.8.8.8" https://placeonthe.net

The above can be useful if you are trying to determine whether an IP address is blocked by a firewall or specific server side rules affecting specific IP addresses are working as expected.

Request method

Setting the request header is also a tool to have in your arsenal. By default, cURL will use the GET method. By providing the -X flag you can change the method to different HTTP Request methods. Sending a post request can significantly change how the server responds to your request, for example with WordPress sites, the xmlrpc.php file will require a POST method to respond correctly. Of note, when using the -d (--data) flag, cURL will automatically set the request method to POST:

Simple POST request:
curl -X POST https://placeonthe.net/contact

POST request using the data flag:
curl -d '<?xml version="1.0" encoding="utf-8"?><methodCall><methodName>system.listMethods</methodName><params></params></methodCall>' https://placeonthe.net/xmlrpc.php -A "cURL Tutorial"

The data example will send a POST request with the methodCall to list all the available/supported methods via xmlrpc.php on a WordPress site.

Authentication

In a nutshell, cURL will allow a number of methods for you to set user credentials for basic authentication on a site. This can be done by directly supplying the username:password combo to the URL or using the -u (--user) option:

Set within the request:
curl https://testUSER:password123@placeonthe.net

Username and password with option:
curl -u 'testUSER:password123' https://placeonthe.net

Username with a blank password will prompt you for the password:
curl -u 'testUSER:' https://placeonthe.net

The last example may be the safest especially on a shared environment as you will be prompted for the password after making the request.

Conclusion

The above is definitely not an exhaustive list, however provides some of the commands I use on a daily basis along with examples and situations I use them in. Further options and use cases can be found on the cURL man pages.

.