DevOps Debugging Part 7: openssl

4 min readDec 2, 2022

This is a multi-part series where we will explore essential unix commands for debugging applications. These skills are critical when an outage occurs or something doesn’t work as expected. This is aimed at DevOps Engineers, SREs and linux sysadmins. Below is a quick navigation if you want to jump to the other parts.

In this part we are going to cover openssl. openssl is a super CLI tool that allows pretty much all functionality to do with SSL certificates and encryption. Here we will look at how we can use it to debug HTTPS websites and verify credentials. Keep in mind that we will not cover the whole usage of the command and what fancy things it can do but rather how to use the command to debug servers and applications.

Installation

To install network on redhat/centos/ubuntu/osx run:

# redhat/centos/amazon linux
$ yum install openssl# ubuntu
$ apt-get install openssl# OSX/Mac
$ brew install openssl# test for installation
$ openssl

If you get a command not found back then please reach out below in the comments section.

Usage

View the certificate information for a HTTPS endpoint.

$ openssl s_client -connect www.google.com:443
CONNECTED(00000006)
depth=2 C = US, O = Google Trust Services LLC, CN = GTS Root R1
verify return:1
depth=1 C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
verify return:1
depth=0 CN = www.google.com
verify return:1
---
Certificate chain
 0 s:CN = www.google.com
   i:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   a:PKEY: id-ecPublicKey, 256 (bit); sigalg: RSA-SHA256
   v:NotBefore: Sep 12 08:19:33 2022 GMT; NotAfter: Dec  5 08:19:32 2022 GMT
 1 s:C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
   i:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   a:PKEY: rsaEncryption, 2048 (bit); sigalg: RSA-SHA256
   v:NotBefore: Aug 13 00:00:42 2020 GMT; NotAfter: Sep 30 00:00:42 2027 GMT
 2 s:C = US, O = Google Trust Services LLC, CN = GTS Root R1
   i:C = BE, O = GlobalSign nv-sa, OU = Root CA, CN = GlobalSign Root CA
   a:PKEY: rsaEncryption, 4096 (bit); sigalg: RSA-SHA256
   v:NotBefore: Jun 19 00:00:42 2020 GMT; NotAfter: Jan 28 00:00:42 2028 GMT
---
...
...
...
subject=CN = www.google.com
issuer=C = US, O = Google Trust Services LLC, CN = GTS CA 1C3
---
No client certificate CA names sent
Peer signing digest: SHA256
Peer signature type: ECDSA
Server Temp Key: X25519, 253 bits
---
SSL handshake has read 4295 bytes and written 400 bytes
Verification: OK

We can connect to any HTTPS endpoint and view the full certificate chain that tells us the expiry date and start date of the certificates. If we scroll further down we can see the CN and that the verification is OK. This is very important as majority of SSL errors are due to the CN being wrong or the full chain is unable to be verified.

$ openssl x509 -noout -modulus -in mango.com.crt | openssl md5
$ openssl rsa -noout -modulus -in mango.com.key | openssl md5

In the above commands we can verify if the certificate matches a particular key. It can be surprising sometimes when the wrong key is used and we get an SSL error when reaching a website.

$ openssl x509 -in mango.com.crt -text

In the above command we can get details about a certificate file to verify that everything is what we expect. The main things to check here are the CN and expiry/start dates.

Debugging

During an outage if there is an SSL error, the best way to determine the root cause is to run the openssl s_connect command to see what certificate is being served. Most likely the full chain is missing or an incorrect certificate has been uploaded. SSH into the server where the certificate and key are and check if they have the right CN and other details. Once root cause is determined and new certificate is uploaded or fixed, check using the openssl s_connect command again to determine if verification is OK.

Alternatives

openssl is a powerful tool but there are alternatives out there that provide almost the same functionality.

$ openssl -a 
LibreSSL 2.6.5

In 2014 there was a fork of openssl called LibreSSL or libssl, this has the same commands as the original openssl but has slight differences. Different Unix operating systems may have LibreSSL as the openssl binary.

$ sudo certbot --nginx

certbot is an alternative to generating certificates using Let’s Encrypt. It is the modern command line tool to get HTTPS setup for free for your website.

$ curl -vvv https://www.google.com

curl has a verbose mode that gives details about the SSL handshake, it can be used to determine if the certificate is invalid or if any errors were encountered during the handshake phase. Usually it is a bit difficult to determine the root cause with just what the curl output but it is a start.

Conclusion

In the next part we are going to cover lsof for debugging applications. These parts will be released on a weekly basis, if you want to skip the queue please buy the book here:

https://www.amazon.com/dp/B0BJC4Y1N1

Please leave comments and share your outage debugging stories.