DevOps Debugging Part 3: dig
This is a multi-part series where we will explore essential unix commands for debugging applications. These skills are critical when an outage occurs or something doesn’t work as expected. This is aimed at DevOps Engineers, SREs and linux sysadmins. Below is a quick navigation if you want to jump to the other parts.
In this part we are going to cover dig . This command is a great way of debugging DNS servers and testing locally deployed services and applications. DNS is a low effort but high impact service, a small error can cause major outages. Being able to thoroughly test and debug DNS servers will always help to identify where something went wrong. dig is part of the bind set of tools for DNS. Keep in mind that we will not cover the whole usage of the command and what fancy things it can do but rather how to use the command to debug servers and applications.
Installation
To install network on redhat/centos/ubuntu/osx run:
# redhat/centos/amazon linux
$ yum install bind-utils# ubuntu
$ apt-get install dns-utils# OSX/Mac (usually already installed)
$ brew install bind# test for installation
$ dig -h
If you get a command not found back then please reach out below in the comments section.
Usage
Resolve a DNS entry.
$ dig www.google.com
; <<>> DiG 9.10.6 <<>> www.google.com
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 21611
;; flags: qr rd ra; QUERY: 1, ANSWER: 1, AUTHORITY: 4, ADDITIONAL: 1;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 4096
;; QUESTION SECTION:
;www.google.com. IN A;; ANSWER SECTION:
www.google.com. 26 IN A 142.250.181.36;; AUTHORITY SECTION:
google.com. 87322 IN NS ns1.google.com.
google.com. 87322 IN NS ns2.google.com.
google.com. 87322 IN NS ns3.google.com.
google.com. 87322 IN NS ns4.google.com.;; Query time: 8 msec
;; SERVER: 192.168.1.1#53(192.168.1.1)
;; WHEN: Sat Oct 15 13:21:39 PKT 2022
;; MSG SIZE rcvd: 131
The above output can look at bit daunting at first but it is gives very detailed information. If we focus on the Question Section, we can see we requested an A entry by default and got an answer back from the Google nameservers. Near the bottom we see the SERVER and it is my local router which probably in turn is getting its DNS from Google DNS servers, 8.8.8.8. Lets see what else we can do:
# will return empty answer since there are no CNAME on google.com
$ dig cname www.google.com
...$ dig aaaa www.google.com
...
;; ANSWER SECTION:
www.google.com. 292 IN AAAA 2a00:1450:4019:80a::2004
...$ dig txt www.google.com
...$ dig ns www.google.com
...$ dig mx www.google.com
Let’s say we have bought up or provisioned a set of DNS servers and would like to test them out.
# query my own nameserver, I added 1.1.1.1 as www.google.com
$ dig @ns1.myowndns.com www.google.com
...
;; ANSWER SECTION:
www.google.com. 3600 IN A 1.1.1.1
...$ dig @8.8.8.8 mycodingacademia.com
; <<>> DiG 9.10.6 <<>> @8.8.8.8 mycodingacademia.com
; (1 server found)
;; global options: +cmd
;; Got answer:
;; ->>HEADER<<- opcode: QUERY, status: NOERROR, id: 53725
;; flags: qr rd ra; QUERY: 1, ANSWER: 4, AUTHORITY: 0, ADDITIONAL: 1;; OPT PSEUDOSECTION:
; EDNS: version: 0, flags:; udp: 512
;; QUESTION SECTION:
;mycodingacademia.com. IN A;; ANSWER SECTION:
mycodingacademia.com. 14400 IN A 198.185.159.145
mycodingacademia.com. 14400 IN A 198.49.23.144
mycodingacademia.com. 14400 IN A 198.49.23.145
mycodingacademia.com. 14400 IN A 198.185.159.144;; Query time: 145 msec
;; SERVER: 8.8.8.8#53(8.8.8.8)
;; WHEN: Sat Oct 15 13:31:32 PKT 2022
;; MSG SIZE rcvd: 113
Notice the server at the bottom is 8.8.8.8 and not my local router. This is a great way to test if my hosted DNS is behaving as expected. Perfect for testing out DNS delegation and private DNS servers.
# to restrict only IPV4 only
$ dig -4 www.google.com# to restrict only IPV6 only
$ dig -6 www.google.com
Debugging
During an outage dig is your best friend. If the website cannot be resolved then try to dig the endpoint, if the entry is blank then most likely google DNS has stopped caching your DNS entry. Try to use dig with the @ to target your DNS servers, it is very plausible that the DNS entry got deleted. if the entry exists but you still cannot resolve then maybe the wrong entry is being advertised. Try to check if the DNS is meant to be that IP address.
Alternatives
Dig is super powerful and I have rarely found alternatives that work as well as dig does at debugging DNS but there are out there. For example getent:
$ getent hosts www.google.com
172.217.194.139 google.com
172.217.194.102 google.com
...nslookup is a good alternative and much more brief than dig.
$ nslookup www.google.com
Server: 192.168.1.1
Address: 192.168.1.1#53Non-authoritative answer:
Name: www.google.com
Address: 142.250.181.36
host is very quick and simple.
$ host www.google.com
www.google.com has address 142.250.181.36
www.google.com has IPv6 address 2a00:1450:4019:80a::2004Conclusion
In the next part we are going to cover ps for debugging applications. These parts will be released on a weekly basis, if you want to skip the queue please buy the book here:
https://www.amazon.com/dp/B0BJC4Y1N1
Please leave comments and share your outage debugging stories.
