DevOps Debugging Part 9: netstat

4 min readDec 16, 2022

This is a multi-part series where we will explore essential unix commands for debugging applications. These skills are critical when an outage occurs or something doesn’t work as expected. This is aimed at DevOps Engineers, SREs and linux sysadmins. Below is a quick navigation if you want to jump to the other parts.

In this part we are going to cover netstat. netstat is a tool that allows us to debug the local networking of our server. It can be used to determine if a process is listening on a particular port or socket and to see if we have routes to reach a destination. Keep in mind that we will not cover the whole usage of the command and what fancy things it can do but rather how to use the command to debug servers and applications.

Installation

To install network on redhat/centos/ubuntu/osx run:

# redhat/centos/amazon linux
$ yum install net-tools# ubuntu
$ apt-get install net-tools# OSX/Mac (usually already installed)# test for installation
$ netstat

If you get a command not found back then please reach out below in the comments section.

Usage

See what routes we have available.

$ netstat -ar
Kernel IP routing table
Destination     Gateway         Genmask         Flags   MSS Window  irtt Iface
default         ip-172-31-32-1. 0.0.0.0         UG        0 0          0 eth0
172.31.32.0     0.0.0.0         255.255.240.0   U         0 0          0 eth0
ip-172-31-32-1. 0.0.0.0         255.255.255.255 UH        0 0          0 eth0
192.168.236.0   0.0.0.0         255.255.255.0   U         0 0          0 tun0

With these routes we can see that we have 4 routes that should be reachable from this server via the eth0 and the tun0 interfaces. tun0 interfaces are usually VPN tunnels. This most likely means that this EC2 instance has a VPN tunnel running on it, which is true in my case.

$ sudo netstat -anlp | grep mongo
tcp        0      0 127.0.0.1:27017         0.0.0.0:*               LISTEN      449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:41344         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:59992         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:60024         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:60000         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:41360         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:41362         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:60008         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:35286         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:41348         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:60016         ESTABLISHED 449/mongod
tcp        0      0 127.0.0.1:27017         127.0.0.1:54532         ESTABLISHED 449/mongod
unix  2      [ ACC ]     STREAM     LISTENING     23459    449/mongod           /tmp/mongodb-27017.sock
unix  3      [ ]         STREAM     CONNECTED     21913    449/mongod

With the output above we can see that mongodb is listening on port 27017 and that it is connected to a bunch of local ports, probably an application. Near the bottom we can also see that mongodb is also listening on a unix socket.

Debugging

During an outage SSH into the server in question and check if it is listening on the correct ports. Check what is the state of the connections, if the state is ESTABLISHED, it means the process can reach where ever it wanted to, but if the connection is SYN or any other state, there might be a firewall rule or egress issue. Maybe a route is missing, maybe an iptables rule needs to be removed. Double check if the IPs that the process is trying to connect to is something you recognise, if it is a random IP on the internet then it could be malicious, someone could be copying over your database dump etc.

Alternatives

netstat is a powerful tool but there are alternatives out there that provide almost the same functionality.

$ lsof | grep mongo

lsof gives the same information on listening unix processes.

$ ss

ss gives the same information on listening and established TCP connections. In fact according to the netstat man page:

This program is mostly obsolete.  Replacement for netstat is ss.  Replacement for netstat -r is ip route.

Conclusion

In the next part we are going to cover iostat for debugging applications. These parts will be released on a weekly basis, if you want to skip the queue please buy the book here:

https://www.amazon.com/dp/B0BJC4Y1N1

Please leave comments and share your outage debugging stories.