DevOps Debugging Part 8: lsof

3 min readDec 9, 2022

This is a multi-part series where we will explore essential unix commands for debugging applications. These skills are critical when an outage occurs or something doesn’t work as expected. This is aimed at DevOps Engineers, SREs and linux sysadmins. Below is a quick navigation if you want to jump to the other parts.

In this part we are going to cover lsof. lsof stands for “list open files”, it is a tool to list all open files that are opened by processes. This command allows us to debug if a particular application is reading the configuration files or logging the right information. It could also be used to detect any rogue processes. Keep in mind that we will not cover the whole usage of the command and what fancy things it can do but rather how to use the command to debug servers and applications.

Installation

To install network on redhat/centos/ubuntu/osx run:

# redhat/centos/amazon linux
$ yum install lsof# ubuntu
$ apt-get install lsof# OSX/Mac
$ brew install lsof# test for installation
$ lsof

If you get a command not found back then please reach out below in the comments section.

Usage

See if a process is logging to the log file.

$ lsof | grep mongo | grep log
...
mongod      449                          mongodb    4w      REG              202,1    720545     258265 /var/log/mongodb/mongod.log
...
mongod      449                          mongodb   11u     unix 0xffff9856513e1dc0       0t0      23459 /tmp/mongodb-27017.sock type=STREAM
mongod      449                          mongodb   12u     IPv4              23460       0t0        TCP localhost:27017 (LISTEN)

As we can see above, mongod process has the /var/log/mongodb/mongod.log file open. If we remove the last grep, we can see in detail what binary is being run, what libraries are being used and if the process is listening on any sockets or ports locally.

Debugging

During an outage after determining that network and disk is working fine, we have to check the process itself. Run lsof and see if the process is not opening some crazy amount of files. Then check if process is listening on the right ports, double check if all the files that are being opened are sane, it is very easy for a malicious hacking attempt to exploit some vulnerability in a process such as listening on random ports or opening other system files. For example a mongodb process has got no business looking at the /etc/passwd file.

Alternatives

lsof is a powerful tool but there are alternatives out there that provide almost the same functionality.

$ fuser -m -u /var/log

fuser can be used to find out what user has a particular path amongst other functionality.

$ pfiles /proc/*

pfiles is another alternative on Solaris Linux.

netstat is another alternative to determine routes and where a process is listening on.

Conclusion

In the next part we are going to cover netstat for debugging applications. These parts will be released on a weekly basis, if you want to skip the queue please buy the book here:

https://www.amazon.com/dp/B0BJC4Y1N1

Please leave comments and share your outage debugging stories.