Colocation to Virtualization

Linux: lsof Interaction With Networking

Most Linux and Unix people are already using lsof to determine what devices or files a process is binding to. Here, I will go over using lsof to pull networking relevant information. We will be using the ‘-i’ option with lsof throughout this article. Below is direct from the lsof man page.

“This option selects the listing of files any of whose Internet address matches the address specified in i. If no address is specified, this option selects the listing of all Internet and x.25 (HP-UX) network files. If -i4 or -i6 is specified with no following address, only files of the indicated IP version, IPv4 or IPv6, are displayed.”

List all processes utilizing any protocol communicating over port 80(http)
root@laptop22:~# lsof -i :80

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
apache2	5989	root	3u	IPv6	20339	TCP *:www	(LISTEN)
apache2	6059	www-data	3u	IPv6	20339	TCP *:www	(LISTEN)
apache2	6060	www-data	3u	IPv6	20339	TCP *:www	(LISTEN)
apache2	6061	www-data	3u	IPv6	20339	TCP *:www	(LISTEN)
apache2	6063	www-data	3u	IPv6	20339	TCP *:www	(LISTEN)
apache2	6064	www-data	3u	IPv6	20339	TCP *:www	(LISTEN)
TweetDeck	12074	usern221	9u	IPv4	228736	TCP 172.16.1.100:46847->cup-www.apple.com:www (CLOSE_WAIT)
TweetDeck	12074	usern221	10u	IPv4	228821	TCP 172.16.1.100:39635->ks357799.kimsufi.com:www (ESTABLISHED)
TweetDeck	12074	usern221	11u	IPv4	229694	TCP 172.16.1.100:58130->gs01.gridserver.com:www (CLOSE_WAIT)
TweetDeck	12074	usern221	14u	IPv4	464706	TCP 172.16.1.100:50025->s3.amazonaws.com:www (ESTABLISHED)
TweetDeck	12074	usern221	15u	IPv4	464707	TCP 172.16.1.100:50026->s3.amazonaws.com:www (ESTABLISHED)
firefox	14945	usern221	43u	IPv4	465318	TCP 172.16.1.100:50796->yw-in-f17.google.com:www (ESTABLISHED)

More from the man page
“An Internet address is specified in the form (Items in square brackets are optional.):

[46][protocol][@hostname|hostaddr][:service|port]

where:
46 specifies the IP version, IPv4 or IPv6
that applies to the following address.
`6’ may be be specified only if the UNIX
dialect supports IPv6. If neither ’4’ nor
`6’ is specified, the following address
applies to all IP versions.
protocol is a protocol name – TCP or UDP.
hostname is an Internet host name. Unless a
specific IP version is specified, open
network files associated with host names
of all versions will be selected.
hostaddr is a numeric Internet IPv4 address in
dot form; or an IPv6 numeric address in
colon form, enclosed in brackets, if the
UNIX dialect supports IPv6. When an IP
version is selected, only its numeric
addresses may be specified.
service is an /etc/services name – e.g., smtp -
or a list of them.
port is a port number, or a list of them.”

Now lets pass a few more parameters to lsof. Below I am looking to connections listening/binding to the ssh servers via port 22 using TCP
root@laptop22:~# lsof -i tcp:22

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
sshd	4709	root	3u	IPv6	15052	TCP *:ssh (LISTEN)
sshd	4709	root	4u	IPv4	15054	TCP *:ssh (LISTEN)
ssh	10268	root	3u	IPv4	557118	TCP localhost:46828->localhost:ssh (ESTABLISHED)
sshd	10270	root	3r	IPv4	557119	TCP localhost:ssh->localhost:46828 (ESTABLISHED)
sshd	10347	usern221	3u	IPv4	557119	TCP localhost:ssh->localhost:46828 (ESTABLISHED)

Query using the services name for ssh, which is mapped to port 22. The service name has to be in /etc/services for this to work on Linux
root@laptop22:~# lsof -i tcp:ssh

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
sshd	4709	root	3u	IPv6	15052	TCP *:ssh (LISTEN)
sshd	4709	root	4u	IPv4	15054	TCP *:ssh (LISTEN)
ssh	19093	root	3u	IPv4	604093	TCP 172.16.1.100:56717->172.16.1.122:ssh (ESTABLISHED)
sshd	19094	root	3r	IPv4	604094	TCP 172.16.1.122:ssh->172.16.1.100:56717 (ESTABLISHED)
sshd	1955	7 usern221	3u	IPv4	604094	TCP 172.16.1.122:ssh->172.16.1.100:56717 (ESTABLISHED)

This shows that user root has an ssh session open to a server on process ID 19093. Here, we can use lsof to get a little more detail:
root@laptop22:~# lsof +p 19093

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
ssh	19093	root	cwd	DIR	8,1	4096	466945 /root
ssh	19093	root	rtd	DIR	8,1	4096	2 /
ssh	19093	root	txt	REG	8,1	318812	410715 /usr/bin/ssh
ssh	19093	root	0u	CHR	136,0	2	/dev/pts/0
ssh	19093	root	1u	CHR	136,0	2	/dev/pts/0
ssh	19093	root	2u	CHR	136,0	2	/dev/pts/0
ssh	19093	root	3u	IPv4	604093		TCP 172.16.1.100:56717->172.16.1.22:ssh (ESTABLISHED)
ssh	19093	root	4u	CHR	136,0	2	/dev/pts/0

This confirms that the ssh initiator is root. Also, this tells us that the root user is on console using pts0. Now it can be killed using root.
root@laptop22:~# kill -9 19093

View from the console being killed
usern221@sshserv:~$ Killed
root@laptop22:~#

Since that side note is out of the way now, lets continue. If you do not know the actual port that a standard service runs on, just try the following.
root@laptop22:~# cat /etc/services | grep tftp
tftp 69/udp

root@laptop22:~# lsof -i udp:69

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
xinetd	7253	root	5u	IPv6	905077	UDP *:tftp

root@laptop22:~# lsof -i udp:tftp

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
xinetd	7253	root	5u	IPv6	905077	UDP *:tftp

Linux: lsof Interaction With Networking
[root@proxy ~]# cat /etc/services | grep squid
squid 3128/tcp # squid web proxy

The services file shows that we only need to check out port 3128 TCP.
[root@proxy ~]# lsof -i tcp:3128

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
squid	29223	squid	11u	IPv4	13410986	TCP *:squid (LISTEN)
squid	29223	squid	14u	IPv4	13630757	TCP proxy.server:squid->172.16.1.112:51974 (ESTABLISHED)
squid	29223	squid	16u	IPv4	13630691	TCP proxy.server:squid->172.16.1.110:38125 (ESTABLISHED)
squid	29223	squid	20u	IPv4	13630803	TCP proxy.server:squid->172.16.1.110:38128 (ESTABLISHED)
squid	29223	squid	21u	IPv4	13630804	TCP proxy.server:squid->172.16.1.110:38129 (ESTABLISHED)
squid	29223	squid	22u	IPv4	13630807	TCP proxy.server:squid->172.16.1.110:38130 (ESTABLISHED)
squid	29223	squid	25u	IPv4	13628848	TCP proxy.server:squid->172.16.1.110:38122 (ESTABLISHED)

[root@proxy ~]# lsof -i tcp:squid

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
squid	29223	squid	11u	IPv4	13410986	TCP *:squid (LISTEN)
squid	29223	squid	14u	IPv4	13630757	TCP proxy.server:squid->172.16.1.112:51974 (ESTABLISHED)
squid	29223	squid	16u	IPv4	13630691	TCP proxy.server:squid->172.16.1.110:38125 (ESTABLISHED)
squid	29223	squid	20u	IPv4	13630803	TCP proxy.server:squid->172.16.1.110:38128 (ESTABLISHED)
squid	29223	squid	21u	IPv4	13630804	TCP proxy.server:squid->172.16.1.110:38129 (ESTABLISHED)
squid	29223	squid	22u	IPv4	13630807	TCP proxy.server:squid->172.16.1.110:38130 (ESTABLISHED)
squid	29223	squid	25u	IPv4	13628848	TCP proxy.server:squid->172.16.1.110:38122 (ESTABLISHED)

I think that people might get the point here. You can either use the port name or port number, IP or hostname to help locate information. A few more examples await.

[root@proxy ~]# lsof -i udp@192.168.50.1:ntp

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
ntpd	28102	ntp	23u	IPv4	9902640	UDP 192.168.50.1:ntp

[root@proxy ~]# lsof -i tcp@client.x.com

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
squid	29223	squid	16u	IPv4	13633658	TCP proxy.server:squid->client.x.com:34361 (ESTABLISHED)
squid	29223	squid	17u	IPv4	13633660	TCP proxy.server:squid->client.x.com:34362 (ESTABLISHED)
squid	29223	squid	18u	IPv4	13633317	TCP proxy.server:squid->client.x.com:34354 (CLOSE_WAIT)
squid	29223	squid	25u	IPv4	13634425	TCP proxy.server:squid->client.x.com:34367 (ESTABLISHED)
squid	29223	squid	27u	IPv4	13632975	TCP proxy.server:squid->client.x.com:58761 (CLOSE_WAIT)

[root@proxy ~]# lsof -i tcp@172.16.1.110:34354

COMMAND	PID	USER	FD	TYPE	DEVICE SIZE	NODE NAME
squid	29223	squid	18u	IPv4	13633317	TCP proxy.server:squid->client.x.com:34354 (CLOSE_WAIT)

Last example will be IBM WebSphere. What I hate about IBM is that most of their services are generically named. In this case, it is ‘java’.

Find out what processes the WebSphere user is running
[root@websphere ~]# ps -u ibmuser
PID TTY TIME CMD
14394 ? 00:27:30 java

Now user lsof to check the java process and pass that data over to grep. This shows any ports that the java (WebSphere) process is listening on.
[root@websphere ~]# lsof +p 14394 | grep LISTEN

java	14394 ibmuser	26u	IPv4	149692	TCP *:58100 (LISTEN)
java	14394 ibmuser	28u	IPv4	149698	TCP *:58809 (LISTEN)
java	14394 ibmuser	29u	IPv4	149701	TCP *:58880 (LISTEN)
java	14394 ibmuser	37u	IPv4	149736	TCP 172.16.1.63:58276 (LISTEN)
java	14394 ibmuser	40u	IPv4	149741	TCP 172.16.1.63:58286 (LISTEN)
java	14394 ibmuser	68u	IPv4	150337	TCP *:58090 (LISTEN)
java	14394 ibmuser	69u	IPv4	150340	TCP *:58080 (LISTEN)
java	14394 ibmuser	70u	IPv4	150343	TCP *:58043 (LISTEN)
java	14394 ibmuser	71u	IPv4	150346	TCP *:58443 (LISTEN)
java	14394 ibmuser	74u	IPv4	150349	TCP *:58081 (LISTEN)

To see what is currently connected, just replace the LISTEN with ESTABLISHED in the grep statement.

Note: Examples can go on and on. These are just the ones I could obtain quickly. The cool thing is that you can pass port ranges and multiple ports to lsof. Example “lsof -i tcp:20-24,squid” this will show you any services listening on ports 20 through 24 and port 3128(squid). If you would like any other examples or have questions, please leave the in the comments.