Colocation to Virtualization

When troubleshooting a *nix box, working knowledge of file system, network, and process utilities are a necessity. The main ones for me are mount, lsof, dd, ps, fsck, netstat, tcpdump, and fuser. All of these tools are very basic, but most admins seem to not know about or utilise fuser. All of fusers functionality can be accomplished by using any of the above commands together. In the below examples, the same end results can be archived by using kill and lsof together, but why not just use one tool?

To begin, we will be passing fuser the -m switch to specify both a device and file(system).

       -m     name specifies a file on a mounted file system or a block device
              that is mounted. All processes accessing files on that file sys-
              tem  are  listed.  If a directory file is specified, it is auto-
              matically changed to name/. to use any file system that might be
              mounted on that directory

In it’s basic form, fuser will provide only the process id(s) (PID) that are currently utilising the specified file/device

root@testbox:~# fuser -m /media/disk/
/dev/sdb1:            5535c  5589c

The above example showed that process 5355 is currently utilizing something on the /media/disk file system. Below is the same example, but specifying the actual device that is mounted on /media/disk.

root@testbox:~# fuser -m /dev/sdb1
/dev/sdb1:            5535c  5589c
From there, we can use 'ps' to reveal what's going on

root@testbox:~# ps -axc | grep 5589
 5589 pts/1    S+     0:00 vi
root@testbox:~# ps -axc | grep 5535
 5535 pts/1    Ss     0:00 bash

Fuser also gives us the ability to terminate the any processes that are utilizing the filesystem/device. The following commands utilises -k to initiate the kill, -m to set the target, -i to prompt the user for verification, and -TERM to send the term signal to the process(es).

root@testbox:~# fuser -kmi -TERM /dev/sdb1
/dev/sdb1:            5535c  5589c
Kill process 5535 ? (y/N) y
Kill process 5589 ? (y/N) y

This is what it looks like from there users end when the process is killed

user01@testbox:/media/disk$
vi File.txt
Vim: Caught deadly signal TERM
Vim: Finished.
Terminated

Here we will be adding the -u and -v options to obtain more details from fuser.
Direct from the help output:

    -u        display user IDs
    -v        verbose output

Doing so displays the username, process id (PID), type of access, and the running command. In this case, we are looking at the actual device.

root@testbox:~# fuser -muv /dev/sdb1
                     USER        PID ACCESS COMMAND
/dev/sdb1:           user01   6457 F.... (user01)vi

When trying to unmount a device while a process is currently utilizing a file, the unmount will fail.

Below will kill all the processes running on the sdb1 device and allow it to be unmounted.

root@testbox:~# fuser -km -TERM /dev/sdb1
/dev/sdb1:            6403

Passing the -TERM parameter causes fuser to perform the same as doing “kill -9 6403″. Below is the list of signals that fuser can send when -k is used

root@testbox:~# fuser -l
HUP INT QUIT ILL TRAP ABRT IOT BUS FPE KILL USR1 SEGV USR2 PIPE ALRM TERM
STKFLT CHLD CONT STOP TSTP TTIN TTOU URG XCPU XFSZ VTALRM PROF WINCH IO PWR SYS
UNUSED

For the last examples, fuser will be used to see what command is bound to TCP port 22.

root@testbox:~# fuser -uv 22/tcp
                     USER        PID ACCESS COMMAND
22/tcp:              user01   7288 F.... (root)sshd

The same can be accomplished as follows

root@testbox:~# fuser -nuv tcp 22
                     USER        PID ACCESS COMMAND
22/tcp:              root       2267 F.... (root)sshd

Full help output

Usage: fuser [ -a | -s | -c ] [ -n SPACE ] [ -SIGNAL ] [ -kimuv ] NAME...
             [ - ] [ -n SPACE ] [ -SIGNAL ] [ -kimuv ] NAME...
       fuser -l
       fuser -V
Show which processes use the named files, sockets, or filesystems.

    -a        display unused files too
    -c        mounted FS
    -f        silently ignored (for POSIX compatibility)
    -i        ask before killing (ignored without -k)
    -k        kill processes accessing the named file
    -l        list available signal names
    -m        show all processes using the named filesystems
    -n SPACE  search in this name space (file, udp, or tcp)
    -s        silent operation
    -SIGNAL   send this signal instead of SIGKILL
    -u        display user IDs
    -v        verbose output
    -V        display version information
    -4        search IPv4 sockets only
    -6        search IPv6 sockets only
    -         reset options

  udp/tcp names: [local_port][,[rmt_host][,[rmt_port]]]

Notes: As you can see, fuser can be used in many different ways and features overlap in some areas with ‘ps’ and ‘lsof’. The main use for me is the ability to look in on what is currently running on filesystems and devices.