Below is an excerpt of the log coming from a dual fiber connected server. In this case, the max active paths would be 2. If the server had quad fiber connectivity it could have a max of 4 paths in our environment. As seen from the log, multipathd reported the loss of a path to each LUN an logged it “remaining active paths: 1″. Once the errors were corrected in the failed switch, multipathd reported the restoration “remaining active paths: 2“.

[root@testsrv01 ~]# cat /var/log/messages | grep multipathd
May  6 10:30:55 testsrv01 multipathd: TempLUN1: remaining active paths: 1
May  6 10:30:55 testsrv01 multipathd: 8:224: readsector0 checker reports path is up
May  6 10:30:55 testsrv01 multipathd: 8:224: reinstated
May  6 10:30:55 testsrv01 multipathd: RanSanLUN2: remaining active paths: 1
May  6 10:30:56 testsrv01 multipathd: 65:128: readsector0 checker reports path is up
May  6 10:30:56 testsrv01 multipathd: 65:128: reinstated
May  6 10:30:57 testsrv01 multipathd: TempLUN1: remaining active paths: 2
May  6 10:30:58 testsrv01 multipathd: 65:224: readsector0 checker reports path is up
May  6 10:30:58 testsrv01 multipathd: 65:224: reinstated
May  6 10:30:58 testsrv01 multipathd: RanSanLUN2: remaining active paths: 2

To be safe, the ‘multipath’ command was used via the Command Line Interface (CLI) on the server(s) to verify that the storage systems were visable to the OS.

NAME
       multipath - Device mapper target autoconfig
SYNOPSIS
       multipath [options] [device]

       -l     show the current multipath topology from information fetched in sysfs and the device mapper
       -ll    show the current multipath topology from all available information (sysfs, the device mapper ...)

Truncated results are below that shows connectivity to the RamSans and EMC. This server is not mapped to the NetApp.

[root@testsrv01 ~]# multipath -ll
ramsan_DramCache (25050c24001094501)
[size=9 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:0  sdah 66:16  [active][ready]
 \_ 2:0:4:0  sdai 66:32  [active][ready]
 \_ 1:0:3:0  sdal 66:80  [active][ready]
 \_ 1:0:4:0  sdan 66:112 [active][ready]

ramsan_FlashSSD (25050c24021094501)
[size=1853 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:1  sdaj 66:48  [active][ready]
 \_ 2:0:4:1  sdak 66:64  [active][ready]
 \_ 1:0:3:1  sdam 66:96  [active][ready]
 \_ 1:0:4:1  sdao 66:128 [active][ready]

EMC_LUN1 (461a963004637699a05bs3464f76)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][active]
 \_ 2:0:2:8  sdab 65:176 [active][ready]
 \_ 1:0:2:8  sdl  8:176  [active][ready]

Notes: Make sure to verify that all servers have multipathing setup and working correctly, all of the Linux servers never missed a beat! The database (Oracle) and Websphere servers kept on going. The same can not be said about the Exchange environment. Due to a configuration problem there, multipathing was not functioning and it lost connectivity to its storage LUNs.  The VMware cluster survived fine as well.

Create a test file

user01@testsrv:~$ touch testfile

Below, “ls” is used to display the inode number (2009418) of the test file

user01@testsrv:~$ ls -i /home/user01/testfile
2009418 /home/user01/testfile
Check the inode record for the newly created empty file

user01@testsrv:~$ stat /home/user01/testfile
File: `/home/user01/testfile'
Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fe01h/65025d	Inode: 2009418     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-04-10 17:52:59.000000000 -0400
Modify: 2009-04-10 17:52:58.000000000 -0400
Change: 2009-04-10 17:52:58.000000000 -0400

In the following example, a file is being checked that is solely owned by the root user

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 root root 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Review the root owned ISO file inode information. The user and group ownership is referenced as Uid and Gid respectively

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-03-28 13:15:33.000000000 -0400

Change the permissions of the ISO file via the root user (current owner of the file)

root@testsrv:/home/user01# chown user01:user01 /home/user01/w2k3-2.iso

Re-check tje ISO file permission from the initial user (user01) via the ls command

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 user01 user01 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Re-stat the ISO file to view the changes in the inode record. As you can see, the file ownership is now set to user user01 and group user01

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 17:57:35.000000000 -0400

For the next example user01 is kept as the user, but the group is changed to ‘admin’. Since the current user user01 now has full rights to the file, it can be done from that account

user01@testsrv:~$ chown :admin /home/user01/w2k3-2.iso

Re-stat the file once more to see if the group (Gid) has been changed in the inode for this file

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: (  119/   admin)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 18:00:32.000000000 -0400

Now with the basic understanding of finding an retrieving inode numbers and information, we will proceed to removing files using the “rm” command. Specifically, the following will detail how to remove a file by referencing the inode number. All of which is done by using “find” and passing the results to “rm”.

There have only been a few occasions that I can remember where I have needed to know the following, but it is worth knowing! The main benefit is when a file has been created with a control character in the name

Create a test file

user01@testsrv:/tmp/testdir$ touch '`'
user01@testsrv:/tmp/testdir$ ls
`

Try to remove the file without enclosing the “`” filename

user01@testsrv:/tmp/testdir$ rm `
>

Above you can see that the command did not work. Instead it dropped to the “>” sub-prompt

Locate the inode number for the “`” file. Tab based auto-complete does wonders escaping the special character filename on the command line

user01@testsrv:/tmp/testdir$ ls -i /tmp/testdir/`
1163285 /tmp/testdir/`

Now that the inode number is known, the “find” command can be used to retrieve the filename associated with it

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285
/tmp/testdir/`

Putting it all together, we can remove the file passing the “find” results to “rm”. Below uses a command line pipe (“|”) and xargs to accomplish this

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 | xargs rm

Verify the file has been removed

user01@testsrv:/tmp/testdir$ ls /tmp/testdir/

The same can be accomplished via find commands built-in exec feature

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 -exec rm {} ;

In my opinion, the best way to remove a file via the inode number is as follows

user01@testsrv:~$ find /tmp -inum 1163285 -exec rm -i {} ;
rm: remove regular empty file `/tmp/testdir/`'? y

Specifying “-i” after rm makes the user verify the file to be removed.

Notes: There are other ways around removing files with certain characters. In this chase, the “`” file could have been removed with:

user01@testsrv:/tmp/testdir$ rm /tmp/testdir/`

If any information in this post is found to be outdated, incorrect, useful, or needs further detail, please leave a comment or email me.

So here is the really bad part. The inodes on ext2 and ext3 (Linux default type) are statically set when the file system is formatted. You can not go back and change the max inode settings. The exceptions to this that I know of are as follows:

- Reiser4
- VxFS
- XFS
- JFS
- WAFL (NetApp proprietary)
- XZFS

If you are running one of the above and have max inodes issue, you can correct it.

I have been working with computers for over 15 years and have only ran into this problem once. Luckily, it occurred on a NetApp NAS device that had the ability to increase this value on the live file system. The main killer here are tons of small files. In this case, the file system for that NFS share was 40 gigabytes in size and default was ~1 million inode limit. The quick fix for the issue was to increase this to 3 million.

As far as a ext2 and 3 go, the following shows how to query a file system for relevant inode information

root@testbox:~# tune2fs -l /dev/sda1
tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          56161dd8-9d1d-4c54-851d-938bb88ce6d4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4685824
Block count:              18731782
Reserved block count:     936589
Free blocks:              15534374
Free inodes:              4459463
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1019
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   256
Filesystem created:       Mon Sep 29 16:25:20 2008
Last mount time:          Fri Jan 23 14:27:02 2009
Last write time:          Fri Jan 23 14:27:02 2009
Mount count:              4
Maximum mount count:      33
Last checked:             Thu Jan 15 09:00:37 2009
Check interval:           15552000 (6 months)
Next check after:         Tue Jul 14 10:00:37 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          128
Journal inode:            8
First orphan inode:       2908742
Default directory hash:   tea
Directory Hash Seed:      a6544c5xxxxxxxxxxxxxxxxx
Journal backup:           inode blocks

The above is good to know, especially to check an un-mounted file system. The command below shows a friendlier formatted output

root@testbox:~# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            4685824  226361 4459463    5% /
tmpfs                 222201       4  222197    1% /lib/init/rw
varrun                222201      64  222137    1% /var/run
varlock               222201       5  222196    1% /var/lock
udev                  222201    5142  217059    3% /dev
tmpfs                 222201       5  222196    1% /dev/shm
/dev/sdb1            61063168    1116 61062052    1% /media/disk

As you can see, there are no issues to be worried about on this test computer. Most systems administrators perform centralized monitoring of disk usage at a disk space level (capacity). On highly used servers that utilize locally stored and/or direct attached storage, it is a good idea to have a script check and report on the available inodes.

PDF Version

Notes: This information is provided for a high level overview concerning inodes. More in-depth information will be provided in up-coming posts.

http://splunk.com

Below are a few rules or “searches” that I have written more tailored to my collocations networks. Of course xxxx is replaced with the actual log name, and these are starting on a schedule.  Using the startminutesago function as not to begin parsing the logs for the beginning of time and getting data I do not want.  “maxspan” tells it to only go from 5 minutes in the past to now basically.

Searches:

Cisco MDS Fiber Switch SANOS
source=”/var/log/xxxx” IF_DOWN_LINK_FAILURE startminutesago=5 | transaction maxspan=5m

Cisco Ethernet Switch
source=”/var/log/xxxx” punct=”__::_…_:___::._:_%–:__//,____” startminutesago=5 | transaction maxspan=5m

Linux Ethernet Down
source=”/var/log/xxxx” link down startminutesago=5 | transaction maxspan=5m

Cisco Failed Login
source=”/var/log/xxxx” Authentication failure for illegal user startminutesago=5 | transaction maxspan=5m

Cisco Config Save
source=”/var/log/xxxx” CFGWRITE startminutesago=5 | transaction maxspan=5m

Cisco Config Mode
source=”/var/log/xxxx” Configuring console from pts startminutesago=5 | transaction maxspan=5m

Here is an example email of what was sent to me via Splunks built in alerting system:

Name: ‘Cisco Config Mode’

Query Terms: ‘now=1224079500 source=”/var/log/messages” Configuring console from pts startminutesago=5 | transaction maxspan=5m’

Alert was triggered because of: ‘Saved Search [Cisco Config Mode]: number of events(1) greater than 0′

Attached to the email is a CSV file that contained all the fields needed for auditing, as well as what was alerted on:

Oct 15 10:00:32 192.x.x.x : 2008 Oct 15 06:49:00 UTC: %VSHD-5-VSHD_SYSLOG_CONFIG_I: Configuring console from pts/1 192.x.x.x

Will provide schreen shots of Splunk in action at a later date.