Colocation to Virtualization » Filesystems

Storage, SAN, Linux: EMC PowerPath Configuration On Cisco UCS

Kevin Goodman — Tue, 04 May 2010 14:39:42 +0000

The following is a walk through of installing EMC PowerPath software on RedHat based Linux hosts (CentOS/Fedora). This is required to fully utilize multiple paths to EMC SANs. The test server used here is a Cisco UCS B250-M1 blade running FCOE over 10gb Ethernet. The configuration steps work for ISCSI, Fiber Channel, and FCOE connectivity to Clariion systems.

First, copy the RPM installation package over to the server. Below shows the package to be installed.

[root@test_server01 user01]# ll
total 7036
-rw-r--r-- 1 user01 user01 7191661 Apr 27 09:24 EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm

Install the package via “rpm -i”.

[root@test_server01 user01]# rpm -i EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm
All trademarks used herein are the property of their respective owners.
NOTE:License registration is not required to manage the CLARiiON AX series array.

Before powerpath can be used, a license key must be installed.

[root@test_server01 user01]# emcpreg -list
unable to open license key file: No such file or directory

Overview of the “emcpreg -add” syntax.

[root@test_server01 user01]# emcpreg -add
Missing option parameter.
Usage:
    emcpreg [opts] -add key [key ...]
    emcpreg [opts] -remove key [key ...]
    emcpreg [opts] -check key [key ...]
    emcpreg [opts] -list
    emcpreg [opts] -edit
    emcpreg [opts] -install
Options:
    -f file     license file

Now we add the license key to powerpath. The following key is fake! You must obtain yours from EMC.

[root@test_server01 user01]# emcpreg -add AGE4-DFD3-89842-DSAF-JIJ0-WKG50
1 key(s) successfully added.

Make sure the license was installed correctly.
[root@test_server01 user01]# emcpreg -list

Key AGE4-DFD3-89842-DSAF-JIJ0-WKG50
  Product: PowerPath
  Capabilities: All
[root@test_server01 user01]#

Next, start the Power Path service.

[root@test_server01 user01]# /etc/init.d/PowerPath start
Starting PowerPath:  done

Display the current paths to storage via “powermt”. Since this server is booting from SAN and just being installed, there is currently only one path to storage.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=Unknown, current=SP A    Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sda       SP A0     active  alive      0      0

Now that powerpath is installed, we need to edit fstab to boot off of the Power Path device.

Origional fstab using labels for “/boot”.

/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Below is the edited fstab with “LABEL=/boot” commented out and /boot changed to use “/dev/emcpowera1″

[root@test_server01 ~]# vi /etc/fstab
/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
/dev/emcpowera1         /boot                   ext3    defaults        0 0
#LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Filesystem options were changed to “0 0″ on emcpowera due to RedHat trying to do filesystem scans before the Power Path driver is started.

All paths now need to be zoned in the fiber switch, initiators registered in Navisphere, and paths added to the host in it’s storage group. This will not be covered here.

After zoning both paths on one switch, “powermt” now shows a path to both Clariion SPA and SPB. If not, try either stopping and restartaring “/etc/init.d/PowerPath” or restarting the server.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP A       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0

Configuration has now been completed on fiber switch 2 and both SPA and SPB in the Clariion. Reboot the server. Again, “powermt” is used to list the paths.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0
   1 fnic                      sde       SP B0     active  alive      0      0
   1 fnic                      sdf       SP A1     active  alive      0      0

From above, you can see that we now have 4 paths definied. Both fnic interfaces can see SPA and SPB. Each fnic is attached to a seperage fiber switch, so we have redundant paths to both Clariion heads (SP’s). Once rebooted, the server should load fine with no issues and see all paths via powermt.

Notes: “/boot” is the storage label used in this example. If your mount point is different, modify it’s entry instead. “/dev/emcpowera1″ is used since there is only one LUN mapped to this host. Like anything else, if there are more than one, each would have it’s own device.

Filed under: EMC, Filesystems, Linux, SAN (Storage Area Network)

Linux, Filesystem: GNOME Virtual File System (GVFS) Remote Connectivity CLI

Kevin Goodman — Mon, 07 Dec 2009 17:47:46 +0000

When not using NFS, Linux administrators generally move files from one server to the next via SFTP or FTP. This can sometimes be a headache when needing to move large amounts of files between the systems. This is where I like GVFS (GNOME Virtual File System). This subsystem allows you to mount remote systems via the following protocols to a local directory tree:

SSH

FTP

CIFS (Windows shares)

WebDav (HTTP)

Secure WebDav (HTTPS)

Above are the common protocols supported, but there is support for more. Using GVFS to mount the remote filesystem to yours allows you to create and move files to and from the remote system using typical “cp”, “rm”, and “mv” commands. This makes things even easier if you are working through an X windows console. Just bring up the remote directory structure through a file manager application and work from there. Gnome also uses GVFS to manage USB based storage. The following will go through manually connecting to a server using GVFS.Move into the “.gvfs” filesystem in the users home directory. Unless Gnome has automatically mounted a device, this filesystem should be empty.

user01@LinuxDesk:~$ cd ~/.gvfs

In the below example, a remote servers filesystem will be mounted over an SSH/SFTP session.

user01@LinuxDesk:~/.gvfs$ gvfs-mount ssh://user05@SftpServer02
Enter password
Password:

Verify that the location has been mounted.

user01@LinuxDesk:~/.gvfs$ ls
sftp for user05 on SftpServer02

The SFTP was mounted and we can now traverse the remote servers filesystem as if it were our own.

user01@LinuxDesk:~/.gvfs$ cd sftp\ for\ user05\ on\ SftpServer02/

user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02$ ls
app  boot  etc   hs_err_pid15240.log  lib         media  mnt  opt   relay  sbin     srv  tmp  var
bin  dev   home  hs_err_pid8660.log   lost+found  misc   net  proc  root   selinux  sys  usr

Since we logged into the SSH/SFTP system using user “user05″, we can write to any direcotry that remote user has access to.

user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02$ cd home/user05/

Below creates a new file “asdf” containing the text “asdfasdf”. Here we are just testing write capability to the remote server

user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02/home/user05$ echo "asdfasdf" > asdf
user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02/home/user05$ cat asdf
asdfasdf

“gvfs-mount” can also be used to list all currently mounted gvfs systems. Below shows only the sftp session.

user01@LinuxDesk:~$ gvfs-mount -l
Mount(0): sftp on SftpServer02 -> sftp://SftpServer02/
  Type: GDaemonMount

For reference, the following shows my 4gig USB drive that was automatically mounted when attached to the workstation through Gnome.

user01@LinuxDesk:~$ gvfs-mount -l
Drive(0): USB Drive
  Type: GProxyDrive (GProxyVolumeMonitorHal)
  Volume(0): 4.1 GB Media
    Type: GProxyVolume (GProxyVolumeMonitorHal)
    Mount(0): 4.1 GB Media -> file:///media/disk
      Type: GProxyMount (GProxyVolumeMonitorHal)
Mount(0): sftp on SftpServer02 -> sftp://SftpServer02/
  Type: GDaemonMount

GVFS mount points can be un-mounted using the “-u” argument. Below will un-mount the remote ssh server.

user01@LinuxDesk:~/.gvfs$ gvfs-mount -u ssh://user05@SftpServer02

Notes: GVFS contains one master daemon (gvfsd) which tracks current GVFS mounts. Each mount is created as an individual daemon with it’s own process. Knowing this, we can find the actual gvfsd process ID that the sftp connection is running under.

user01@LinuxDesk:~/.gvfs$  ps -ef | grep gvfsd-sftp
user01  8022     1  0 10:34 ?        00:00:00 /usr/lib/gvfs/gvfsd-sftp --spawner :1.8 /org/gtk/gvfs/exec_spaw/21

Posted in Filesystems, Linux, Networking

Linux / Storage: Memory – Huge Pages Overview

Kevin Goodman — Tue, 13 Oct 2009 14:39:16 +0000

A page is really virtual memory which is managed by the Translation Lookaside Buffers(TLB) in the CPU. The TLB controls the mapping of the virtual memory pages to physical memory addresses. In doing so, it bypasses the kernel virtual memory manager.

Per RedHat,

The TLB is a limited hardware resource, so utilising a huge amount of physical memory with the default page size consumes the TLB and adds processing overhead – many pages of size 4096 Bytes equates to many TLB resources consumed.

This is where Huge Pages come in. Pages are created at a larger size than the default 4096 bytes, and each page will consume only one TLB resource. So you can see this is a huge benefit. Using Huge Pages decrease the number of TLB resources required.

Side Affect
This is great, depending on what you are trying to accomplish. Once the physical memory is mapped to a Huge Page, it can no longer be used for “normal” memory allocation. This is because the memory is no longer mapped by the kernel virtual memory manager. The applications that you want to dedicate the Huge Pages to have to have support for them.

Benefit
So here is the best part of Huge Pages. It is dedicated memory to be used by only applications that request them. This dedicated memory is stored in physical RAM and will NEVER be swapped out! Thus, guaranteeing a level of performance. When memory is swapped to disk, it’s a lot slower than RAM and decreases the performance of the process(s)/program(s) gets pushed there.

Now knowing that Huge Pages are stored in RAM, this also means that the allocated RAM is dedicated. This is a little bit redundant to the above, but I want to make sure this point is clear.

Example: If a server has 8gigs of RAM and 5gigs are allocated to Huge Pages, that only leaves 3gigs for all other processes, programs, and underlining operating system to use.

Below shows my Linux desktop that has the default page size of 4096 set

user@workstation:~$ cat /proc/meminfo | grep Huge
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       4096 kB

So as you can see, I have no Huge Pages reserved or in use. The next example is from a production Oracle database server

[root@OracleServer1 ~]# cat /proc/meminfo | grep Huge
HugePages_Total: 12200
HugePages_Free:     85
Hugepagesize:     2048 kB

So to calculate the space dedicated to Huge Pages from above, it is 12,200 x 2048 kB which gives us

24 985 600 kilobytes = 23.828125 gigabytes

In the 2.6x Linux kenel, Huge Pages are enabled using the CONFIG_HUGETLB_PAGE feature when compiling the kernel. Most “Enterprise” Linux OSs by default have this enabled. The ones that I know of are RedHat, CentOS, and possibly Fedora from version 4+.

Notes: Again, applications that you want to dedicate Huge Pages to must have support for them. Most memory intensive ones do, but check for this first.

Posted in Filesystems, Linux

VMWare: vSphere / ESX 4 Server Partitioning

Kevin Goodman — Thu, 06 Aug 2009 19:07:07 +0000

This will review the partitioning scheme that I am currently using for VMware vSphere (ESX 4). For information concerning partitioning for VMware ESX 3.x, please refer to the following link:

http://blog.colovirt.com/2008/10/31/vmware-esx-server-partitioning/

The majority of the partitioning structure is the same what was used for 3.x. The only real changes is the fact that the installation process auto-creates two of the partitions that were to be manually created on 3.x. Those two partitions are:

/boot (260mb)

vmkcore (100mb)

As in the 3.x structure, again I still maintain that creating a seperate mount point for /var/core should be used. For the reasons stated below:From the 3.x post

“I have had a few servers core dump and drop over 5 gigs of data to /var/core. Before, per “best practices” a vendor recommended around 4 gigs for /var. I upped that to 6 gigs originally, but after 2 servers had /var 100% utilized I and revising that. /var is still 6 gigs but /var/core has been broken out into its own mount point. 15 gigs is a little high, but these servers had raid 1 – 73 gig hard drives. At least now if the servers core dump it will affect only its mount point. I highly recommend doing this!”

Below is how I am partitioning vSphere 4 servers

Mount Point	Size(m)	Partition type
/	10240	Primary
swap	1600	Primary *max
/var	6142	Extended
/var/core	15360	Extended
/opt	2048	Extended
/home	2048	Extended
/tmp	1024	Extended

Notes: Yellow-Bricks (Duncan Epping), as well as VMETC have good articles as well.

Posted in Filesystems, VMWare

SAN/Networking/Linux: Using Multipath To Verify and Troubleshoot Connectivity To FC LUNs

Kevin Goodman — Mon, 11 May 2009 13:33:11 +0000

A few days ago there was in error on one of our Cisco MDS 9120 fiber switches. The current environment at this datacenter consists of two Cisco MDS 9120 SAN switches with servers redundantly connected between the two. These switches are used to connect the servers to our fiber channel (FC) storage systems. In this case, the servers are generally mapped to an EMC, NetApp, and two RamSans. Below outlines a basic example of what can be expected from multipathing and the Linux environment when there is loss of connectivity to one leg of the fiber network.

Below is an excerpt of the log coming from a dual fiber connected server. In this case, the max active paths would be 2. If the server had quad fiber connectivity it could have a max of 4 paths in our environment. As seen from the log, multipathd reported the loss of a path to each LUN an logged it “remaining active paths: 1″. Once the errors were corrected in the failed switch, multipathd reported the restoration “remaining active paths: 2“.

[root@testsrv01 ~]# cat /var/log/messages | grep multipathd
May  6 10:30:55 testsrv01 multipathd: TempLUN1: remaining active paths: 1
May  6 10:30:55 testsrv01 multipathd: 8:224: readsector0 checker reports path is up
May  6 10:30:55 testsrv01 multipathd: 8:224: reinstated
May  6 10:30:55 testsrv01 multipathd: RanSanLUN2: remaining active paths: 1
May  6 10:30:56 testsrv01 multipathd: 65:128: readsector0 checker reports path is up
May  6 10:30:56 testsrv01 multipathd: 65:128: reinstated
May  6 10:30:57 testsrv01 multipathd: TempLUN1: remaining active paths: 2
May  6 10:30:58 testsrv01 multipathd: 65:224: readsector0 checker reports path is up
May  6 10:30:58 testsrv01 multipathd: 65:224: reinstated
May  6 10:30:58 testsrv01 multipathd: RanSanLUN2: remaining active paths: 2

To be safe, the ‘multipath’ command was used via the Command Line Interface (CLI) on the server(s) to verify that the storage systems were visable to the OS.

NAME
       multipath - Device mapper target autoconfig
SYNOPSIS
       multipath [options] [device]

       -l     show the current multipath topology from information fetched in sysfs and the device mapper
       -ll    show the current multipath topology from all available information (sysfs, the device mapper ...)

Truncated results are below that shows connectivity to the RamSans and EMC. This server is not mapped to the NetApp.

[root@testsrv01 ~]# multipath -ll
ramsan_DramCache (25050c24001094501)
[size=9 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:0  sdah 66:16  [active][ready]
 \_ 2:0:4:0  sdai 66:32  [active][ready]
 \_ 1:0:3:0  sdal 66:80  [active][ready]
 \_ 1:0:4:0  sdan 66:112 [active][ready]

ramsan_FlashSSD (25050c24021094501)
[size=1853 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:1  sdaj 66:48  [active][ready]
 \_ 2:0:4:1  sdak 66:64  [active][ready]
 \_ 1:0:3:1  sdam 66:96  [active][ready]
 \_ 1:0:4:1  sdao 66:128 [active][ready]

EMC_LUN1 (461a963004637699a05bs3464f76)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][active]
 \_ 2:0:2:8  sdab 65:176 [active][ready]
 \_ 1:0:2:8  sdl  8:176  [active][ready]

Notes: Make sure to verify that all servers have multipathing setup and working correctly, all of the Linux servers never missed a beat! The database (Oracle) and Websphere servers kept on going. The same can not be said about the Exchange environment. Due to a configuration problem there, multipathing was not functioning and it lost connectivity to its storage LUNs. The VMware cluster survived fine as well.

Posted in EMC, Filesystems, Linux, Monitoring, Networking, SAN (Storage Area Network), VMWare

DataDomain/NAS/Filesystems/Linux: Remove Files From A DataDomain’s /ddvar

Kevin Goodman — Thu, 16 Apr 2009 15:19:59 +0000

A few days ago we received alerts for the /ddvar fileystem from one of our Data Domain units. Normally, I would not manually remove any files from this filesystem due to it being mainly used by the underlining OS and not for NAS storage. In this case, the problem was the “core” subdirectory. I tried to remove the files from a Windows machine, but due to permission issues, I was unable to do so. Even through Windows (CIFS) to the DataDomain, I was able to modify the permissions of the file(s) and still would receive a permission denied issue when trying to remove. The quick solution here was to add my Linux box to the NFS access list, mount “/ddvar”, and remove the files as the root user. Below details the process that worked for me.

DataDomain Web Alert

SSH to the DataDomain and add my Linux systems IP address

admin1@DataDomain01# nfs add /ddvar 172.16.100.6
Mount the DataDomain /ddvar export on the Linux system
root@tstbox02:~# mount DataDomain01:/ddvar /mnt/DataDomain01/

Move into the core sub-directory under the ddvar mount point

root@tstbox02:~# cd /mnt/DataDomain01/core/

Check the directory for files. As you can see below, the core directory contained 47 gig worth of data

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 47G
drwxrwsr-x  2 root group 4.0K 2009-01-14 22:43 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 2.2M 2008-04-10 14:23 cmdb.core.26956.1207851782.gz
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 121K 2008-04-08 07:01 ddfs.core.14547.1207652507.gz
-rw-------  1 root group 352K 2008-04-07 22:56 ddfs.core.15098.1207618361.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:30 ddfs.core.15100.1207747799.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:37 ddfs.core.15125.1207665449.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:38 ddfs.core.15505.1207665497.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:31 ddfs.core.15506.1207747848.gz
-rw-r--r--  1 root group 527M 2008-09-10 16:27 ddfs.core.1560.1221078099.gz
-rw-r--r--  1 root group 166M 2008-04-08 07:32 ddfs.core.15675.1207654304.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:39 ddfs.core.15795.1207665554.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:32 ddfs.core.15813.1207747905.gz
-rw-r--r--  1 root group 636M 2008-07-28 15:20 ddfs.core.15991.1217272516.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:40 ddfs.core.16055.1207665605.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:33 ddfs.core.16095.1207747957.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:32 ddfs.core.18047.1207654349.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:33 ddfs.core.18231.1207654404.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:34 ddfs.core.18591.1207654458.gz
-rw-r--r--  1 root group 548M 2008-09-20 02:56 ddfs.core.22290.1221893385.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22705.1207750958.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22875.1207751006.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:24 ddfs.core.23008.1207751055.gz
-rw-r--r--  1 root group 553M 2008-08-19 15:26 ddfs.core.23183.1219173638.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:25 ddfs.core.23269.1207751113.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:06 ddfs.core.28347.1207746385.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:07 ddfs.core.28507.1207746432.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:08 ddfs.core.28694.1207746490.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:09 ddfs.core.28989.1207746540.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:14 ddfs.core.29883.1207746837.gz
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Since I verified that we do not need the archived (.gz) files, they were removed

root@tstbox02:/mnt/DataDomain01/core# rm *.gz

By only retaining the newer files in this directory, the size is now down to 1.6 gigs.

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 1.6G
drwxrwsr-x  2 root group 4.0K 2009-04-15 14:29 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Once this was completed, the alert in this DataDomain cleared in the web interface.

Data Domain Web Alert Cleared

Notes: Always verify files that are being removed are not needed. If there are core files present, it generally means the system has hit an error. Verify with your DataDomain support before modifying the ddvar filesystem.

Posted in Data Domain, Filesystems, Linux, NAS

Linux/Unix/File Systems: Inodes (Part 2) – File Level Inode Information And Removal

Kevin Goodman — Mon, 13 Apr 2009 08:00:03 +0000

Inodes “Part 1″ went into locating filesystem level inode information. Here we will move from the main filesystem to the individual file. Besides reviewing how the inode record reflects permissions modification using chown, file removal based on inode number will be covered.

Create a test file

user01@testsrv:~$ touch testfile

Below, “ls” is used to display the inode number (2009418) of the test file

user01@testsrv:~$ ls -i /home/user01/testfile
2009418 /home/user01/testfile
Check the inode record for the newly created empty file

user01@testsrv:~$ stat /home/user01/testfile
File: `/home/user01/testfile'
Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fe01h/65025d	Inode: 2009418     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-04-10 17:52:59.000000000 -0400
Modify: 2009-04-10 17:52:58.000000000 -0400
Change: 2009-04-10 17:52:58.000000000 -0400

In the following example, a file is being checked that is solely owned by the root user

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 root root 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Review the root owned ISO file inode information. The user and group ownership is referenced as Uid and Gid respectively

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-03-28 13:15:33.000000000 -0400

Change the permissions of the ISO file via the root user (current owner of the file)

root@testsrv:/home/user01# chown user01:user01 /home/user01/w2k3-2.iso

Re-check tje ISO file permission from the initial user (user01) via the ls command

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 user01 user01 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Re-stat the ISO file to view the changes in the inode record. As you can see, the file ownership is now set to user user01 and group user01

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 17:57:35.000000000 -0400

For the next example user01 is kept as the user, but the group is changed to ‘admin’. Since the current user user01 now has full rights to the file, it can be done from that account

user01@testsrv:~$ chown :admin /home/user01/w2k3-2.iso

Re-stat the file once more to see if the group (Gid) has been changed in the inode for this file

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: (  119/   admin)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 18:00:32.000000000 -0400

Now with the basic understanding of finding an retrieving inode numbers and information, we will proceed to removing files using the “rm” command. Specifically, the following will detail how to remove a file by referencing the inode number. All of which is done by using “find” and passing the results to “rm”.

There have only been a few occasions that I can remember where I have needed to know the following, but it is worth knowing! The main benefit is when a file has been created with a control character in the name

Create a test file

user01@testsrv:/tmp/testdir$ touch '`'
user01@testsrv:/tmp/testdir$ ls
`

Try to remove the file without enclosing the “`” filename

user01@testsrv:/tmp/testdir$ rm `
>

Above you can see that the command did not work. Instead it dropped to the “>” sub-prompt

Locate the inode number for the “`” file. Tab based auto-complete does wonders escaping the special character filename on the command line

user01@testsrv:/tmp/testdir$ ls -i /tmp/testdir/`
1163285 /tmp/testdir/`

Now that the inode number is known, the “find” command can be used to retrieve the filename associated with it

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285
/tmp/testdir/`

Putting it all together, we can remove the file passing the “find” results to “rm”. Below uses a command line pipe (“|”) and xargs to accomplish this

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 | xargs rm

Verify the file has been removed

user01@testsrv:/tmp/testdir$ ls /tmp/testdir/

The same can be accomplished via find commands built-in exec feature

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 -exec rm {} ;

In my opinion, the best way to remove a file via the inode number is as follows

user01@testsrv:~$ find /tmp -inum 1163285 -exec rm -i {} ;
rm: remove regular empty file `/tmp/testdir/`'? y

Specifying “-i” after rm makes the user verify the file to be removed.

Notes: There are other ways around removing files with certain characters. In this chase, the “`” file could have been removed with:

user01@testsrv:/tmp/testdir$ rm /tmp/testdir/`

If any information in this post is found to be outdated, incorrect, useful, or needs further detail, please leave a comment or email me.

Posted in Filesystems, Linux, Monitoring, Uncategorized

Linux, Unix, NAS, File Systems: Inodes (Part 1) – Checking Availability And High Level Overview

Kevin Goodman — Fri, 10 Apr 2009 19:37:40 +0000

Inodes really tell you how many file handles (files) that can be created on a file system. Most people will never exceed the default setting when the file system is created, nor even know that one is set. I will eventually go into more detail concerning this topic here on the blog. The majority (not all) of file systems that are used on Linux and Unix do not support dynamic inode allocation. What this means is that if you exceed the inode limit of a file system before the storage space, the remainder will be un-usable. That is until some of the current files are removed.

So here is the really bad part. The inodes on ext2 and ext3 (Linux default type) are statically set when the file system is formatted. You can not go back and change the max inode settings. The exceptions to this that I know of are as follows:

- Reiser4
- VxFS
- XFS
- JFS
- WAFL (NetApp proprietary)
- XZFS

If you are running one of the above and have max inodes issue, you can correct it.

I have been working with computers for over 15 years and have only ran into this problem once. Luckily, it occurred on a NetApp NAS device that had the ability to increase this value on the live file system. The main killer here are tons of small files. In this case, the file system for that NFS share was 40 gigabytes in size and default was ~1 million inode limit. The quick fix for the issue was to increase this to 3 million.

As far as a ext2 and 3 go, the following shows how to query a file system for relevant inode information

root@testbox:~# tune2fs -l /dev/sda1
tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   
Last mounted on:          
Filesystem UUID:          56161dd8-9d1d-4c54-851d-938bb88ce6d4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4685824
Block count:              18731782
Reserved block count:     936589
Free blocks:              15534374
Free inodes:              4459463
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1019
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   256
Filesystem created:       Mon Sep 29 16:25:20 2008
Last mount time:          Fri Jan 23 14:27:02 2009
Last write time:          Fri Jan 23 14:27:02 2009
Mount count:              4
Maximum mount count:      33
Last checked:             Thu Jan 15 09:00:37 2009
Check interval:           15552000 (6 months)
Next check after:         Tue Jul 14 10:00:37 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          128
Journal inode:            8
First orphan inode:       2908742
Default directory hash:   tea
Directory Hash Seed:      a6544c5xxxxxxxxxxxxxxxxx
Journal backup:           inode blocks

The above is good to know, especially to check an un-mounted file system. The command below shows a friendlier formatted output

root@testbox:~# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            4685824  226361 4459463    5% /
tmpfs                 222201       4  222197    1% /lib/init/rw
varrun                222201      64  222137    1% /var/run
varlock               222201       5  222196    1% /var/lock
udev                  222201    5142  217059    3% /dev
tmpfs                 222201       5  222196    1% /dev/shm
/dev/sdb1            61063168    1116 61062052    1% /media/disk

As you can see, there are no issues to be worried about on this test computer. Most systems administrators perform centralized monitoring of disk usage at a disk space level (capacity). On highly used servers that utilize locally stored and/or direct attached storage, it is a good idea to have a script check and report on the available inodes.

PDF Version

Notes: This information is provided for a high level overview concerning inodes. More in-depth information will be provided in up-coming posts.

Posted in Filesystems, Linux, Monitoring, NAS

VMWare: ESX Server Partitioning

Kevin Goodman — Fri, 31 Oct 2008 14:50:28 +0000

I have had a few servers core dump and drop over 5 gigs of data to /var/core. Before, per “best practices” a vendor recommended around 4 gigs for /var. I upped that to 6 gigs originally, but after 2 servers had /var 100% utilized I and revising that. /var is still 6 gigs but /var/core has been broken out into its own mount point. 15 gigs is a little high, but these servers had raid 1 – 73 gig hard drives. At least now if the servers core dump it will affect only its mount point. I highly recommend doing this!

Below is how I am partitioning ESX servers during the local installs from now on

Mount Point	Size(m)	Partition type
/boot	250	Primary
/	10240	Primary
swap	1600	Primary *max
/var	6142	Extended
/var/core	15360	Extended
/opt	2048	Extended
/home	2048	Extended
/tmp	1024	Extended
vmkcore	109	*max

Note: The *max references to the mount point being at the max size limit for ESX v3*. The total size of this install is ~36 gigs. If you have smaller drives, just decrease /var/core. All the rest, I would keep close to the same.

Update: Here is an example of why /var/core now has its own filesystem

[user@vm user]$ df -h

Filesystem	Size	Used	Avail	Use%	Mounted on
/dev/sda2	9.9G	1.2G	8.3G	13%	/
/dev/sda1	244M	27M	205M	12%	/boot
/dev/sda8	2.0G	341M	1.6G	18%	/home
/dev/sda7	2.0G	81M	1.8G	5%	/opt
none	391M	0	391M	0%	/dev/shm
/dev/sda9	1012M	33M	928M	4%	/tmp
/dev/sda6	6.0G	327M	5.3G	6%	/var
/dev/sda5	15G	15G	0	100%	/var/core

Never lost vpxa or mgmt-vmware (hostd) this time!

Posted in Filesystems, VMWare