Now all that we need to do is get some data over to it.  We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit.  You can not change the order in which each device is racked.  Also, the controllers (SPs) are built into the drive tray.  Best part is that this will be wired up via 10gbE connectivity over fiber.

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

I will be using “naviseccli” for this, but the same syntax should work with “navicli”. Both are available through EMC Powerlink site.

Below will query the Clariion and return a list of all LUNs that are trespassed. The -h specifies which host (SP) to connect to. You only need to run this on one SP. The other would report back the exact same results.

C:\>naviseccli -h 127.0.0.12 getlun -trespass
LOGICAL UNIT NUMBER 7
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 15
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 118
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 17
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 204
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 619
Default Owner:              SP B
Current owner:              SP A

....

I have some Linux style utilities installed on my Windows machine. Below shows that we have a total of 55 LUNs that are currently trespassed. Yeah, don’t ask.

C:\>naviseccli -h 127.0.0.13 getlun -trespass | grep LOGICAL | wc -l
     55

I thought that a list of all LUN IDs would be needed and then have to make a script to run through them. Luckily that is not the case. The syntax is as follows:C:\>naviseccli -h trespass mine

The above command has to be run on both SPs (Service Processors).
C:\>naviseccli -h 127.0.0.12 trespass mine

C:\>naviseccli -h 127.0.0.13 trespass mine

Use the “getlun -trespass” command once again to make sure that there are no more trespassed LUNs. A few things to keep in mind are that trespassed snapshots will stay trespassed. CLI or even the Navisphere GUI will not let you move them. Also, I had a few LUNs that would not move via the CLI, but when done manually in Navisphere they moved fine.

Below is the output of help for naviseccli.
C:\>naviseccli -help

@(#)Navisphere naviseccli Revision 6.29.5.0.66 on Wed Jan 20 22:47:06 2010
Copyright (C) 1997-2009, EMC Corporation
  Usage:
    [-User ] [-Password]
    [-Scope <0 - global; 1 - local; 2 - LDAP>]
    [-Address  | -h ]
    [-Port] [-Timeout  | -t ]
    [-AddUserSecurity | -RemoveUserSecurity | -DeleteSecurityEntry]
    [-Parse | -p] [-NoPoll | -np] [-cmdtime]
    [-Xml] [-f ] [-Help] CMD

If you were using “navicli” the following commands should work for you:

navicli -h 127.0.0.12 getlun -trespass
navicli -h 127.0.0.12 trespass mine
navicli -h 127.0.0.13 trespass mine

Notes: Feel free to comment / email (kevin <at> colovirt.com) / twitter me with any questions!

First, copy the RPM installation package over to the server. Below shows the package to be installed.

[root@test_server01 user01]# ll
total 7036
-rw-r--r-- 1 user01 user01 7191661 Apr 27 09:24 EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm

Install the package via “rpm -i”.

[root@test_server01 user01]# rpm -i EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm
All trademarks used herein are the property of their respective owners.
NOTE:License registration is not required to manage the CLARiiON AX series array.

Before powerpath can be used, a license key must be installed.

[root@test_server01 user01]# emcpreg -list
unable to open license key file: No such file or directory

Overview of the “emcpreg -add” syntax.

[root@test_server01 user01]# emcpreg -add
Missing option parameter.
Usage:
    emcpreg [opts] -add key [key ...]
    emcpreg [opts] -remove key [key ...]
    emcpreg [opts] -check key [key ...]
    emcpreg [opts] -list
    emcpreg [opts] -edit
    emcpreg [opts] -install
Options:
    -f file     license file

Now we add the license key to powerpath. The following key is fake! You must obtain yours from EMC.

[root@test_server01 user01]# emcpreg -add AGE4-DFD3-89842-DSAF-JIJ0-WKG50
1 key(s) successfully added.

Make sure the license was installed correctly.
[root@test_server01 user01]# emcpreg -list

Key AGE4-DFD3-89842-DSAF-JIJ0-WKG50
  Product: PowerPath
  Capabilities: All
[root@test_server01 user01]#

Next, start the Power Path service.

[root@test_server01 user01]# /etc/init.d/PowerPath start
Starting PowerPath:  done

Display the current paths to storage via “powermt”. Since this server is booting from SAN and just being installed, there is currently only one path to storage.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=Unknown, current=SP A    Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sda       SP A0     active  alive      0      0

Now that powerpath is installed, we need to edit fstab to boot off of the Power Path device.

Origional fstab using labels for “/boot”.

/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Below is the edited fstab with “LABEL=/boot” commented out and /boot changed to use “/dev/emcpowera1″

[root@test_server01 ~]# vi /etc/fstab
/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
/dev/emcpowera1         /boot                   ext3    defaults        0 0
#LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Filesystem options were changed to “0 0″ on emcpowera due to RedHat trying to do filesystem scans before the Power Path driver is started.

All paths now need to be zoned in the fiber switch, initiators registered in Navisphere, and paths added to the host in it’s storage group. This will not be covered here.

After zoning both paths on one switch, “powermt” now shows a path to both Clariion SPA and SPB. If not, try either stopping and restartaring “/etc/init.d/PowerPath” or restarting the server.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP A       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0

Configuration has now been completed on fiber switch 2 and both SPA and SPB in the Clariion. Reboot the server. Again, “powermt” is used to list the paths.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0
   1 fnic                      sde       SP B0     active  alive      0      0
   1 fnic                      sdf       SP A1     active  alive      0      0

From above, you can see that we now have 4 paths definied. Both fnic interfaces can see SPA and SPB. Each fnic is attached to a seperage fiber switch, so we have redundant paths to both Clariion heads (SP’s). Once rebooted, the server should load fine with no issues and see all paths via powermt.

Notes:  “/boot” is the storage label used in this example.  If your mount point is different, modify it’s entry instead.  “/dev/emcpowera1″ is used since there is only one LUN mapped to this host.  Like anything else, if there are more than one, each would have it’s own device.

First check and see if there are any current licenses installed. This being a new switch, there aren’t any.

nexus_5000# sh license
nexus_5000#

Licenses are normally tied to “host IDs”. Below shows the host ID of the switch.

nexus_5000# sh license host-id
License hostid: VDH=3WI246599Z

The license file is not local to the switch yet, so below I fetch it via TFTP.nexus_5000# copy tftp://127.0.0.1:69/Cisco_license/NX5KLIC.lic bootflash: vrf default_vrf

Trying to connect to tftp server......
Connection to Server Established.
[                         ]         0.50KB
TFTP get operation was successful

Next, the license is installed with the “install license” command.

nexus_5000# install license bootflash:NX5KLIC.lic
Installing license .....done

Now that the license has been added, use “show license” to confirm.

nexus_5000# show license
NX5KLIC.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT ENTERPRISE_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
1 \
RMA license" SIGN=DDWD62626W7f
INCREMENT FC_FEATURES_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
2 \
RMA license" SIGN=VVSVDS848SFW

Another way is to see the names of the installed licenses.

nexus_5000# show license brief
NX5KLIC.lic

To be safe, copy the running configuration to startup.

nexus_5000# copy running-config startup-config

Also, here I copy the configuration to TFTP backup.

nexus_5000# copy running-config tftp://127.0.0.1:69/Cisco_config/ns5k2_startup vrf default_vrf
Trying to connect to tftp server......
Connection to Server Established.
[#                        ]         4.50KB
TFTP put operation was successful

Notes:  All IP, license files, and hashes were changed.  The installation process is pretty strait forward.  The managment console (GUI) should also be able to provide this function.

First, make sure you put a copy of the license onto a tftp, ftp, or sftp server. The MDS switch supports all of those protocols. Here we will be using tftp.

Copy the license from TFTP server to bootflash (persistent storage)

mds9124# copy tftp://172.0.0.1/MDS20090209112333135513.lic bootflash:
Trying to connect to tftp server......
|
 TFTP get operation was successful

Install the license

mds9124# install license bootflash:MDS20090209112333135513.lic
Installing license .......done

Now that the new port license is installed we need to verify that it is working. Below shows the default licensing that came with the unit.

mds9124# show license default
Feature                               Default License Count
-----------------------------------------------------------------------------
FM_SERVER_PKG                         -
ENTERPRISE_PKG                        -
PORT_ACTIVATION_PKG                   8
10G_PORT_ACTIVATION_PKG               0
-----------------------------------------------------------------------------

The new one contained licensing for an additional 8 ports. Below you can see that now there are 16 ports licensed.

mds9124# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
PORT_ACTIVATION_PKG           Yes  16   In use never       -
10G_PORT_ACTIVATION_PKG       No    0   Unused             -
-------------------------------------------------------------------------------

<pre>mds9124# copy tftp://172.0.0.1/MDS20090209112333135513.lic bootflash:
Trying to connect to tftp server......
|
TFTP get operation was successful</pre>
Install the license
<pre>mds9124# install license bootflash:MDS20090209112333135513.lic
Installing license .......done<pre>

<pre>mds9124# show license default
Feature                               Default License Count
-----------------------------------------------------------------------------
FM_SERVER_PKG                         -
ENTERPRISE_PKG                        -
PORT_ACTIVATION_PKG                   8
10G_PORT_ACTIVATION_PKG               0
-----------------------------------------------------------------------------<pre>

<pre>mds9124# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
PORT_ACTIVATION_PKG           Yes  16   In use never       -
10G_PORT_ACTIVATION_PKG       No    0   Unused             -
-------------------------------------------------------------------------------</pre>

Quick way to find your serial number.

tstSwitch01# show license host-id
License hostid: VDH=SOZ115568P9

The following will also get the information that you need. I truncated some of the output. The serial number under the “Common block” is what we need.

tstSwitch01# show sprom backplane 1
DISPLAY backplane sprom contents:
Common block:
 EEPROM Size     : 1024
 Block Count     : 5
 FRU Major Type  : 0x6003
 FRU Minor Type  : 0x0
 OEM String      : Cisco Systems, Inc.
 Product Number  : DS-C9124-K9
 Serial Number   : SOZ115568P9
 Part Number     : 73-10565-03
 Part Revision   : A9
 Mfg Deviation   : 0
 H/W Version     : 1.0
 Mfg Bits        : 0
Chassis specific block:
 Block Signature : 0x5601
 MAC Addresses   : 00-0g-tr-46-n3-u6
 Number of MACs  : 64

This is a little easier to read. Here, an include statement is passed to only return lines including “Serial”. We need the first, not second serial number.

tstSwitch01# show sprom backplane 1 | include Serial
 Serial Number   : SOZ115568P9
Second Serial number specific block:
 Serial Number   : JFH2486G4DR

Notes: All actual serial numbers were changed. This process should be the same for all Cisco MDS 9000 series. If using a chassis based MDS switch, make sure to verify if you need the serial of the unit or the actual blade module for licensing.

If a cable change or SFP swap is all that is needed, than beaconing can help out!  Below is a quick run-though of enabling and disabling beaconing “blinky mode”.
Before we enable beaconing, check the interface to verify the beacon status

mds9000# sh interface fc1/20 | inc Beacon
    Beacon is turned off

Connect to the Cisco switch and move into configuration mode.

mds9000# config t
Enter configuration commands, one per line.  End with CNTL/Z.

Once in config mode, go into the specific port that needs to be located.

mds9000(config)# interface fc 1/20

General sub-parameters are listed below.

mds9000(config-if)# switchport ?
beacon        Disable/enable the beacon for an interface
description   Enter description of maximum 80 characters
encap         Configure encapsulation for the port
fcbbscn       Disable/enable buffer-to-buffer state change notification
fcrxbbcredit  Configure receive BB_credit for the port
fcrxbufsize   Configure receive data field size for the port
ignore        Enter parameter to be ignored
mode          Enter the port mode
rate-mode     Configure the rate mode for an interface
speed         Enter the port speed
trunk         Configure trunking parameters on an interface

Easy enough, tell the system to enable beacon.

mds9000(config-if)# switchport beacon

Exit config mode.

mds9000(config-if)# end

Re-check the interface to verify that beaconing is on

mds9000# sh interface fc1/20 | inc Beacon
    Beacon is turned on

So now the LEDs below the port should be flashing pretty lights and easily be located by anyone.  Once the fiber/SFP swap is completed, go back into config mode and disable beaconing.

mds9000# config t
Enter configuration commands, one per line.  End with CNTL/Z.
mds9000(config)# interface fc 1/20
mds9000(config-if)# no switchport beacon
mds9000(config-if)# end

Or you can use the GUI.  The following image shows how to enable and disable beacon from Cisco Device Manager.  It's easy as right-clicking on the port and going to beacon.

Beacon

Notes:  Pretty easy and basic command, but extremely useful!

About 4 months ago, we were looking to add another RamSan into our environment after deciding EMCs flash DAEs were not for us (at this time).  I was initially a little weary of flash based storage, but had faith in RamSan.  After reviewing the product literature more and getting some hands on with the unit, those worries are gone.  Our RamSan 500 was purchased with 2 terabytes of Flash storage and the DRAM write cache was expanded to 64 gigabytes.

With the write cache expansion, this new units cache was literally 1/2 the size of the current RamSan 400 DRAM capacity.  With this in place and the high IO that the flash disks can maintain, there has not been any noticeable speed differences between the two units.  The 500 also has an internal battery for power backup.  If power is lost, the battery will keep the unit going long enough to copy the cache’s un-written data to disk.

There are two main features that I love.  First is tha ability to lock a LUN on the 500 series into the cache, so even-though it is mainly a flash storage system, extreme IO can be accomplished.  By locking the lun in cache, the data will reside in RAM and never go to disk unless power is lost.  The second is the ability to centrally manage other RamSans on the network (auto discovered) with a specific license key.  Almost forgot to mention, the unit has quad 4 gigabit fiber connectivity.

Notes:  If you have any questions, please ask!  Again, these units are amazing.

Per the title, this is really about the EMC Clariion, not RamSans.  Since the RamSan 500 was fronted with the DRAM cache, and the EMC CX4 series contains cache as well, I was curious.  I already knew that each Service Processor (SP) in the EMC has 4 gig of cache, and that a LUN can only be active on one SP at a time.  Also, per a previous blog post, each DAE has a theoretical max throughput of 8 gigabit per second, 4 gigabit if a single LUN stripes across the whole shelf.

CX4 DAE (general) information
http://blog.colovirt.com/2009/05/29/san-emc-cx4-dae-drive-shelf-information/

As the conversation continued, I was told that typically the Service Processors caching would be disabled on LUNS that reside on the solid state flash drives.  In all actuality, I think it should.  Being that the typical Clariion implementation will not be dedicated as a solid state SAN, they will have to co-mingle with spindle based hard drives(regular).  The throughput is a LOT faster and latency a lot lower on the flash drives.  Enabling SP caching on LUNs contained in flash could possibly have a really bad affect on the rest of the system.  Being that the cache really speeds up IO for spindle based disks.

Most vendors are toting a 20 physical disks(15kRPM) to 1 Flash disk consolidation

What I am getting at is there seems to be the possibility of flooding the Service Processors cache with flash based storage data, depending on change rate.  Since flash is a lot faster, their LUNs would typically be used for databases or high volume services.  Even with the ability to do QOS in the CX4 series, disabling Service Processor caching for the flash LUNS looks to be the best solution.

Each DAE contains 2 LCC interfaces.  LCC is the acronymn for “Link Control Card”.  Typically one LCC goes to each Service Processor.  So that means that each DAE has two 4 gig links for a total of 8gb theoretical throughput.

This is where I will be reading more information.  Just because you have 2 LCC cards, each LUN can only be assigned to 1 Service Processor!  If my theory is correct, that means if all drives in one shelf is dedicated to a single LUN, and that LUN can only be active on one SP, does that mean that the actual throughput will be limited to 4gb instead of 8?  This being due to the other LCC loop being connected to the SP that is not assigned control of that LUN.

Any thoughts?  Feel free to comment!

Below is an excerpt of the log coming from a dual fiber connected server. In this case, the max active paths would be 2. If the server had quad fiber connectivity it could have a max of 4 paths in our environment. As seen from the log, multipathd reported the loss of a path to each LUN an logged it “remaining active paths: 1″. Once the errors were corrected in the failed switch, multipathd reported the restoration “remaining active paths: 2“.

[root@testsrv01 ~]# cat /var/log/messages | grep multipathd
May  6 10:30:55 testsrv01 multipathd: TempLUN1: remaining active paths: 1
May  6 10:30:55 testsrv01 multipathd: 8:224: readsector0 checker reports path is up
May  6 10:30:55 testsrv01 multipathd: 8:224: reinstated
May  6 10:30:55 testsrv01 multipathd: RanSanLUN2: remaining active paths: 1
May  6 10:30:56 testsrv01 multipathd: 65:128: readsector0 checker reports path is up
May  6 10:30:56 testsrv01 multipathd: 65:128: reinstated
May  6 10:30:57 testsrv01 multipathd: TempLUN1: remaining active paths: 2
May  6 10:30:58 testsrv01 multipathd: 65:224: readsector0 checker reports path is up
May  6 10:30:58 testsrv01 multipathd: 65:224: reinstated
May  6 10:30:58 testsrv01 multipathd: RanSanLUN2: remaining active paths: 2

To be safe, the ‘multipath’ command was used via the Command Line Interface (CLI) on the server(s) to verify that the storage systems were visable to the OS.

NAME
       multipath - Device mapper target autoconfig
SYNOPSIS
       multipath [options] [device]

       -l     show the current multipath topology from information fetched in sysfs and the device mapper
       -ll    show the current multipath topology from all available information (sysfs, the device mapper ...)

Truncated results are below that shows connectivity to the RamSans and EMC. This server is not mapped to the NetApp.

[root@testsrv01 ~]# multipath -ll
ramsan_DramCache (25050c24001094501)
[size=9 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:0  sdah 66:16  [active][ready]
 \_ 2:0:4:0  sdai 66:32  [active][ready]
 \_ 1:0:3:0  sdal 66:80  [active][ready]
 \_ 1:0:4:0  sdan 66:112 [active][ready]

ramsan_FlashSSD (25050c24021094501)
[size=1853 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:1  sdaj 66:48  [active][ready]
 \_ 2:0:4:1  sdak 66:64  [active][ready]
 \_ 1:0:3:1  sdam 66:96  [active][ready]
 \_ 1:0:4:1  sdao 66:128 [active][ready]

EMC_LUN1 (461a963004637699a05bs3464f76)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][active]
 \_ 2:0:2:8  sdab 65:176 [active][ready]
 \_ 1:0:2:8  sdl  8:176  [active][ready]

Notes: Make sure to verify that all servers have multipathing setup and working correctly, all of the Linux servers never missed a beat! The database (Oracle) and Websphere servers kept on going. The same can not be said about the Exchange environment. Due to a configuration problem there, multipathing was not functioning and it lost connectivity to its storage LUNs.  The VMware cluster survived fine as well.

[root@tftpsrv ~]# yum install tftp
Resolving Dependencies
--> Running transaction check
---> Package tftp-server.i386 0:0.42-3.1.el5.centos set to be updated
--> Processing Dependency: xinetd for package: tftp-server
--> Running transaction check
---> Package xinetd.i386 2:2.3.14-10.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 tftp-server             i386       0.42-3.1.el5.centos  base               27 k
Installing for dependencies:
 xinetd                  i386       2:2.3.14-10.el5  base              124 k

Transaction Summary
=============================================================================
Install      2 Package(s)
Update       0 Package(s)
Remove       0 Package(s)         

Total download size: 151 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): tftp-server-0.42-3 100% |=========================|  27 kB    00:00
(2/2): xinetd-2.3.14-10.e 100% |=========================| 124 kB    00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: xinetd                       ######################### [1/2]
  Installing: tftp-server                  ######################### [2/2]

Installed: tftp-server.i386 0:0.42-3.1.el5.centos
Dependency Installed: xinetd.i386 2:2.3.14-10.el5
Complete!

Edit configuration to enable tftp

[root@tftpsrv ~]# vi /etc/xinetd.d/tftp
# default: off
# description: The tftp server serves files using the trivial file transfer \
#       protocol.  The tftp protocol is often used to boot diskless \
#       workstations, download configuration files to network-aware printers, \
#       and to start the installation process for some operating systems.
service tftp
{
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /tftpboot        -> some directory (/tftpfiles)
        disable                 = yes            -> no
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

Create directory specified in tftp configuration file

[root@tftpsrv xinetd.d]# mkdir /tftpfiles

Start up xinetd. This is used to call tftp

[root@tftpsrv ~]# /etc/init.d/xinetd start
Starting xinetd:                                           [  OK  ]

[root@tftpsrv xinetd.d]# iptables-save > /etc/init.d/iptables
[root@tftpsrv xinetd.d]# iptables -F

Below, the tftp put will fail. This is due to the file needing to be created on the TFTP server before the client can write to it. This is the only real security there is to TFTP. You at least need to know the filename before the file can be written or read.

C9124SW5# copy running-config tftp:CISCSCOCFG1
Enter hostname for the tftp server: 172.16.100.6
Trying to connect to tftp server......

TFTP put operation failed:Undefined error code (2)

Create the file to be saved from switch and change the permissions

[root@tftpsrv ~]# touch /tftpfiles/CISCSCOCFG1
[root@tftpsrv ~]# chmod 777 /tftpfiles/CISCSCOCFG1

Tell the switch to save the file

C9124SW5# copy running-config tftp:CISCSCOCFG1
Enter hostname for the tftp server: 172.16.100.6
Trying to connect to tftp server......
|
TFTP put operation was successful

Check the services file to find the TFTP port and protocol information

[root@tftpsrv]# cat /etc/services | grep tftp
tftp        69/tcp
tftp        69/udp

Bring the firewall back up so we can insert rules to allow TFTP in

[root@tftpsrv]# /etc/init.d/iptables restart

On my test server, the firewall chain is “RH-Firewall-1-INPUT”. I always prefer inserting new firewall rules as the first rule. Most servers keep a few custom reject rules and most are explicit allow with the default deny at the end. Inserting the new rule as the first will normally bypass those that might reject before it ever gets to the tftp rule.

[root@tftpsrv]# iptables -I RH-Firewall-1-INPUT 1 -s 172.16.100.98 -p tcp --dport 69 -j ACCEPT
[root@tftpsrv]# iptables -I RH-Firewall-1-INPUT 1 -s 172.16.100.98 -p udp --dport 69 -j ACCEPT

The above statements tell iptables to insert “-I” the new rule into the chain “RH-Firewall-1-INPUT” as rule number “1″. The -s is specifying the source, -p the protocol –dport the destination port and -j allows the connection to establish by jumping over to ACCEPT.

Verify the rules are there

[root@tftpsrv]# iptables -L
Chain RH-Firewall-1-INPUT
target     prot opt source               destination
ACCEPT     udp  --  172.16.100.98        anywhere            udp dpt:tftp
ACCEPT     tcp  --  172.16.100.98        anywhere            tcp dpt:tftp

Save the rules in sysconfig so they will be persistent through reboots

[root@tftpsrv]# iptables-save > /etc/sysconfig/iptables

Notes: Never flush your iptables rules “iptables -F” on production systems that are not protected by a firewall or are on are public IP. Always be sure to backup/save your iptables configuration when testing. Also, if you are not familiar with security, or there is someone else responsible for security in the company, as them before or have them modify the local iptables rules. Another good rule for servers running TFTP, FTP, Telnet, DNS, and mail is to have servers dedicated for each. These are some of the most exploited servers out there.

I was outside hooking the kid bike trailer to the 12 speed when my wife came out holding my cell phone

Wife> Eric just called you twice
Me> Startup my laptop for me
... Call Eric
Eric> Sorry to bother you but  .... .

Luckily he noticed a couple of VMs dropped off the face of the earth. To be exact, we lost 4 virtual machines. When told to power on, they would error with “Can not open disk: PathToVmdk.. The parent virtual disk has been modified since the child was created“.

I never trust the GUI, so I went strait to the ESX console. Sure enough, things failed there also.

Per /var/log/vmkernel

Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.271 cpu6:1215)<6>Debug scsi underrun
Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.342 cpu6:1183)<6>Debug scsi underrun
Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.413 cpu6:1098)<6>Debug scsi underrun
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.898 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1497
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.898 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1501
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.899 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1499
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.899 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1504
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.900 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1496
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.901 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1494
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.901 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1502
Mar  5 23:10:06 vmsrv06 vmkernel: 28:09:47:04.375 cpu2:1051)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts
Mar  5 23:10:06 vmsrv06 vmkernel: 28:09:47:04.375 cpu2:1051)WARNING: FS3: 4785: Reservation error: SCSI reservation conflict

Next we will look at the NetApp. Below is from a console session. As you can see, the CPUs are hit hard.
From the NetApp

NAS>stats show system
system:system:nfs_ops:536/s
system:system:cifs_ops:0/s
system:system:http_ops:0/s
system:system:dafs_ops:0/s
system:system:fcp_ops:481/s
system:system:iscsi_ops:97/s
system:system:net_data_recv:136KB/s
system:system:net_data_sent:2323KB/s
system:system:disk_data_read:63752KB/s
system:system:disk_data_written:51640KB/s
system:system:cpu_busy:50%
system:system:avg_processor_busy:50%
system:system:total_processor_busy:100%
system:system:num_processors:2
system:system:time:1236313212s
system:system:uptime:9164385s

The conclusion is as follows: When VMware creates a snapshot, a new vmdk is created that references the new snapshot. The main vmx files will then be modified to link to the snapshot as well. When a snapshot is deleted, VMware is not just deleting a file from the file system. VMware gathers all the snapshots for that VM, collapses all their contained data into the main vmdk file, modifies the vmx files, then deletes the selected snapshots files. So in essence, when going down the list of all VMs that had snapshots(9-12) and deleting them, this caused the NetApp to overload. During this, the ESX hosts intermittently lost connectivity to the VMFS LUNS. This happened at such a fast rate, that no alerts were generated, but caused the VMs that were currently writing the snapshot to the main vmdk files to corrupt their configurations.

Moral of the story: Rapidly deleting snapshots can DOS older/slower storage systems.