Colocation to Virtualization » EMC

EMC Celerra NX4 NAS Install In Pictures

Kevin Goodman — Tue, 27 Jul 2010 19:34:07 +0000

So we received our 12tb raw EMC Celerra NX4 system(s) about 3 weeks ago. Eager to get going, I went ahead and racked the units. Below are pictures showing the different pieces that make up the NX4

Now all that we need to do is get some data over to it. We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit. You can not change the order in which each device is racked. Also, the controllers (SPs) are built into the drive tray. Best part is that this will be wired up via 10gbE connectivity over fiber.

Filed under: Colocation, EMC, Hardware, NAS, SAN (Storage Area Network)

Cisco UCS and Nexus 5000 DataCenter – Our Implementation

Kevin Goodman — Wed, 07 Jul 2010 19:07:07 +0000

Our Cisco 10gbE implementation consists of 2 Chassis fully populated UCS with a mix of full and half width blades. The servers are all boot from SAN with no local disks. “PALO” cards are used in all servers which allow us to do FCOE. 7 of the blades are running VMware ESX 4 (vSphere) and the rest are a mix of RedHat Linux and Windows 2008.

Storage:

All servers boot from SAN over FCOE

Primary storage is via Fibre Channel to an EMC CX4-240

A few servers are also connected via Fibre Channel to RamSan SSD SANs

The CX4-240 is also connected to the UCS environment via quad 10gbE links over fiberNetwork:

Core 10gbE switching is provided via Cisco Nexus 5010s

1 Gigabit Ethernet to legacy servers are provided by dual 48 port C2148 Fabric Extenders

Each C2148 is connected to a Nexus 5010 by dual 10gbE uplinks

Each 5010 is connected via quad 4gig fiber to the CX4-240 to provide FCoE to physical servers outside of the UCS environment

Each 5010 is also connected to the CX4-240 by dual 10gbE fiber ports to provide ISCSI to all of the network

Each UCS switch (6120s) are uplinked to the 5010s by dual 10gbE interfaces

Each UCS chassis is uplinked to the 6120s by 8 qty 10gbE ports (4 connections per IO module)

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon. This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120. I definitely would love feedback on this. How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Filed under: Colocation, EMC, Hardware, Networking, RamSan, SAN (Storage Area Network)

Storage, SAN: EMC Clariion LUN Trespass

Kevin Goodman — Mon, 14 Jun 2010 10:31:50 +0000

So we are in the process of working with EMC to have 10gb ISCSI interfaces installed in our CX4. I knew that we had a lot of trespassed LUNs and that those had to be corrected before the install could take place. This post will go through the process of using the CLI (Command Line Interface) to find out what is trespassed and to move them back.

I will be using “naviseccli” for this, but the same syntax should work with “navicli”. Both are available through EMC Powerlink site.

Below will query the Clariion and return a list of all LUNs that are trespassed. The -h specifies which host (SP) to connect to. You only need to run this on one SP. The other would report back the exact same results.

C:\>naviseccli -h 127.0.0.12 getlun -trespass
LOGICAL UNIT NUMBER 7
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 15
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 118
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 17
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 204
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 619
Default Owner:              SP B
Current owner:              SP A

....

I have some Linux style utilities installed on my Windows machine. Below shows that we have a total of 55 LUNs that are currently trespassed. Yeah, don’t ask.

C:\>naviseccli -h 127.0.0.13 getlun -trespass | grep LOGICAL | wc -l
     55

I thought that a list of all LUN IDs would be needed and then have to make a script to run through them. Luckily that is not the case. The syntax is as follows:C:\>naviseccli -h trespass mine

The above command has to be run on both SPs (Service Processors).
C:\>naviseccli -h 127.0.0.12 trespass mine

C:\>naviseccli -h 127.0.0.13 trespass mine

Use the “getlun -trespass” command once again to make sure that there are no more trespassed LUNs. A few things to keep in mind are that trespassed snapshots will stay trespassed. CLI or even the Navisphere GUI will not let you move them. Also, I had a few LUNs that would not move via the CLI, but when done manually in Navisphere they moved fine.

Below is the output of help for naviseccli.
C:\>naviseccli -help

@(#)Navisphere naviseccli Revision 6.29.5.0.66 on Wed Jan 20 22:47:06 2010
Copyright (C) 1997-2009, EMC Corporation
  Usage:
    [-User ] [-Password]
    [-Scope <0 - global; 1 - local; 2 - LDAP>]
    [-Address  | -h ]
    [-Port] [-Timeout  | -t ]
    [-AddUserSecurity | -RemoveUserSecurity | -DeleteSecurityEntry]
    [-Parse | -p] [-NoPoll | -np] [-cmdtime]
    [-Xml] [-f ] [-Help] CMD

If you were using “navicli” the following commands should work for you:

navicli -h 127.0.0.12 getlun -trespass
navicli -h 127.0.0.12 trespass mine
navicli -h 127.0.0.13 trespass mine

Notes: Feel free to comment / email (kevin colovirt.com) / twitter me with any questions!

Filed under: EMC, SAN (Storage Area Network)

Storage, SAN, Linux: EMC PowerPath Configuration On Cisco UCS

Kevin Goodman — Tue, 04 May 2010 14:39:42 +0000

The following is a walk through of installing EMC PowerPath software on RedHat based Linux hosts (CentOS/Fedora). This is required to fully utilize multiple paths to EMC SANs. The test server used here is a Cisco UCS B250-M1 blade running FCOE over 10gb Ethernet. The configuration steps work for ISCSI, Fiber Channel, and FCOE connectivity to Clariion systems.

First, copy the RPM installation package over to the server. Below shows the package to be installed.

[root@test_server01 user01]# ll
total 7036
-rw-r--r-- 1 user01 user01 7191661 Apr 27 09:24 EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm

Install the package via “rpm -i”.

[root@test_server01 user01]# rpm -i EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm
All trademarks used herein are the property of their respective owners.
NOTE:License registration is not required to manage the CLARiiON AX series array.

Before powerpath can be used, a license key must be installed.

[root@test_server01 user01]# emcpreg -list
unable to open license key file: No such file or directory

Overview of the “emcpreg -add” syntax.

[root@test_server01 user01]# emcpreg -add
Missing option parameter.
Usage:
    emcpreg [opts] -add key [key ...]
    emcpreg [opts] -remove key [key ...]
    emcpreg [opts] -check key [key ...]
    emcpreg [opts] -list
    emcpreg [opts] -edit
    emcpreg [opts] -install
Options:
    -f file     license file

Now we add the license key to powerpath. The following key is fake! You must obtain yours from EMC.

[root@test_server01 user01]# emcpreg -add AGE4-DFD3-89842-DSAF-JIJ0-WKG50
1 key(s) successfully added.

Make sure the license was installed correctly.
[root@test_server01 user01]# emcpreg -list

Key AGE4-DFD3-89842-DSAF-JIJ0-WKG50
  Product: PowerPath
  Capabilities: All
[root@test_server01 user01]#

Next, start the Power Path service.

[root@test_server01 user01]# /etc/init.d/PowerPath start
Starting PowerPath:  done

Display the current paths to storage via “powermt”. Since this server is booting from SAN and just being installed, there is currently only one path to storage.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=Unknown, current=SP A    Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sda       SP A0     active  alive      0      0

Now that powerpath is installed, we need to edit fstab to boot off of the Power Path device.

Origional fstab using labels for “/boot”.

/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Below is the edited fstab with “LABEL=/boot” commented out and /boot changed to use “/dev/emcpowera1″

[root@test_server01 ~]# vi /etc/fstab
/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
/dev/emcpowera1         /boot                   ext3    defaults        0 0
#LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Filesystem options were changed to “0 0″ on emcpowera due to RedHat trying to do filesystem scans before the Power Path driver is started.

All paths now need to be zoned in the fiber switch, initiators registered in Navisphere, and paths added to the host in it’s storage group. This will not be covered here.

After zoning both paths on one switch, “powermt” now shows a path to both Clariion SPA and SPB. If not, try either stopping and restartaring “/etc/init.d/PowerPath” or restarting the server.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP A       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0

Configuration has now been completed on fiber switch 2 and both SPA and SPB in the Clariion. Reboot the server. Again, “powermt” is used to list the paths.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0
   1 fnic                      sde       SP B0     active  alive      0      0
   1 fnic                      sdf       SP A1     active  alive      0      0

From above, you can see that we now have 4 paths definied. Both fnic interfaces can see SPA and SPB. Each fnic is attached to a seperage fiber switch, so we have redundant paths to both Clariion heads (SP’s). Once rebooted, the server should load fine with no issues and see all paths via powermt.

Notes: “/boot” is the storage label used in this example. If your mount point is different, modify it’s entry instead. “/dev/emcpowera1″ is used since there is only one LUN mapped to this host. Like anything else, if there are more than one, each would have it’s own device.

Filed under: EMC, Filesystems, Linux, SAN (Storage Area Network)

Storage, Network: What I have Been Doing (EMC,Cisco UCS)

Kevin Goodman — Tue, 02 Mar 2010 19:41:19 +0000

This is more of an informational update of things that I have going on right now. I normally do not publish day-to-day type of things, but here we go.

Storage

We have received both replacement drives for our EMC Clariion CX340 and four new DAEs (disk shelves) for our CX4-240

Clariion CX3

The CX3 was originally bought to speed up our Oracle implementation. This was accomplished by ordering lots of fast disks (spindles) that were small. We wound up with 6 DAEs filled with 73gig 15kRPM disks, totalling 90 dedicated drives for Oracle.

This was great for the original purpose but the unit was replaced a year after initial deployment with a RamSan and EMC CX4. Having been decommissioned from production and moved to the tier 2 site, the need for space over IOPS (speed) drastically increased. Trying to keep performance and space requirements in balance, the decision has been made to go with a smaller RamSan for Oracle at the tier 2 site. This gives us the ability to replace the small 73 gigabyte drives with bigger 600 gigabyte 10kRPM disks. Replacing those disk with the same quantity of 600 gig ones will give us ~8 times as much space.

The RamSan will almost double the IOPS capacity that the CX3 is able to achieve and speed up our data warehouse even more.

Clariion CX4

So last year we went with implementing EMC Recoverpoint SAN based replication. This has been great and served us well! The only downfall was that we were doing “CRR” remote replication only. In a case of a failure and data needed to be recovered, there were no local copies. The snapshot or “point in time” would have to be loaded from the tier 2 site and transferred across the datacenter interconnect. The interconnect being 150 megs slowed this process down.

As planned from the beginning, we are implementing “CLR” local replication as well. This means that there will be a local copy of snapshots saved locally to the CX4. This will give us almost immediate access to the snapshots without being slowed down by the interconnect. The problem with RecoverPoint is that if you have a terrabyte LUN that you want to connect, you must have an extra terrabyte worth of space to save it. This is not really a problem, but a major consideration on the number of drives to buy and the overall expense of the implementation.

In our case, a terrabyte oracle LUN will wind up costing 3 terrabytes in the end. 1 terrabyte for the original data, 1 terrabyte for the local copy (CLR), and 1 terrabyte at the remote tier 2 site (CX3).

Our virtualization effort is continuing and this is another huge factor on the storage expansion. Currently we have 16 LUNs dedicated to the VMware environment. Each is 320 gigs in size. Moving forward, we will be doing a virtual desktop deployment as well. The leftover ~400 gigs will not cut it. So in the new 60 disks, 15 or more will have to be dedicated to VMware.

Cisco UCS

We have begun our UCS voyage. As of last weekend, we did a “rip and replace of our network”. This included rewiring the main network rack and configuring a new network core. Also, the Cisco Nexus 5010, 10 gigabit Ethernet switches are in. Uplinked to them are two 48 port gigabit fabric extenders.

The VMware environment is now connected via dual 10gigE links per server through this infrastructure. Reducing the cable count from 6 to 2 per server. So far verything is stable! A purchase order has been sent out and we should hopefully have two Cisco UCS Blade chassis and switching infrastructure show up within about 30 days.

Filed under: EMC, Networking, RamSan, SAN (Storage Area Network), VMWare

SAN / EMC: Clariion CX4 Solid State DAEs (Shelves)

Kevin Goodman — Wed, 03 Jun 2009 07:59:20 +0000

Going over the solid state offerings for the EMC Clariion lines, Texas Memory RamSans came into the conversation. This was due to the fact that we currently run 2 RamSans in our Environment and consider them the highest tier storage in our datacenters. One is 128 gigs of solid state DRAM storage and the other is 2 terabyte solid state Flash storage with a 64 gig DRAM cache.

Per the title, this is really about the EMC Clariion, not RamSans. Since the RamSan 500 was fronted with the DRAM cache, and the EMC CX4 series contains cache as well, I was curious. I already knew that each Service Processor (SP) in the EMC has 4 gig of cache, and that a LUN can only be active on one SP at a time. Also, per a previous blog post, each DAE has a theoretical max throughput of 8 gigabit per second, 4 gigabit if a single LUN stripes across the whole shelf.

CX4 DAE (general) information
http://blog.colovirt.com/2009/05/29/san-emc-cx4-dae-drive-shelf-information/

As the conversation continued, I was told that typically the Service Processors caching would be disabled on LUNS that reside on the solid state flash drives. In all actuality, I think it should. Being that the typical Clariion implementation will not be dedicated as a solid state SAN, they will have to co-mingle with spindle based hard drives(regular). The throughput is a LOT faster and latency a lot lower on the flash drives. Enabling SP caching on LUNs contained in flash could possibly have a really bad affect on the rest of the system. Being that the cache really speeds up IO for spindle based disks.

Most vendors are toting a 20 physical disks(15kRPM) to 1 Flash disk consolidation

What I am getting at is there seems to be the possibility of flooding the Service Processors cache with flash based storage data, depending on change rate. Since flash is a lot faster, their LUNs would typically be used for databases or high volume services. Even with the ability to do QOS in the CX4 series, disabling Service Processor caching for the flash LUNS looks to be the best solution.

Posted in EMC, Hardware, SAN (Storage Area Network)

SAN / EMC: CX4 DAE (Drive Shelf) Information

Kevin Goodman — Sat, 30 May 2009 02:34:19 +0000

This will not get very detailed, but I figured I would share the following information. In light of not being happy with the typical “each shelf has a 4 Gig interconnect” statement, I kept checking until there was a better answer. So, anyone working with EMC SANs typically knows that every shelf is connected to each SP (Service Processor – 2 per SAN), daisy chained in a specific loop, and assigned a shelf id. Next is the LCC.

Each DAE contains 2 LCC interfaces. LCC is the acronymn for “Link Control Card”. Typically one LCC goes to each Service Processor. So that means that each DAE has two 4 gig links for a total of 8gb theoretical throughput.

This is where I will be reading more information. Just because you have 2 LCC cards, each LUN can only be assigned to 1 Service Processor! If my theory is correct, that means if all drives in one shelf is dedicated to a single LUN, and that LUN can only be active on one SP, does that mean that the actual throughput will be limited to 4gb instead of 8? This being due to the other LCC loop being connected to the SP that is not assigned control of that LUN.

Any thoughts? Feel free to comment!

Posted in EMC, Hardware, SAN (Storage Area Network)

SAN/Networking/Linux: Using Multipath To Verify and Troubleshoot Connectivity To FC LUNs

Kevin Goodman — Mon, 11 May 2009 13:33:11 +0000

A few days ago there was in error on one of our Cisco MDS 9120 fiber switches. The current environment at this datacenter consists of two Cisco MDS 9120 SAN switches with servers redundantly connected between the two. These switches are used to connect the servers to our fiber channel (FC) storage systems. In this case, the servers are generally mapped to an EMC, NetApp, and two RamSans. Below outlines a basic example of what can be expected from multipathing and the Linux environment when there is loss of connectivity to one leg of the fiber network.

Below is an excerpt of the log coming from a dual fiber connected server. In this case, the max active paths would be 2. If the server had quad fiber connectivity it could have a max of 4 paths in our environment. As seen from the log, multipathd reported the loss of a path to each LUN an logged it “remaining active paths: 1″. Once the errors were corrected in the failed switch, multipathd reported the restoration “remaining active paths: 2“.

[root@testsrv01 ~]# cat /var/log/messages | grep multipathd
May  6 10:30:55 testsrv01 multipathd: TempLUN1: remaining active paths: 1
May  6 10:30:55 testsrv01 multipathd: 8:224: readsector0 checker reports path is up
May  6 10:30:55 testsrv01 multipathd: 8:224: reinstated
May  6 10:30:55 testsrv01 multipathd: RanSanLUN2: remaining active paths: 1
May  6 10:30:56 testsrv01 multipathd: 65:128: readsector0 checker reports path is up
May  6 10:30:56 testsrv01 multipathd: 65:128: reinstated
May  6 10:30:57 testsrv01 multipathd: TempLUN1: remaining active paths: 2
May  6 10:30:58 testsrv01 multipathd: 65:224: readsector0 checker reports path is up
May  6 10:30:58 testsrv01 multipathd: 65:224: reinstated
May  6 10:30:58 testsrv01 multipathd: RanSanLUN2: remaining active paths: 2

To be safe, the ‘multipath’ command was used via the Command Line Interface (CLI) on the server(s) to verify that the storage systems were visable to the OS.

NAME
       multipath - Device mapper target autoconfig
SYNOPSIS
       multipath [options] [device]

       -l     show the current multipath topology from information fetched in sysfs and the device mapper
       -ll    show the current multipath topology from all available information (sysfs, the device mapper ...)

Truncated results are below that shows connectivity to the RamSans and EMC. This server is not mapped to the NetApp.

[root@testsrv01 ~]# multipath -ll
ramsan_DramCache (25050c24001094501)
[size=9 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:0  sdah 66:16  [active][ready]
 \_ 2:0:4:0  sdai 66:32  [active][ready]
 \_ 1:0:3:0  sdal 66:80  [active][ready]
 \_ 1:0:4:0  sdan 66:112 [active][ready]

ramsan_FlashSSD (25050c24021094501)
[size=1853 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:1  sdaj 66:48  [active][ready]
 \_ 2:0:4:1  sdak 66:64  [active][ready]
 \_ 1:0:3:1  sdam 66:96  [active][ready]
 \_ 1:0:4:1  sdao 66:128 [active][ready]

EMC_LUN1 (461a963004637699a05bs3464f76)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][active]
 \_ 2:0:2:8  sdab 65:176 [active][ready]
 \_ 1:0:2:8  sdl  8:176  [active][ready]

Notes: Make sure to verify that all servers have multipathing setup and working correctly, all of the Linux servers never missed a beat! The database (Oracle) and Websphere servers kept on going. The same can not be said about the Exchange environment. Due to a configuration problem there, multipathing was not functioning and it lost connectivity to its storage LUNs. The VMware cluster survived fine as well.

Posted in EMC, Filesystems, Linux, Monitoring, Networking, SAN (Storage Area Network), VMWare