Once we got over the fear of FCoE reliability, almost everything is now being ordered with 10gbE. The only reason of worry was during this implementation over a year ago, FCoE was not in the main stream yet. It was almost “bleeding edge” in the market.

Where are we a year later?

One of the big killer of ports in our environment is having to tie the Gigabit Ethernet switches into them. This is wasting 4 ports that could be 10gb pre switch. Hopefully that will be resolved with a future purchase of Nexus 7000 switches.  We also have a pair of fabric extenders connected into the 5010s.  This also takes up two 10gbE ports per 5010.

Below is a port listing from one of our Nexus 5010 switches. This is after we removed 3 ports (per switch) from physical servers that were tied into the 10gbE infrastructure. As you can see from below, we only have 5 ports per switch left (10 total if you count the redundant switch)! That is only 5 more devices at the most.

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        x     eth  trunk  up      none                       1000(D) 1
Eth1/2        x     eth  trunk  up      none                       1000(D) 1
Eth1/3        x     eth  trunk  up      none                       1000(D) 1
Eth1/4        x     eth  trunk  up      none                       1000(D) 1
Eth1/5        x     eth  trunk  up      none                        10G(D) --
Eth1/6        x     eth  trunk  up      none                        10G(D) --
Eth1/7        x     eth  access up      none                        10G(D) --
Eth1/8        x     eth  access up      none                        10G(D) --
Eth1/9        x     eth  trunk  down    Link not connected          10G(D) --
Eth1/10       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/11       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/12       x     eth  access down    SFP not inserted            10G(D) --
Eth1/13       x     eth  access up      none                        10G(D) --
Eth1/14       x     eth  access down    SFP not inserted            10G(D) --
Eth1/15       x     eth  trunk  up      none                        10G(D) --
Eth1/16       x     eth  trunk  up      none                        10G(D) --
Eth1/17       x     eth  fabric up      none                        10G(D) --
Eth1/18       x     eth  fabric up      none                        10G(D) --
Eth1/19       x     eth  trunk  up      none                        10G(D) 2
Eth1/20       x     eth  trunk  up      none                        10G(D) 2

At the 6120′s end (UCS switches) we are fine. Currently have 14 ports on each switch available. That should give us the ability to wire in 7 more chassis with 2 – 10gbE uplinks per IOM. A total of 4 links / 40gb throughput per chassis. Our CIFS and NFS access is a lot faster running through the EMC NX4′s 10gbE.

So if you are looking into 10gbE for our network, it would help to have at least a rough draft of where you want your datacenter to be a few years from now. Pretty much all major storage companies are doing some sort of 10gbE connectivity. Using it makes administration a lot easier. FCoE simplifies things even more, including eliminating the need for separate fiber switches. To sum it up, I love 10gbE and would recommend it even if not planning on incorporating FCoE or UCS in your environment.

So can someone explain to me why EMC sells systems with non-matching face plates.  Like our RecoverPoint systems attached to our CX4 actually have CX3 style faceplates.  Same with the NX4, the 3 1RU components at the bottom faceplates are completely different than the rest.  Either way, it still is a nice system.

Also to note, the tests were done with blades using FCoE boot from SAN to a fiber connected EMC CX4-240 Clariion. The storage on the CX4 supporting these and 10 other boot from SAN blades LUNs is as follows:

Qty 4 – 600gb 10kRPM disks setup in a RAID 10 group.

The following firmware versions are the same between both blades being tested:

Storage path: FCoE through Palo interface -> Cisco Nexus 6120 -> fiber -> EMC CX4

Test 1 Consists of a B200-M1 blade with the following configuration

Bootup and shutdown:

Service Profile associations:

Test 2 consists of a B200-M1 blade with the following configuration

Bootup and shutdown:

Service Profile associations:

Note(s): The testing was done with blades that were already at the service profiles required firmware level.  More time would have to be added if a firmware change was needed. Also, boot times will be dependent on the IOPs achievalbe to your storage sytem.

Below are a few videos of the equipment.  Be careful with the sound!  These were taken in the datacenter and you will hear the typical rumbling of servers and AC units.

Video of the NetApp FAS 3020 once removed

Video of the EMC NX4 waiting to go in

DD01# filesys show compression /backup/db/xxx/xxx/xx6e_1_1
Total files: 1;  bytes/storage_used: 59.1
       Original Bytes:       16,252,249,684
  Globally Compressed:          954,839,915
   Locally Compressed:          272,055,847
            Meta-data:            3,009,088

bytes/storage_used: 59.1 =	Compression and Deduplication ratio
Globally Compressed Factor = 	Pre-Comp / (Size after de-dupe)
Locally Compressed Factor = 	(Size after de-dupe) / Post-Comp

From above you should be getting an idea of the deduplication, but lets follow DataDomain’s formula

Reduction % = ((Pre-Comp – Post-Comp) / Pre-Comp) * 100

Now put in values from our above example

16,252,249,684 – 272,055,847 / 16,252,249,684 * 100 = 98.326%

That gives us a 98.326% reduction!

Basically what I am trying to say is that if you are not doing some sort of deduplication + compression on your Oracle backups, you should look into it.

Note(s): The above information is from a system running Data Domain OS 4.8.1.0-175623.  Also, remember that deduplication savings grow as the same types of blocks are written.  This system has been in production for over a year.  We have about 1tb raw of Oracle data being written to the DataDomain which helps the dedupe ratio.

Below is an overview of the DataDomain 565 we have at our “Production” datacenter.

Production DataDomain 565
24hr max pre-compressed           =          1869.5
30 day max pre-compressed       =          4278.81
60 day max pre-compressed       =          4387.35
120 day max pre-compressed     =          4922.56
* above is in gigabyte(s)

So with that in mind, we prefer to hit an 8hr backup window.  We currently are not doing that now with our 565 but that is our goal.

Max of 4922.56 / 8hr window       =          ~615.32 gb/hr

615.32 gigabyte = 4 922.56 gigabit
10gbE = 36,000 gigabit an hour

So from above, it would seem that we could theoretically hit our window if using 10gbE

We are using ~4236.5 gig used after deduplication is done I doing so, would be pushing the throughput and maximum capacity limitations of the DD610.  The DD630 would allow ~6tb worth of expansion and we would meet an 8 hour backup window. With the above information, you can see we want to go with 10gbE connectivity.  I currently love our Cisco UCS and 10gb network and plan to hook in all critical infrastructure to it via 10gbE.

This was our main limitation here! * Only the DD670 offers 10gbE connectivity (fiber)

So would the DD630 work for us?  Yes it would, but who wants to trunk a bunch of gigE ports when you have a 10gb network already?  Unless the cost point goes up horrifically, the DD670 would be a better fit in our datacenter.

DataDomain 6xx Overview

Notes: We could expand our DD565 another shelf, but we are getting low on storage at the off-site replication locations for the DataDomains.  Win – win but buying a new unit and shipping the 565 to increase capacity off-site.  DD510/610s can not expand beyond their initial shelf.

Just some technical specs on this model:

- 7 DAEs (drive trays)
- Total of 105 drives / spindles
- 2 gigabit fiber connectivity

So approximately 13 terabytes is still a decent amount.  Taken into account that we ram mainly RAID 10 on this array, that usable space dwindled dramatically.  Of course, this was built at a time where the industry was still throwing disks “spindles” at Oracle databases.  Back then there was no “enterprise” solid state in the mainstream market.

Having severed it’s purpose as first a “production” services storage array, it was replaced by a CX3-40.  At that time, the CX500 was moved to a disaster recovery site and used there for it’s primary storage.  Within about a year the CX3-40 was replaced at production site with a CX4-240.  Since then, the CX3-40 was moved to the disaster recover site as primary storage.  The CX500 had its data migrated off to either the RamSan 630 there or the EMC CX3-40.  We have now almost outgrown our CX4-240 and soon it’s fate might be sealed as well.

Hope you liked the above history lesson and now comes the pictures.  Also please note that these DAEs are full length DAEs, not the smaller shelves like the CX3+ series.  Hence the reason why Bill (co-worker) came to help!

Notes: Check out the popped (flat) tire on the cart.  The Trailblazer was definitely riding low on the way home.  Didn’t want the rack rails messing up my trunk, so those were all put in the back seat.  Surprisingly, I was able to fit the whole SAN into the back compartment without folding the rear seats down.

Since then I have installed a RamSan for the Oracle environment at this datacenter and have no need for a lot of tiny drives.  So we replaced them with 600gig 10kRPM drives to be used in our VMware environment and RecoverPoint systems.  How about some EMC drive replacement pics until I can get some more content available?

Funny thing is that I forgot to take pictures of the drives once installed.  Might add them next trip/

Now all that we need to do is get some data over to it.  We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit.  You can not change the order in which each device is racked.  Also, the controllers (SPs) are built into the drive tray.  Best part is that this will be wired up via 10gbE connectivity over fiber.

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

I will be using “naviseccli” for this, but the same syntax should work with “navicli”. Both are available through EMC Powerlink site.

Below will query the Clariion and return a list of all LUNs that are trespassed. The -h specifies which host (SP) to connect to. You only need to run this on one SP. The other would report back the exact same results.

C:\>naviseccli -h 127.0.0.12 getlun -trespass
LOGICAL UNIT NUMBER 7
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 15
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 118
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 17
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 204
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 619
Default Owner:              SP B
Current owner:              SP A

....

I have some Linux style utilities installed on my Windows machine. Below shows that we have a total of 55 LUNs that are currently trespassed. Yeah, don’t ask.

C:\>naviseccli -h 127.0.0.13 getlun -trespass | grep LOGICAL | wc -l
     55

I thought that a list of all LUN IDs would be needed and then have to make a script to run through them. Luckily that is not the case. The syntax is as follows:C:\>naviseccli -h trespass mine

The above command has to be run on both SPs (Service Processors).
C:\>naviseccli -h 127.0.0.12 trespass mine

C:\>naviseccli -h 127.0.0.13 trespass mine

Use the “getlun -trespass” command once again to make sure that there are no more trespassed LUNs. A few things to keep in mind are that trespassed snapshots will stay trespassed. CLI or even the Navisphere GUI will not let you move them. Also, I had a few LUNs that would not move via the CLI, but when done manually in Navisphere they moved fine.

Below is the output of help for naviseccli.
C:\>naviseccli -help

@(#)Navisphere naviseccli Revision 6.29.5.0.66 on Wed Jan 20 22:47:06 2010
Copyright (C) 1997-2009, EMC Corporation
  Usage:
    [-User ] [-Password]
    [-Scope <0 - global; 1 - local; 2 - LDAP>]
    [-Address  | -h ]
    [-Port] [-Timeout  | -t ]
    [-AddUserSecurity | -RemoveUserSecurity | -DeleteSecurityEntry]
    [-Parse | -p] [-NoPoll | -np] [-cmdtime]
    [-Xml] [-f ] [-Help] CMD

If you were using “navicli” the following commands should work for you:

navicli -h 127.0.0.12 getlun -trespass
navicli -h 127.0.0.12 trespass mine
navicli -h 127.0.0.13 trespass mine

Notes: Feel free to comment / email (kevin <at> colovirt.com) / twitter me with any questions!

First, copy the RPM installation package over to the server. Below shows the package to be installed.

[root@test_server01 user01]# ll
total 7036
-rw-r--r-- 1 user01 user01 7191661 Apr 27 09:24 EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm

Install the package via “rpm -i”.

[root@test_server01 user01]# rpm -i EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm
All trademarks used herein are the property of their respective owners.
NOTE:License registration is not required to manage the CLARiiON AX series array.

Before powerpath can be used, a license key must be installed.

[root@test_server01 user01]# emcpreg -list
unable to open license key file: No such file or directory

Overview of the “emcpreg -add” syntax.

[root@test_server01 user01]# emcpreg -add
Missing option parameter.
Usage:
    emcpreg [opts] -add key [key ...]
    emcpreg [opts] -remove key [key ...]
    emcpreg [opts] -check key [key ...]
    emcpreg [opts] -list
    emcpreg [opts] -edit
    emcpreg [opts] -install
Options:
    -f file     license file

Now we add the license key to powerpath. The following key is fake! You must obtain yours from EMC.

[root@test_server01 user01]# emcpreg -add AGE4-DFD3-89842-DSAF-JIJ0-WKG50
1 key(s) successfully added.

Make sure the license was installed correctly.
[root@test_server01 user01]# emcpreg -list

Key AGE4-DFD3-89842-DSAF-JIJ0-WKG50
  Product: PowerPath
  Capabilities: All
[root@test_server01 user01]#

Next, start the Power Path service.

[root@test_server01 user01]# /etc/init.d/PowerPath start
Starting PowerPath:  done

Display the current paths to storage via “powermt”. Since this server is booting from SAN and just being installed, there is currently only one path to storage.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=Unknown, current=SP A    Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sda       SP A0     active  alive      0      0

Now that powerpath is installed, we need to edit fstab to boot off of the Power Path device.

Origional fstab using labels for “/boot”.

/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Below is the edited fstab with “LABEL=/boot” commented out and /boot changed to use “/dev/emcpowera1″

[root@test_server01 ~]# vi /etc/fstab
/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
/dev/emcpowera1         /boot                   ext3    defaults        0 0
#LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Filesystem options were changed to “0 0″ on emcpowera due to RedHat trying to do filesystem scans before the Power Path driver is started.

All paths now need to be zoned in the fiber switch, initiators registered in Navisphere, and paths added to the host in it’s storage group. This will not be covered here.

After zoning both paths on one switch, “powermt” now shows a path to both Clariion SPA and SPB. If not, try either stopping and restartaring “/etc/init.d/PowerPath” or restarting the server.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP A       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0

Configuration has now been completed on fiber switch 2 and both SPA and SPB in the Clariion. Reboot the server. Again, “powermt” is used to list the paths.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0
   1 fnic                      sde       SP B0     active  alive      0      0
   1 fnic                      sdf       SP A1     active  alive      0      0

From above, you can see that we now have 4 paths definied. Both fnic interfaces can see SPA and SPB. Each fnic is attached to a seperage fiber switch, so we have redundant paths to both Clariion heads (SP’s). Once rebooted, the server should load fine with no issues and see all paths via powermt.

Notes:  “/boot” is the storage label used in this example.  If your mount point is different, modify it’s entry instead.  “/dev/emcpowera1″ is used since there is only one LUN mapped to this host.  Like anything else, if there are more than one, each would have it’s own device.

Per the title, this is really about the EMC Clariion, not RamSans.  Since the RamSan 500 was fronted with the DRAM cache, and the EMC CX4 series contains cache as well, I was curious.  I already knew that each Service Processor (SP) in the EMC has 4 gig of cache, and that a LUN can only be active on one SP at a time.  Also, per a previous blog post, each DAE has a theoretical max throughput of 8 gigabit per second, 4 gigabit if a single LUN stripes across the whole shelf.

CX4 DAE (general) information
http://blog.colovirt.com/2009/05/29/san-emc-cx4-dae-drive-shelf-information/

As the conversation continued, I was told that typically the Service Processors caching would be disabled on LUNS that reside on the solid state flash drives.  In all actuality, I think it should.  Being that the typical Clariion implementation will not be dedicated as a solid state SAN, they will have to co-mingle with spindle based hard drives(regular).  The throughput is a LOT faster and latency a lot lower on the flash drives.  Enabling SP caching on LUNs contained in flash could possibly have a really bad affect on the rest of the system.  Being that the cache really speeds up IO for spindle based disks.

Most vendors are toting a 20 physical disks(15kRPM) to 1 Flash disk consolidation

What I am getting at is there seems to be the possibility of flooding the Service Processors cache with flash based storage data, depending on change rate.  Since flash is a lot faster, their LUNs would typically be used for databases or high volume services.  Even with the ability to do QOS in the CX4 series, disabling Service Processor caching for the flash LUNS looks to be the best solution.

Each DAE contains 2 LCC interfaces.  LCC is the acronymn for “Link Control Card”.  Typically one LCC goes to each Service Processor.  So that means that each DAE has two 4 gig links for a total of 8gb theoretical throughput.

This is where I will be reading more information.  Just because you have 2 LCC cards, each LUN can only be assigned to 1 Service Processor!  If my theory is correct, that means if all drives in one shelf is dedicated to a single LUN, and that LUN can only be active on one SP, does that mean that the actual throughput will be limited to 4gb instead of 8?  This being due to the other LCC loop being connected to the SP that is not assigned control of that LUN.

Any thoughts?  Feel free to comment!