General Specs:

Dual controllers
RAW 12tb
Cache (SSD) RAW 320gb
Quad 10gbE connectivity (2 per controller)

Below shows the Nimble device being un-boxed. I highlighted the SSD drives in green

Nimble Un-boxing

Even though the unit is 3ru in size, they were able to stuff two controllers (heads) into it. Primary is highlighted blue and secondary green.

A small gallery including the racked unit. Notice it is in good company!

Uncorrected ECC event detected on boards 4,6

The DRAM cache fronts the Flash storage drives in the unit.  Basically, loosing these two boards took down the unit completely.  To get the unit back up, boards 4 and 6 needed to be removed.  RamSan actually had me remove 4 of the boards instead of just the two that were bad.  Guessing it has something to do with not being able to have slots not filled between active modules.  
Doing so took the DRAM cache from 64 gigs down to 32.  All data was lost on the FLASH RAID array, making us have to restore the Oracle database from backup.  After the cards were removed, a low level format of the RAID array was done, the unit was operational again.   Below are pictures from within the RamSan 500 for all the storage people out there.

Best part of this is we had racked two RamSan 630′s that were being configured as a mirrored pair.  The plan was to implement them in two weeks.  When the 500 failed, they were implemented that night.

Once we got over the fear of FCoE reliability, almost everything is now being ordered with 10gbE. The only reason of worry was during this implementation over a year ago, FCoE was not in the main stream yet. It was almost “bleeding edge” in the market.

Where are we a year later?

One of the big killer of ports in our environment is having to tie the Gigabit Ethernet switches into them. This is wasting 4 ports that could be 10gb pre switch. Hopefully that will be resolved with a future purchase of Nexus 7000 switches.  We also have a pair of fabric extenders connected into the 5010s.  This also takes up two 10gbE ports per 5010.

Below is a port listing from one of our Nexus 5010 switches. This is after we removed 3 ports (per switch) from physical servers that were tied into the 10gbE infrastructure. As you can see from below, we only have 5 ports per switch left (10 total if you count the redundant switch)! That is only 5 more devices at the most.

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        x     eth  trunk  up      none                       1000(D) 1
Eth1/2        x     eth  trunk  up      none                       1000(D) 1
Eth1/3        x     eth  trunk  up      none                       1000(D) 1
Eth1/4        x     eth  trunk  up      none                       1000(D) 1
Eth1/5        x     eth  trunk  up      none                        10G(D) --
Eth1/6        x     eth  trunk  up      none                        10G(D) --
Eth1/7        x     eth  access up      none                        10G(D) --
Eth1/8        x     eth  access up      none                        10G(D) --
Eth1/9        x     eth  trunk  down    Link not connected          10G(D) --
Eth1/10       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/11       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/12       x     eth  access down    SFP not inserted            10G(D) --
Eth1/13       x     eth  access up      none                        10G(D) --
Eth1/14       x     eth  access down    SFP not inserted            10G(D) --
Eth1/15       x     eth  trunk  up      none                        10G(D) --
Eth1/16       x     eth  trunk  up      none                        10G(D) --
Eth1/17       x     eth  fabric up      none                        10G(D) --
Eth1/18       x     eth  fabric up      none                        10G(D) --
Eth1/19       x     eth  trunk  up      none                        10G(D) 2
Eth1/20       x     eth  trunk  up      none                        10G(D) 2

At the 6120′s end (UCS switches) we are fine. Currently have 14 ports on each switch available. That should give us the ability to wire in 7 more chassis with 2 – 10gbE uplinks per IOM. A total of 4 links / 40gb throughput per chassis. Our CIFS and NFS access is a lot faster running through the EMC NX4′s 10gbE.

So if you are looking into 10gbE for our network, it would help to have at least a rough draft of where you want your datacenter to be a few years from now. Pretty much all major storage companies are doing some sort of 10gbE connectivity. Using it makes administration a lot easier. FCoE simplifies things even more, including eliminating the need for separate fiber switches. To sum it up, I love 10gbE and would recommend it even if not planning on incorporating FCoE or UCS in your environment.

So can someone explain to me why EMC sells systems with non-matching face plates.  Like our RecoverPoint systems attached to our CX4 actually have CX3 style faceplates.  Same with the NX4, the 3 1RU components at the bottom faceplates are completely different than the rest.  Either way, it still is a nice system.

Also to note, the tests were done with blades using FCoE boot from SAN to a fiber connected EMC CX4-240 Clariion. The storage on the CX4 supporting these and 10 other boot from SAN blades LUNs is as follows:

Qty 4 – 600gb 10kRPM disks setup in a RAID 10 group.

The following firmware versions are the same between both blades being tested:

Storage path: FCoE through Palo interface -> Cisco Nexus 6120 -> fiber -> EMC CX4

Test 1 Consists of a B200-M1 blade with the following configuration

Bootup and shutdown:

Service Profile associations:

Test 2 consists of a B200-M1 blade with the following configuration

Bootup and shutdown:

Service Profile associations:

Note(s): The testing was done with blades that were already at the service profiles required firmware level.  More time would have to be added if a firmware change was needed. Also, boot times will be dependent on the IOPs achievalbe to your storage sytem.

Some specs on the system:

RamSan Box

Turns out that the hot-spare flash module was shipped separately so I had to install that myself.  Pictures from the racking and module installation is below.

Below are a few videos of the equipment.  Be careful with the sound!  These were taken in the datacenter and you will hear the typical rumbling of servers and AC units.

Video of the NetApp FAS 3020 once removed

Video of the EMC NX4 waiting to go in

DD01# filesys show compression /backup/db/xxx/xxx/xx6e_1_1
Total files: 1;  bytes/storage_used: 59.1
       Original Bytes:       16,252,249,684
  Globally Compressed:          954,839,915
   Locally Compressed:          272,055,847
            Meta-data:            3,009,088

bytes/storage_used: 59.1 =	Compression and Deduplication ratio
Globally Compressed Factor = 	Pre-Comp / (Size after de-dupe)
Locally Compressed Factor = 	(Size after de-dupe) / Post-Comp

From above you should be getting an idea of the deduplication, but lets follow DataDomain’s formula

Reduction % = ((Pre-Comp – Post-Comp) / Pre-Comp) * 100

Now put in values from our above example

16,252,249,684 – 272,055,847 / 16,252,249,684 * 100 = 98.326%

That gives us a 98.326% reduction!

Basically what I am trying to say is that if you are not doing some sort of deduplication + compression on your Oracle backups, you should look into it.

Note(s): The above information is from a system running Data Domain OS 4.8.1.0-175623.  Also, remember that deduplication savings grow as the same types of blocks are written.  This system has been in production for over a year.  We have about 1tb raw of Oracle data being written to the DataDomain which helps the dedupe ratio.

Below is an overview of the DataDomain 565 we have at our “Production” datacenter.

Production DataDomain 565
24hr max pre-compressed           =          1869.5
30 day max pre-compressed       =          4278.81
60 day max pre-compressed       =          4387.35
120 day max pre-compressed     =          4922.56
* above is in gigabyte(s)

So with that in mind, we prefer to hit an 8hr backup window.  We currently are not doing that now with our 565 but that is our goal.

Max of 4922.56 / 8hr window       =          ~615.32 gb/hr

615.32 gigabyte = 4 922.56 gigabit
10gbE = 36,000 gigabit an hour

So from above, it would seem that we could theoretically hit our window if using 10gbE

We are using ~4236.5 gig used after deduplication is done I doing so, would be pushing the throughput and maximum capacity limitations of the DD610.  The DD630 would allow ~6tb worth of expansion and we would meet an 8 hour backup window. With the above information, you can see we want to go with 10gbE connectivity.  I currently love our Cisco UCS and 10gb network and plan to hook in all critical infrastructure to it via 10gbE.

This was our main limitation here! * Only the DD670 offers 10gbE connectivity (fiber)

So would the DD630 work for us?  Yes it would, but who wants to trunk a bunch of gigE ports when you have a 10gb network already?  Unless the cost point goes up horrifically, the DD670 would be a better fit in our datacenter.

DataDomain 6xx Overview

Notes: We could expand our DD565 another shelf, but we are getting low on storage at the off-site replication locations for the DataDomains.  Win – win but buying a new unit and shipping the 565 to increase capacity off-site.  DD510/610s can not expand beyond their initial shelf.

Just some technical specs on this model:

- 7 DAEs (drive trays)
- Total of 105 drives / spindles
- 2 gigabit fiber connectivity

So approximately 13 terabytes is still a decent amount.  Taken into account that we ram mainly RAID 10 on this array, that usable space dwindled dramatically.  Of course, this was built at a time where the industry was still throwing disks “spindles” at Oracle databases.  Back then there was no “enterprise” solid state in the mainstream market.

Having severed it’s purpose as first a “production” services storage array, it was replaced by a CX3-40.  At that time, the CX500 was moved to a disaster recovery site and used there for it’s primary storage.  Within about a year the CX3-40 was replaced at production site with a CX4-240.  Since then, the CX3-40 was moved to the disaster recover site as primary storage.  The CX500 had its data migrated off to either the RamSan 630 there or the EMC CX3-40.  We have now almost outgrown our CX4-240 and soon it’s fate might be sealed as well.

Hope you liked the above history lesson and now comes the pictures.  Also please note that these DAEs are full length DAEs, not the smaller shelves like the CX3+ series.  Hence the reason why Bill (co-worker) came to help!

Notes: Check out the popped (flat) tire on the cart.  The Trailblazer was definitely riding low on the way home.  Didn’t want the rack rails messing up my trunk, so those were all put in the back seat.  Surprisingly, I was able to fit the whole SAN into the back compartment without folding the rear seats down.

Since then I have installed a RamSan for the Oracle environment at this datacenter and have no need for a lot of tiny drives.  So we replaced them with 600gig 10kRPM drives to be used in our VMware environment and RecoverPoint systems.  How about some EMC drive replacement pics until I can get some more content available?

Funny thing is that I forgot to take pictures of the drives once installed.  Might add them next trip/

Now all that we need to do is get some data over to it.  We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit.  You can not change the order in which each device is racked.  Also, the controllers (SPs) are built into the drive tray.  Best part is that this will be wired up via 10gbE connectivity over fiber.

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

I will be using “naviseccli” for this, but the same syntax should work with “navicli”. Both are available through EMC Powerlink site.

Below will query the Clariion and return a list of all LUNs that are trespassed. The -h specifies which host (SP) to connect to. You only need to run this on one SP. The other would report back the exact same results.

C:\>naviseccli -h 127.0.0.12 getlun -trespass
LOGICAL UNIT NUMBER 7
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 15
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 118
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 17
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 204
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 619
Default Owner:              SP B
Current owner:              SP A

....

I have some Linux style utilities installed on my Windows machine. Below shows that we have a total of 55 LUNs that are currently trespassed. Yeah, don’t ask.

C:\>naviseccli -h 127.0.0.13 getlun -trespass | grep LOGICAL | wc -l
     55

I thought that a list of all LUN IDs would be needed and then have to make a script to run through them. Luckily that is not the case. The syntax is as follows:C:\>naviseccli -h trespass mine

The above command has to be run on both SPs (Service Processors).
C:\>naviseccli -h 127.0.0.12 trespass mine

C:\>naviseccli -h 127.0.0.13 trespass mine

Use the “getlun -trespass” command once again to make sure that there are no more trespassed LUNs. A few things to keep in mind are that trespassed snapshots will stay trespassed. CLI or even the Navisphere GUI will not let you move them. Also, I had a few LUNs that would not move via the CLI, but when done manually in Navisphere they moved fine.

Below is the output of help for naviseccli.
C:\>naviseccli -help

@(#)Navisphere naviseccli Revision 6.29.5.0.66 on Wed Jan 20 22:47:06 2010
Copyright (C) 1997-2009, EMC Corporation
  Usage:
    [-User ] [-Password]
    [-Scope <0 - global; 1 - local; 2 - LDAP>]
    [-Address  | -h ]
    [-Port] [-Timeout  | -t ]
    [-AddUserSecurity | -RemoveUserSecurity | -DeleteSecurityEntry]
    [-Parse | -p] [-NoPoll | -np] [-cmdtime]
    [-Xml] [-f ] [-Help] CMD

If you were using “navicli” the following commands should work for you:

navicli -h 127.0.0.12 getlun -trespass
navicli -h 127.0.0.12 trespass mine
navicli -h 127.0.0.13 trespass mine

Notes: Feel free to comment / email (kevin <at> colovirt.com) / twitter me with any questions!