Colocation to Virtualization » cx4

10gb Ethernet – A Year Later / Buy It Carefully

Kevin Goodman — Wed, 16 Feb 2011 13:00:27 +0000

What I am getting at here is do not underestimate the amount of ports you will be using. We have implemented Cisco UCS with 6120 switches but also installed a pair of Nexus 5010 switches as well. Going in, we had only planned for a few devices to be connected to the 10gb Ethernet network outside of the UCS infrastructure. Looks like we underestimated!

Once we got over the fear of FCoE reliability, almost everything is now being ordered with 10gbE. The only reason of worry was during this implementation over a year ago, FCoE was not in the main stream yet. It was almost “bleeding edge” in the market.

Where are we a year later?

We are expanding our Cisco UCS environment to at least an additional chassis

EMC CX-4 has 10gbE fiber modules for ISCSI (instead of RDMs for some VMs)

DataDomain 670 is 10gbE fiber connected

EMC NX4 NAS has 10gbE fiber connectivity

We are planning a purchase of either a CX480 or VNX 5700 which will be 10gb FCoE

Hoping to order a pair of Cisco Nexus 7000s

One of the big killer of ports in our environment is having to tie the Gigabit Ethernet switches into them. This is wasting 4 ports that could be 10gb pre switch. Hopefully that will be resolved with a future purchase of Nexus 7000 switches. We also have a pair of fabric extenders connected into the 5010s. This also takes up two 10gbE ports per 5010.

Below is a port listing from one of our Nexus 5010 switches. This is after we removed 3 ports (per switch) from physical servers that were tied into the 10gbE infrastructure. As you can see from below, we only have 5 ports per switch left (10 total if you count the redundant switch)! That is only 5 more devices at the most.

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        x     eth  trunk  up      none                       1000(D) 1
Eth1/2        x     eth  trunk  up      none                       1000(D) 1
Eth1/3        x     eth  trunk  up      none                       1000(D) 1
Eth1/4        x     eth  trunk  up      none                       1000(D) 1
Eth1/5        x     eth  trunk  up      none                        10G(D) --
Eth1/6        x     eth  trunk  up      none                        10G(D) --
Eth1/7        x     eth  access up      none                        10G(D) --
Eth1/8        x     eth  access up      none                        10G(D) --
Eth1/9        x     eth  trunk  down    Link not connected          10G(D) --
Eth1/10       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/11       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/12       x     eth  access down    SFP not inserted            10G(D) --
Eth1/13       x     eth  access up      none                        10G(D) --
Eth1/14       x     eth  access down    SFP not inserted            10G(D) --
Eth1/15       x     eth  trunk  up      none                        10G(D) --
Eth1/16       x     eth  trunk  up      none                        10G(D) --
Eth1/17       x     eth  fabric up      none                        10G(D) --
Eth1/18       x     eth  fabric up      none                        10G(D) --
Eth1/19       x     eth  trunk  up      none                        10G(D) 2
Eth1/20       x     eth  trunk  up      none                        10G(D) 2

At the 6120′s end (UCS switches) we are fine. Currently have 14 ports on each switch available. That should give us the ability to wire in 7 more chassis with 2 – 10gbE uplinks per IOM. A total of 4 links / 40gb throughput per chassis. Our CIFS and NFS access is a lot faster running through the EMC NX4′s 10gbE.

So if you are looking into 10gbE for our network, it would help to have at least a rough draft of where you want your datacenter to be a few years from now. Pretty much all major storage companies are doing some sort of 10gbE connectivity. Using it makes administration a lot easier. FCoE simplifies things even more, including eliminating the need for separate fiber switches. To sum it up, I love 10gbE and would recommend it even if not planning on incorporating FCoE or UCS in your environment.

Filed under: Cisco UCS, Data Domain, Datacenter, deduplication, EMC, NAS, SAN (Storage Area Network)

Cisco UCS and Nexus 5000 DataCenter – Our Implementation

Kevin Goodman — Wed, 07 Jul 2010 19:07:07 +0000

Our Cisco 10gbE implementation consists of 2 Chassis fully populated UCS with a mix of full and half width blades. The servers are all boot from SAN with no local disks. “PALO” cards are used in all servers which allow us to do FCOE. 7 of the blades are running VMware ESX 4 (vSphere) and the rest are a mix of RedHat Linux and Windows 2008.

Storage:

All servers boot from SAN over FCOE

Primary storage is via Fibre Channel to an EMC CX4-240

A few servers are also connected via Fibre Channel to RamSan SSD SANs

The CX4-240 is also connected to the UCS environment via quad 10gbE links over fiberNetwork:

Core 10gbE switching is provided via Cisco Nexus 5010s

1 Gigabit Ethernet to legacy servers are provided by dual 48 port C2148 Fabric Extenders

Each C2148 is connected to a Nexus 5010 by dual 10gbE uplinks

Each 5010 is connected via quad 4gig fiber to the CX4-240 to provide FCoE to physical servers outside of the UCS environment

Each 5010 is also connected to the CX4-240 by dual 10gbE fiber ports to provide ISCSI to all of the network

Each UCS switch (6120s) are uplinked to the 5010s by dual 10gbE interfaces

Each UCS chassis is uplinked to the 6120s by 8 qty 10gbE ports (4 connections per IO module)

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon. This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120. I definitely would love feedback on this. How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Filed under: Colocation, EMC, Hardware, Networking, RamSan, SAN (Storage Area Network)

Storage, SAN: EMC Clariion LUN Trespass

Kevin Goodman — Mon, 14 Jun 2010 10:31:50 +0000

So we are in the process of working with EMC to have 10gb ISCSI interfaces installed in our CX4. I knew that we had a lot of trespassed LUNs and that those had to be corrected before the install could take place. This post will go through the process of using the CLI (Command Line Interface) to find out what is trespassed and to move them back.

I will be using “naviseccli” for this, but the same syntax should work with “navicli”. Both are available through EMC Powerlink site.

Below will query the Clariion and return a list of all LUNs that are trespassed. The -h specifies which host (SP) to connect to. You only need to run this on one SP. The other would report back the exact same results.

C:\>naviseccli -h 127.0.0.12 getlun -trespass
LOGICAL UNIT NUMBER 7
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 15
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 118
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 17
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 204
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 619
Default Owner:              SP B
Current owner:              SP A

....

I have some Linux style utilities installed on my Windows machine. Below shows that we have a total of 55 LUNs that are currently trespassed. Yeah, don’t ask.

C:\>naviseccli -h 127.0.0.13 getlun -trespass | grep LOGICAL | wc -l
     55

I thought that a list of all LUN IDs would be needed and then have to make a script to run through them. Luckily that is not the case. The syntax is as follows:C:\>naviseccli -h trespass mine

The above command has to be run on both SPs (Service Processors).
C:\>naviseccli -h 127.0.0.12 trespass mine

C:\>naviseccli -h 127.0.0.13 trespass mine

Use the “getlun -trespass” command once again to make sure that there are no more trespassed LUNs. A few things to keep in mind are that trespassed snapshots will stay trespassed. CLI or even the Navisphere GUI will not let you move them. Also, I had a few LUNs that would not move via the CLI, but when done manually in Navisphere they moved fine.

Below is the output of help for naviseccli.
C:\>naviseccli -help

@(#)Navisphere naviseccli Revision 6.29.5.0.66 on Wed Jan 20 22:47:06 2010
Copyright (C) 1997-2009, EMC Corporation
  Usage:
    [-User ] [-Password]
    [-Scope <0 - global; 1 - local; 2 - LDAP>]
    [-Address  | -h ]
    [-Port] [-Timeout  | -t ]
    [-AddUserSecurity | -RemoveUserSecurity | -DeleteSecurityEntry]
    [-Parse | -p] [-NoPoll | -np] [-cmdtime]
    [-Xml] [-f ] [-Help] CMD

If you were using “navicli” the following commands should work for you:

navicli -h 127.0.0.12 getlun -trespass
navicli -h 127.0.0.12 trespass mine
navicli -h 127.0.0.13 trespass mine

Notes: Feel free to comment / email (kevin colovirt.com) / twitter me with any questions!

Filed under: EMC, SAN (Storage Area Network)

Storage, Network: What I have Been Doing (EMC,Cisco UCS)

Kevin Goodman — Tue, 02 Mar 2010 19:41:19 +0000

This is more of an informational update of things that I have going on right now. I normally do not publish day-to-day type of things, but here we go.

Storage

We have received both replacement drives for our EMC Clariion CX340 and four new DAEs (disk shelves) for our CX4-240

Clariion CX3

The CX3 was originally bought to speed up our Oracle implementation. This was accomplished by ordering lots of fast disks (spindles) that were small. We wound up with 6 DAEs filled with 73gig 15kRPM disks, totalling 90 dedicated drives for Oracle.

This was great for the original purpose but the unit was replaced a year after initial deployment with a RamSan and EMC CX4. Having been decommissioned from production and moved to the tier 2 site, the need for space over IOPS (speed) drastically increased. Trying to keep performance and space requirements in balance, the decision has been made to go with a smaller RamSan for Oracle at the tier 2 site. This gives us the ability to replace the small 73 gigabyte drives with bigger 600 gigabyte 10kRPM disks. Replacing those disk with the same quantity of 600 gig ones will give us ~8 times as much space.

The RamSan will almost double the IOPS capacity that the CX3 is able to achieve and speed up our data warehouse even more.

Clariion CX4

So last year we went with implementing EMC Recoverpoint SAN based replication. This has been great and served us well! The only downfall was that we were doing “CRR” remote replication only. In a case of a failure and data needed to be recovered, there were no local copies. The snapshot or “point in time” would have to be loaded from the tier 2 site and transferred across the datacenter interconnect. The interconnect being 150 megs slowed this process down.

As planned from the beginning, we are implementing “CLR” local replication as well. This means that there will be a local copy of snapshots saved locally to the CX4. This will give us almost immediate access to the snapshots without being slowed down by the interconnect. The problem with RecoverPoint is that if you have a terrabyte LUN that you want to connect, you must have an extra terrabyte worth of space to save it. This is not really a problem, but a major consideration on the number of drives to buy and the overall expense of the implementation.

In our case, a terrabyte oracle LUN will wind up costing 3 terrabytes in the end. 1 terrabyte for the original data, 1 terrabyte for the local copy (CLR), and 1 terrabyte at the remote tier 2 site (CX3).

Our virtualization effort is continuing and this is another huge factor on the storage expansion. Currently we have 16 LUNs dedicated to the VMware environment. Each is 320 gigs in size. Moving forward, we will be doing a virtual desktop deployment as well. The leftover ~400 gigs will not cut it. So in the new 60 disks, 15 or more will have to be dedicated to VMware.

Cisco UCS

We have begun our UCS voyage. As of last weekend, we did a “rip and replace of our network”. This included rewiring the main network rack and configuring a new network core. Also, the Cisco Nexus 5010, 10 gigabit Ethernet switches are in. Uplinked to them are two 48 port gigabit fabric extenders.

The VMware environment is now connected via dual 10gigE links per server through this infrastructure. Reducing the cable count from 6 to 2 per server. So far verything is stable! A purchase order has been sent out and we should hopefully have two Cisco UCS Blade chassis and switching infrastructure show up within about 30 days.

Filed under: EMC, Networking, RamSan, SAN (Storage Area Network), VMWare

SAN / EMC: Clariion CX4 Solid State DAEs (Shelves)

Kevin Goodman — Wed, 03 Jun 2009 07:59:20 +0000

Going over the solid state offerings for the EMC Clariion lines, Texas Memory RamSans came into the conversation. This was due to the fact that we currently run 2 RamSans in our Environment and consider them the highest tier storage in our datacenters. One is 128 gigs of solid state DRAM storage and the other is 2 terabyte solid state Flash storage with a 64 gig DRAM cache.

Per the title, this is really about the EMC Clariion, not RamSans. Since the RamSan 500 was fronted with the DRAM cache, and the EMC CX4 series contains cache as well, I was curious. I already knew that each Service Processor (SP) in the EMC has 4 gig of cache, and that a LUN can only be active on one SP at a time. Also, per a previous blog post, each DAE has a theoretical max throughput of 8 gigabit per second, 4 gigabit if a single LUN stripes across the whole shelf.

CX4 DAE (general) information
http://blog.colovirt.com/2009/05/29/san-emc-cx4-dae-drive-shelf-information/

As the conversation continued, I was told that typically the Service Processors caching would be disabled on LUNS that reside on the solid state flash drives. In all actuality, I think it should. Being that the typical Clariion implementation will not be dedicated as a solid state SAN, they will have to co-mingle with spindle based hard drives(regular). The throughput is a LOT faster and latency a lot lower on the flash drives. Enabling SP caching on LUNs contained in flash could possibly have a really bad affect on the rest of the system. Being that the cache really speeds up IO for spindle based disks.

Most vendors are toting a 20 physical disks(15kRPM) to 1 Flash disk consolidation

What I am getting at is there seems to be the possibility of flooding the Service Processors cache with flash based storage data, depending on change rate. Since flash is a lot faster, their LUNs would typically be used for databases or high volume services. Even with the ability to do QOS in the CX4 series, disabling Service Processor caching for the flash LUNS looks to be the best solution.

Posted in EMC, Hardware, SAN (Storage Area Network)

SAN / EMC: CX4 DAE (Drive Shelf) Information

Kevin Goodman — Sat, 30 May 2009 02:34:19 +0000

This will not get very detailed, but I figured I would share the following information. In light of not being happy with the typical “each shelf has a 4 Gig interconnect” statement, I kept checking until there was a better answer. So, anyone working with EMC SANs typically knows that every shelf is connected to each SP (Service Processor – 2 per SAN), daisy chained in a specific loop, and assigned a shelf id. Next is the LCC.

Each DAE contains 2 LCC interfaces. LCC is the acronymn for “Link Control Card”. Typically one LCC goes to each Service Processor. So that means that each DAE has two 4 gig links for a total of 8gb theoretical throughput.

This is where I will be reading more information. Just because you have 2 LCC cards, each LUN can only be assigned to 1 Service Processor! If my theory is correct, that means if all drives in one shelf is dedicated to a single LUN, and that LUN can only be active on one SP, does that mean that the actual throughput will be limited to 4gb instead of 8? This being due to the other LCC loop being connected to the SP that is not assigned control of that LUN.

Any thoughts? Feel free to comment!

Posted in EMC, Hardware, SAN (Storage Area Network)