Cisco UCS and Nexus 5000 DataCenter – Our Implementation

Our Cisco 10gbE implementation consists of 2 Chassis fully populated UCS with a mix of full and half width blades. The servers are all boot from SAN with no local disks. “PALO” cards are used in all servers which allow us to do FCOE. 7 of the blades are running VMware ESX 4 (vSphere) and the rest are a mix of RedHat Linux and Windows 2008.

Storage:

  • All servers boot from SAN over FCOE
  • Primary storage is via Fibre Channel to an EMC CX4-240
  • A few servers are also connected via Fibre Channel to RamSan SSD SANs
  • The CX4-240 is also connected to the UCS environment via quad 10gbE links over fiberNetwork:
  • Core 10gbE switching is provided via Cisco Nexus 5010s
  • 1 Gigabit Ethernet to legacy servers are provided by dual 48 port C2148 Fabric Extenders
  • Each C2148 is connected to a Nexus 5010 by dual 10gbE uplinks
  • Each 5010 is connected via quad 4gig fiber to the CX4-240 to provide FCoE to physical servers outside of the UCS environment
  • Each 5010 is also connected to the CX4-240 by dual 10gbE fiber ports to provide ISCSI to all of the network
  • Each UCS switch (6120s) are uplinked to the 5010s by dual 10gbE interfaces
  • Each UCS chassis is uplinked to the 6120s by 8 qty 10gbE ports (4 connections per IO module)

    • The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

    Below is a diagram of how we are currently setup.

    10gbE DataCenter

    Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

    Possibly related posts: (automatically generated)

    ~ by Kevin Goodman on July 7, 2010.

    Posted in Colocation, EMC, Hardware, Networking, RamSan, SAN (Storage Area Network)
    Tags: , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , , ,

    7 Responses to “Cisco UCS and Nexus 5000 DataCenter – Our Implementation”

    1. Interesting setup, are you using multipath over the 10GB and 4GB links or just over the dual 4GB links for storage?

      jamonation said this on July 7, 2010 at 3:25 pm

    2. Oh, r.e. last comment, I didn’t see the notes on your post ;)

      I’m working on a full 10GbE SFP+ and 10Base-T deployment using Arista 7100T switches for storage and LAN, and Cisco 4900M 10GbE SFP+ uplinks.

      jamonation said this on July 7, 2010 at 3:29 pm

    3. [...] This post was mentioned on Twitter by Brian Gracely. Brian Gracely said: RT @colovirt #Cisco UCS and Nexus 5000 #DataCenter – Our Implementation: http://wp.me/pm3nc-et [...]

      Tweets that mention Cisco UCS and Nexus 5000 DataCenter – Our Implementation « Colocation to Virtualization -- Topsy.com said this on July 7, 2010 at 3:37 pm

    4. cool…. SAN over FCOE
      how good performance?
      i wounder a lot off switch (8) what C2148 use for?

      pual said this on August 17, 2010 at 7:42 pm

    5. FCoE Performance has been fine. I was worried initially because we were doing dedicated dual 4gb fibre uplinks to our physical servers. The C2148 are for legacy gigabit (1gig) ethernet. They are uplinked to the nexus 5010s by dual 10gbE ports (so they have 20gigabit throughput to the 5010 network). Each 2148 can have up to 4 – 10gbE uplinks to the 5010s. They are controlled / admin’ed through the 5010s.

      Kevin Goodman said this on August 18, 2010 at 10:13 am

    6. i would like to copycat configuration design above.
      it’s necessary or not to have SAN clustering?
      my idea do same thing at two location for HA.
      provide me opinion please..in case EMC CX4 broken down
      —-
      thank you for your information very helpful for newbie like me..

      pual said this on August 19, 2010 at 2:36 pm

    7. The EMC CX4 comes with 2 controllers (Service Processors) per array. So you have an SPA and SPB. They are redundant to each other. We also use EMC RecoverPoint replication to replicate the critical LUNs on the CX4 to an off-site location. We have another EMC and vmware cluster at a second datacenter. We can bring up backups or boot systems that were replicated through RecoverPoint at that site.

      Kevin Goodman said this on August 20, 2010 at 2:59 pm

    Leave a Reply