Cisco UCS 6120 Switch Replacement

So back in August of 2010 I had to do a 6120 switch replacement, due to a bad port on the switch. I know I commented on twitter during the process but did not here.

To be safe, we were in a maintenance window, but servers were still up and running while the switch replacement was being done.

A little more information on our setup

  • UCS Manager was running at v1.3(1c)
  • All blades boot from SAN via FCoE
  • Cisco MDS 9124s sits between the 6120s and SAN Storage (EMC)
  • RedHat Linux blades were left running
  • Windows 2008 blade was left running
  • VMware vSphere blades were left running
  • All blades contain the VIC N81kr (PALO) interfaces
  • Two UCS chassis in the environment
  • In “theory” UCS has a lot of redundant built-in, but I had yet to test out taking down a switch. I had at this point pulled IOM modules, blades, SFP cables from an active chassis but not a 6120.
    Just to give you an idea of how many dropped packets were lost during the process, below are pings that were done during the removal.

    Pings to a vSphere host in Cisco UCS

    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
    
    Ping statistics for 127.0.0.59:
        Packets: Sent = 2413, Received = 2410, Lost = 3 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 0ms, Maximum = 75ms, Average = 0ms

    Pings to the cluster IP of 6120s

    Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
    Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
    Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
    Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
    Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
    Reply from 127.0.0.41: bytes=32 time=4ms TTL=63
    Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
    Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
    Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
    Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
    
    Ping statistics for 127.0.0.41:
        Packets: Sent = 2339, Received = 2316, Lost = 23 (0% loss),
    Approximate round trip times in milli-seconds:
        Minimum = 0ms, Maximum = 8ms, Average = 1ms

    So all servers were fine. Nothing went down and no data was lost to the storage going to the SAN. It took about 1 hour to replace switch which entailed:

  • Un-rack bad 6120
  • Rack replacement 6120
  • Upgrade firmware on the new 6120. It was sent to us with version 1.2
  • Join new 6120 to cluster
  • Note(s): Wish that I would have noticed the replacement switch came with an older firmware. I spent a decent amount of time setting it up on a temporary IP (standalone), pushing new firmware to it, and then updating it. Once done, I had to wipe the config and link it to the other 6120 and join the cluster.

    About these ads

    ~ by Kevin Goodman on January 18, 2011.

    One Response to “Cisco UCS 6120 Switch Replacement”

    1. I would also like to comment more on the redundancy of the interconnects. I had to replace a server cage that better suited our datacenter needs. This required shutting down one of the 6120′s and unracking it. I did this during a maintenance window just in case something went wrong. I wanted to test a true failure and I simply unplugged the sucker. It went down hard and not one of 111 VMs on 6 ESX hosts had an issue. All ESX hosts are boot from SAN and many VM’s have iSCSI attached LUN’s. In the end I was very impressed.

    Leave a Reply

    Fill in your details below or click an icon to log in:

    WordPress.com Logo

    You are commenting using your WordPress.com account. Log Out / Change )

    Twitter picture

    You are commenting using your Twitter account. Log Out / Change )

    Facebook photo

    You are commenting using your Facebook account. Log Out / Change )

    Google+ photo

    You are commenting using your Google+ account. Log Out / Change )

    Connecting to %s

     
    Follow

    Get every new post delivered to your Inbox.

    Join 1,372 other followers

    %d bloggers like this: