Cisco UCS IO Module Fabric Port Pinning

Recently we have had issues with our chassis 2 iom (IO Module) 1 port 1. In the process of troubleshooting, Cisco sent out a replacement IO module hoping it would solve the issue. Turns out one thing that I did not know is that once the “fabric ports” on the IOM have been wired, it must keep the same cabling.

I had the great (not really) idea to swap cable 1 and 2 on the IOM while I was replacing the module. I figured that if it was a port issue on the 6120 blade switch that the problem would move to the new port. Turns out the IOM freaked out when this happened and downed the ports that were not where they had been.

The part which made it more difficult to track down was the the “fabric ports” (links from chassis to switch) all showed green. It was 3 of the 8 backend ports on the IOM that were errored.

Here is a server interface that was pinned to a removed port from the UCS Manager CLI. Below shows the downed interfaces as “Error disabled”.

ucsm01-A(nxos)# sh interface brief | inc "Error disabled"
Eth2/1/3      1      eth  access down    Error disabled              10G(D) --

Once the 10g cables (SFPs) were swapped back to their origional locations, this ports came back. Here the status now shows “up”

ucsm01-A(nxos)# sh interface brief | inc Eth2/1/3
Eth2/1/3      1      eth  access up      none                        10G(D) --

Again, the resolution here was to put the cabling back in their origional locations. Even with this being a new IO module, UCS loaded it with the configuration from the origional card. While talking with Cisco TAC, they said that the development group is working on making the IO module switch uplinks swappable (dynamic) and not statically pinned.

This really wasn’t a major issue due to UCS having redundancy built in, we never lost connectivity to a server. It’s just one of those “gotchas” that I ran into while troubleshooting a port.

~ by Kevin Goodman on June 30, 2010.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Get every new post delivered to your Inbox.

Join 1,374 other followers

%d bloggers like this: