So when it comes to maintenance, I time almost everything. I really like being able to estimate how long specific actions should take to complete. That is what this post is all about. I will be going over startup times, dissassociation, and association of service profiles on Cisco B200 M1 blades. The test was done using blades having different RAM configurations across multiple chassis.

Also to note, our blades boot from SAN to a fiber connected EMC CX4-240 Clariion. The storage on the CX4 supporting these and 10 other boot from SAN blades LUNs is as follows:

Qty 4 – 600gb 10kRPM disks setup in a RAID 10 group.

The following firmware versions are the same between both blades being tested:

Storage path: Palo interface -> fiber -> Cisco Nexus 6120 -> fiber -> EMC CX4

Test 1 Consists of a B200-M1 blade with the following configuration

Bootup and shutdown:

Service Profile associations:

Test 2 Consists of a B200-M1 blade with the following configuration

Bootup and shutdown:

Service Profile associations:

Note(s): The testing was done with blades that were already at the service profiles required firmware level. More time would have to be added if an firmware change was needed. Also, boot times will be dependent on the IOPs achievable to your storage system.

General Specs:

Dual controllers
RAW 12tb
Cache (SSD) RAW 320gb
Quad 10gbE connectivity (2 per controller)

Below shows the Nimble device being un-boxed. I highlighted the SSD drives in green

Nimble Un-boxing

Even though the unit is 3ru in size, they were able to stuff two controllers (heads) into it. Primary is highlighted blue and secondary green.

A small gallery including the racked unit. Notice it is in good company!

Once we got over the fear of FCoE reliability, almost everything is now being ordered with 10gbE. The only reason of worry was during this implementation over a year ago, FCoE was not in the main stream yet. It was almost “bleeding edge” in the market.

Where are we a year later?

One of the big killer of ports in our environment is having to tie the Gigabit Ethernet switches into them. This is wasting 4 ports that could be 10gb pre switch. Hopefully that will be resolved with a future purchase of Nexus 7000 switches.  We also have a pair of fabric extenders connected into the 5010s.  This also takes up two 10gbE ports per 5010.

Below is a port listing from one of our Nexus 5010 switches. This is after we removed 3 ports (per switch) from physical servers that were tied into the 10gbE infrastructure. As you can see from below, we only have 5 ports per switch left (10 total if you count the redundant switch)! That is only 5 more devices at the most.

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        x     eth  trunk  up      none                       1000(D) 1
Eth1/2        x     eth  trunk  up      none                       1000(D) 1
Eth1/3        x     eth  trunk  up      none                       1000(D) 1
Eth1/4        x     eth  trunk  up      none                       1000(D) 1
Eth1/5        x     eth  trunk  up      none                        10G(D) --
Eth1/6        x     eth  trunk  up      none                        10G(D) --
Eth1/7        x     eth  access up      none                        10G(D) --
Eth1/8        x     eth  access up      none                        10G(D) --
Eth1/9        x     eth  trunk  down    Link not connected          10G(D) --
Eth1/10       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/11       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/12       x     eth  access down    SFP not inserted            10G(D) --
Eth1/13       x     eth  access up      none                        10G(D) --
Eth1/14       x     eth  access down    SFP not inserted            10G(D) --
Eth1/15       x     eth  trunk  up      none                        10G(D) --
Eth1/16       x     eth  trunk  up      none                        10G(D) --
Eth1/17       x     eth  fabric up      none                        10G(D) --
Eth1/18       x     eth  fabric up      none                        10G(D) --
Eth1/19       x     eth  trunk  up      none                        10G(D) 2
Eth1/20       x     eth  trunk  up      none                        10G(D) 2

At the 6120′s end (UCS switches) we are fine. Currently have 14 ports on each switch available. That should give us the ability to wire in 7 more chassis with 2 – 10gbE uplinks per IOM. A total of 4 links / 40gb throughput per chassis. Our CIFS and NFS access is a lot faster running through the EMC NX4′s 10gbE.

So if you are looking into 10gbE for our network, it would help to have at least a rough draft of where you want your datacenter to be a few years from now. Pretty much all major storage companies are doing some sort of 10gbE connectivity. Using it makes administration a lot easier. FCoE simplifies things even more, including eliminating the need for separate fiber switches. To sum it up, I love 10gbE and would recommend it even if not planning on incorporating FCoE or UCS in your environment.

So can someone explain to me why EMC sells systems with non-matching face plates.  Like our RecoverPoint systems attached to our CX4 actually have CX3 style faceplates.  Same with the NX4, the 3 1RU components at the bottom faceplates are completely different than the rest.  Either way, it still is a nice system.

To be safe, we were in a maintenance window, but servers were still up and running while the switch replacement was being done.

A little more information on our setup

In “theory” UCS has a lot of redundant built-in, but I had yet to test out taking down a switch. I had at this point pulled IOM modules, blades, SFP cables from an active chassis but not a 6120.
Just to give you an idea of how many dropped packets were lost during the process, below are pings that were done during the removal.

Pings to a vSphere host in Cisco UCS

Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63

Ping statistics for 127.0.0.59:
    Packets: Sent = 2413, Received = 2410, Lost = 3 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 75ms, Average = 0ms

Pings to the cluster IP of 6120s

Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
Reply from 127.0.0.41: bytes=32 time=4ms TTL=63
Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63

Ping statistics for 127.0.0.41:
    Packets: Sent = 2339, Received = 2316, Lost = 23 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 8ms, Average = 1ms

So all servers were fine. Nothing went down and no data was lost to the storage going to the SAN. It took about 1 hour to replace switch which entailed:

Note(s): Wish that I would have noticed the replacement switch came with an older firmware. I spent a decent amount of time setting it up on a temporary IP (standalone), pushing new firmware to it, and then updating it. Once done, I had to wipe the config and link it to the other 6120 and join the cluster.

Below is a run through of associating and disassociating a service profile from one blade to the other.

To begin with, lets find out the state of all service profiles on the system.

ucsm01-A# show service-profile assoc
Service Profile Name Association    Server Server Pool
-------------------- -------------- ------ -----------
db05           Associated     2/3
vcb01          Unassociated
vm02           Associated     1/4
vm03           Associated     1/5
vm04           Associated     2/5
vm05           Associated     2/6
esxi01         Unassociated

For testing, I am going to use “vcb01″. The include function is used below to only show me the stats of the vcb01 service profile.

ucsm01-A# show service-profile assoc | include vcb
vcb01          Unassociated

There are two ways to access the service profile. The first method is to find the UUID.

ucsm01-A# show service-profile identity name vcb01
Service Profile Name: vcb01
UUID Suffix Pool: UUID_Pool
Dynamic UUID: 11000000-0000-0000-2100-00000000148f

    VNIC FC Node:
    WWNN Pool: Blade_WWNN
    Dynamic WWNN: 20:11:00:25:B5:00:00:2F

Since we will be associating a profile with a blade, make sure that it is currently available

ucsm01-A# show server assoc
Server  Association  Service Profile
------- ------------ ---------------
1/1     None
1/3     None
1/4     Associated   vm02

Since we know the uuid, “scope” is used to modify its attributes.

ucsm01-A# scope service-profile dynamic-uuid 11000000-0000-0000-2100-00000000148f

Once we are in, associating a server is very easy.

ucsm01-A /org/service-profile # associate server 1/3

Before the above action takes place, “commit-buffer” must be issued. This is identified by the “*” before the “#”

ucsm01-A /org/service-profile* # commit-buffer

Time to check on the status of the association.

ucsm01-A /org/service-profile # show fsm status

Service Profile Name: vcb01

    FSM 1:
        Remote Result: Not Applicable
        Remote Error Code: None
        Remote Error Description:
        Status: Associate Nic Config Pnuos
        Previous Status: Associate Nic Config Pnuos
        Timestamp: 2010-11-19T13:53:15.093
        Try: 1
        Flags: 0
        Progress (%): 32
        Current Task: Configure adapter for pre-boot environment on server 1/3(FSM-STAGE:sam:dme:ComputeBladeAssociate:NicConf
igPnuOS)

Checking for the last time shows progress as 100%.

ucsm01-A /org/service-profile # show fsm status

Service Profile Name: vcb01

    FSM 1:
        Remote Result: Not Applicable
        Remote Error Code: None
        Remote Error Description:
        Status: 0
        Previous Status: Turnup Success
        Timestamp: 2010-11-19T13:56:56.975
        Try: 0
        Flags: 0
        Progress (%): 100
        Current Task:

Make sure that the server was booted

ucsm01-A /org/service-profile # power up
ucsm01-A /org/service-profile* # commit-buffer

To exit the service profile scope, just type “end”.

ucsm01-A /org/service-profile # end

Another way to access the service profile without specifying the UUID is as follows.

First, you have to find out the organization setup in the UCS config.

ucsm01-A# show org

Organizations:
    Name
    ----
    / (root)

Command sytax:

ucsm01-A# scope service-profile ?
  dynamic-uuid  Dynamic UUID
  org           Org
  server        Server

Knowing the org, the syntax changes to the following:

ucsm01-A# scope service-profile org / vcb01

This method is a little faster, but make sure you do not fat finger hostnames if they are similar.

Disassociation is just as easy.  Once in the service profile scope:

ucsm01-A /org/service-profile # disassociate
ucsm01-A /org/service-profile* # commit-buffer

I removed the faulty switch after disconnecting all cabling. The new 6120 was put on the network and firmware updated to match the cluster. After the upgrade was done, I erased the configuration on the switch and connected the fabric interconnects.

At this point, I ran through the CLI setup utility. Everything looked find and I though the configuration push was successful due to the following information being returned:

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.
Configuration file - Ok

When I went into the UCS manager, the switch didn’t look right. Once into the CLI on the new 6120, the following error confirmed there was a problem.

6120-A# scope fabric-interconnect a
Note: SAMCLI process currently not running

That definitely wasn’t a good thing! The 6120-B member did not show 6120-A as working in its “show cluster state”. Below is the needle in the haystack. After connecting to local management on the switch, it threw a configuration error.

6120-A# connect local-mgmt
svcconfig init: Ini file syntax error at line 127.0.0.10

At that time of night ~2am, it didn’t stand out what that IP address was. I called Cisco TAC and waited for an hour. It was getting pretty late so we called off the maintenance. I put the faulty switch back in. It was still functioning and cables moved from the bad ports over to working ones.

I erased to configuration of the replacement switch and brought it back up in standalone mode. The next day I went to check that switch out and all errors were gone. It seemed to be functioning fine when it was not a member of the cluster.

I used Putty client to log all output while I was connected via serial to the switches that nigh. After reviewing them, I came across the syntax error line multiple times. Then the light bulb came on. The IP 127.0.0.10 is the secondary DNS server on our network.

I already had a ticket opened with Cisco TAC on this. Called back in and asked if they had any bugs that were due to DNS issues. Turned out they were two of them. The IDs are as follows:

CSCtg14999
CSCtf95156

Both of the links above require a cisco.com login, but information on them are also listed below:

“CSCtg14999 has been superseded by CSCtf95156 displayed below.

CSCtf95156 Bug Details
secondary DNS entry not properly copied to peer during clustering
Symptom:

If you have 2 DNS servers configured in UCSM and then attempt to peer with a new FI (i.e. a subordinate box is added to the setup or a write erase is done to an existing subordinate) the first DNS entry is copied over properly but the second one is missing the ‘dnsServer’ prefix and ends up in the sam.config file (/opt/db/sam.config) as just an IP address on a line.

As a result the config is not properly read and clustering does not occur.

Workaround:

Deleting the second DNS IP address from the config will allow the FI to cluster (assuming all other criteria are correct and in place).”

So per the above information, ID CSCtg14999 looks have been replaced or is a child bug of CSCtf95156.

The fix: I removed the DNS servers from UCS configuration, and replaced the switch during another maintenance window. This time, it worked great. Was in and out in about 1.5 hours.

Also, the order of the people in the table below is based on preference, just the order I copied an pasted the information. Except for me. I am listed on the top of the list because this is my blog ;)

Login to the primary 6120 via the UCS cluster IP address (SSH).

Verify the current state (roles) of the cluster

ucs01-B# show cluster state
Cluster Id: 0xg7949/79843654he-0x8486a7868s8787785

A: UP, SUBORDINATE
B: UP, PRIMARY

Enter local management on the primary.

ucs01-B# connect local-mgmt
Cisco UCS 6100 Series Fabric Interconnect

Since the “b” 6120 is the current primary, the following will activate “a” as the primary.

ucs01-B(local-mgmt)# cluster lead a
Cluster Id: 0xg7949/79843654he-0x8486a7868s8787785

Verify that the subordinate 6120 has now become the primary.  This was done by SSHing directly to 6120 “a”.

ucs01-A# show cluster state
Cluster Id: 0xg7949/79843654he-0x8486a7868s8787785

A: UP, PRIMARY
B: UP, SUBORDINATE

HA READY

Process to failover
1. Login to the primary 6120 via the UCS cluster IP address.
2. Verify the current primary 6120 in the cluster.
3. Enter local management on the primary via “connect local-mgmt” command.
4. Issue the “cluster lead x” command to make the subordinate switch become the primary. Replace “x” with the correct switch letter.
5. Verify that the role has changed on the previously subordinate switch by SSH’ing into it and issuing the “show cluster state” command. The switch show now show up as the primary.

Notes: I listed the 5 step process above with the “fluff” removed.  Just in case you don’t need to see the command(s) output.

Having limited funds during our initial UCS purchase, we were only able to fill our B200-M1 blades with 4gb RAM modules. Below is an image showing the current RAM to CPU utilization.

vSphere RAM Usage

I have put out a few previous posts concearing consolidation ratios here:

http://blog.colovirt.com/2010/05/24/cisco-vmware-cisco-ucs-b6620-vmware-consolidation-ratio/
http://blog.colovirt.com/2010/06/01/cisco-vmware-cisco-ucs-b250-m1-vmware-consolidation-ratio-oracle-dbs/

A recap of our current setup is below.

Density = ~19.8 VMs per B200-M1 server w/48g of RAM

Again, this is a mixed environment containing host OSs:

• Windows 2003
• Windows 2008
• RedHat Linux

We have stopped deploying any VMs in this cluster due to RAM constraints.  Thankfully our RAM upgrade is being shipped out currently from Cisco.

This will be a total replacement of the 4gig RAM modules currently installed in the B200′s with 8gig sticks.

Ordered 60 x 8gb sticks = 480gig upgrade (total) to ESX hosts

Of course our 4gig modules will not be wasted.  They will be used to fill our B250-M1 servers to maximum capacity (number of RAM sticks).

This upgrade should get us through until another UCS chassis can be purchased next year.  My advice for anyone who is planning on deploying UCS for the sole purpose of running VMware:

• Bite the bullet and go with the 8 gig sticks from the start if deploying only B200 blades
• B250 1/2 populated with 4 gig sticks = 98gigs of RAM.  You can put 192 gigs of RAM in using 4gig sticks or 384gigs with 8gig sticks
• Keep at least one spare server in each UCS chassis for un-planned expansion or to use if another blade dies

Follow me on twitter

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Initial RAM Error - Server 1

“scope server” help output via CLI

ucs_6120-A# scope server ?
  WORD          <chassis-id>/<blade-id>
  dynamic-uuid  Dynamic UUID

Since the problem is currently showing up on chassis 2 blade 1, we set the scope for it below

ucs_6120-A# scope server 2/1

Now we use “show sel” to view the event log, filtering the output with “include”. sel = System Event Log

ucs_6120-A /chassis/server # show sel | include error

4ef | 04/15/2010 10:40:28 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 0, DIMM Socket: 4, Channel: A, Socket: 1 | Asserted
4f1 | 04/15/2010 10:40:29 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 1, DIMM Socket: 5, Channel: A, Socket: 1 | Asserted
4f3 | 04/15/2010 10:40:30 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 2, DIMM Socket: 6, Channel: A, Socket: 1 | Asserted
4f5 | 04/15/2010 10:40:31 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 3, DIMM Socket: 7, Channel: A, Socket: 1 | Asserted

So from the messages above, you can see that there is definitely an issue. I originally thought that this was an actual hardware failure. The server never had a problem and continued ot run. At first only two DIMMs showed errors, but about 3 hours later two more DIMMs errored.

To be safe, I disassociated the service profile from this blade and moved it to a spare. About 30 minutes after the service profile was booted on the new blade DIMM errors showed up on it. This definitely hinted to a bug.

Initial RAM Error - Server 3

It turns out that this is a bug in UCS firmware 1.2

Interestingly Cisco shows the condition to be “Fully populated DIMM slots with X5570 CPU on B250 blade” and with “2 X5570 CPUs”.  In our case they are right with the CPU count, but we are not fully populated with DIMMs. We are only at 1/2 capacity.

I did upgrade the firmware to version 1.3(1c) and the errors went away.  Per cisco this is a “cosmetic” error and did not actually affect the server.

Just in case you do not have access to Cisco.com to view the bug ID, the text is listed below:

Ventura: 1.2(1b): X5570:Sel Events P0V75_DDR3_P2 errors filling up.
Symptom:

Amber LED/Sel Events on B250 blades with 2 X5570 CPU.
A /chassis/server # show sel
17 | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Asserted | Reading
0.71 < Threshold 0.71 Volts 18 | | Platform alert LED_BLADE_STATUS | LED color is amber | Asserted 19 | | Platform alert LED_BLADE_STATUS | LED color is green | Deasserted 1a | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Deasserted | Reading0.73>Threshold 0.71 Volts
Conditions:

Fully populated DIMM slots with X5570 CPU on B250 blade

Workaround:

a. sel events can be cleared for server X/Y by

scope chassis X
scope server Y
clear sel
commit
You can use sel backup policy to clear it automatically.

b. threshold values can be changed by
ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 20 0×36 0×00 0×47 0×44 0×00 0×51 0×55

ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 19 0×36 0×00 0×47 0×44 0×00 0×51 0×55

Before making this change run the following commands and confirm the output equals these values:

ipmitool -H BMC_IP_ADDRESS -U user -P password -I lanplus raw 4 0×27 19

36 00 49 46 00 51 55

ipmitool -H BMC_IP_ADDRESS -U user -P password -II lanplus raw 4 0×27 20

36 00 49 46 00 51 55

If the output does not match please contact TAC for assistance on implementing this workaround.

Threshold needs to be set again after BMC reset.

There are a total of 7 blades in our VMware environment, but only 5 of those are dedicated to our main HA/DRS cluster. That gives us ~240 gigs of RAM for the main cluster. Currently, I am seeing a VM consolidation ratio of about 24 VMs (virtual machines) per B200-M1 blade. The limitation here is definitely the RAM. The CPU itself is less than 25% utilized per blade. Below is a screenshot of a B200-M1 host in our cluster.

Cisco UCS B200-M1 VM Ratio

Notes: The VMs running on the above host are

Operating system utilization

Unpacked B200-M1 Blades

Unpacked Blades B200/B250

B250-M1 Blade Pictures

B250-M1 Guide

B250-M1 Open Front

B250-M1 Open Rear

B250-M1 Internal Top

B250-M1 Top Internal 2

B250-M1 Internal Palo

B250-M1 And B200-M1 In Chassis

B250-M1 Bezel

B200-M1 Blade Pictures

B200-M1 Guide

B200-M1 Internal Air Flow

B200-M1 Top Full

B200-M1 Top CPU / RAM

B200-M1 Rear Palo

B200-M1 Top Internal w/vents

B200-M1 Bezel

First check and see if there are any current licenses installed. This being a new switch, there aren’t any.

nexus_5000# sh license
nexus_5000#

Licenses are normally tied to “host IDs”. Below shows the host ID of the switch.

nexus_5000# sh license host-id
License hostid: VDH=3WI246599Z

The license file is not local to the switch yet, so below I fetch it via TFTP.nexus_5000# copy tftp://127.0.0.1:69/Cisco_license/NX5KLIC.lic bootflash: vrf default_vrf

Trying to connect to tftp server......
Connection to Server Established.
[                         ]         0.50KB
TFTP get operation was successful

Next, the license is installed with the “install license” command.

nexus_5000# install license bootflash:NX5KLIC.lic
Installing license .....done

Now that the license has been added, use “show license” to confirm.

nexus_5000# show license
NX5KLIC.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT ENTERPRISE_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
1 \
RMA license" SIGN=DDWD62626W7f
INCREMENT FC_FEATURES_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
2 \
RMA license" SIGN=VVSVDS848SFW

Another way is to see the names of the installed licenses.

nexus_5000# show license brief
NX5KLIC.lic

To be safe, copy the running configuration to startup.

nexus_5000# copy running-config startup-config

Also, here I copy the configuration to TFTP backup.

nexus_5000# copy running-config tftp://127.0.0.1:69/Cisco_config/ns5k2_startup vrf default_vrf
Trying to connect to tftp server......
Connection to Server Established.
[#                        ]         4.50KB
TFTP put operation was successful

Notes:  All IP, license files, and hashes were changed.  The installation process is pretty strait forward.  The managment console (GUI) should also be able to provide this function.