Time to shutdown: ~30 seconds

From boot:

It seems that SSH access came in at almost exactly the same time as the CLI console.  Some of you might find this useful, some not. Just figured I would post it either way.

Having limited funds during our initial UCS purchase, we were only able to fill our B200-M1 blades with 4gb RAM modules. Below is an image showing the current RAM to CPU utilization.

vSphere RAM Usage

I have put out a few previous posts concearing consolidation ratios here:

http://blog.colovirt.com/2010/05/24/cisco-vmware-cisco-ucs-b6620-vmware-consolidation-ratio/
http://blog.colovirt.com/2010/06/01/cisco-vmware-cisco-ucs-b250-m1-vmware-consolidation-ratio-oracle-dbs/

A recap of our current setup is below.

Density = ~19.8 VMs per B200-M1 server w/48g of RAM

Again, this is a mixed environment containing host OSs:

• Windows 2003
• Windows 2008
• RedHat Linux

We have stopped deploying any VMs in this cluster due to RAM constraints.  Thankfully our RAM upgrade is being shipped out currently from Cisco.

This will be a total replacement of the 4gig RAM modules currently installed in the B200′s with 8gig sticks.

Ordered 60 x 8gb sticks = 480gig upgrade (total) to ESX hosts

Of course our 4gig modules will not be wasted.  They will be used to fill our B250-M1 servers to maximum capacity (number of RAM sticks).

This upgrade should get us through until another UCS chassis can be purchased next year.  My advice for anyone who is planning on deploying UCS for the sole purpose of running VMware:

• Bite the bullet and go with the 8 gig sticks from the start if deploying only B200 blades
• B250 1/2 populated with 4 gig sticks = 98gigs of RAM.  You can put 192 gigs of RAM in using 4gig sticks or 384gigs with 8gig sticks
• Keep at least one spare server in each UCS chassis for un-planned expansion or to use if another blade dies

Follow me on twitter

Now all that we need to do is get some data over to it.  We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit.  You can not change the order in which each device is racked.  Also, the controllers (SPs) are built into the drive tray.  Best part is that this will be wired up via 10gbE connectivity over fiber.

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

I had the great (not really) idea to swap cable 1 and 2 on the IOM while I was replacing the module. I figured that if it was a port issue on the 6120 blade switch that the problem would move to the new port. Turns out the IOM freaked out when this happened and downed the ports that were not where they had been.

The part which made it more difficult to track down was the the “fabric ports” (links from chassis to switch) all showed green. It was 3 of the 8 backend ports on the IOM that were errored.

Here is a server interface that was pinned to a removed port from the UCS Manager CLI. Below shows the downed interfaces as “Error disabled”.

ucsm01-A(nxos)# sh interface brief | inc "Error disabled"
Eth2/1/3      1      eth  access down    Error disabled              10G(D) --

Once the 10g cables (SFPs) were swapped back to their origional locations, this ports came back. Here the status now shows “up”

ucsm01-A(nxos)# sh interface brief | inc Eth2/1/3
Eth2/1/3      1      eth  access up      none                        10G(D) --

Again, the resolution here was to put the cabling back in their origional locations. Even with this being a new IO module, UCS loaded it with the configuration from the origional card. While talking with Cisco TAC, they said that the development group is working on making the IO module switch uplinks swappable (dynamic) and not statically pinned.

This really wasn’t a major issue due to UCS having redundancy built in, we never lost connectivity to a server. It’s just one of those “gotchas” that I ran into while troubleshooting a port.

Initial RAM Error - Server 1

“scope server” help output via CLI

ucs_6120-A# scope server ?
  WORD          <chassis-id>/<blade-id>
  dynamic-uuid  Dynamic UUID

Since the problem is currently showing up on chassis 2 blade 1, we set the scope for it below

ucs_6120-A# scope server 2/1

Now we use “show sel” to view the event log, filtering the output with “include”. sel = System Event Log

ucs_6120-A /chassis/server # show sel | include error

4ef | 04/15/2010 10:40:28 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 0, DIMM Socket: 4, Channel: A, Socket: 1 | Asserted
4f1 | 04/15/2010 10:40:29 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 1, DIMM Socket: 5, Channel: A, Socket: 1 | Asserted
4f3 | 04/15/2010 10:40:30 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 2, DIMM Socket: 6, Channel: A, Socket: 1 | Asserted
4f5 | 04/15/2010 10:40:31 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 3, DIMM Socket: 7, Channel: A, Socket: 1 | Asserted

So from the messages above, you can see that there is definitely an issue. I originally thought that this was an actual hardware failure. The server never had a problem and continued ot run. At first only two DIMMs showed errors, but about 3 hours later two more DIMMs errored.

To be safe, I disassociated the service profile from this blade and moved it to a spare. About 30 minutes after the service profile was booted on the new blade DIMM errors showed up on it. This definitely hinted to a bug.

Initial RAM Error - Server 3

It turns out that this is a bug in UCS firmware 1.2

Interestingly Cisco shows the condition to be “Fully populated DIMM slots with X5570 CPU on B250 blade” and with “2 X5570 CPUs”.  In our case they are right with the CPU count, but we are not fully populated with DIMMs. We are only at 1/2 capacity.

I did upgrade the firmware to version 1.3(1c) and the errors went away.  Per cisco this is a “cosmetic” error and did not actually affect the server.

Just in case you do not have access to Cisco.com to view the bug ID, the text is listed below:

Ventura: 1.2(1b): X5570:Sel Events P0V75_DDR3_P2 errors filling up.
Symptom:

Amber LED/Sel Events on B250 blades with 2 X5570 CPU.
A /chassis/server # show sel
17 | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Asserted | Reading
0.71 < Threshold 0.71 Volts 18 | | Platform alert LED_BLADE_STATUS | LED color is amber | Asserted 19 | | Platform alert LED_BLADE_STATUS | LED color is green | Deasserted 1a | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Deasserted | Reading0.73>Threshold 0.71 Volts
Conditions:

Fully populated DIMM slots with X5570 CPU on B250 blade

Workaround:

a. sel events can be cleared for server X/Y by

scope chassis X
scope server Y
clear sel
commit
You can use sel backup policy to clear it automatically.

b. threshold values can be changed by
ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 20 0×36 0×00 0×47 0×44 0×00 0×51 0×55

ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 19 0×36 0×00 0×47 0×44 0×00 0×51 0×55

Before making this change run the following commands and confirm the output equals these values:

ipmitool -H BMC_IP_ADDRESS -U user -P password -I lanplus raw 4 0×27 19

36 00 49 46 00 51 55

ipmitool -H BMC_IP_ADDRESS -U user -P password -II lanplus raw 4 0×27 20

36 00 49 46 00 51 55

If the output does not match please contact TAC for assistance on implementing this workaround.

Threshold needs to be set again after BMC reset.

Cisco UCS PALO Details

Per UCS the PALO cards are listed as

Boot times based on firmware version:

PALO Boot Time Firmware 1.2(1b)

PALO Boot Time Firmware 1.3(1c)

So the fix is to upgrade.  Not the easiest process, but not that hard either.  As from the stats above, v1.3(1c) is definitely worth the upgrade!  No more waiting around for 2008 to boot.

Notes: We also run RedHat Linux and VMware ESX in this environment.  They boot without issues or delay with the same PALO adapters. Also, it only took 2 minutes and 16 seconds to get network connectivity up on 2008 running version 1.3(1c) firmware.  Having no other good options I timed the boot using my iPhone.  The above images are from screen-captures on the phone.  UCS is not 3G connected :).

I will be using “naviseccli” for this, but the same syntax should work with “navicli”. Both are available through EMC Powerlink site.

Below will query the Clariion and return a list of all LUNs that are trespassed. The -h specifies which host (SP) to connect to. You only need to run this on one SP. The other would report back the exact same results.

C:\>naviseccli -h 127.0.0.12 getlun -trespass
LOGICAL UNIT NUMBER 7
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 15
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 118
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 17
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 204
Default Owner:              SP B
Current owner:              SP A

LOGICAL UNIT NUMBER 619
Default Owner:              SP B
Current owner:              SP A

....

I have some Linux style utilities installed on my Windows machine. Below shows that we have a total of 55 LUNs that are currently trespassed. Yeah, don’t ask.

C:\>naviseccli -h 127.0.0.13 getlun -trespass | grep LOGICAL | wc -l
     55

I thought that a list of all LUN IDs would be needed and then have to make a script to run through them. Luckily that is not the case. The syntax is as follows:C:\>naviseccli -h trespass mine

The above command has to be run on both SPs (Service Processors).
C:\>naviseccli -h 127.0.0.12 trespass mine

C:\>naviseccli -h 127.0.0.13 trespass mine

Use the “getlun -trespass” command once again to make sure that there are no more trespassed LUNs. A few things to keep in mind are that trespassed snapshots will stay trespassed. CLI or even the Navisphere GUI will not let you move them. Also, I had a few LUNs that would not move via the CLI, but when done manually in Navisphere they moved fine.

Below is the output of help for naviseccli.
C:\>naviseccli -help

@(#)Navisphere naviseccli Revision 6.29.5.0.66 on Wed Jan 20 22:47:06 2010
Copyright (C) 1997-2009, EMC Corporation
  Usage:
    [-User ] [-Password]
    [-Scope <0 - global; 1 - local; 2 - LDAP>]
    [-Address  | -h ]
    [-Port] [-Timeout  | -t ]
    [-AddUserSecurity | -RemoveUserSecurity | -DeleteSecurityEntry]
    [-Parse | -p] [-NoPoll | -np] [-cmdtime]
    [-Xml] [-f ] [-Help] CMD

If you were using “navicli” the following commands should work for you:

navicli -h 127.0.0.12 getlun -trespass
navicli -h 127.0.0.12 trespass mine
navicli -h 127.0.0.13 trespass mine

Notes: Feel free to comment / email (kevin <at> colovirt.com) / twitter me with any questions!

After connecting to the console, the following command was ran. nexus1kv# sh interface counters errors

--------------------------------------------------------------------------------
Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards
--------------------------------------------------------------------------------
Veth30            --          --          --          --          --     1127136
Veth69            --          --          --          --          --       29654
Veth70            --          --          --          --          --       31966
Veth71            --          --          --          --          --       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifying the VM is indeed “w2k_serv0″
nexus1kv# sh interface Veth30 description

-------------------------------------------------------------------------------
Interface                Description
-------------------------------------------------------------------------------
Vethernet30              w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported
nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).
nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1
Veth70                   linux_host01, Network Adapter 2
Veth71                   linux_host01, Network Adapter 3

After a little more investigation, it turned out that “w2k_serv01″ was deployed from a legacy template that was created on our old AMD cluster. Looking at the network interface in Windows 2003, it showed up as an “AMD …” network adapter.

To try and correct these errors, I removed the current network adapter through VMware vCenter and added a new one. This forced the OS in seeing the network interface as a new device and install fresh drivers. Now in Windows it shows up as an Intel network inteface.

Now that the new network adapter has been installed, I reset the statistics (and errors) on all the interfaces.
nexus1kv# clear counters

This command will clear "show interface" counters on all interfaces
Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.
nexus1kv# sh interface counters errors | inc Veth30

Veth30            --          --          --          --          --           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error filed there. Yeah, that was a run-on sentence.  Sorry.

Below are pictures of the full rack that includes:

• Two Cisco UCS Chassis fully populated
• Two Cisco Nexus 5010 switches
• Two Cisco 6120XP switches
• Two horizontal PDUs (Circuit 1)
• Two vertical PDUs (Circuit 2)

Just in case you wanted a pretty view, the last picture is from my mountain trip last weekend

The server itself is running mainly Oracle database VMs.  Blade specs are as follows:

98 gigs of RAMSince the Oracle VMs are running semi-intensive databases, the RAM allocated to the heavy hitters are between 8-10 gigs.

B250m1 Stats From UCS

B250m1 Stats From Virtual Center

Now for the actual utilization statistics.  The following is from Virtual Center showing the physical B250-M1 database server.  You can see that the CPUs are more highly used in this environment.

B250m1 VC Utilization

Again, this is a mix between Windows Remote desktop and Oracle (RedHat Linux) database servers.  The following shows utilization information perm VM.

B250m1_2 VM Utilization

Hopefully this gives you a little more information concerning Oracle, VMware, Cisco UCS, and consolidation ratios for the full-width blades. Again, this server is running 98gigs of RAM.  Next purchase cycle, we will be upgrading the amount of RAM in all of our blades to get better consolidation.

On the source, I have already setup a directory tree for “/backup/europe_data”. All files destined for our Euroupe office will be placed here. On the DataDomain devices, replicated folders are added manually. By default none are replicated.
*********************
Be aware that if you are viewing this through a web browser that a lot of the command might be hidden due to formatting of the WordPress theme.  It is best viewed by an RSS reader.  If you would like the full text, email or leave a comment and I can send you a PDF version.
*********************

Below is the help output for “replication add” command.

user01@us-dd565# replication add  ?
Commands matching "replication add":
  replication add source  destination
                 Add a replication pair

I wish the syntax was easy as what was stated above, but below is the actual command used to setup replication on the source DataDomain

user01@us-dd565# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once the source is setup, the destination endpoint/DataDomain also must be configured. The command is identical to what was entered at the source DataDomain

user01@eu-dd01# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once replication is setup on both DataDomains, log back into the source and start the replication.

user01@us-dd565# replication initialize
Incorrect syntax "replication initialize", showing help:
  replication initialize
                 Initialize replication on the source
                 (configure both source and destination
                 first)

Below is the actual syntax to startup replication for the directory that we just configured.

user01@us-dd565# replication initialize dir://eu-dd01.testdomain.com/backup/europe_data
(00:04) Waiting for initialize to start...
(00:06) Initialize started.
Use 'replication watch dir://eu-dd01.testdomain.com/backup/europe_data' to monitor progress.

Since I am impatient and want to know for sure this is working, I use “replication watch”. This shows the current progress of the replication between DataDomains.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
    Use Control-C to stop monitoring.

(00:57) Replication initialize started...
(00:59) initializing 3/3:
(10:35)     : pre-comp:   0%    27 KB/s, network    26 KB/s, 0 of 6 files

When new data is written to the mount point, replication automatically starts. Below is what shows if no data is currently being replicated.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
**** Replication initialize or recover already completed.

“replication show config” is used to display all the configured replication points.

user01@us-dd565# replication show config
CTX   Source                                                    Destination                                                 Connection Host and Port                       Enabled
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------
11    dir://us-dd565.testdomain.com/backup/mysql-rep       	dir://us-t2-dd510.testdomain.com/backup/mysql-rep	us-t2-dd510.testdomain.com   (default)     yes
12    dir://us-dd565.testdomain.com/backup/europe_data        	dir://eu-dd510.testdomain.com/backup/europe_data	eu-dd510.testdomain.com   (default)     yes
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------

Sorry for all of the spacing being off. All the tabs did not come over when copying data from the console.

user01

There are a total of 7 blades in our VMware environment, but only 5 of those are dedicated to our main HA/DRS cluster. That gives us ~240 gigs of RAM for the main cluster. Currently, I am seeing a VM consolidation ratio of about 24 VMs (virtual machines) per B200-M1 blade. The limitation here is definitely the RAM. The CPU itself is less than 25% utilized per blade. Below is a screenshot of a B200-M1 host in our cluster.

Cisco UCS B200-M1 VM Ratio

Notes: The VMs running on the above host are

Operating system utilization

Unpacked B200-M1 Blades

Unpacked Blades B200/B250

B250-M1 Blade Pictures

B250-M1 Guide

B250-M1 Open Front

B250-M1 Open Rear

B250-M1 Internal Top

B250-M1 Top Internal 2

B250-M1 Internal Palo

B250-M1 And B200-M1 In Chassis

B250-M1 Bezel

B200-M1 Blade Pictures

B200-M1 Guide

B200-M1 Internal Air Flow

B200-M1 Top Full

B200-M1 Top CPU / RAM

B200-M1 Rear Palo

B200-M1 Top Internal w/vents

B200-M1 Bezel

Doing so allowed us to free up fiber ports on our Cisco MDS 9124 switches needed later on in the UCS blade chassis implementation.  I love pictures of equipment so I figured I would share.

Floor Jack Loader

Nexus 5010 Un-boxed

Nexus 5010s Racked

Nexus 5010s With Fabric Extenders

Nexus 5010s Racked Front - Cold Aisle

5010's Rear Wired - Hot Aisle

N2K-C2148T-1GE Fabric Extenders In Network Rack