Uncorrected ECC event detected on boards 4,6

The DRAM cache fronts the Flash storage drives in the unit.  Basically, loosing these two boards took down the unit completely.  To get the unit back up, boards 4 and 6 needed to be removed.  RamSan actually had me remove 4 of the boards instead of just the two that were bad.  Guessing it has something to do with not being able to have slots not filled between active modules.  
Doing so took the DRAM cache from 64 gigs down to 32.  All data was lost on the FLASH RAID array, making us have to restore the Oracle database from backup.  After the cards were removed, a low level format of the RAID array was done, the unit was operational again.   Below are pictures from within the RamSan 500 for all the storage people out there.

Best part of this is we had racked two RamSan 630′s that were being configured as a mirrored pair.  The plan was to implement them in two weeks.  When the 500 failed, they were implemented that night.

So can someone explain to me why EMC sells systems with non-matching face plates.  Like our RecoverPoint systems attached to our CX4 actually have CX3 style faceplates.  Same with the NX4, the 3 1RU components at the bottom faceplates are completely different than the rest.  Either way, it still is a nice system.

Some specs on the system:

RamSan Box

Turns out that the hot-spare flash module was shipped separately so I had to install that myself.  Pictures from the racking and module installation is below.

Below are a few videos of the equipment.  Be careful with the sound!  These were taken in the datacenter and you will hear the typical rumbling of servers and AC units.

Video of the NetApp FAS 3020 once removed

Video of the EMC NX4 waiting to go in

Just some technical specs on this model:

- 7 DAEs (drive trays)
- Total of 105 drives / spindles
- 2 gigabit fiber connectivity

So approximately 13 terabytes is still a decent amount.  Taken into account that we ram mainly RAID 10 on this array, that usable space dwindled dramatically.  Of course, this was built at a time where the industry was still throwing disks “spindles” at Oracle databases.  Back then there was no “enterprise” solid state in the mainstream market.

Having severed it’s purpose as first a “production” services storage array, it was replaced by a CX3-40.  At that time, the CX500 was moved to a disaster recovery site and used there for it’s primary storage.  Within about a year the CX3-40 was replaced at production site with a CX4-240.  Since then, the CX3-40 was moved to the disaster recover site as primary storage.  The CX500 had its data migrated off to either the RamSan 630 there or the EMC CX3-40.  We have now almost outgrown our CX4-240 and soon it’s fate might be sealed as well.

Hope you liked the above history lesson and now comes the pictures.  Also please note that these DAEs are full length DAEs, not the smaller shelves like the CX3+ series.  Hence the reason why Bill (co-worker) came to help!

Notes: Check out the popped (flat) tire on the cart.  The Trailblazer was definitely riding low on the way home.  Didn’t want the rack rails messing up my trunk, so those were all put in the back seat.  Surprisingly, I was able to fit the whole SAN into the back compartment without folding the rear seats down.

I removed the faulty switch after disconnecting all cabling. The new 6120 was put on the network and firmware updated to match the cluster. After the upgrade was done, I erased the configuration on the switch and connected the fabric interconnects.

At this point, I ran through the CLI setup utility. Everything looked find and I though the configuration push was successful due to the following information being returned:

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.
Configuration file - Ok

When I went into the UCS manager, the switch didn’t look right. Once into the CLI on the new 6120, the following error confirmed there was a problem.

6120-A# scope fabric-interconnect a
Note: SAMCLI process currently not running

That definitely wasn’t a good thing! The 6120-B member did not show 6120-A as working in its “show cluster state”. Below is the needle in the haystack. After connecting to local management on the switch, it threw a configuration error.

6120-A# connect local-mgmt
svcconfig init: Ini file syntax error at line 127.0.0.10

At that time of night ~2am, it didn’t stand out what that IP address was. I called Cisco TAC and waited for an hour. It was getting pretty late so we called off the maintenance. I put the faulty switch back in. It was still functioning and cables moved from the bad ports over to working ones.

I erased to configuration of the replacement switch and brought it back up in standalone mode. The next day I went to check that switch out and all errors were gone. It seemed to be functioning fine when it was not a member of the cluster.

I used Putty client to log all output while I was connected via serial to the switches that nigh. After reviewing them, I came across the syntax error line multiple times. Then the light bulb came on. The IP 127.0.0.10 is the secondary DNS server on our network.

I already had a ticket opened with Cisco TAC on this. Called back in and asked if they had any bugs that were due to DNS issues. Turned out they were two of them. The IDs are as follows:

CSCtg14999
CSCtf95156

Both of the links above require a cisco.com login, but information on them are also listed below:

“CSCtg14999 has been superseded by CSCtf95156 displayed below.

CSCtf95156 Bug Details
secondary DNS entry not properly copied to peer during clustering
Symptom:

If you have 2 DNS servers configured in UCSM and then attempt to peer with a new FI (i.e. a subordinate box is added to the setup or a write erase is done to an existing subordinate) the first DNS entry is copied over properly but the second one is missing the ‘dnsServer’ prefix and ends up in the sam.config file (/opt/db/sam.config) as just an IP address on a line.

As a result the config is not properly read and clustering does not occur.

Workaround:

Deleting the second DNS IP address from the config will allow the FI to cluster (assuming all other criteria are correct and in place).”

So per the above information, ID CSCtg14999 looks have been replaced or is a child bug of CSCtf95156.

The fix: I removed the DNS servers from UCS configuration, and replaced the switch during another maintenance window. This time, it worked great. Was in and out in about 1.5 hours.

Having limited funds during our initial UCS purchase, we were only able to fill our B200-M1 blades with 4gb RAM modules. Below is an image showing the current RAM to CPU utilization.

vSphere RAM Usage

I have put out a few previous posts concearing consolidation ratios here:

http://blog.colovirt.com/2010/05/24/cisco-vmware-cisco-ucs-b6620-vmware-consolidation-ratio/
http://blog.colovirt.com/2010/06/01/cisco-vmware-cisco-ucs-b250-m1-vmware-consolidation-ratio-oracle-dbs/

A recap of our current setup is below.

Density = ~19.8 VMs per B200-M1 server w/48g of RAM

Again, this is a mixed environment containing host OSs:

• Windows 2003
• Windows 2008
• RedHat Linux

We have stopped deploying any VMs in this cluster due to RAM constraints.  Thankfully our RAM upgrade is being shipped out currently from Cisco.

This will be a total replacement of the 4gig RAM modules currently installed in the B200′s with 8gig sticks.

Ordered 60 x 8gb sticks = 480gig upgrade (total) to ESX hosts

Of course our 4gig modules will not be wasted.  They will be used to fill our B250-M1 servers to maximum capacity (number of RAM sticks).

This upgrade should get us through until another UCS chassis can be purchased next year.  My advice for anyone who is planning on deploying UCS for the sole purpose of running VMware:

• Bite the bullet and go with the 8 gig sticks from the start if deploying only B200 blades
• B250 1/2 populated with 4 gig sticks = 98gigs of RAM.  You can put 192 gigs of RAM in using 4gig sticks or 384gigs with 8gig sticks
• Keep at least one spare server in each UCS chassis for un-planned expansion or to use if another blade dies

Follow me on twitter

Now all that we need to do is get some data over to it.  We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit.  You can not change the order in which each device is racked.  Also, the controllers (SPs) are built into the drive tray.  Best part is that this will be wired up via 10gbE connectivity over fiber.

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Initial RAM Error - Server 1

“scope server” help output via CLI

ucs_6120-A# scope server ?
  WORD          <chassis-id>/<blade-id>
  dynamic-uuid  Dynamic UUID

Since the problem is currently showing up on chassis 2 blade 1, we set the scope for it below

ucs_6120-A# scope server 2/1

Now we use “show sel” to view the event log, filtering the output with “include”. sel = System Event Log

ucs_6120-A /chassis/server # show sel | include error

4ef | 04/15/2010 10:40:28 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 0, DIMM Socket: 4, Channel: A, Socket: 1 | Asserted
4f1 | 04/15/2010 10:40:29 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 1, DIMM Socket: 5, Channel: A, Socket: 1 | Asserted
4f3 | 04/15/2010 10:40:30 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 2, DIMM Socket: 6, Channel: A, Socket: 1 | Asserted
4f5 | 04/15/2010 10:40:31 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 3, DIMM Socket: 7, Channel: A, Socket: 1 | Asserted

So from the messages above, you can see that there is definitely an issue. I originally thought that this was an actual hardware failure. The server never had a problem and continued ot run. At first only two DIMMs showed errors, but about 3 hours later two more DIMMs errored.

To be safe, I disassociated the service profile from this blade and moved it to a spare. About 30 minutes after the service profile was booted on the new blade DIMM errors showed up on it. This definitely hinted to a bug.

Initial RAM Error - Server 3

It turns out that this is a bug in UCS firmware 1.2

Interestingly Cisco shows the condition to be “Fully populated DIMM slots with X5570 CPU on B250 blade” and with “2 X5570 CPUs”.  In our case they are right with the CPU count, but we are not fully populated with DIMMs. We are only at 1/2 capacity.

I did upgrade the firmware to version 1.3(1c) and the errors went away.  Per cisco this is a “cosmetic” error and did not actually affect the server.

Just in case you do not have access to Cisco.com to view the bug ID, the text is listed below:

Ventura: 1.2(1b): X5570:Sel Events P0V75_DDR3_P2 errors filling up.
Symptom:

Amber LED/Sel Events on B250 blades with 2 X5570 CPU.
A /chassis/server # show sel
17 | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Asserted | Reading
0.71 < Threshold 0.71 Volts 18 | | Platform alert LED_BLADE_STATUS | LED color is amber | Asserted 19 | | Platform alert LED_BLADE_STATUS | LED color is green | Deasserted 1a | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Deasserted | Reading0.73>Threshold 0.71 Volts
Conditions:

Fully populated DIMM slots with X5570 CPU on B250 blade

Workaround:

a. sel events can be cleared for server X/Y by

scope chassis X
scope server Y
clear sel
commit
You can use sel backup policy to clear it automatically.

b. threshold values can be changed by
ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 20 0×36 0×00 0×47 0×44 0×00 0×51 0×55

ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 19 0×36 0×00 0×47 0×44 0×00 0×51 0×55

Before making this change run the following commands and confirm the output equals these values:

ipmitool -H BMC_IP_ADDRESS -U user -P password -I lanplus raw 4 0×27 19

36 00 49 46 00 51 55

ipmitool -H BMC_IP_ADDRESS -U user -P password -II lanplus raw 4 0×27 20

36 00 49 46 00 51 55

If the output does not match please contact TAC for assistance on implementing this workaround.

Threshold needs to be set again after BMC reset.

Cisco UCS PALO Details

Per UCS the PALO cards are listed as

Boot times based on firmware version:

PALO Boot Time Firmware 1.2(1b)

PALO Boot Time Firmware 1.3(1c)

So the fix is to upgrade.  Not the easiest process, but not that hard either.  As from the stats above, v1.3(1c) is definitely worth the upgrade!  No more waiting around for 2008 to boot.

Notes: We also run RedHat Linux and VMware ESX in this environment.  They boot without issues or delay with the same PALO adapters. Also, it only took 2 minutes and 16 seconds to get network connectivity up on 2008 running version 1.3(1c) firmware.  Having no other good options I timed the boot using my iPhone.  The above images are from screen-captures on the phone.  UCS is not 3G connected :).

After connecting to the console, the following command was ran. nexus1kv# sh interface counters errors

--------------------------------------------------------------------------------
Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards
--------------------------------------------------------------------------------
Veth30            --          --          --          --          --     1127136
Veth69            --          --          --          --          --       29654
Veth70            --          --          --          --          --       31966
Veth71            --          --          --          --          --       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifying the VM is indeed “w2k_serv0″
nexus1kv# sh interface Veth30 description

-------------------------------------------------------------------------------
Interface                Description
-------------------------------------------------------------------------------
Vethernet30              w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported
nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).
nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1
Veth70                   linux_host01, Network Adapter 2
Veth71                   linux_host01, Network Adapter 3

After a little more investigation, it turned out that “w2k_serv01″ was deployed from a legacy template that was created on our old AMD cluster. Looking at the network interface in Windows 2003, it showed up as an “AMD …” network adapter.

To try and correct these errors, I removed the current network adapter through VMware vCenter and added a new one. This forced the OS in seeing the network interface as a new device and install fresh drivers. Now in Windows it shows up as an Intel network inteface.

Now that the new network adapter has been installed, I reset the statistics (and errors) on all the interfaces.
nexus1kv# clear counters

This command will clear "show interface" counters on all interfaces
Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.
nexus1kv# sh interface counters errors | inc Veth30

Veth30            --          --          --          --          --           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error filed there. Yeah, that was a run-on sentence.  Sorry.

Below are pictures of the full rack that includes:

• Two Cisco UCS Chassis fully populated
• Two Cisco Nexus 5010 switches
• Two Cisco 6120XP switches
• Two horizontal PDUs (Circuit 1)
• Two vertical PDUs (Circuit 2)

Just in case you wanted a pretty view, the last picture is from my mountain trip last weekend

There are a total of 7 blades in our VMware environment, but only 5 of those are dedicated to our main HA/DRS cluster. That gives us ~240 gigs of RAM for the main cluster. Currently, I am seeing a VM consolidation ratio of about 24 VMs (virtual machines) per B200-M1 blade. The limitation here is definitely the RAM. The CPU itself is less than 25% utilized per blade. Below is a screenshot of a B200-M1 host in our cluster.

Cisco UCS B200-M1 VM Ratio

Notes: The VMs running on the above host are

Operating system utilization

Unpacked B200-M1 Blades

Unpacked Blades B200/B250

B250-M1 Blade Pictures

B250-M1 Guide

B250-M1 Open Front

B250-M1 Open Rear

B250-M1 Internal Top

B250-M1 Top Internal 2

B250-M1 Internal Palo

B250-M1 And B200-M1 In Chassis

B250-M1 Bezel

B200-M1 Blade Pictures

B200-M1 Guide

B200-M1 Internal Air Flow

B200-M1 Top Full

B200-M1 Top CPU / RAM

B200-M1 Rear Palo

B200-M1 Top Internal w/vents

B200-M1 Bezel