Having limited funds during our initial UCS purchase, we were only able to fill our B200-M1 blades with 4gb RAM modules. Below is an image showing the current RAM to CPU utilization.

vSphere RAM Usage

I have put out a few previous posts concearing consolidation ratios here:

http://blog.colovirt.com/2010/05/24/cisco-vmware-cisco-ucs-b6620-vmware-consolidation-ratio/
http://blog.colovirt.com/2010/06/01/cisco-vmware-cisco-ucs-b250-m1-vmware-consolidation-ratio-oracle-dbs/

A recap of our current setup is below.

Density = ~19.8 VMs per B200-M1 server w/48g of RAM

Again, this is a mixed environment containing host OSs:

• Windows 2003
• Windows 2008
• RedHat Linux

We have stopped deploying any VMs in this cluster due to RAM constraints.  Thankfully our RAM upgrade is being shipped out currently from Cisco.

This will be a total replacement of the 4gig RAM modules currently installed in the B200′s with 8gig sticks.

Ordered 60 x 8gb sticks = 480gig upgrade (total) to ESX hosts

Of course our 4gig modules will not be wasted.  They will be used to fill our B250-M1 servers to maximum capacity (number of RAM sticks).

This upgrade should get us through until another UCS chassis can be purchased next year.  My advice for anyone who is planning on deploying UCS for the sole purpose of running VMware:

• Bite the bullet and go with the 8 gig sticks from the start if deploying only B200 blades
• B250 1/2 populated with 4 gig sticks = 98gigs of RAM.  You can put 192 gigs of RAM in using 4gig sticks or 384gigs with 8gig sticks
• Keep at least one spare server in each UCS chassis for un-planned expansion or to use if another blade dies

Follow me on twitter

Now all that we need to do is get some data over to it.  We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit.  You can not change the order in which each device is racked.  Also, the controllers (SPs) are built into the drive tray.  Best part is that this will be wired up via 10gbE connectivity over fiber.

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Initial RAM Error - Server 1

“scope server” help output via CLI

ucs_6120-A# scope server ?
  WORD          <chassis-id>/<blade-id>
  dynamic-uuid  Dynamic UUID

Since the problem is currently showing up on chassis 2 blade 1, we set the scope for it below

ucs_6120-A# scope server 2/1

Now we use “show sel” to view the event log, filtering the output with “include”. sel = System Event Log

ucs_6120-A /chassis/server # show sel | include error

4ef | 04/15/2010 10:40:28 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 0, DIMM Socket: 4, Channel: A, Socket: 1 | Asserted
4f1 | 04/15/2010 10:40:29 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 1, DIMM Socket: 5, Channel: A, Socket: 1 | Asserted
4f3 | 04/15/2010 10:40:30 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 2, DIMM Socket: 6, Channel: A, Socket: 1 | Asserted
4f5 | 04/15/2010 10:40:31 | Memory 0×02 | Uncorrectable ECC/other uncorrectable memory error | Rank: 3, DIMM Socket: 7, Channel: A, Socket: 1 | Asserted

So from the messages above, you can see that there is definitely an issue. I originally thought that this was an actual hardware failure. The server never had a problem and continued ot run. At first only two DIMMs showed errors, but about 3 hours later two more DIMMs errored.

To be safe, I disassociated the service profile from this blade and moved it to a spare. About 30 minutes after the service profile was booted on the new blade DIMM errors showed up on it. This definitely hinted to a bug.

Initial RAM Error - Server 3

It turns out that this is a bug in UCS firmware 1.2

Interestingly Cisco shows the condition to be “Fully populated DIMM slots with X5570 CPU on B250 blade” and with “2 X5570 CPUs”.  In our case they are right with the CPU count, but we are not fully populated with DIMMs. We are only at 1/2 capacity.

I did upgrade the firmware to version 1.3(1c) and the errors went away.  Per cisco this is a “cosmetic” error and did not actually affect the server.

Just in case you do not have access to Cisco.com to view the bug ID, the text is listed below:

Ventura: 1.2(1b): X5570:Sel Events P0V75_DDR3_P2 errors filling up.
Symptom:

Amber LED/Sel Events on B250 blades with 2 X5570 CPU.
A /chassis/server # show sel
17 | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Asserted | Reading
0.71 < Threshold 0.71 Volts 18 | | Platform alert LED_BLADE_STATUS | LED color is amber | Asserted 19 | | Platform alert LED_BLADE_STATUS | LED color is green | Deasserted 1a | | Voltage P0V75_DDR3_P2 | Lower critical – going low | Deasserted | Reading0.73>Threshold 0.71 Volts
Conditions:

Fully populated DIMM slots with X5570 CPU on B250 blade

Workaround:

a. sel events can be cleared for server X/Y by

scope chassis X
scope server Y
clear sel
commit
You can use sel backup policy to clear it automatically.

b. threshold values can be changed by
ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 20 0×36 0×00 0×47 0×44 0×00 0×51 0×55

ipmitool -H BMC_IP_ADDRESS -U user -P password -I
lanplus raw 4 0×26 19 0×36 0×00 0×47 0×44 0×00 0×51 0×55

Before making this change run the following commands and confirm the output equals these values:

ipmitool -H BMC_IP_ADDRESS -U user -P password -I lanplus raw 4 0×27 19

36 00 49 46 00 51 55

ipmitool -H BMC_IP_ADDRESS -U user -P password -II lanplus raw 4 0×27 20

36 00 49 46 00 51 55

If the output does not match please contact TAC for assistance on implementing this workaround.

Threshold needs to be set again after BMC reset.

Cisco UCS PALO Details

Per UCS the PALO cards are listed as

Boot times based on firmware version:

PALO Boot Time Firmware 1.2(1b)

PALO Boot Time Firmware 1.3(1c)

So the fix is to upgrade.  Not the easiest process, but not that hard either.  As from the stats above, v1.3(1c) is definitely worth the upgrade!  No more waiting around for 2008 to boot.

Notes: We also run RedHat Linux and VMware ESX in this environment.  They boot without issues or delay with the same PALO adapters. Also, it only took 2 minutes and 16 seconds to get network connectivity up on 2008 running version 1.3(1c) firmware.  Having no other good options I timed the boot using my iPhone.  The above images are from screen-captures on the phone.  UCS is not 3G connected :).

After connecting to the console, the following command was ran. nexus1kv# sh interface counters errors

--------------------------------------------------------------------------------
Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards
--------------------------------------------------------------------------------
Veth30            --          --          --          --          --     1127136
Veth69            --          --          --          --          --       29654
Veth70            --          --          --          --          --       31966
Veth71            --          --          --          --          --       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifying the VM is indeed “w2k_serv0″
nexus1kv# sh interface Veth30 description

-------------------------------------------------------------------------------
Interface                Description
-------------------------------------------------------------------------------
Vethernet30              w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported
nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).
nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1
Veth70                   linux_host01, Network Adapter 2
Veth71                   linux_host01, Network Adapter 3

After a little more investigation, it turned out that “w2k_serv01″ was deployed from a legacy template that was created on our old AMD cluster. Looking at the network interface in Windows 2003, it showed up as an “AMD …” network adapter.

To try and correct these errors, I removed the current network adapter through VMware vCenter and added a new one. This forced the OS in seeing the network interface as a new device and install fresh drivers. Now in Windows it shows up as an Intel network inteface.

Now that the new network adapter has been installed, I reset the statistics (and errors) on all the interfaces.
nexus1kv# clear counters

This command will clear "show interface" counters on all interfaces
Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.
nexus1kv# sh interface counters errors | inc Veth30

Veth30            --          --          --          --          --           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error filed there. Yeah, that was a run-on sentence.  Sorry.

Below are pictures of the full rack that includes:

• Two Cisco UCS Chassis fully populated
• Two Cisco Nexus 5010 switches
• Two Cisco 6120XP switches
• Two horizontal PDUs (Circuit 1)
• Two vertical PDUs (Circuit 2)

Just in case you wanted a pretty view, the last picture is from my mountain trip last weekend

There are a total of 7 blades in our VMware environment, but only 5 of those are dedicated to our main HA/DRS cluster. That gives us ~240 gigs of RAM for the main cluster. Currently, I am seeing a VM consolidation ratio of about 24 VMs (virtual machines) per B200-M1 blade. The limitation here is definitely the RAM. The CPU itself is less than 25% utilized per blade. Below is a screenshot of a B200-M1 host in our cluster.

Cisco UCS B200-M1 VM Ratio

Notes: The VMs running on the above host are

Operating system utilization

Unpacked B200-M1 Blades

Unpacked Blades B200/B250

B250-M1 Blade Pictures

B250-M1 Guide

B250-M1 Open Front

B250-M1 Open Rear

B250-M1 Internal Top

B250-M1 Top Internal 2

B250-M1 Internal Palo

B250-M1 And B200-M1 In Chassis

B250-M1 Bezel

B200-M1 Blade Pictures

B200-M1 Guide

B200-M1 Internal Air Flow

B200-M1 Top Full

B200-M1 Top CPU / RAM

B200-M1 Rear Palo

B200-M1 Top Internal w/vents

B200-M1 Bezel

This is not a major issue, just an annoyance.  We also use RedHat Satellite server and can not join to the patch management system with the incorrect date. So my question is where does the OS get it’s bad time from? I figure that the OS gets the time from the bios and that the bios would obtaion the information from UCS Manger. That does not appear to be the case.

Below is an example of the issue:

Cisco UCS Time

Above in the bottom right corner shows the correct time and date in the Cisco UCS Manager.

Below you can see that the date shows 2009 on a freshly installed server in UCS.  The difference of the time and day are due to screen shots taking a day apart, but you get the idea.  It’s not 2009.

Cisco UCS Bios Time

Server has just been installed and booted. The command “date” is issued to see the current time.

[root@test-server02 build]# date
Sat Aug 15 02:37:24 EDT 2009

As seen from above, the date is incorrect (2009). NTP needs to be updated, but the service must be stopped first, or else it will not accept the new time.

[root@test-server02 build]# /etc/init.d/ntpd stop
Shutting down ntpd:                                        [  OK  ]

Once stopped, “ntpdate” is used to query an NTP server and adjust the local server time.

[root@test-server02 build]# ntpdate pool.ntp.org
27 Apr 14:36:23 ntpdate[5214]: step time server 70.86.250.6 offset 22075095.967480 sec

Now that the system time is correct, NTP service is restarted to

[root@test-server02 build]# /etc/init.d/ntpd start
Starting ntpd:                                             [  OK  ]

From here on out, the time will be correct and sync with the NTP service withouth issue.

[root@test-server02 build]# date
Tue Apr 27 14:36:40 EDT 2010

First check and see if there are any current licenses installed. This being a new switch, there aren’t any.

nexus_5000# sh license
nexus_5000#

Licenses are normally tied to “host IDs”. Below shows the host ID of the switch.

nexus_5000# sh license host-id
License hostid: VDH=3WI246599Z

The license file is not local to the switch yet, so below I fetch it via TFTP.nexus_5000# copy tftp://127.0.0.1:69/Cisco_license/NX5KLIC.lic bootflash: vrf default_vrf

Trying to connect to tftp server......
Connection to Server Established.
[                         ]         0.50KB
TFTP get operation was successful

Next, the license is installed with the “install license” command.

nexus_5000# install license bootflash:NX5KLIC.lic
Installing license .....done

Now that the license has been added, use “show license” to confirm.

nexus_5000# show license
NX5KLIC.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT ENTERPRISE_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
1 \
RMA license" SIGN=DDWD62626W7f
INCREMENT FC_FEATURES_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
2 \
RMA license" SIGN=VVSVDS848SFW

Another way is to see the names of the installed licenses.

nexus_5000# show license brief
NX5KLIC.lic

To be safe, copy the running configuration to startup.

nexus_5000# copy running-config startup-config

Also, here I copy the configuration to TFTP backup.

nexus_5000# copy running-config tftp://127.0.0.1:69/Cisco_config/ns5k2_startup vrf default_vrf
Trying to connect to tftp server......
Connection to Server Established.
[#                        ]         4.50KB
TFTP put operation was successful

Notes:  All IP, license files, and hashes were changed.  The installation process is pretty strait forward.  The managment console (GUI) should also be able to provide this function.

About 4 months ago, we were looking to add another RamSan into our environment after deciding EMCs flash DAEs were not for us (at this time).  I was initially a little weary of flash based storage, but had faith in RamSan.  After reviewing the product literature more and getting some hands on with the unit, those worries are gone.  Our RamSan 500 was purchased with 2 terabytes of Flash storage and the DRAM write cache was expanded to 64 gigabytes.

With the write cache expansion, this new units cache was literally 1/2 the size of the current RamSan 400 DRAM capacity.  With this in place and the high IO that the flash disks can maintain, there has not been any noticeable speed differences between the two units.  The 500 also has an internal battery for power backup.  If power is lost, the battery will keep the unit going long enough to copy the cache’s un-written data to disk.

There are two main features that I love.  First is tha ability to lock a LUN on the 500 series into the cache, so even-though it is mainly a flash storage system, extreme IO can be accomplished.  By locking the lun in cache, the data will reside in RAM and never go to disk unless power is lost.  The second is the ability to centrally manage other RamSans on the network (auto discovered) with a specific license key.  Almost forgot to mention, the unit has quad 4 gigabit fiber connectivity.

Notes:  If you have any questions, please ask!  Again, these units are amazing.

Per the title, this is really about the EMC Clariion, not RamSans.  Since the RamSan 500 was fronted with the DRAM cache, and the EMC CX4 series contains cache as well, I was curious.  I already knew that each Service Processor (SP) in the EMC has 4 gig of cache, and that a LUN can only be active on one SP at a time.  Also, per a previous blog post, each DAE has a theoretical max throughput of 8 gigabit per second, 4 gigabit if a single LUN stripes across the whole shelf.

CX4 DAE (general) information
http://blog.colovirt.com/2009/05/29/san-emc-cx4-dae-drive-shelf-information/

As the conversation continued, I was told that typically the Service Processors caching would be disabled on LUNS that reside on the solid state flash drives.  In all actuality, I think it should.  Being that the typical Clariion implementation will not be dedicated as a solid state SAN, they will have to co-mingle with spindle based hard drives(regular).  The throughput is a LOT faster and latency a lot lower on the flash drives.  Enabling SP caching on LUNs contained in flash could possibly have a really bad affect on the rest of the system.  Being that the cache really speeds up IO for spindle based disks.

Most vendors are toting a 20 physical disks(15kRPM) to 1 Flash disk consolidation

What I am getting at is there seems to be the possibility of flooding the Service Processors cache with flash based storage data, depending on change rate.  Since flash is a lot faster, their LUNs would typically be used for databases or high volume services.  Even with the ability to do QOS in the CX4 series, disabling Service Processor caching for the flash LUNS looks to be the best solution.

Each DAE contains 2 LCC interfaces.  LCC is the acronymn for “Link Control Card”.  Typically one LCC goes to each Service Processor.  So that means that each DAE has two 4 gig links for a total of 8gb theoretical throughput.

This is where I will be reading more information.  Just because you have 2 LCC cards, each LUN can only be assigned to 1 Service Processor!  If my theory is correct, that means if all drives in one shelf is dedicated to a single LUN, and that LUN can only be active on one SP, does that mean that the actual throughput will be limited to 4gb instead of 8?  This being due to the other LCC loop being connected to the SP that is not assigned control of that LUN.

Any thoughts?  Feel free to comment!

First, start DD and pass it into the background using “&”

root@workstation:~# dd if=/dev/urandom of=/dev/sdd2 bs=4K &
[1] 25263

The process id (PID) was shown on the second line and in this case is “25263″

Now use “kill” to pass the user defined signal 1 to the process (dd)

root@workstation:~# kill -USR1 25263

Below is what follows after the kill is initiated

root@workstation:~# 9770295+0 records in
9770294+0 records out
40019124224 bytes (40 GB) copied, 9721.18 s, 4.1 MB/s

Now I know that the drive (device) that is being filled with random data has a 160 gig capacity. This tells me that dd has a long way to go.

Another Source For Information. Especially check where it shows using “watch” to keep a loop going for “kill”.

Note: in this case, “kill” does not actually stop the running process. “dd” will continue the copy in the background until complete. This is really only useful if you know the size of the destination drive. Same theory works if just coping data from one drive to another, just figure out the size of the source and use this to show the progress.  Also, the speed is not bad for a USB attached disk!  Last thing, just because this signal works on dd, it does not mean it will on any other commands.  I tested it with “cp” and it did kill the process insted of giving feedback.