Colocation to Virtualization » Networking

Cisco UCS and Nexus 5000 DataCenter – Our Implementation

Kevin Goodman — Wed, 07 Jul 2010 19:07:07 +0000

Our Cisco 10gbE implementation consists of 2 Chassis fully populated UCS with a mix of full and half width blades. The servers are all boot from SAN with no local disks. “PALO” cards are used in all servers which allow us to do FCOE. 7 of the blades are running VMware ESX 4 (vSphere) and the rest are a mix of RedHat Linux and Windows 2008.

Storage:

All servers boot from SAN over FCOE

Primary storage is via Fibre Channel to an EMC CX4-240

A few servers are also connected via Fibre Channel to RamSan SSD SANs

The CX4-240 is also connected to the UCS environment via quad 10gbE links over fiberNetwork:

Core 10gbE switching is provided via Cisco Nexus 5010s

1 Gigabit Ethernet to legacy servers are provided by dual 48 port C2148 Fabric Extenders

Each C2148 is connected to a Nexus 5010 by dual 10gbE uplinks

Each 5010 is connected via quad 4gig fiber to the CX4-240 to provide FCoE to physical servers outside of the UCS environment

Each 5010 is also connected to the CX4-240 by dual 10gbE fiber ports to provide ISCSI to all of the network

Each UCS switch (6120s) are uplinked to the 5010s by dual 10gbE interfaces

Each UCS chassis is uplinked to the 6120s by 8 qty 10gbE ports (4 connections per IO module)

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon. This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120. I definitely would love feedback on this. How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Filed under: Colocation, EMC, Hardware, Networking, RamSan, SAN (Storage Area Network)

Cisco, VMware: Nexus 1000v Tracking VM Interface Errors

Kevin Goodman — Mon, 07 Jun 2010 10:38:48 +0000

So I have found the first use of our new Cisco Nexus 1000v virtual switch. The counters (stats) were reset about a week agon and this is the first time they have been reviewed since. Almost all virtual machines show no errors, but there were a few that were high.

After connecting to the console, the following command was ran. nexus1kv# sh interface counters errors

--------------------------------------------------------------------------------
Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards
--------------------------------------------------------------------------------
Veth30            --          --          --          --          --     1127136
Veth69            --          --          --          --          --       29654
Veth70            --          --          --          --          --       31966
Veth71            --          --          --          --          --       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifying the VM is indeed “w2k_serv0″
nexus1kv# sh interface Veth30 description

-------------------------------------------------------------------------------
Interface                Description
-------------------------------------------------------------------------------
Vethernet30              w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported
nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).
nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1
Veth70                   linux_host01, Network Adapter 2
Veth71                   linux_host01, Network Adapter 3

After a little more investigation, it turned out that “w2k_serv01″ was deployed from a legacy template that was created on our old AMD cluster. Looking at the network interface in Windows 2003, it showed up as an “AMD …” network adapter.

To try and correct these errors, I removed the current network adapter through VMware vCenter and added a new one. This forced the OS in seeing the network interface as a new device and install fresh drivers. Now in Windows it shows up as an Intel network inteface.

Now that the new network adapter has been installed, I reset the statistics (and errors) on all the interfaces.
nexus1kv# clear counters

This command will clear "show interface" counters on all interfaces
Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.
nexus1kv# sh interface counters errors | inc Veth30

Veth30            --          --          --          --          --           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error filed there. Yeah, that was a run-on sentence. Sorry.

After connecting to the console, the following command was ran.

nexus1kv# sh interface counters errors

——————————————————————————–

Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards

——————————————————————————–

Veth30            –          –          –          –          –     1127136

Veth69            –          –          –          –          –       29654

Veth70            –          –          –          –          –       31966

Veth71            –          –          –          –          –       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifing the VM is indeed “w2k_serv0″

nexus1kv# sh interface Veth30 description

——————————————————————————-
Interface Description
——————————————————————————-
Vethernet30 w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported

nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).

nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1

Veth70                   linux_host01, Network Adapter 2

Veth71                   linux_host01, Network Adapter 3

Now that the new network adapater has been installed, I reset the statistics (and errors) on all the interfaces.

nexus1kv# clear counters

This command will clear “show interface” counters on all interfaces

Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.

t1prd-nexus01# sh interface counters errors | inc Veth30

Veth30            –          –          –          –          –           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error fied there. Yeah, that was a run-on scentance. Sorry.

Filed under: Hardware, Networking, VMWare

Cisco UCS Implementation In Pictures: Part 3 The Finished Product

Kevin Goodman — Thu, 03 Jun 2010 12:02:42 +0000

We finished up our Cisco UCS installation about 2 months ago, but have been to busy to get the last set of pictures up. Sorry for the delay! Just in case you don’t follow me on Twitter, UCS is working out nicely.

Below are pictures of the full rack that includes:

Two Cisco UCS Chassis fully populated
Two Cisco Nexus 5010 switches
Two Cisco 6120XP switches
Two horizontal PDUs (Circuit 1)
Two vertical PDUs (Circuit 2)

Just in case you wanted a pretty view, the last picture is from my mountain trip last weekend

Filed under: Colocation, Hardware, Networking

Storage, Networking, EMC: DataDomain Replication

Kevin Goodman — Wed, 26 May 2010 14:49:41 +0000

We have a total of 3 DataDomains currently in production. One at our “Tier 1″ site, “Tier 2″(DR) and one in Europe. All DataDomain appliances have the ability to replicate data among themselves. This will be a general overview of how to setup replication between two DataDomains

On the source, I have already setup a directory tree for “/backup/europe_data”. All files destined for our Euroupe office will be placed here. On the DataDomain devices, replicated folders are added manually. By default none are replicated.
*********************
Be aware that if you are viewing this through a web browser that a lot of the command might be hidden due to formatting of the WordPress theme. It is best viewed by an RSS reader. If you would like the full text, email or leave a comment and I can send you a PDF version.
*********************

Below is the help output for “replication add” command.

user01@us-dd565# replication add  ?
Commands matching "replication add":
  replication add source  destination
                 Add a replication pair

I wish the syntax was easy as what was stated above, but below is the actual command used to setup replication on the source DataDomain

user01@us-dd565# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once the source is setup, the destination endpoint/DataDomain also must be configured. The command is identical to what was entered at the source DataDomain

user01@eu-dd01# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once replication is setup on both DataDomains, log back into the source and start the replication.

user01@us-dd565# replication initialize
Incorrect syntax "replication initialize", showing help:
  replication initialize
                 Initialize replication on the source
                 (configure both source and destination
                 first)

Below is the actual syntax to startup replication for the directory that we just configured.

user01@us-dd565# replication initialize dir://eu-dd01.testdomain.com/backup/europe_data
(00:04) Waiting for initialize to start...
(00:06) Initialize started.
Use 'replication watch dir://eu-dd01.testdomain.com/backup/europe_data' to monitor progress.

Since I am impatient and want to know for sure this is working, I use “replication watch”. This shows the current progress of the replication between DataDomains.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
    Use Control-C to stop monitoring.

(00:57) Replication initialize started...
(00:59) initializing 3/3:
(10:35)     : pre-comp:   0%    27 KB/s, network    26 KB/s, 0 of 6 files

When new data is written to the mount point, replication automatically starts. Below is what shows if no data is currently being replicated.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
**** Replication initialize or recover already completed.

“replication show config” is used to display all the configured replication points.

user01@us-dd565# replication show config
CTX   Source                                                    Destination                                                 Connection Host and Port                       Enabled
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------
11    dir://us-dd565.testdomain.com/backup/mysql-rep       	dir://us-t2-dd510.testdomain.com/backup/mysql-rep	us-t2-dd510.testdomain.com   (default)     yes
12    dir://us-dd565.testdomain.com/backup/europe_data        	dir://eu-dd510.testdomain.com/backup/europe_data	eu-dd510.testdomain.com   (default)     yes
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------

Sorry for all of the spacing being off. All the tabs did not come over when copying data from the console.

user01

Filed under: Data Domain, deduplication, NAS, Networking

Cisco UCS Implementation In Pictures: Part 2 Nexus B250 And B200 External/Internals

Kevin Goodman — Mon, 10 May 2010 13:25:55 +0000

Below is a collection of photos from phase 2 of our Cisco UCS implementation. This consisted of installing the Cisco UCS Blade Chassis, blade switches and server blades.

Unpacked B200-M1 Blades

Unpacked Blades B200/B250

B250-M1 Blade Pictures

B250-M1 Guide

B250-M1 Open Front

B250-M1 Open Rear

B250-M1 Internal Top

B250-M1 Top Internal 2

B250-M1 Internal Palo

B250-M1 And B200-M1 In Chassis

B250-M1 Bezel

B200-M1 Blade Pictures

B200-M1 Guide

B200-M1 Internal Air Flow

B200-M1 Top Full

B200-M1 Top CPU / RAM

B200-M1 Rear Palo

B200-M1 Top Internal w/vents

B200-M1 Bezel

Filed under: Colocation, Hardware, Networking

Cisco, Networking: VMware Cisco Nexus 1000v Increasing Default Max Ports

Kevin Goodman — Wed, 05 May 2010 13:26:58 +0000

Earlier today we ran into an issue with exceeding the default max number of ports (virtual) on a port profile. The default is set to 32 and the max is 1024 per vethernet port profile. I really can’t believe how low the default setting is. Most VMware people I know consolidate waaayyyy more than 32 host per datacenter.

Below shows the error in VMware Virtual Center.

Here is the profile “iSCSI” before modification. Note the max 32 and available count.

The only way to increase this value is through command line in the active Nexus 1000v.

nexus01# configure t
nexus01(config)# port-profile type vethernet iSCSI
nexus01(config-port-prof)# vmware max-ports 1024
nexus01(config-port-prof)# end

Once the above command is issued, Virtual Center Client should show a reconfiguration task for the distributed virtual switch

Once the tasks is completed, go back into Home -> Inventory -> Networking and select the port profile in the distributed switch. The “total ports” should now be set to 1024.

Make sure to save your configuration!

nexus01# copy running-config startup-config

Notes: This should be non-intrussive, but make sure to test it in your environment!

Filed under: Colocation, Networking, VMWare

Hardware, Linux, Networking: Cisco UCS Time Problem

Kevin Goodman — Wed, 28 Apr 2010 14:12:55 +0000

So we have our new Cisco UCS system installed and a weird problem is showing up. The Cisco UCS Manager console shows the correct date (2010), but when setting up a new server, the date is incorrect. Also, an NTP server (working correctly) is set. Since we mainly run Linux here, the NTP service will not update the date/time from the NTP server because of how long the difference is between the system clock and NTP. Also, on a Windows 2008 install we had to manually adjust the time/date as well

This is not a major issue, just an annoyance. We also use RedHat Satellite server and can not join to the patch management system with the incorrect date. So my question is where does the OS get it’s bad time from? I figure that the OS gets the time from the bios and that the bios would obtaion the information from UCS Manger. That does not appear to be the case.

Below is an example of the issue:

Cisco UCS Time

Above in the bottom right corner shows the correct time and date in the Cisco UCS Manager.

Below you can see that the date shows 2009 on a freshly installed server in UCS. The difference of the time and day are due to screen shots taking a day apart, but you get the idea. It’s not 2009.

Cisco UCS Bios Time

Server has just been installed and booted. The command “date” is issued to see the current time.

[root@test-server02 build]# date
Sat Aug 15 02:37:24 EDT 2009

As seen from above, the date is incorrect (2009). NTP needs to be updated, but the service must be stopped first, or else it will not accept the new time.

[root@test-server02 build]# /etc/init.d/ntpd stop
Shutting down ntpd:                                        [  OK  ]

Once stopped, “ntpdate” is used to query an NTP server and adjust the local server time.

[root@test-server02 build]# ntpdate pool.ntp.org
27 Apr 14:36:23 ntpdate[5214]: step time server 70.86.250.6 offset 22075095.967480 sec

Now that the system time is correct, NTP service is restarted to

[root@test-server02 build]# /etc/init.d/ntpd start
Starting ntpd:                                             [  OK  ]

From here on out, the time will be correct and sync with the NTP service withouth issue.

[root@test-server02 build]# date
Tue Apr 27 14:36:40 EDT 2010

Filed under: Hardware, Linux, Networking

Networking, UCS: Cisco Nexus License Installation

Kevin Goodman — Thu, 08 Apr 2010 18:06:48 +0000

The following is a quick run through of installing a license file for a Cisco Nexus device.

First check and see if there are any current licenses installed. This being a new switch, there aren’t any.

nexus_5000# sh license
nexus_5000#

Licenses are normally tied to “host IDs”. Below shows the host ID of the switch.

nexus_5000# sh license host-id
License hostid: VDH=3WI246599Z

The license file is not local to the switch yet, so below I fetch it via TFTP.nexus_5000# copy tftp://127.0.0.1:69/Cisco_license/NX5KLIC.lic bootflash: vrf default_vrf

Trying to connect to tftp server......
Connection to Server Established.
[                         ]         0.50KB
TFTP get operation was successful

Next, the license is installed with the “install license” command.

nexus_5000# install license bootflash:NX5KLIC.lic
Installing license .....done

Now that the license has been added, use “show license” to confirm.

nexus_5000# show license
NX5KLIC.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT ENTERPRISE_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
1 \
RMA license" SIGN=DDWD62626W7f
INCREMENT FC_FEATURES_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
2 \
RMA license" SIGN=VVSVDS848SFW

Another way is to see the names of the installed licenses.

nexus_5000# show license brief
NX5KLIC.lic

To be safe, copy the running configuration to startup.

nexus_5000# copy running-config startup-config

Also, here I copy the configuration to TFTP backup.

nexus_5000# copy running-config tftp://127.0.0.1:69/Cisco_config/ns5k2_startup vrf default_vrf
Trying to connect to tftp server......
Connection to Server Established.
[#                        ]         4.50KB
TFTP put operation was successful

Notes: All IP, license files, and hashes were changed. The installation process is pretty strait forward. The managment console (GUI) should also be able to provide this function.

Filed under: Hardware, Networking, SAN (Storage Area Network)

Linux, Networking, Security: Get Remote SSL Certificate From Command Line

Kevin Goodman — Wed, 03 Mar 2010 20:31:00 +0000

Easy way to get the SSL certificate of a server from the command line in Linux. The nice thing about it is that you get the full certificate chain. Nice for troubleshooting issues. After the “-connect”, specify the host and port you want to connect to. TCP port 443 is the default https port.

[user1@testserver ~]$ openssl s_client -connect mail.google.com:443

CONNECTED(00000003)
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDIjCCAougAwIBAgIQHxn23jXdY6FCkYrVLMCrEjANBgkqhkiG9w0BAQUFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wOTEyMTgwMDAwMDBaFw0x
MTEyMTgyMzU5NTlaMGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHFA1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKFApHb29nbGUgSW5jMRgw
FgYDVQQDFA9tYWlsLmdvb2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ
AoGBANknyBHye+RFyUa2Y3WDsXd+F0GJgDjxRSegPNnoqABL2QfQut7t9CymrNwn
E+wMwaaZF0LmjSfSgRSwS4L6ssXQuyBZYiijlrVh9nbBbUbS/brGDz3RyXeaWDP2
BnYyrVFfKV9u+BKLrebFCDmzQ0OpW5Ed1+PPUd91WY6NgKtTAgMBAAGjgecwgeQw
DAYDVR0TAQH/BAIwADA2BgNVHR8ELzAtMCugKaAnhiVodHRwOi8vY3JsLnRoYXd0
ZS5jb20vVGhhd3RlU0dDQ0EuY3JsMCgGA1UdJQQhMB8GCCsGAQUFBwMBBggrBgEF
BQcDAgYJYIZIAYb4QgQBMHIGCCsGAQUFBwEBBGYwZDAiBggrBgEFBQcwAYYWaHR0
cDovL29jc3AudGhhd3RlLmNvbTA+BggrBgEFBQcwAoYyaHR0cDovL3d3dy50aGF3
dGUuY29tL3JlcG9zaXRvcnkvVGhhd3RlX1NHQ19DQS5jcnQwDQYJKoZIhvcNAQEF
BQADgYEAicju7fexy+yRP2drx57Tcqo+BElR1CiHNZ1nhPmS9QSZaudDA8jy25IP
VWvjEgaq13Hro0Hg32ZNVK53qcXwjWtnCAReojvNwj6/x1Ciq5B6D7E6eiYDSfXJ
8/a2vR5IbgY89nq+wuHaA6vspH6vNR848xO3z1PQ7BrIjnYQ1A0=
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 1778 bytes and written 316 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: DEB23CF699255054E08F69181B2342E9F6D6DF0D02B399C36034E0D8BE18AC0C
    Session-ID-ctx:
    Master-Key: D696A99CEC2FDD9535FE2EC936531AD129FD97E56441E37AE7A143C40304E395EA7DA039797B948B009B42DA5377E668
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1267560715
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=UTF-8
Content-Length: 1350
Date: Tue, 02 Mar 2010 20:11:57 GMT
Server: GFE/2.0
X-XSS-Protection: 0

Filed under: Linux, Middle Ware, Networking, Security

Storage, Network: What I have Been Doing (EMC,Cisco UCS)

Kevin Goodman — Tue, 02 Mar 2010 19:41:19 +0000

This is more of an informational update of things that I have going on right now. I normally do not publish day-to-day type of things, but here we go.

Storage

We have received both replacement drives for our EMC Clariion CX340 and four new DAEs (disk shelves) for our CX4-240

Clariion CX3

The CX3 was originally bought to speed up our Oracle implementation. This was accomplished by ordering lots of fast disks (spindles) that were small. We wound up with 6 DAEs filled with 73gig 15kRPM disks, totalling 90 dedicated drives for Oracle.

This was great for the original purpose but the unit was replaced a year after initial deployment with a RamSan and EMC CX4. Having been decommissioned from production and moved to the tier 2 site, the need for space over IOPS (speed) drastically increased. Trying to keep performance and space requirements in balance, the decision has been made to go with a smaller RamSan for Oracle at the tier 2 site. This gives us the ability to replace the small 73 gigabyte drives with bigger 600 gigabyte 10kRPM disks. Replacing those disk with the same quantity of 600 gig ones will give us ~8 times as much space.

The RamSan will almost double the IOPS capacity that the CX3 is able to achieve and speed up our data warehouse even more.

Clariion CX4

So last year we went with implementing EMC Recoverpoint SAN based replication. This has been great and served us well! The only downfall was that we were doing “CRR” remote replication only. In a case of a failure and data needed to be recovered, there were no local copies. The snapshot or “point in time” would have to be loaded from the tier 2 site and transferred across the datacenter interconnect. The interconnect being 150 megs slowed this process down.

As planned from the beginning, we are implementing “CLR” local replication as well. This means that there will be a local copy of snapshots saved locally to the CX4. This will give us almost immediate access to the snapshots without being slowed down by the interconnect. The problem with RecoverPoint is that if you have a terrabyte LUN that you want to connect, you must have an extra terrabyte worth of space to save it. This is not really a problem, but a major consideration on the number of drives to buy and the overall expense of the implementation.

In our case, a terrabyte oracle LUN will wind up costing 3 terrabytes in the end. 1 terrabyte for the original data, 1 terrabyte for the local copy (CLR), and 1 terrabyte at the remote tier 2 site (CX3).

Our virtualization effort is continuing and this is another huge factor on the storage expansion. Currently we have 16 LUNs dedicated to the VMware environment. Each is 320 gigs in size. Moving forward, we will be doing a virtual desktop deployment as well. The leftover ~400 gigs will not cut it. So in the new 60 disks, 15 or more will have to be dedicated to VMware.

Cisco UCS

We have begun our UCS voyage. As of last weekend, we did a “rip and replace of our network”. This included rewiring the main network rack and configuring a new network core. Also, the Cisco Nexus 5010, 10 gigabit Ethernet switches are in. Uplinked to them are two 48 port gigabit fabric extenders.

The VMware environment is now connected via dual 10gigE links per server through this infrastructure. Reducing the cable count from 6 to 2 per server. So far verything is stable! A purchase order has been sent out and we should hopefully have two Cisco UCS Blade chassis and switching infrastructure show up within about 30 days.

Filed under: EMC, Networking, RamSan, SAN (Storage Area Network), VMWare

VMware, Linux: Install VMware Tools On RedHat Based Systems

Kevin Goodman — Tue, 12 Jan 2010 18:58:20 +0000

The following is a quick overview of installing VMware Tools on RedHat, CentOS, and Fedora systems. Specifically for VMware ESX, ESXi, and vSphere systems.

First, go into the VMware console and right-click on the VM (Virtual Machine) that you are going to install VMware tools on. Select “Install/Upgrade VMware Tools” option from the list. Below is a screen shot of the menu.

VMware Tools Menu

By default, most CDROM devices are symbolically linked to /dev/cdrom by the operating system.

Just in case, you can search the messages file to see the actual device. This is needed only if /dev/cdrom is not automatically linked or you have setup multiple cdrom devices on the VM (Virtual Machine).

[root@RHserver01 media]# cat /var/log/messages | grep CDROM
Jan 10 10:59:03 RHserver01 kernel: hda: VMware Virtual IDE CDROM Drive, ATAPI CD/DVD-ROM drive

From above, you can see that the actual device is hda, specifically /dev/hda. If you are just curious you can do an “ll” on the /dev/cdrom device to see where is it linked to. In this case again, it’s going to hda.

[root@RHserver01 ~]# ll /dev/cdrom
lrwxrwxrwx 1 root root 3 Jan 12 13:18 /dev/cdrom -> hda

Mount the cdrom device to an empty or non-mounted point on the filesystem. Here I use the defaultly present /media location.

[root@RHserver01 /]# mount /dev/cdrom /media/
mount: block device /dev/cdrom is write-protected, mounting read-only

Below we move into the /media location using “cd”.

[root@RHserver01 /]# cd /media/

“ls” is used to display what files are present. Here we see both an RPM (native RedHat based OS package) and a gzip archive. If you were installing VMware Tools on a non-RedHat derived distribution, you would use the .gz package.

[root@RHserver01 media]# ls
VMwareTools-3.5.0-143128.i386.rpm  VMwareTools-3.5.0-143128.tar.gz

Since we are on RedHat, this is simple. Pass “-i” to the rpm command then the package name to be installed.

[root@RHserver01 media]# rpm -i VMwareTools-3.5.0-143128.i386.rpm

Immediately after installing the RPM, you might see the following errors to your console, or in /var/log/messages.

Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda
Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda

If you are getting these to the console, it makes it hard to continue working form the command line. This is easy to stop. First, cd out of the /media/ mount point

[root@RHserver01 ~]# cd ..

Next, unmount the cdrom device. After doing so, the messages will stop

[root@RHserver01 ~]# umount /dev/cdrom

The “vmware-config-tools.pl” command must be ran from the VMware console. Below is the output you would get if it was tried through a remote session (SSH).

[root@RHserver01 ~]# vmware-config-tools.pl

It looks like you are trying to run this program in a remote session. This
program will temporarily shut down your network connection, so you should only
run it from a local console session. Are you SURE you want to continue?
[no]
Please re-run this program from a local console shell.
Execution aborted.

There is a good reason for this. vmware-config-tools.pl drops networking on the server to install the VMware network drives. In doing so, you loose remote connectivity.

Below shows the actual output from vmware-config-tools.pl on the console

[root@RHserver01 ~]# vmware-config-tools.pl
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Stopping VMware Tools services in the virtual machine:
   Guest operating system daemon:                          [  OK  ]
   Unmounting HGFS shares:                                 [  OK  ]
   Guest filesystem driver:                                [  OK  ]
   Guest memory manager:                                   [  OK  ]
Trying to find a suitable vmmemctl module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmhgfs module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmxnet module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmblock module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

No X install found.

Starting VMware Tools services in the virtual machine:
   Switching to guest configuration:                       [  OK  ]
   Guest memory manager:                                   [  OK  ]
   Guest vmxnet fast network device:                       [  OK  ]
   DMA setup:                                              [  OK  ]
   Guest operating system daemon:                          [  OK  ]

The configuration of VMware Tools 3.5.0 build-143128 for Linux for this running
kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking the following command:
"/usr/bin/vmware-toolbox" during an X server session.

To use the vmxnet driver, restart networking using the following commands:
/etc/rc.d/init.d/network stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/rc.d/init.d/network start

If you wish to configure any experimental features, please run the following
command: "vmware-config-tools.pl --experimental".

Enjoy,

--the VMware team

Notes: From my experience, restart of networking via init.d scripts or rebooting the server is always needed. I personally always reboot the server to be safe.

Posted in Linux, Networking, VMWare

Linux, Security, LDAP: Local Authentication Fallback

Kevin Goodman — Wed, 16 Dec 2009 17:49:33 +0000

I have been setting up and integrating an LDAP authentication system into our infrastructure over the past few days. This is just one small “got-cha” that I ran into. The default setting in the OpenLDAP configuration (/etc/ldap.conf) is to continuously try reconnecting to the LDAP server on failure. This is definitely not what I want to happen if we loose LDAP. In this scenario, when connecting to the server via SSH, the session will hang and eventually timeout. This even removes the ability to login with a local system account.
Example of the timeout when LDAP server is down:

testuser@workstation4-l:~$ ssh test123@ldapclientsrv
Connection closed by 172.16.0.192

To begin, lets look at a typical error that you would get on the system if LDAP communication was down.

Dec 13 12:52:58 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:52:58 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 4 seconds)...
Dec 13 12:53:02 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:53:02 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 8 seconds)...
Dec 13 12:53:10 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:53:10 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 16 seconds)...

As noted before, I was unable to login with a local account. Turns out that the problem was with the default “bind_policy” in /etc/ldap.conf. Per the document:

# Reconnect policy: hard (default) will retry connecting to
# the software with exponential backoff, soft will fail
# immediately.
#bind_policy hard

This was changed to:

bind_policy soft

Once this was changed, I brought up the firewall on the LDAP server and refused connections. Ability to login via LDAP was gone, but the server did fail back to local system authentication

Note(s): When failing back to local authentication, there is no error sent back to the client trying to login, only errors go to /var/log/secure file. The server will just keep rejecting the users login until LDAP is back up. At least this gives you the ability to get in with a local system account in an emergency.

Example error to /var/log/secure when LDAP server is down and local authentication is rejecting the LDAP user received from the client:

Dec 13 12:59:59 ldapServer sshd[2588]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=172.16.0.22

Posted in Linux, Networking, Security

Linux, Filesystem: GNOME Virtual File System (GVFS) Remote Connectivity CLI

Kevin Goodman — Mon, 07 Dec 2009 17:47:46 +0000

When not using NFS, Linux administrators generally move files from one server to the next via SFTP or FTP. This can sometimes be a headache when needing to move large amounts of files between the systems. This is where I like GVFS (GNOME Virtual File System). This subsystem allows you to mount remote systems via the following protocols to a local directory tree:

SSH

FTP

CIFS (Windows shares)

WebDav (HTTP)

Secure WebDav (HTTPS)

Above are the common protocols supported, but there is support for more. Using GVFS to mount the remote filesystem to yours allows you to create and move files to and from the remote system using typical “cp”, “rm”, and “mv” commands. This makes things even easier if you are working through an X windows console. Just bring up the remote directory structure through a file manager application and work from there. Gnome also uses GVFS to manage USB based storage. The following will go through manually connecting to a server using GVFS.Move into the “.gvfs” filesystem in the users home directory. Unless Gnome has automatically mounted a device, this filesystem should be empty.

user01@LinuxDesk:~$ cd ~/.gvfs

In the below example, a remote servers filesystem will be mounted over an SSH/SFTP session.

user01@LinuxDesk:~/.gvfs$ gvfs-mount ssh://user05@SftpServer02
Enter password
Password:

Verify that the location has been mounted.

user01@LinuxDesk:~/.gvfs$ ls
sftp for user05 on SftpServer02

The SFTP was mounted and we can now traverse the remote servers filesystem as if it were our own.

user01@LinuxDesk:~/.gvfs$ cd sftp\ for\ user05\ on\ SftpServer02/

user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02$ ls
app  boot  etc   hs_err_pid15240.log  lib         media  mnt  opt   relay  sbin     srv  tmp  var
bin  dev   home  hs_err_pid8660.log   lost+found  misc   net  proc  root   selinux  sys  usr

Since we logged into the SSH/SFTP system using user “user05″, we can write to any direcotry that remote user has access to.

user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02$ cd home/user05/

Below creates a new file “asdf” containing the text “asdfasdf”. Here we are just testing write capability to the remote server

user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02/home/user05$ echo "asdfasdf" > asdf
user01@LinuxDesk:~/.gvfs/sftp for user05 on SftpServer02/home/user05$ cat asdf
asdfasdf

“gvfs-mount” can also be used to list all currently mounted gvfs systems. Below shows only the sftp session.

user01@LinuxDesk:~$ gvfs-mount -l
Mount(0): sftp on SftpServer02 -> sftp://SftpServer02/
  Type: GDaemonMount

For reference, the following shows my 4gig USB drive that was automatically mounted when attached to the workstation through Gnome.

user01@LinuxDesk:~$ gvfs-mount -l
Drive(0): USB Drive
  Type: GProxyDrive (GProxyVolumeMonitorHal)
  Volume(0): 4.1 GB Media
    Type: GProxyVolume (GProxyVolumeMonitorHal)
    Mount(0): 4.1 GB Media -> file:///media/disk
      Type: GProxyMount (GProxyVolumeMonitorHal)
Mount(0): sftp on SftpServer02 -> sftp://SftpServer02/
  Type: GDaemonMount

GVFS mount points can be un-mounted using the “-u” argument. Below will un-mount the remote ssh server.

user01@LinuxDesk:~/.gvfs$ gvfs-mount -u ssh://user05@SftpServer02

Notes: GVFS contains one master daemon (gvfsd) which tracks current GVFS mounts. Each mount is created as an individual daemon with it’s own process. Knowing this, we can find the actual gvfsd process ID that the sftp connection is running under.

user01@LinuxDesk:~/.gvfs$  ps -ef | grep gvfsd-sftp
user01  8022     1  0 10:34 ?        00:00:00 /usr/lib/gvfs/gvfsd-sftp --spawner :1.8 /org/gtk/gvfs/exec_spaw/21

Posted in Filesystems, Linux, Networking

Networking / SAN: Cisco MDS 9000 License Installation

Kevin Goodman — Thu, 03 Sep 2009 19:06:45 +0000

First, make sure you put a copy of the license onto a tftp, ftp, or sftp server. The MDS switch supports all of those protocols. Here we will be using tftp.

Copy the license from TFTP server to bootflash (persistent storage)

mds9124# copy tftp://172.0.0.1/MDS20090209112333135513.lic bootflash:
Trying to connect to tftp server......
|
 TFTP get operation was successful

Install the license

mds9124# install license bootflash:MDS20090209112333135513.lic
Installing license .......done

Now that the new port license is installed we need to verify that it is working. Below shows the default licensing that came with the unit.

mds9124# show license default
Feature                               Default License Count
-----------------------------------------------------------------------------
FM_SERVER_PKG                         -
ENTERPRISE_PKG                        -
PORT_ACTIVATION_PKG                   8
10G_PORT_ACTIVATION_PKG               0
-----------------------------------------------------------------------------

The new one contained licensing for an additional 8 ports. Below you can see that now there are 16 ports licensed.

mds9124# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
                                 Count
--------------------------------------------------------------------------------
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
PORT_ACTIVATION_PKG           Yes  16   In use never       -
10G_PORT_ACTIVATION_PKG       No    0   Unused             -
-------------------------------------------------------------------------------

This one will be quick and easy! Below is how to install a new port license on a Cisco MDS 9000 switch from the Cisco CLI (Command Line Interface). Doing this does not remove the current license, just adds it to the configuration. As always though, back up your configuration and make sure if there is a current license that you also have a backup copy of it.First, make sure you put a copy of the license onto a tftp, ftp, or sftp server. The MDS switch supports all of those protocols. Here we will be using tftp.Copy the license from TFTP server to bootflash (persistent storage)

mds9124# copy tftp://172.0.0.1/MDS20090209112333135513.lic bootflash:
Trying to connect to tftp server......
|
TFTP get operation was successful
Install the license
mds9124# install license bootflash:MDS20090209112333135513.lic
Installing license .......done
Now that the new port license is installed we need to verify that it is working.  Below shows the default licensing that came with the unit.
mds9124# show license default
Feature                               Default License Count
-----------------------------------------------------------------------------
FM_SERVER_PKG                         -
ENTERPRISE_PKG                        -
PORT_ACTIVATION_PKG                   8
10G_PORT_ACTIVATION_PKG               0
-----------------------------------------------------------------------------
The new one contained licensing for an additional 8 ports.  Below you can see that now there are 16 ports licensed.
mds9124# show license usage
Feature                      Ins  Lic   Status Expiry Date Comments
Count
--------------------------------------------------------------------------------
FM_SERVER_PKG                 No    -   Unused             -
ENTERPRISE_PKG                No    -   Unused             -
PORT_ACTIVATION_PKG           Yes  16   In use never       -
10G_PORT_ACTIVATION_PKG       No    0   Unused             -
-------------------------------------------------------------------------------

Posted in Networking, SAN (Storage Area Network)

Networking / SAN: Cisco MDS 9000 Serial Number (Licensing)

Kevin Goodman — Fri, 07 Aug 2009 18:23:50 +0000

So you need to find the serial number on your Cisco MDS 9000 series fiber switch? This is easy enough, although “show serial number” would have been better.

Quick way to find your serial number.

tstSwitch01# show license host-id
License hostid: VDH=SOZ115568P9

The following will also get the information that you need. I truncated some of the output. The serial number under the “Common block” is what we need.

tstSwitch01# show sprom backplane 1
DISPLAY backplane sprom contents:
Common block:
 EEPROM Size     : 1024
 Block Count     : 5
 FRU Major Type  : 0x6003
 FRU Minor Type  : 0x0
 OEM String      : Cisco Systems, Inc.
 Product Number  : DS-C9124-K9
 Serial Number   : SOZ115568P9
 Part Number     : 73-10565-03
 Part Revision   : A9
 Mfg Deviation   : 0
 H/W Version     : 1.0
 Mfg Bits        : 0
Chassis specific block:
 Block Signature : 0x5601
 MAC Addresses   : 00-0g-tr-46-n3-u6
 Number of MACs  : 64

This is a little easier to read. Here, an include statement is passed to only return lines including “Serial”. We need the first, not second serial number.

tstSwitch01# show sprom backplane 1 | include Serial
 Serial Number   : SOZ115568P9
Second Serial number specific block:
 Serial Number   : JFH2486G4DR

Notes: All actual serial numbers were changed. This process should be the same for all Cisco MDS 9000 series. If using a chassis based MDS switch, make sure to verify if you need the serial of the unit or the actual blade module for licensing.

Posted in Networking, SAN (Storage Area Network)