Colocation to Virtualization » Networking

Cisco Nexus 1000v 4.2 Upgrade Problems And Fix

Kevin Goodman — Tue, 01 Mar 2011 13:33:41 +0000

Update

I’m glad Jeff (see comments) caught where I went wrong. After his comment I went back through the n1000v_upgrade_software PDF from cisco. For version 4.0(4)SV1(3a), the VEMs should have been upgraded first. It is on page 9 of the document.

Upgrading from Release 4.0(4)SV1(3, 3a, or 3b) to Release 4.2(1)SV1(4)

Step 1 Upgrading the VEMs: Release 4.0(4)SV1(2, 3, 3a, 3b) to Release 4.2(1)SV1(4), page 20

Step 2 Upgrading the VSMs to Release 4.2(1)SV1(4) Using the Upgrade Application, page 33

I was following lower down in the document that showed an upgrade of the VSM first then the VEM. Looking closer that was for one that required an intermediate upgrade between the current running and the newer 4.2(1)SV1(4) firmware. Thanks for the catch!

Original post:

So last weekend was our Cisco Nexus 1000v upgrade. We were upgrading from 4.0(4)SV1(3a) to 4.2(1)SV1(4). What should have been an easy upgrade really turned into a huge headache. Below is a walk-through of the upgrade process with notes on where things went wrong.

To start, lets verify the current running state of the VSM (1000v switch) and the VEM (ESX host modules). Below you will see that they are all on version 4.0(4)SV1(3a). Also note the standby VSM module number is 2. This will be the first one reloaded.

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
2    0      Virtual Supervisor Module        Nexus1000V         ha-standby
3    248    Virtual Ethernet Module          NA                 ok
4    248    Virtual Ethernet Module          NA                 ok
5    248    Virtual Ethernet Module          NA                 ok
6    248    Virtual Ethernet Module          NA                 ok
7    248    Virtual Ethernet Module          NA                 ok
8    248    Virtual Ethernet Module          NA                 ok
9    248    Virtual Ethernet Module          NA                 ok

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.0(4)SV1(3a)      0.0
2    4.0(4)SV1(3a)      0.0
3    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
4    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
5    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
6    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
7    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
8    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)
9    4.0(4)SV1(3a)      VMware ESXi 4.1.0 Releasebuild-348481 (2.0)

Show boot displays the current location/firmware being used on bootup. This will be changed with the upgrade.

1000vSW# sh boot
sup-1
kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin
system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin
sup-2
kickstart variable = bootflash:/nexus-1000v-kickstart-mz.4.0.4.SV1.3a.bin
system variable = bootflash:/nexus-1000v-mz.4.0.4.SV1.3a.bin
No module boot variable set

Just like any other devices, make sure there is enough room on the bootflash device to store the new firmware

1000vSW# dir

Usage for bootflash://
  481083392 bytes used
 1907187712 bytes free
 2388271104 bytes total

After adding up the file size of the both files, there is enough on the bootflash to hold the new firmware. I already had the files on a host running scp, but the files can be loaded from any hosts running the following protocols

TFTP

FTP

SCP

SFTP
The syntax for copying over SCP is as follows

copy scp://user01@127.0.0.15/IOS/1000v/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin bootflash://
copy scp://user01@127.0.0.15/IOS/1000v/nexus-1000v-mz.4.2.1.SV1.4.bin bootflash://

From the active VSM (module 1 in this case), the new firmware is loaded into the configuration via the “install” command.

1000vSW# install all system bootflash:nexus-1000v-mz.4.2.1.SV1.4.bin kickstart bootflash:nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin
System image sync to standby is in progress...
System image is synced to standby.
Kickstart image sync to Standby is in progress...
Kickstart image is synced to standby.
Boot variables are updated to running configuration.

The boot variables were updated per the above, but it is still worth verifying. Below shows that the running configuration was updated to load the new firmware.

1000vSW# sh running-config | inc boot
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-1
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-2
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-2

I almost skipped this command the first time. Make sure to save the configuration to startup-config or the reload of standby module will load up the startup-configs boot variables. That would bring it up on the same 4.0(4)SV1(3a) version.

1000vSW# copy running-config startup-config
[########################################] 100%

We saw the module status earlier in this post, but I will reiterate it here. Module 2 shows as the standby VSM here, so it will be reloaded first.

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
2    0      Virtual Supervisor Module        Nexus1000V         ha-standby

Time to reboot the standby module. This is done as follows.

1000vSW# reload module 2
This command will reboot standby supervisor module. (y/n)?  [n] y
about to reset standby sup
1000vSW# 2011 Feb 25 19:39:33 1000vSW %PLATFORM-2-PFM_MODULE_RESET: Manual restart of Module 2 from Command Line Interface

Module 2 is still restarting, notice it missing under “Mod”

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
3    248    Virtual Ethernet Module          NA                 ok
4    248    Virtual Ethernet Module          NA                 ok

* you can bring up the console of the standby switch in vCenter to check progress

Ok, this is where everything went wrong for us. A typical upgrade would have the module connect back in showing the new version of code running on it. For us, this did not happen. Instead module 1 and 2 were not able to communicated with each other. Both decided to become active.

This became a big problem. All VMs running were still able to pass traffic across the network, but none of ESX hosts showed up as modules in the VSM, thus isolating them. First symptom I saw was connectivity loss via SSH to the VSM IP. It seemed to happen about every two minutes. luckily I saw the firmware version changing each time I did a show module after reconnecting. So the two 1000v modules were in IP conflict with each other, stealing it away from one another every few minutes.

I waited for the module running version 4.0(4)SV1(3a) to grab the IP again and issued a reload. This was module 1. Once it restarted and came up on the new firmware, it started talking with module 2.

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          ha-standby
2    0      Virtual Supervisor Module         Nexus1000V          active *

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Now that we are on the same firmware, it was time to make module 1 primary/active once more. This was non-intrusive.

1000vSW# system switchover

Once more to verify that module 1 has become active:

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Also to be safe, have the VSM reconnect with Virtual Center.

1000vSW## configure t
Enter configuration commands, one per line.  End with CNTL/Z.
1000vSW#(config)# svs connection VirtualCenter
1000vSW#(config-svs-conn)# connect
1000vSW#(config-svs-conn)# end

Verify the connection as well.

1000vSW#  show svs connections

connection vcenter:
    ip address: 127.0.0.99
    remote port: 80
    protocol: vmware-vim https
    certificate: default
    datacenter name: Test
    DVS uuid: 55 55 55 55 55 55 55 55
    config status: Enabled
    operational status: Connected
    sync status: Complete
    version: VMware vCenter Server

At this point we should be good. At least that is what I thought. Going back to what I said before
“but none of ESX hosts showed up as modules in the VSM.”
Yeah, I didn’t notice that untill after the fact. Normally I vmotion over an individual host to a different server to test connectivity, but this time I didn’t.

I selected our first vSphere (ESX) host and entered maintenance mode, thus vmotioning all VMs off to other hosts. Immediately they fell off of the network. All pings were lost. These were running critical/production servers. Of course we were in a maintenance window, but this still was worst case scenario. Normally each hosts VEM would show up in the show module, but they did not after the upgrade.

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-00-00-00-00-00 to 00-00-00-00-00-00  NA
2    00-00-00-00-00-00 to 00-00-00-00-00-00  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    127.0.0.55     NA                                    NA
2    127.0.0.55     NA                                    NA

The above resembled what I was seeing. Just the VSM modules. The mac addresses were correct, I just changed them to all 0′s here. We tried shutting down a VM and moving it to different hosts in the cluster, but still no pings. I also tried stopping and starting the VEM via command line on the ESX host.

The only thing that fixed this was pushing the VEM upgrades. This was done using VMware Update Manager. Very easy, but we had to put the hosts into maintenance mode to do so. This wound up being very time-consuming. Once server 1′s VEM was updated, we were able to vMotion VMs to it without losing network connectivity to them. It went like this:

Update ESX host 1 VEM to 4.2

vMotion ESX host 2 VMs to host 1

Update ESX host 2 VEM to 4.2

vMotion ESX host 3 VMs to host 2

Update ESX host 3 VEM to 4.2

vMotion ESX host 4 VMs to host 3

Update ESX host 4 VEM to 4.2

Repeat until done!

It would have been a lot easier if the hosts could have been placed into maintenance mode and VMs automatically migrated to other members in the cluster. Once completed, everything looked fine in the VSM. Even the hosts showed up!

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    127.0.0.55       NA                                    NA
2    127.0.0.55       NA                                    NA
3    127.0.0.24       22222222-2222-2222-2222-22222222221g  vm_host01 
4    127.0.0.26       22222222-2222-2222-2222-22222222222g  vm_host03 
5    127.0.0.27       22222222-2222-2222-2222-22222222223g  vm_host04 
6    127.0.0.28       22222222-2222-2222-2222-22222222224g  vm_host05 
7    127.0.0.25       22222222-2222-2222-2222-22222222225g  vm_host02 
8    127.0.0.29       22222222-2222-2222-2222-22222222226g  vm_host06
9    127.0.0.120      22222222-2222-2222-2222-22222222228w  vm_host07

A new rule for us to now follow: Always put one host into maintenance mode before upgrading!
This would have allowed us to upgrade without downtime in this scenario.

Filed under: Cisco UCS, Networking, VMWare

Cisco UCS 6120 Switch Replacement

Kevin Goodman — Tue, 18 Jan 2011 12:00:54 +0000

So back in August of 2010 I had to do a 6120 switch replacement, due to a bad port on the switch. I know I commented on twitter during the process but did not here.

To be safe, we were in a maintenance window, but servers were still up and running while the switch replacement was being done.

A little more information on our setup

UCS Manager was running at v1.3(1c)

All blades boot from SAN via FCoE

Cisco MDS 9124s sits between the 6120s and SAN Storage (EMC)

RedHat Linux blades were left running

Windows 2008 blade was left running

VMware vSphere blades were left running

All blades contain the VIC N81kr (PALO) interfaces

Two UCS chassis in the environment

In “theory” UCS has a lot of redundant built-in, but I had yet to test out taking down a switch. I had at this point pulled IOM modules, blades, SFP cables from an active chassis but not a 6120.
Just to give you an idea of how many dropped packets were lost during the process, below are pings that were done during the removal.

Pings to a vSphere host in Cisco UCS

Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63
Reply from 127.0.0.59: bytes=32 time<1ms TTL=63

Ping statistics for 127.0.0.59:
    Packets: Sent = 2413, Received = 2410, Lost = 3 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 75ms, Average = 0ms

Pings to the cluster IP of 6120s

Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63
Reply from 127.0.0.41: bytes=32 time=4ms TTL=63
Reply from 127.0.0.41: bytes=32 time=3ms TTL=63
Reply from 127.0.0.41: bytes=32 time=2ms TTL=63
Reply from 127.0.0.41: bytes=32 time=1ms TTL=63
Reply from 127.0.0.41: bytes=32 time<1ms TTL=63

Ping statistics for 127.0.0.41:
    Packets: Sent = 2339, Received = 2316, Lost = 23 (0% loss),
Approximate round trip times in milli-seconds:
    Minimum = 0ms, Maximum = 8ms, Average = 1ms

So all servers were fine. Nothing went down and no data was lost to the storage going to the SAN. It took about 1 hour to replace switch which entailed:

Un-rack bad 6120

Rack replacement 6120

Upgrade firmware on the new 6120. It was sent to us with version 1.2

Join new 6120 to cluster

Note(s): Wish that I would have noticed the replacement switch came with an older firmware. I spent a decent amount of time setting it up on a temporary IP (standalone), pushing new firmware to it, and then updating it. Once done, I had to wipe the config and link it to the other 6120 and join the cluster.

Filed under: Cisco UCS, Datacenter, Networking

Cisco Nexus 1000v and vSphere VEM Manual Uninstall

Kevin Goodman — Fri, 13 Aug 2010 13:00:36 +0000

I found the need when troubleshooting a few days ago to manually remove the Cisco 1000v VEM from a host. The distributed virtual switch (1000v) was not functioning correctly on this individal machine. Here I will go through the process of doing so form the command line

Fist, SSH to the ESX (vSphere) host.

Make sure that the VEM is listed on this server.

[root@vm09 ~]# esxupdate --vib-view query | grep cisco
cross_cisco-vem-v121-esx_4.0.4.1.3.1.0-1.20.2        installed     2010-08-10T15:41:37.946711-04:00

Next, the server will have to be removed from the 1000v switch in Virtual Center. To do this, all VMs that has a nic on the 1000v switch will have to be migrated off.

To remove the host from the switch, navigate to the following.
Home -> Inventory -> Networking -> select your 1000v switch -> Host tab

From there, right-click the ESX host and tell it to “Remove from Distributed Virtual Switch”

Go back to the CLI. This is actually easy enough, just use the vem-remove command. Below is the help information from the command.

[root@vm09 ~]# vem-remove -h

vem-remove [options]
Options:

    -c                          Remove /usr/lib/ext/cisco and modules from visor system
    -d                          Developer removal option
    -h                          Help
    -r                          Remove VIBs
    -s                          Stop DPA and unload modules before removing
    -v                          Verbose
    -u                          Cleanup stale dvs information
    -z                          Dry run (don't really remove)

Per the cisco documentation, the “-d” option was used.

[root@vm09 ~]# vem-remove -d
watchdog-vemdpa: Terminating watchdog with PID 1551
Removing CIsco VEM VIB from COS system
Removing VIB cross_cisco-vem-v121-esx_4.0.4.1.3.1.0-1.20.2
Removing cisco-vem-v121-esx             ############## [100%]

Running [/usr/sbin/vmkmod-install.sh]...
ok.

Re-query esxupdate and the VEM should now be in the “retired” status.

[root@vm09 ~]# esxupdate --vib-view query | grep cisco
cross_cisco-vem-v121-esx_4.0.4.1.3.1.0-1.20.2        retired       2010-08-10T15:41:37.946711-04:00

The re-install of the VEM is easy enough through Virtual Center. I always reboot the host before, then once it is back in Virtual Center, add the host back to the 1000v Distributed Virtual Switch. In the proccess of adding it back, VC will re-load the VEM and reconfigure the networking on the host. Once done, you now have 1000v VEM re-installed on the host.

Filed under: Cisco UCS, Networking

Cisco UCS And 1000v Weirdness v1.3(1c) and VEM v121-esx_4.0.4.1.3.1.0-1.20.2 – Isolated Issue

Kevin Goodman — Thu, 12 Aug 2010 13:04:51 +0000

Note: This is not a bug, just an isolated issue with one of our blades

So we had one of our ESX host that did not want to funtion correctly once 1000v and vSphere was patched. The virtual switch worked fine, but the deistributed Cisco 1000v switch did not. When a VM was migrated over, the pings would drop. VLAN settings and NIC assignments were all verified. I tried to the following to fix the issue

1.    Rebooted VM – Still no network
2.    Remove vSphere host from distributed switch and re-added – Still no network
3.    Removed 1000v VEM from the host manually, rebooted, and VEM added – Still no network
4.    Removed 1000v VEM, shutdown the host, disassociated service profile, re-associated, booted, and added back to the distributed virtual switch (1000v) – worked!

It was the 4th attempt that worked. For some reason 1000v seemed to not be picking up the VLANs. I know that there is a bug out there for this, but it was for an older version of 1000v. Either way, just wanted to say if all else fails, disassocate the service profile, then re-associate it back to the same blade. By doing this, it re-pushes the service profile configuration and basically re-burns the config to the blade.

Filed under: Cisco UCS, Networking

Cisco UCS and Nexus 5000 DataCenter – Our Implementation

Kevin Goodman — Wed, 07 Jul 2010 19:07:07 +0000

Our Cisco 10gbE implementation consists of 2 Chassis fully populated UCS with a mix of full and half width blades. The servers are all boot from SAN with no local disks. “PALO” cards are used in all servers which allow us to do FCOE. 7 of the blades are running VMware ESX 4 (vSphere) and the rest are a mix of RedHat Linux and Windows 2008.

Storage:

All servers boot from SAN over FCOE

Primary storage is via Fibre Channel to an EMC CX4-240

A few servers are also connected via Fibre Channel to RamSan SSD SANs

The CX4-240 is also connected to the UCS environment via quad 10gbE links over fiberNetwork:

Core 10gbE switching is provided via Cisco Nexus 5010s

1 Gigabit Ethernet to legacy servers are provided by dual 48 port C2148 Fabric Extenders

Each C2148 is connected to a Nexus 5010 by dual 10gbE uplinks

Each 5010 is connected via quad 4gig fiber to the CX4-240 to provide FCoE to physical servers outside of the UCS environment

Each 5010 is also connected to the CX4-240 by dual 10gbE fiber ports to provide ISCSI to all of the network

Each UCS switch (6120s) are uplinked to the 5010s by dual 10gbE interfaces

Each UCS chassis is uplinked to the 6120s by 8 qty 10gbE ports (4 connections per IO module)

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon. This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120. I definitely would love feedback on this. How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Filed under: Colocation, EMC, Hardware, Networking, RamSan, SAN (Storage Area Network)

Cisco, VMware: Nexus 1000v Tracking VM Interface Errors

Kevin Goodman — Mon, 07 Jun 2010 10:38:48 +0000

So I have found the first use of our new Cisco Nexus 1000v virtual switch. The counters (stats) were reset about a week agon and this is the first time they have been reviewed since. Almost all virtual machines show no errors, but there were a few that were high.

After connecting to the console, the following command was ran. nexus1kv# sh interface counters errors

--------------------------------------------------------------------------------
Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards
--------------------------------------------------------------------------------
Veth30            --          --          --          --          --     1127136
Veth69            --          --          --          --          --       29654
Veth70            --          --          --          --          --       31966
Veth71            --          --          --          --          --       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifying the VM is indeed “w2k_serv0″
nexus1kv# sh interface Veth30 description

-------------------------------------------------------------------------------
Interface                Description
-------------------------------------------------------------------------------
Vethernet30              w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported
nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).
nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1
Veth70                   linux_host01, Network Adapter 2
Veth71                   linux_host01, Network Adapter 3

After a little more investigation, it turned out that “w2k_serv01″ was deployed from a legacy template that was created on our old AMD cluster. Looking at the network interface in Windows 2003, it showed up as an “AMD …” network adapter.

To try and correct these errors, I removed the current network adapter through VMware vCenter and added a new one. This forced the OS in seeing the network interface as a new device and install fresh drivers. Now in Windows it shows up as an Intel network inteface.

Now that the new network adapter has been installed, I reset the statistics (and errors) on all the interfaces.
nexus1kv# clear counters

This command will clear "show interface" counters on all interfaces
Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.
nexus1kv# sh interface counters errors | inc Veth30

Veth30            --          --          --          --          --           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error filed there. Yeah, that was a run-on sentence. Sorry.

After connecting to the console, the following command was ran.

nexus1kv# sh interface counters errors

——————————————————————————–

Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards

——————————————————————————–

Veth30            –          –          –          –          –     1127136

Veth69            –          –          –          –          –       29654

Veth70            –          –          –          –          –       31966

Veth71            –          –          –          –          –       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifing the VM is indeed “w2k_serv0″

nexus1kv# sh interface Veth30 description

——————————————————————————-
Interface Description
——————————————————————————-
Vethernet30 w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported

nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).

nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1

Veth70                   linux_host01, Network Adapter 2

Veth71                   linux_host01, Network Adapter 3

Now that the new network adapater has been installed, I reset the statistics (and errors) on all the interfaces.

nexus1kv# clear counters

This command will clear “show interface” counters on all interfaces

Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.

t1prd-nexus01# sh interface counters errors | inc Veth30

Veth30            –          –          –          –          –           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error fied there. Yeah, that was a run-on scentance. Sorry.

Filed under: Hardware, Networking, VMWare

Cisco UCS Implementation In Pictures: Part 3 The Finished Product

Kevin Goodman — Thu, 03 Jun 2010 12:02:42 +0000

We finished up our Cisco UCS installation about 2 months ago, but have been to busy to get the last set of pictures up. Sorry for the delay! Just in case you don’t follow me on Twitter, UCS is working out nicely.

Below are pictures of the full rack that includes:

Two Cisco UCS Chassis fully populated
Two Cisco Nexus 5010 switches
Two Cisco 6120XP switches
Two horizontal PDUs (Circuit 1)
Two vertical PDUs (Circuit 2)

Just in case you wanted a pretty view, the last picture is from my mountain trip last weekend

Filed under: Colocation, Hardware, Networking

Storage, Networking, EMC: DataDomain Replication

Kevin Goodman — Wed, 26 May 2010 14:49:41 +0000

We have a total of 3 DataDomains currently in production. One at our “Tier 1″ site, “Tier 2″(DR) and one in Europe. All DataDomain appliances have the ability to replicate data among themselves. This will be a general overview of how to setup replication between two DataDomains

On the source, I have already setup a directory tree for “/backup/europe_data”. All files destined for our Euroupe office will be placed here. On the DataDomain devices, replicated folders are added manually. By default none are replicated.
*********************
Be aware that if you are viewing this through a web browser that a lot of the command might be hidden due to formatting of the WordPress theme. It is best viewed by an RSS reader. If you would like the full text, email or leave a comment and I can send you a PDF version.
*********************

Below is the help output for “replication add” command.

user01@us-dd565# replication add  ?
Commands matching "replication add":
  replication add source  destination
                 Add a replication pair

I wish the syntax was easy as what was stated above, but below is the actual command used to setup replication on the source DataDomain

user01@us-dd565# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once the source is setup, the destination endpoint/DataDomain also must be configured. The command is identical to what was entered at the source DataDomain

user01@eu-dd01# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once replication is setup on both DataDomains, log back into the source and start the replication.

user01@us-dd565# replication initialize
Incorrect syntax "replication initialize", showing help:
  replication initialize
                 Initialize replication on the source
                 (configure both source and destination
                 first)

Below is the actual syntax to startup replication for the directory that we just configured.

user01@us-dd565# replication initialize dir://eu-dd01.testdomain.com/backup/europe_data
(00:04) Waiting for initialize to start...
(00:06) Initialize started.
Use 'replication watch dir://eu-dd01.testdomain.com/backup/europe_data' to monitor progress.

Since I am impatient and want to know for sure this is working, I use “replication watch”. This shows the current progress of the replication between DataDomains.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
    Use Control-C to stop monitoring.

(00:57) Replication initialize started...
(00:59) initializing 3/3:
(10:35)     : pre-comp:   0%    27 KB/s, network    26 KB/s, 0 of 6 files

When new data is written to the mount point, replication automatically starts. Below is what shows if no data is currently being replicated.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
**** Replication initialize or recover already completed.

“replication show config” is used to display all the configured replication points.

user01@us-dd565# replication show config
CTX   Source                                                    Destination                                                 Connection Host and Port                       Enabled
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------
11    dir://us-dd565.testdomain.com/backup/mysql-rep       	dir://us-t2-dd510.testdomain.com/backup/mysql-rep	us-t2-dd510.testdomain.com   (default)     yes
12    dir://us-dd565.testdomain.com/backup/europe_data        	dir://eu-dd510.testdomain.com/backup/europe_data	eu-dd510.testdomain.com   (default)     yes
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------

Sorry for all of the spacing being off. All the tabs did not come over when copying data from the console.

user01

Filed under: Data Domain, deduplication, NAS, Networking

Cisco UCS Implementation In Pictures: Part 2 Nexus B250 And B200 External/Internals

Kevin Goodman — Mon, 10 May 2010 13:25:55 +0000

Below is a collection of photos from phase 2 of our Cisco UCS implementation. This consisted of installing the Cisco UCS Blade Chassis, blade switches and server blades.

Unpacked B200-M1 Blades

Unpacked Blades B200/B250

B250-M1 Blade Pictures

B250-M1 Guide

B250-M1 Open Front

B250-M1 Open Rear

B250-M1 Internal Top

B250-M1 Top Internal 2

B250-M1 Internal Palo

B250-M1 And B200-M1 In Chassis

B250-M1 Bezel

B200-M1 Blade Pictures

B200-M1 Guide

B200-M1 Internal Air Flow

B200-M1 Top Full

B200-M1 Top CPU / RAM

B200-M1 Rear Palo

B200-M1 Top Internal w/vents

B200-M1 Bezel

Filed under: Colocation, Hardware, Networking

Cisco, Networking: VMware Cisco Nexus 1000v Increasing Default Max Ports

Kevin Goodman — Wed, 05 May 2010 13:26:58 +0000

Earlier today we ran into an issue with exceeding the default max number of ports (virtual) on a port profile. The default is set to 32 and the max is 1024 per vethernet port profile. I really can’t believe how low the default setting is. Most VMware people I know consolidate waaayyyy more than 32 host per datacenter.

Below shows the error in VMware Virtual Center.

Here is the profile “iSCSI” before modification. Note the max 32 and available count.

The only way to increase this value is through command line in the active Nexus 1000v.

nexus01# configure t
nexus01(config)# port-profile type vethernet iSCSI
nexus01(config-port-prof)# vmware max-ports 1024
nexus01(config-port-prof)# end

Once the above command is issued, Virtual Center Client should show a reconfiguration task for the distributed virtual switch

Once the tasks is completed, go back into Home -> Inventory -> Networking and select the port profile in the distributed switch. The “total ports” should now be set to 1024.

Make sure to save your configuration!

nexus01# copy running-config startup-config

Notes: This should be non-intrussive, but make sure to test it in your environment!

Filed under: Colocation, Networking, VMWare

Hardware, Linux, Networking: Cisco UCS Time Problem

Kevin Goodman — Wed, 28 Apr 2010 14:12:55 +0000

So we have our new Cisco UCS system installed and a weird problem is showing up. The Cisco UCS Manager console shows the correct date (2010), but when setting up a new server, the date is incorrect. Also, an NTP server (working correctly) is set. Since we mainly run Linux here, the NTP service will not update the date/time from the NTP server because of how long the difference is between the system clock and NTP. Also, on a Windows 2008 install we had to manually adjust the time/date as well

This is not a major issue, just an annoyance. We also use RedHat Satellite server and can not join to the patch management system with the incorrect date. So my question is where does the OS get it’s bad time from? I figure that the OS gets the time from the bios and that the bios would obtaion the information from UCS Manger. That does not appear to be the case.

Below is an example of the issue:

Cisco UCS Time

Above in the bottom right corner shows the correct time and date in the Cisco UCS Manager.

Below you can see that the date shows 2009 on a freshly installed server in UCS. The difference of the time and day are due to screen shots taking a day apart, but you get the idea. It’s not 2009.

Cisco UCS Bios Time

Server has just been installed and booted. The command “date” is issued to see the current time.

[root@test-server02 build]# date
Sat Aug 15 02:37:24 EDT 2009

As seen from above, the date is incorrect (2009). NTP needs to be updated, but the service must be stopped first, or else it will not accept the new time.

[root@test-server02 build]# /etc/init.d/ntpd stop
Shutting down ntpd:                                        [  OK  ]

Once stopped, “ntpdate” is used to query an NTP server and adjust the local server time.

[root@test-server02 build]# ntpdate pool.ntp.org
27 Apr 14:36:23 ntpdate[5214]: step time server 70.86.250.6 offset 22075095.967480 sec

Now that the system time is correct, NTP service is restarted to

[root@test-server02 build]# /etc/init.d/ntpd start
Starting ntpd:                                             [  OK  ]

From here on out, the time will be correct and sync with the NTP service withouth issue.

[root@test-server02 build]# date
Tue Apr 27 14:36:40 EDT 2010

Filed under: Hardware, Linux, Networking

Networking, UCS: Cisco Nexus License Installation

Kevin Goodman — Thu, 08 Apr 2010 18:06:48 +0000

The following is a quick run through of installing a license file for a Cisco Nexus device.

First check and see if there are any current licenses installed. This being a new switch, there aren’t any.

nexus_5000# sh license
nexus_5000#

Licenses are normally tied to “host IDs”. Below shows the host ID of the switch.

nexus_5000# sh license host-id
License hostid: VDH=3WI246599Z

The license file is not local to the switch yet, so below I fetch it via TFTP.nexus_5000# copy tftp://127.0.0.1:69/Cisco_license/NX5KLIC.lic bootflash: vrf default_vrf

Trying to connect to tftp server......
Connection to Server Established.
[                         ]         0.50KB
TFTP get operation was successful

Next, the license is installed with the “install license” command.

nexus_5000# install license bootflash:NX5KLIC.lic
Installing license .....done

Now that the license has been added, use “show license” to confirm.

nexus_5000# show license
NX5KLIC.lic:
SERVER this_host ANY
VENDOR cisco
INCREMENT ENTERPRISE_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
1 \
RMA license" SIGN=DDWD62626W7f
INCREMENT FC_FEATURES_PKG cisco 1.0 permanent uncounted \
        VENDOR_STRING=
MDS_SWIFTN5010= \
        HOSTID=VDH=3WI246529Z \
        NOTICE="
200903360467890433
2 \
RMA license" SIGN=VVSVDS848SFW

Another way is to see the names of the installed licenses.

nexus_5000# show license brief
NX5KLIC.lic

To be safe, copy the running configuration to startup.

nexus_5000# copy running-config startup-config

Also, here I copy the configuration to TFTP backup.

nexus_5000# copy running-config tftp://127.0.0.1:69/Cisco_config/ns5k2_startup vrf default_vrf
Trying to connect to tftp server......
Connection to Server Established.
[#                        ]         4.50KB
TFTP put operation was successful

Notes: All IP, license files, and hashes were changed. The installation process is pretty strait forward. The managment console (GUI) should also be able to provide this function.

Filed under: Hardware, Networking, SAN (Storage Area Network)

Linux, Networking, Security: Get Remote SSL Certificate From Command Line

Kevin Goodman — Wed, 03 Mar 2010 20:31:00 +0000

Easy way to get the SSL certificate of a server from the command line in Linux. The nice thing about it is that you get the full certificate chain. Nice for troubleshooting issues. After the “-connect”, specify the host and port you want to connect to. TCP port 443 is the default https port.

[user1@testserver ~]$ openssl s_client -connect mail.google.com:443

CONNECTED(00000003)
---
Certificate chain
 0 s:/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com
   i:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
 1 s:/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
   i:/C=US/O=VeriSign, Inc./OU=Class 3 Public Primary Certification Authority
---
Server certificate
-----BEGIN CERTIFICATE-----
MIIDIjCCAougAwIBAgIQHxn23jXdY6FCkYrVLMCrEjANBgkqhkiG9w0BAQUFADBM
MQswCQYDVQQGEwJaQTElMCMGA1UEChMcVGhhd3RlIENvbnN1bHRpbmcgKFB0eSkg
THRkLjEWMBQGA1UEAxMNVGhhd3RlIFNHQyBDQTAeFw0wOTEyMTgwMDAwMDBaFw0x
MTEyMTgyMzU5NTlaMGkxCzAJBgNVBAYTAlVTMRMwEQYDVQQIEwpDYWxpZm9ybmlh
MRYwFAYDVQQHFA1Nb3VudGFpbiBWaWV3MRMwEQYDVQQKFApHb29nbGUgSW5jMRgw
FgYDVQQDFA9tYWlsLmdvb2dsZS5jb20wgZ8wDQYJKoZIhvcNAQEBBQADgY0AMIGJ
AoGBANknyBHye+RFyUa2Y3WDsXd+F0GJgDjxRSegPNnoqABL2QfQut7t9CymrNwn
E+wMwaaZF0LmjSfSgRSwS4L6ssXQuyBZYiijlrVh9nbBbUbS/brGDz3RyXeaWDP2
BnYyrVFfKV9u+BKLrebFCDmzQ0OpW5Ed1+PPUd91WY6NgKtTAgMBAAGjgecwgeQw
DAYDVR0TAQH/BAIwADA2BgNVHR8ELzAtMCugKaAnhiVodHRwOi8vY3JsLnRoYXd0
ZS5jb20vVGhhd3RlU0dDQ0EuY3JsMCgGA1UdJQQhMB8GCCsGAQUFBwMBBggrBgEF
BQcDAgYJYIZIAYb4QgQBMHIGCCsGAQUFBwEBBGYwZDAiBggrBgEFBQcwAYYWaHR0
cDovL29jc3AudGhhd3RlLmNvbTA+BggrBgEFBQcwAoYyaHR0cDovL3d3dy50aGF3
dGUuY29tL3JlcG9zaXRvcnkvVGhhd3RlX1NHQ19DQS5jcnQwDQYJKoZIhvcNAQEF
BQADgYEAicju7fexy+yRP2drx57Tcqo+BElR1CiHNZ1nhPmS9QSZaudDA8jy25IP
VWvjEgaq13Hro0Hg32ZNVK53qcXwjWtnCAReojvNwj6/x1Ciq5B6D7E6eiYDSfXJ
8/a2vR5IbgY89nq+wuHaA6vspH6vNR848xO3z1PQ7BrIjnYQ1A0=
-----END CERTIFICATE-----
subject=/C=US/ST=California/L=Mountain View/O=Google Inc/CN=mail.google.com issuer=/C=ZA/O=Thawte Consulting (Pty) Ltd./CN=Thawte SGC CA
---
No client certificate CA names sent
---
SSL handshake has read 1778 bytes and written 316 bytes
---
New, TLSv1/SSLv3, Cipher is AES256-SHA
Server public key is 1024 bit
Compression: NONE
Expansion: NONE
SSL-Session:
    Protocol  : TLSv1
    Cipher    : AES256-SHA
    Session-ID: DEB23CF699255054E08F69181B2342E9F6D6DF0D02B399C36034E0D8BE18AC0C
    Session-ID-ctx:
    Master-Key: D696A99CEC2FDD9535FE2EC936531AD129FD97E56441E37AE7A143C40304E395EA7DA039797B948B009B42DA5377E668
    Key-Arg   : None
    Krb5 Principal: None
    Start Time: 1267560715
    Timeout   : 300 (sec)
    Verify return code: 0 (ok)
---
HTTP/1.0 400 Bad Request
Content-Type: text/html; charset=UTF-8
Content-Length: 1350
Date: Tue, 02 Mar 2010 20:11:57 GMT
Server: GFE/2.0
X-XSS-Protection: 0

Filed under: Linux, Middle Ware, Networking, Security

Storage, Network: What I have Been Doing (EMC,Cisco UCS)

Kevin Goodman — Tue, 02 Mar 2010 19:41:19 +0000

This is more of an informational update of things that I have going on right now. I normally do not publish day-to-day type of things, but here we go.

Storage

We have received both replacement drives for our EMC Clariion CX340 and four new DAEs (disk shelves) for our CX4-240

Clariion CX3

The CX3 was originally bought to speed up our Oracle implementation. This was accomplished by ordering lots of fast disks (spindles) that were small. We wound up with 6 DAEs filled with 73gig 15kRPM disks, totalling 90 dedicated drives for Oracle.

This was great for the original purpose but the unit was replaced a year after initial deployment with a RamSan and EMC CX4. Having been decommissioned from production and moved to the tier 2 site, the need for space over IOPS (speed) drastically increased. Trying to keep performance and space requirements in balance, the decision has been made to go with a smaller RamSan for Oracle at the tier 2 site. This gives us the ability to replace the small 73 gigabyte drives with bigger 600 gigabyte 10kRPM disks. Replacing those disk with the same quantity of 600 gig ones will give us ~8 times as much space.

The RamSan will almost double the IOPS capacity that the CX3 is able to achieve and speed up our data warehouse even more.

Clariion CX4

So last year we went with implementing EMC Recoverpoint SAN based replication. This has been great and served us well! The only downfall was that we were doing “CRR” remote replication only. In a case of a failure and data needed to be recovered, there were no local copies. The snapshot or “point in time” would have to be loaded from the tier 2 site and transferred across the datacenter interconnect. The interconnect being 150 megs slowed this process down.

As planned from the beginning, we are implementing “CLR” local replication as well. This means that there will be a local copy of snapshots saved locally to the CX4. This will give us almost immediate access to the snapshots without being slowed down by the interconnect. The problem with RecoverPoint is that if you have a terrabyte LUN that you want to connect, you must have an extra terrabyte worth of space to save it. This is not really a problem, but a major consideration on the number of drives to buy and the overall expense of the implementation.

In our case, a terrabyte oracle LUN will wind up costing 3 terrabytes in the end. 1 terrabyte for the original data, 1 terrabyte for the local copy (CLR), and 1 terrabyte at the remote tier 2 site (CX3).

Our virtualization effort is continuing and this is another huge factor on the storage expansion. Currently we have 16 LUNs dedicated to the VMware environment. Each is 320 gigs in size. Moving forward, we will be doing a virtual desktop deployment as well. The leftover ~400 gigs will not cut it. So in the new 60 disks, 15 or more will have to be dedicated to VMware.

Cisco UCS

We have begun our UCS voyage. As of last weekend, we did a “rip and replace of our network”. This included rewiring the main network rack and configuring a new network core. Also, the Cisco Nexus 5010, 10 gigabit Ethernet switches are in. Uplinked to them are two 48 port gigabit fabric extenders.

The VMware environment is now connected via dual 10gigE links per server through this infrastructure. Reducing the cable count from 6 to 2 per server. So far verything is stable! A purchase order has been sent out and we should hopefully have two Cisco UCS Blade chassis and switching infrastructure show up within about 30 days.

Filed under: EMC, Networking, RamSan, SAN (Storage Area Network), VMWare

VMware, Linux: Install VMware Tools On RedHat Based Systems

Kevin Goodman — Tue, 12 Jan 2010 18:58:20 +0000

The following is a quick overview of installing VMware Tools on RedHat, CentOS, and Fedora systems. Specifically for VMware ESX, ESXi, and vSphere systems.

First, go into the VMware console and right-click on the VM (Virtual Machine) that you are going to install VMware tools on. Select “Install/Upgrade VMware Tools” option from the list. Below is a screen shot of the menu.

VMware Tools Menu

By default, most CDROM devices are symbolically linked to /dev/cdrom by the operating system.

Just in case, you can search the messages file to see the actual device. This is needed only if /dev/cdrom is not automatically linked or you have setup multiple cdrom devices on the VM (Virtual Machine).

[root@RHserver01 media]# cat /var/log/messages | grep CDROM
Jan 10 10:59:03 RHserver01 kernel: hda: VMware Virtual IDE CDROM Drive, ATAPI CD/DVD-ROM drive

From above, you can see that the actual device is hda, specifically /dev/hda. If you are just curious you can do an “ll” on the /dev/cdrom device to see where is it linked to. In this case again, it’s going to hda.

[root@RHserver01 ~]# ll /dev/cdrom
lrwxrwxrwx 1 root root 3 Jan 12 13:18 /dev/cdrom -> hda

Mount the cdrom device to an empty or non-mounted point on the filesystem. Here I use the defaultly present /media location.

[root@RHserver01 /]# mount /dev/cdrom /media/
mount: block device /dev/cdrom is write-protected, mounting read-only

Below we move into the /media location using “cd”.

[root@RHserver01 /]# cd /media/

“ls” is used to display what files are present. Here we see both an RPM (native RedHat based OS package) and a gzip archive. If you were installing VMware Tools on a non-RedHat derived distribution, you would use the .gz package.

[root@RHserver01 media]# ls
VMwareTools-3.5.0-143128.i386.rpm  VMwareTools-3.5.0-143128.tar.gz

Since we are on RedHat, this is simple. Pass “-i” to the rpm command then the package name to be installed.

[root@RHserver01 media]# rpm -i VMwareTools-3.5.0-143128.i386.rpm

Immediately after installing the RPM, you might see the following errors to your console, or in /var/log/messages.

Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda
Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda

If you are getting these to the console, it makes it hard to continue working form the command line. This is easy to stop. First, cd out of the /media/ mount point

[root@RHserver01 ~]# cd ..

Next, unmount the cdrom device. After doing so, the messages will stop

[root@RHserver01 ~]# umount /dev/cdrom

The “vmware-config-tools.pl” command must be ran from the VMware console. Below is the output you would get if it was tried through a remote session (SSH).

[root@RHserver01 ~]# vmware-config-tools.pl

It looks like you are trying to run this program in a remote session. This
program will temporarily shut down your network connection, so you should only
run it from a local console session. Are you SURE you want to continue?
[no]
Please re-run this program from a local console shell.
Execution aborted.

There is a good reason for this. vmware-config-tools.pl drops networking on the server to install the VMware network drives. In doing so, you loose remote connectivity.

Below shows the actual output from vmware-config-tools.pl on the console

[root@RHserver01 ~]# vmware-config-tools.pl
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Stopping VMware Tools services in the virtual machine:
   Guest operating system daemon:                          [  OK  ]
   Unmounting HGFS shares:                                 [  OK  ]
   Guest filesystem driver:                                [  OK  ]
   Guest memory manager:                                   [  OK  ]
Trying to find a suitable vmmemctl module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmhgfs module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmxnet module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmblock module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

No X install found.

Starting VMware Tools services in the virtual machine:
   Switching to guest configuration:                       [  OK  ]
   Guest memory manager:                                   [  OK  ]
   Guest vmxnet fast network device:                       [  OK  ]
   DMA setup:                                              [  OK  ]
   Guest operating system daemon:                          [  OK  ]

The configuration of VMware Tools 3.5.0 build-143128 for Linux for this running
kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking the following command:
"/usr/bin/vmware-toolbox" during an X server session.

To use the vmxnet driver, restart networking using the following commands:
/etc/rc.d/init.d/network stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/rc.d/init.d/network start

If you wish to configure any experimental features, please run the following
command: "vmware-config-tools.pl --experimental".

Enjoy,

--the VMware team

Notes: From my experience, restart of networking via init.d scripts or rebooting the server is always needed. I personally always reboot the server to be safe.

Posted in Linux, Networking, VMWare