After connecting to the console, the following command was ran. nexus1kv# sh interface counters errors

--------------------------------------------------------------------------------
Port       Align-Err     FCS-Err    Xmit-Err     Rcv-Err   UnderSize OutDiscards
--------------------------------------------------------------------------------
Veth30            --          --          --          --          --     1127136
Veth69            --          --          --          --          --       29654
Veth70            --          --          --          --          --       31966
Veth71            --          --          --          --          --       39625

Above is a subset of the entries due to most of the rest having so little or no errors at all. The main problem interface is Veth30, which turns out to be w2k_serv01 (Windows).

Verifying the VM is indeed “w2k_serv0″
nexus1kv# sh interface Veth30 description

-------------------------------------------------------------------------------
Interface                Description
-------------------------------------------------------------------------------
Vethernet30              w2k_serv01, Network Adapter 1

Below is just to verify what the “sh interface counters errors” reported
nexus1kv# sh interface Veth30 | inc Drops

63 Input Packet Drops 1127136 Output Packet Drops

The other 3 ports are all assigned to linux_host01 (Linux).
nexus1kv# sh interface description | inc ops01

Veth69                   linux_host01, Network Adapter 1
Veth70                   linux_host01, Network Adapter 2
Veth71                   linux_host01, Network Adapter 3

After a little more investigation, it turned out that “w2k_serv01″ was deployed from a legacy template that was created on our old AMD cluster. Looking at the network interface in Windows 2003, it showed up as an “AMD …” network adapter.

To try and correct these errors, I removed the current network adapter through VMware vCenter and added a new one. This forced the OS in seeing the network interface as a new device and install fresh drivers. Now in Windows it shows up as an Intel network inteface.

Now that the new network adapter has been installed, I reset the statistics (and errors) on all the interfaces.
nexus1kv# clear counters

This command will clear "show interface" counters on all interfaces
Do you want to continue? (y/n)  [n] y

In the Nexus 1000v, the interface still shows up as Veth30. It has been a few days since the new network adapter was added and no errors are present.
nexus1kv# sh interface counters errors | inc Veth30

Veth30            --          --          --          --          --           0

I wrote this up just to show that even virtual network interfaces can have errors. Most likely the cause is the VMs OS or driver. Previously, I would have had to use esxcfg-info from the ESX host CLI, try to grep out/sort through to find a VMs network interface, and hope to find an error filed there. Yeah, that was a run-on sentence.  Sorry.

The server itself is running mainly Oracle database VMs.  Blade specs are as follows:

98 gigs of RAMSince the Oracle VMs are running semi-intensive databases, the RAM allocated to the heavy hitters are between 8-10 gigs.

B250m1 Stats From UCS

B250m1 Stats From Virtual Center

Now for the actual utilization statistics.  The following is from Virtual Center showing the physical B250-M1 database server.  You can see that the CPUs are more highly used in this environment.

B250m1 VC Utilization

Again, this is a mix between Windows Remote desktop and Oracle (RedHat Linux) database servers.  The following shows utilization information perm VM.

B250m1_2 VM Utilization

Hopefully this gives you a little more information concerning Oracle, VMware, Cisco UCS, and consolidation ratios for the full-width blades. Again, this server is running 98gigs of RAM.  Next purchase cycle, we will be upgrading the amount of RAM in all of our blades to get better consolidation.

There are a total of 7 blades in our VMware environment, but only 5 of those are dedicated to our main HA/DRS cluster. That gives us ~240 gigs of RAM for the main cluster. Currently, I am seeing a VM consolidation ratio of about 24 VMs (virtual machines) per B200-M1 blade. The limitation here is definitely the RAM. The CPU itself is less than 25% utilized per blade. Below is a screenshot of a B200-M1 host in our cluster.

Cisco UCS B200-M1 VM Ratio

Notes: The VMs running on the above host are

Operating system utilization

Below shows the error in VMware Virtual Center.

Here is the profile “iSCSI” before modification. Note the max 32 and available count.

The only way to increase this value is through command line in the active Nexus 1000v.

nexus01# configure t
nexus01(config)# port-profile type vethernet iSCSI
nexus01(config-port-prof)# vmware max-ports 1024
nexus01(config-port-prof)# end

Once the above command is issued, Virtual Center Client should show a reconfiguration task for the distributed virtual switch

Once the tasks is completed, go back into Home -> Inventory -> Networking and select the port profile in the distributed switch.  The “total ports” should now be set to 1024.

Make sure to save your configuration!

nexus01# copy running-config startup-config

Notes:  This should be non-intrussive, but make sure to test it in your environment!

First, go into the VMware console and right-click on the VM (Virtual Machine) that you are going to install VMware tools on.  Select “Install/Upgrade VMware Tools” option from the list.  Below is a screen shot of the menu.

VMware Tools Menu

By default, most CDROM devices are symbolically linked to /dev/cdrom by the operating system.

Just in case, you can search the messages file to see the actual device. This is needed only if /dev/cdrom is not automatically linked or you have setup multiple cdrom devices on the VM (Virtual Machine).

[root@RHserver01 media]# cat /var/log/messages | grep CDROM
Jan 10 10:59:03 RHserver01 kernel: hda: VMware Virtual IDE CDROM Drive, ATAPI CD/DVD-ROM drive

From above, you can see that the actual device is hda, specifically /dev/hda. If you are just curious you can do an “ll” on the /dev/cdrom device to see where is it linked to. In this case again, it’s going to hda.

[root@RHserver01 ~]# ll /dev/cdrom
lrwxrwxrwx 1 root root 3 Jan 12 13:18 /dev/cdrom -> hda

Mount the cdrom device to an empty or non-mounted point on the filesystem. Here I use the defaultly present /media location.

[root@RHserver01 /]# mount /dev/cdrom /media/
mount: block device /dev/cdrom is write-protected, mounting read-only

Below we move into the /media location using “cd”.

[root@RHserver01 /]# cd /media/

“ls” is used to display what files are present. Here we see both an RPM (native RedHat based OS package) and a gzip archive. If you were installing VMware Tools on a non-RedHat derived distribution, you would use the .gz package.

[root@RHserver01 media]# ls
VMwareTools-3.5.0-143128.i386.rpm  VMwareTools-3.5.0-143128.tar.gz

Since we are on RedHat, this is simple. Pass “-i” to the rpm command then the package name to be installed.

[root@RHserver01 media]# rpm -i VMwareTools-3.5.0-143128.i386.rpm

Immediately after installing the RPM, you might see the following errors to your console, or in /var/log/messages.

Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda
Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda

If you are getting these to the console, it makes it hard to continue working form the command line. This is easy to stop. First, cd out of the /media/ mount point

[root@RHserver01 ~]# cd ..

Next, unmount the cdrom device. After doing so, the messages will stop

[root@RHserver01 ~]# umount /dev/cdrom

The “vmware-config-tools.pl” command must be ran from the VMware console. Below is the output you would get if it was tried through a remote session (SSH).

[root@RHserver01 ~]# vmware-config-tools.pl

It looks like you are trying to run this program in a remote session. This
program will temporarily shut down your network connection, so you should only
run it from a local console session. Are you SURE you want to continue?
[no]
Please re-run this program from a local console shell.
Execution aborted.

There is a good reason for this. vmware-config-tools.pl drops networking on the server to install the VMware network drives. In doing so, you loose remote connectivity.

Below shows the actual output from vmware-config-tools.pl on the console

[root@RHserver01 ~]# vmware-config-tools.pl
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Stopping VMware Tools services in the virtual machine:
   Guest operating system daemon:                          [  OK  ]
   Unmounting HGFS shares:                                 [  OK  ]
   Guest filesystem driver:                                [  OK  ]
   Guest memory manager:                                   [  OK  ]
Trying to find a suitable vmmemctl module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmhgfs module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmxnet module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmblock module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

No X install found.

Starting VMware Tools services in the virtual machine:
   Switching to guest configuration:                       [  OK  ]
   Guest memory manager:                                   [  OK  ]
   Guest vmxnet fast network device:                       [  OK  ]
   DMA setup:                                              [  OK  ]
   Guest operating system daemon:                          [  OK  ]

The configuration of VMware Tools 3.5.0 build-143128 for Linux for this running
kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking the following command:
"/usr/bin/vmware-toolbox" during an X server session.

To use the vmxnet driver, restart networking using the following commands:
/etc/rc.d/init.d/network stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/rc.d/init.d/network start

If you wish to configure any experimental features, please run the following
command: "vmware-config-tools.pl --experimental".

Enjoy,

--the VMware team

Notes: From my experience, restart of networking via init.d scripts or rebooting the server is always needed. I personally always reboot the server to be safe.

http://blog.colovirt.com/2008/10/31/vmware-esx-server-partitioning/

The majority of the partitioning structure is the same what was used for 3.x. The only real changes is the fact that the installation process auto-creates two of the partitions that were to be manually created on 3.x. Those two partitions are:

As in the 3.x structure, again I still maintain that creating a seperate mount point for /var/core should be used. For the reasons stated below:From the 3.x post

“I have had a few servers core dump and drop over 5 gigs of data to /var/core. Before, per “best practices” a vendor recommended around 4 gigs for /var. I upped that to 6 gigs originally, but after 2 servers had /var 100% utilized I and revising that. /var is still 6 gigs but /var/core has been broken out into its own mount point. 15 gigs is a little high, but these servers had raid 1 – 73 gig hard drives. At least now if the servers core dump it will affect only its mount point. I highly recommend doing this!”

Below is how I am partitioning vSphere 4 servers

Notes: Yellow-Bricks (Duncan Epping), as well as VMETC have good articles as well.

Even though this is being done using Linux, the directory structure and location of data files will be helpful in doing this from Windows.

Below is a link to Yellow Bricks (Duncan Epping) VMware blog.  He goes through the process of creating the USB boot device for Windows.  If you are doing this from Linux, continue on from this section.
http://www.yellow-bricks.com/2009/06/09/vsphere-esxi-on-a-usb-memory-stick/

Create a temporary mount point where the downloaded ESX4i ISO will be placed

root@laptop:~# mkdir /mnt/temp

Next, “mount -o loop” is used to mount the ISO to the /mnt/tmp location.  Using the loop option, it allows us to view and traverse the ISOs structure.

root@laptop:~# mount -o loop /home/user01/Desktop/VMware-VMvisor-Installer-4.0.0-164009.x86_64.iso /mnt/temp

Change into the temp directory and check the contents

root@laptop:~# cd /mnt/temp
root@laptop:/mnt/temp# ls
boot.cat    ienviron.tgz  isolinux.bin  menu.c32    vmkboot.gz
cimstg.tgz  image.tgz     isolinux.cfg  README.txt  vmkernel.gz
cim.vgz     install.tgz   mboot.c32     sys.vgz

For ESX4i, the file that actually contains what we need is image.tgz.  I created the “/root/build/vi4″ subdirectory for building this configuration.  the tgz file is copied there shown below since /mnt/tmp is read only.

root@laptop:/mnt/temp# cp image.tgz /root/build/vi4

Move into the vi4 directory and verify the file was copied

root@laptop:/mnt/temp# cd /root/build/vi4
root@laptop:~/build/vi4# ls
image.tgz

Un-compress and extract the contents of image.tgz into the current directory

root@laptop:~/build/vi4# tar -xzvf image.tgz
usr/
usr/lib/
usr/lib/vmware/
usr/lib/vmware/installer/
usr/lib/vmware/installer/VMware-VMvisor-big-164009-x86_64.dd.bz2

Above lists the directories and files created when the contents were extracted.  So we see that the dd file was listed in the output.

Move into the directory containing the dd file and list the contents

root@laptop:~/build/vi4# cd usr/lib/vmware/installer/
root@laptop:~/build/vi4/usr/lib/vmware/installer# ls
VMware-VMvisor-big-164009-x86_64.dd.bz2

Here, bunzip2 is used to de-compress the VMware-VMvisor-big-164009-x86_64.dd.bz2 file.  This took about 10 minutes on a slow laptop.

root@laptop:~/build/vi4/usr/lib/vmware/installer# bunzip2 -d VMware-VMvisor-big-164009-x86_64.dd.bz2

Now we can see the fully decompressed dd file, which is almost 1 gig in size.

root@laptop:~/build/vi4/usr/lib/vmware/installer# ls -lah
total 901M
drwxr-xr-x 2 201 201 4.0K 2009-05-25 15:16 .
drwxr-xr-x 3 201 201 4.0K 2009-04-30 22:08 ..
-rw-r--r-- 1 201 201 900M 2009-04-30 22:11 VMware-VMvisor-big-164009-x86_64.dd

I already removed all of the partitions from the memory stick being used here.  I used gparted to accomplish this, but fdisk can be used as well.  Once the memory stick to be used is connected, check /var/log/messages to see what device has been assigned to the memory stick.

Here, I already knew that /dev/sdb was assigned to the USB memory stick.  The “dd” command thats standard on Linux is used to identically copy the VMware-VMvisor file to the memory stick.

root@laptop:~/build/vi4/usr/lib/vmware/installer# dd if=VMware-VMvisor-big-164009-x86_64.dd of=/dev/sdb
9-x86_64.dd of=/dev/sdb
1843200+0 records in
1843200+0 records out
943718400 bytes (944 MB) copied, 322.119 s, 2.9 MB/s

Once this was complete, the USB memory stick was ready to go.  I went over to the server, attached it, and ESX4i was immediately booted with no issues.  Configure the management IP and user from the console and you are ready to go! The same process needs to be repeated for each memory stick that needs to be created.  You could just use “dd” to copy the ESXi memory stick to another memory stick, but that would take longer.

Notes: When mounting an ISO in loop mode, you will not be able to write the the directory structure or modify files located there.  It is read only.

Comment or email me if you have any questions. kevin@colovirt.com

Below is an excerpt of the log coming from a dual fiber connected server. In this case, the max active paths would be 2. If the server had quad fiber connectivity it could have a max of 4 paths in our environment. As seen from the log, multipathd reported the loss of a path to each LUN an logged it “remaining active paths: 1″. Once the errors were corrected in the failed switch, multipathd reported the restoration “remaining active paths: 2“.

[root@testsrv01 ~]# cat /var/log/messages | grep multipathd
May  6 10:30:55 testsrv01 multipathd: TempLUN1: remaining active paths: 1
May  6 10:30:55 testsrv01 multipathd: 8:224: readsector0 checker reports path is up
May  6 10:30:55 testsrv01 multipathd: 8:224: reinstated
May  6 10:30:55 testsrv01 multipathd: RanSanLUN2: remaining active paths: 1
May  6 10:30:56 testsrv01 multipathd: 65:128: readsector0 checker reports path is up
May  6 10:30:56 testsrv01 multipathd: 65:128: reinstated
May  6 10:30:57 testsrv01 multipathd: TempLUN1: remaining active paths: 2
May  6 10:30:58 testsrv01 multipathd: 65:224: readsector0 checker reports path is up
May  6 10:30:58 testsrv01 multipathd: 65:224: reinstated
May  6 10:30:58 testsrv01 multipathd: RanSanLUN2: remaining active paths: 2

To be safe, the ‘multipath’ command was used via the Command Line Interface (CLI) on the server(s) to verify that the storage systems were visable to the OS.

NAME
       multipath - Device mapper target autoconfig
SYNOPSIS
       multipath [options] [device]

       -l     show the current multipath topology from information fetched in sysfs and the device mapper
       -ll    show the current multipath topology from all available information (sysfs, the device mapper ...)

Truncated results are below that shows connectivity to the RamSans and EMC. This server is not mapped to the NetApp.

[root@testsrv01 ~]# multipath -ll
ramsan_DramCache (25050c24001094501)
[size=9 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:0  sdah 66:16  [active][ready]
 \_ 2:0:4:0  sdai 66:32  [active][ready]
 \_ 1:0:3:0  sdal 66:80  [active][ready]
 \_ 1:0:4:0  sdan 66:112 [active][ready]

ramsan_FlashSSD (25050c24021094501)
[size=1853 GB][features="0"][hwhandler="0"]
\_ round-robin 0 [prio=4][active]
 \_ 2:0:3:1  sdaj 66:48  [active][ready]
 \_ 2:0:4:1  sdak 66:64  [active][ready]
 \_ 1:0:3:1  sdam 66:96  [active][ready]
 \_ 1:0:4:1  sdao 66:128 [active][ready]

EMC_LUN1 (461a963004637699a05bs3464f76)
[size=100 GB][features="1 queue_if_no_path"][hwhandler="0"]
\_ round-robin 0 [prio=2][active]
 \_ 2:0:2:8  sdab 65:176 [active][ready]
 \_ 1:0:2:8  sdl  8:176  [active][ready]

Notes: Make sure to verify that all servers have multipathing setup and working correctly, all of the Linux servers never missed a beat! The database (Oracle) and Websphere servers kept on going. The same can not be said about the Exchange environment. Due to a configuration problem there, multipathing was not functioning and it lost connectivity to its storage LUNs.  The VMware cluster survived fine as well.

The following example will show how to query a standard Linux based server running SNMP with version 1 queries allowed. I am passing the “ifPhysAddress” parameter to return only the interface(s) MAC addresses.
snmpwalk basic usage

USAGE: snmpwalk [OPTIONS] AGENT [OID]

Below is specifying version 1, community string “PUBLIC”, server to query “dbserv01.testdomain.com”, and information to return “ifPhysAddress”

root@testbox01:~# snmpwalk -v 1 -c PUBLIC dbserv01.testdomain.com ifPhysAddress
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.3 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.4 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.5 = STRING: 0:0:0:0:36:28

Above shows all the listed MACs that can be referenced by SNMP. As you can see, the same MAC “0:12:39:65:36:28″ is listed 3 times. This typically means that the network interfaces are bonded.

Below shows the actual interface information on the queried Linux server. As noted, the interfaces are bonded (bond0), and the MAC address (HWaddr) listed is the same as what was retrieved by snmpwalk.

[root@dbserv01 ~]# ifconfig
bond0     Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          inet addr:127.1.1.182  Bcast:127.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:1033814138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:833777834 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1051206111974 (979.0 GiB)  TX bytes:602216674869 (560.8 GiB)

eth0      Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1025047842 errors:0 dropped:0 overruns:0 frame:0
          TX packets:833777834 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1050502485293 (978.3 GiB)  TX bytes:602216674869 (560.8 GiB)
          Interrupt:193 Memory:e4000000-e4012100 

eth1      Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:8766296 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:703626681 (671.0 MiB)  TX bytes:0 (0.0 b)
          Interrupt:193 Memory:e2000000-e2012100

Here’s the main “gotcha”. Snmpwalk shows the MAC as “0:12:39:65:36:28″, but ifconfig on the local server shows “00:12:39:65:36:28″. For better comparison, both are listed again below:

0:12:39:65:36:28
00:12:39:65:36:28

As you will probably see, a “0″ is missing. This is not a true issue, but a design in SNMP. Most applications will auto correct/translate this. Of course, some do not.

Here are a few more examples

IF-MIB::ifPhysAddress.2 = STRING: 0:13:99:65:c:80
IF-MIB::ifPhysAddress.3 = STRING: 0:13:99:65:c:81

Each one of the above also has “0″s removed.

0:13:99:65:c:80
is really
00:13:99:65:0c:80

I think the point has been made. “0″s are removed from all fields (“:” separates the fields) if it is the first character. So “00″ is displayed as “0″, “0c” displayed as “c”, etc.

The best information that I have found on this is that each field represents a byte. Logically, 01 is the same as 1, 0d is the same as d, and so on. Also, remember that bonded interfaces will also list the same MAC on all interfaces.

This will be the same on most Linux/Unix based distributions. I was able to successfully retrive MAC information from Linux servers, DataDomain appliances(RedHat based), and NetApp NAS (BSD based).

Same results when running a check on a VMware ESX server

root@testbox01:~# snmpwalk -v 1 -c PUBLIC vmserv01.testdomain.com ifPhysAddress
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 0:1a:54:eb:12:96
IF-MIB::ifPhysAddress.3 = STRING: 0:1a:54:eb:12:94
IF-MIB::ifPhysAddress.4 = STRING: 0:13:26:84:7a:26
IF-MIB::ifPhysAddress.5 = STRING: 0:13:26:84:7a:27
IF-MIB::ifPhysAddress.6 = STRING: 0:13:26:84:7a:24

Notes: This is not an issue/design on all SNMP clients or daemons. Just don’t be freaked out if you stumble across this. Most SNMP clients/MIB browsers automatically add in the 0s that are left out.

I was outside hooking the kid bike trailer to the 12 speed when my wife came out holding my cell phone

Wife> Eric just called you twice
Me> Startup my laptop for me
... Call Eric
Eric> Sorry to bother you but  .... .

Luckily he noticed a couple of VMs dropped off the face of the earth. To be exact, we lost 4 virtual machines. When told to power on, they would error with “Can not open disk: PathToVmdk.. The parent virtual disk has been modified since the child was created“.

I never trust the GUI, so I went strait to the ESX console. Sure enough, things failed there also.

Per /var/log/vmkernel

Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.271 cpu6:1215)<6>Debug scsi underrun
Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.342 cpu6:1183)<6>Debug scsi underrun
Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.413 cpu6:1098)<6>Debug scsi underrun
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.898 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1497
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.898 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1501
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.899 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1499
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.899 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1504
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.900 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1496
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.901 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1494
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.901 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1502
Mar  5 23:10:06 vmsrv06 vmkernel: 28:09:47:04.375 cpu2:1051)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts
Mar  5 23:10:06 vmsrv06 vmkernel: 28:09:47:04.375 cpu2:1051)WARNING: FS3: 4785: Reservation error: SCSI reservation conflict

Next we will look at the NetApp. Below is from a console session. As you can see, the CPUs are hit hard.
From the NetApp

NAS>stats show system
system:system:nfs_ops:536/s
system:system:cifs_ops:0/s
system:system:http_ops:0/s
system:system:dafs_ops:0/s
system:system:fcp_ops:481/s
system:system:iscsi_ops:97/s
system:system:net_data_recv:136KB/s
system:system:net_data_sent:2323KB/s
system:system:disk_data_read:63752KB/s
system:system:disk_data_written:51640KB/s
system:system:cpu_busy:50%
system:system:avg_processor_busy:50%
system:system:total_processor_busy:100%
system:system:num_processors:2
system:system:time:1236313212s
system:system:uptime:9164385s

The conclusion is as follows: When VMware creates a snapshot, a new vmdk is created that references the new snapshot. The main vmx files will then be modified to link to the snapshot as well. When a snapshot is deleted, VMware is not just deleting a file from the file system. VMware gathers all the snapshots for that VM, collapses all their contained data into the main vmdk file, modifies the vmx files, then deletes the selected snapshots files. So in essence, when going down the list of all VMs that had snapshots(9-12) and deleting them, this caused the NetApp to overload. During this, the ESX hosts intermittently lost connectivity to the VMFS LUNS. This happened at such a fast rate, that no alerts were generated, but caused the VMs that were currently writing the snapshot to the main vmdk files to corrupt their configurations.

Moral of the story: Rapidly deleting snapshots can DOS older/slower storage systems.

You have to su to root for this command to work

[root@esxserverroot]# vmware-cmd -l
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/exchangeserv/exchangeserv.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/sql01/sql01.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/domaincontrol01/domaincontrol01.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/tstxp04/tstxp04.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/monitor5/monitor5.vmx

This commands lists all the currently running virtual machines on the individual ESX host. Also displayed is the full path to the vmx file, which is very useful when using vmware-cmd.

Now lets pass -axfww to ps. This will give us the

[root@esxserverroot]# ps -axfww | grep exchangeserv

30008 pts/0    S      0:00                      \_ grep exchangeserv
 2941 ?        S<     0:04 /usr/lib/vmware/bin/vmkload_app /usr/lib/vmware/bin
/vmware-vmx -ssched.group=host/user -# name=VMware ESX Server;version=3.5.0;
licensename=VMware ESX Server;licenseversion=2.0 build-120512; -@
pipe=/tmp/vmhsdaemon-0/vmxce7cca7370d3034d; /vmfs/volumes/
4844ace-b448s7d4-74e-000055858ed/exchangeserv/exchangeserv.vmx

Here is the same example, not using the BSD derivative arguments. Instead we will use -efww

[root@esxserverroot]# ps -efww | grep exchangeserv
root      2941     1  0 Feb02 ?        00:00:04 /usr/lib/vmware/bin/vmkload_app
/usr/lib/vmware/bin/vmware-vmx -ssched.group=host/user -# name=VMware ESX Server;
version=3.5.0;licensename=VMware ESX Server;licenseversion=2.0 build-120512;
-@ pipe=/tmp/vmhsdaemon-0/vmxce7cca7370d3034d; /vmfs/volumes/
4844ace-b448s7d4-74e-000055858ed/exchangeserv/exchangeserv.vmx
root     30029 29692  0 16:08 pts/0    00:00:00 grep exchangeserv

To forcefully stop the hung machine, just kill the process passing -9 to send term signal. This is not recommended unless the machine will NOT resopond to anything else.

[root@esxserver root]# kill -9 2941

Each time I have had to do this, the machine showed powered off in Virtual Center, but the getstate showed on.

[root@esxserverroot]# vmware-cmd /vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/
exchangeserv/exchangeserv.vmx getstate
getstate() = on

Try to force it off before going through the above steps to kill the process.

[root@esxserverroot]# vmware-cmd /vmfs/volumes/
4844ace-b448s7d4-74e-000055858ed/exchangeserv/exchangeserv.vmx stop force

If that errors, you must kill the processes.  Once done, try and restart the virtual machine.

[root@esxserverroot]# vmware-cmd /vmfs/volumes/
4844ace-b448s7d4-74e-000055858ed/exchangeserv/exchangeserv.vmx start

Note: Again, this is a last resort and could possible corrupt the vmdk disk. I have done this successfully about 6 times with no problmes.

Turns out this can definitely save a lot of money! Check out  Redhat pricing.

The licensing comes in the following forms:

Now for the fun part:

These are licensed per socket of the ESX host, not per virtual cpu. So for us, each of our ESX hosts has 4 sockets (physical cpu). So this means that the cheaper basic and standard subscriptions are out for us. Now, we will have to buy “Red Hat Enterprise Linux Advanced Platform for VMware”. Even though the unlimited sockets is the higher costs of all the “VMware” licensing, we still come out a LOT cheaper!

Note: Be careful if you inquire about this licensing. We almost sent in our order with CDW that was quoted using virtual CPUs instead of physical ESX host CPUs. The bad thing is it was CDW that told us that it was based on VCPU.  Also, to make it clear, the number of guests is how many installs of RedHat you can have under that license.  So if you get the 10 guests Advanced Platform license, you get to run 10 Redhat virtual machines on your ESX host under that license.

My observations are as follows:
- Works well with Virtual Center
- When adding an ESX host outside of VC, it will list running VMs on that server
* The alerting function looks great!
- The console works well for the VMs

Veeam Monitor Main View

Veeam Monitor Top VMs (utilization)

Veeam Monitor Individual VM Summary

Veeam Monitor VM Console

Below are a few screen shots of attaching alerts to a VM

Veeam Monitor Alert Setup

Veeam Monitor Alert Setup 2

Veeam Monitor Alert Setup 3

Veeam Monitor Alert Setup 4

Veeam Monitor Alert Setup 5

Please see the following post concerning some limitations and requirements of the Veeam Monitor.

Note: This offering adds better alerting capabilities than what is provided within Virtual Center.

I will update this post more over the next few days of testing this software.

Veeam Monitor supports the following VMware products:
VMware Infrastructure 3:
ESX/ESXi Server 3.x; VirtualCenter 2.x

System requirements are as follows:
Hardware
Processor: Pentium III CPU or higher preferred

Memory: 256MB RAM (512MB when using local SQL Express installation)

Free Hard Disk Space:
6MB (Client Only), 10 MB (Server installation)
Additional 260 MB for Microsoft SQL Server Express installation

Resolution: 1024×768 or higher preferred
Network adapter: 10 mbit/sec or higher preferred

OS: Windows XP (SP2 or later), Windows Vista, Windows Server 2003

Software: Microsoft .NET Framework 2.0 or later

SQL Database:
Microsoft SQL Server 2005 Express, or SQL Server 2005
An upgrade to a licensed version is needed for the following:
- Performance history
- Trend analysis and capacity planning
- Unlimited alarms and alarm modelling
- Drill-down into a vm (view running processes)

Note:  If you do not have a SQL server, the Veeam Monitor installer has the ability to install a Microsoft SQL Lite server.