Colocation to Virtualization

copy scp://user01@127.0.0.15/IOS/1000v/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin bootflash://
copy scp://user01@127.0.0.15/IOS/1000v/nexus-1000v-mz.4.2.1.SV1.4.bin bootflash://

From the active VSM (module 1 in this case), the new firmware is loaded into the configuration via the “install” command.

1000vSW# install all system bootflash:nexus-1000v-mz.4.2.1.SV1.4.bin kickstart bootflash:nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin
System image sync to standby is in progress...
System image is synced to standby.
Kickstart image sync to Standby is in progress...
Kickstart image is synced to standby.
Boot variables are updated to running configuration.

The boot variables were updated per the above, but it is still worth verifying.  Below shows that the running configuration was updated to load the new firmware.

1000vSW# sh running-config | inc boot
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-1
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-1
boot kickstart bootflash:/nexus-1000v-kickstart-mz.4.2.1.SV1.4.bin sup-2
boot system bootflash:/nexus-1000v-mz.4.2.1.SV1.4.bin sup-2

I almost skipped this command the first time. Make sure to save the configuration to startup-config or the reload of standby module will load up the startup-configs boot variables. That would bring it up on the same 4.0(4)SV1(3a) version.

1000vSW# copy running-config startup-config
[########################################] 100%

We saw the module status earlier in this post, but I will reiterate it here. Module 2 shows as the standby VSM here, so it will be reloaded first.

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
2    0      Virtual Supervisor Module        Nexus1000V         ha-standby

Time to reboot the standby module. This is done as follows.

1000vSW# reload module 2
This command will reboot standby supervisor module. (y/n)?  [n] y
about to reset standby sup
1000vSW# 2011 Feb 25 19:39:33 1000vSW %PLATFORM-2-PFM_MODULE_RESET: Manual restart of Module 2 from Command Line Interface

Module 2 is still restarting, notice it missing under “Mod”

1000vSW# sh module
Mod  Ports  Module-Type                      Model              Status
---  -----  -------------------------------- ------------------ ------------
1    0      Virtual Supervisor Module        Nexus1000V         active *
3    248    Virtual Ethernet Module          NA                 ok
4    248    Virtual Ethernet Module          NA                 ok

* you can bring up the console of the standby switch in vCenter to check progress

Ok, this is where everything went wrong for us.  A typical upgrade would have the module connect back in showing the new version of code running on it. For us, this did not happen. Instead module 1 and 2 were not able to communicated with each other. Both decided to become active.

This became a big problem. All VMs running were still able to pass traffic across the network, but none of ESX hosts showed up as modules in the VSM, thus isolating them. First symptom I saw was connectivity loss via SSH to the VSM IP. It seemed to happen about every two minutes. luckily I saw the firmware version changing each time I did a show module after reconnecting. So the two 1000v modules were in IP conflict with each other, stealing it away from one another every few minutes.

I waited for the module running version 4.0(4)SV1(3a) to grab the IP again and issued a reload. This was module 1. Once it restarted and came up on the new firmware, it started talking with module 2.

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          ha-standby
2    0      Virtual Supervisor Module         Nexus1000V          active *

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Now that we are on the same firmware, it was time to make module 1 primary/active once more. This was non-intrusive.

1000vSW# system switchover

Once more to verify that module 1 has become active:

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Also to be safe, have the VSM reconnect with Virtual Center.

1000vSW## configure t
Enter configuration commands, one per line.  End with CNTL/Z.
1000vSW#(config)# svs connection VirtualCenter
1000vSW#(config-svs-conn)# connect
1000vSW#(config-svs-conn)# end

Verify the connection as well.

1000vSW#  show svs connections

connection vcenter:
    ip address: 127.0.0.99
    remote port: 80
    protocol: vmware-vim https
    certificate: default
    datacenter name: Test
    DVS uuid: 55 55 55 55 55 55 55 55
    config status: Enabled
    operational status: Connected
    sync status: Complete
    version: VMware vCenter Server

At this point we should be good. At least that is what I thought.  Going back to what I said before
“but none of ESX hosts showed up as modules in the VSM.”
Yeah, I didn’t notice that untill after the fact. Normally I vmotion over an individual host to a different server to test connectivity, but this time I didn’t.

I selected our first vSphere (ESX) host and entered maintenance mode, thus vmotioning all VMs off to other hosts. Immediately they fell off of the network. All pings were lost. These were running critical/production servers. Of course we were in a maintenance window, but this still was worst case scenario. Normally each hosts VEM would show up in the show module, but they did not after the upgrade.

1000vSW# sh module
Mod  Ports  Module-Type                       Model               Status
---  -----  --------------------------------  ------------------  ------------
1    0      Virtual Supervisor Module         Nexus1000V          active *
2    0      Virtual Supervisor Module         Nexus1000V          ha-standby

Mod  Sw                Hw
---  ----------------  ------------------------------------------------
1    4.2(1)SV1(4)      0.0
2    4.2(1)SV1(4)      0.0

Mod  MAC-Address(es)                         Serial-Num
---  --------------------------------------  ----------
1    00-00-00-00-00-00 to 00-00-00-00-00-00  NA
2    00-00-00-00-00-00 to 00-00-00-00-00-00  NA

Mod  Server-IP        Server-UUID                           Server-Name
---  ---------------  ------------------------------------  --------------------
1    127.0.0.55     NA                                    NA
2    127.0.0.55     NA                                    NA

The above resembled what I was seeing. Just the VSM modules. The mac addresses were correct, I just changed them to all 0′s here. We tried shutting down a VM and moving it to different hosts in the cluster, but still no pings. I also tried stopping and starting the VEM via command line on the ESX host.

The only thing that fixed this was pushing the VEM upgrades. This was done using VMware Update Manager. Very easy, but we had to put the hosts into maintenance mode to do so. This wound up being very time-consuming. Once server 1′s VEM was updated, we were able to vMotion VMs to it without losing network connectivity to them. It went like this: