Login to the primary 6120 via the UCS cluster IP address (SSH).

Verify the current state (roles) of the cluster

ucs01-B# show cluster state
Cluster Id: 0xg7949/79843654he-0x8486a7868s8787785

A: UP, SUBORDINATE
B: UP, PRIMARY

Enter local management on the primary.

ucs01-B# connect local-mgmt
Cisco UCS 6100 Series Fabric Interconnect

Since the “b” 6120 is the current primary, the following will activate “a” as the primary.

ucs01-B(local-mgmt)# cluster lead a
Cluster Id: 0xg7949/79843654he-0x8486a7868s8787785

Verify that the subordinate 6120 has now become the primary.  This was done by SSHing directly to 6120 “a”.

ucs01-A# show cluster state
Cluster Id: 0xg7949/79843654he-0x8486a7868s8787785

A: UP, PRIMARY
B: UP, SUBORDINATE

HA READY

Process to failover
1. Login to the primary 6120 via the UCS cluster IP address.
2. Verify the current primary 6120 in the cluster.
3. Enter local management on the primary via “connect local-mgmt” command.
4. Issue the “cluster lead x” command to make the subordinate switch become the primary. Replace “x” with the correct switch letter.
5. Verify that the role has changed on the previously subordinate switch by SSH’ing into it and issuing the “show cluster state” command. The switch show now show up as the primary.

Notes: I listed the 5 step process above with the “fluff” removed.  Just in case you don’t need to see the command(s) output.

Instead, I just erased the configuration (needed to pull the permanent config from the cluster anyways) and rebooted the unit. This took ~15 minutes to reset and re-configure the management interface.

I was able to find the process in a Cisco document (Linked to PDF below) and tested it on another switch.

First, connect into the Command Line Interface (CLI) of the 6120.

Change the “scope” to match the 6120 that you want to modify. We will be working with 6120 / fabric-interconnect a.

New6120-A# scope fabric-interconnect a

IP information is changed using the “set” command. It is as easy as below

New6120-A /fabric-interconnect # set out-of-band ip 172.16.0.67 netmask 255.255.255.0 gw 172.16.0.1
Warning: When committed, this change may disconnect the current CLI session

Once you have made the changes, “commit-buffer” must be executed to activate the changes

New6120-A /fabric-interconnect* # commit-buffer

The following is from a continuous ping that was running from a local machine. Pings started completing immediately after the changes were committed.

Request timed out.
Request timed out.
Request timed out.
Request timed out.
Reply from 172.16.0.67: bytes=32 time=5ms TTL=63
Reply from 172.16.0.67: bytes=32 time=1ms TTL=63
Reply from 172.16.0.67: bytes=32 time<1ms TTL=63
Reply from 172.16.0.67: bytes=32 time=4ms TTL=63

This process can be referenced in Cisco’s official documentation (Large PDF) linked below.
Page 55 in UCS CLI Configuration Guide 1.3.1

Follow me on Twitter

Time to shutdown: ~30 seconds

From boot:

It seems that SSH access came in at almost exactly the same time as the CLI console.  Some of you might find this useful, some not. Just figured I would post it either way.

I had the great (not really) idea to swap cable 1 and 2 on the IOM while I was replacing the module. I figured that if it was a port issue on the 6120 blade switch that the problem would move to the new port. Turns out the IOM freaked out when this happened and downed the ports that were not where they had been.

The part which made it more difficult to track down was the the “fabric ports” (links from chassis to switch) all showed green. It was 3 of the 8 backend ports on the IOM that were errored.

Here is a server interface that was pinned to a removed port from the UCS Manager CLI. Below shows the downed interfaces as “Error disabled”.

ucsm01-A(nxos)# sh interface brief | inc "Error disabled"
Eth2/1/3      1      eth  access down    Error disabled              10G(D) --

Once the 10g cables (SFPs) were swapped back to their origional locations, this ports came back. Here the status now shows “up”

ucsm01-A(nxos)# sh interface brief | inc Eth2/1/3
Eth2/1/3      1      eth  access up      none                        10G(D) --

Again, the resolution here was to put the cabling back in their origional locations. Even with this being a new IO module, UCS loaded it with the configuration from the origional card. While talking with Cisco TAC, they said that the development group is working on making the IO module switch uplinks swappable (dynamic) and not statically pinned.

This really wasn’t a major issue due to UCS having redundancy built in, we never lost connectivity to a server. It’s just one of those “gotchas” that I ran into while troubleshooting a port.

Doing so allowed us to free up fiber ports on our Cisco MDS 9124 switches needed later on in the UCS blade chassis implementation.  I love pictures of equipment so I figured I would share.

Floor Jack Loader

Nexus 5010 Un-boxed

Nexus 5010s Racked

Nexus 5010s With Fabric Extenders

Nexus 5010s Racked Front - Cold Aisle

5010's Rear Wired - Hot Aisle

N2K-C2148T-1GE Fabric Extenders In Network Rack

I am very picky when it comes to my Cisco devices. A lot of information I read on this had the schedule execute “copy running-config startup” and would only backup one configuration. This is not a good thing, especially when there are multiple device managers. Below will go through setting up two jobs that backup both the running and saved configurations to different files daily.

Note: This assumes that you already have a TFTP server running on the network.

After logging into the switch, move into configuration mode

FiberSw01# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.

Enable the scheduler

FiberSw01(config)# scheduler enable

Crate a job named “backup_running”

FiberSw01(config)# scheduler job name backup_running

Enter the syntax used to copy the running configuration to your TFTP server

FiberSw01(config-job)# copy running-config tftp://172.0.0.22:69/config/FiberSw01_running

Exit configuration mode

FiberSw01(config-job)# end

Now that we have a job defined, make sure it is listed with the scheduler

FiberSw01# show scheduler job
Job Name: backup_running
------------------------
   copy running-config tftp://172.0.0.22:69/config/FiberSw01_running
==============================================================================

With the job defined, we can go back in and set when we want it executed

FiberSw01# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.

Define a new schedule name and set execution time(s)

FiberSw01(config)# scheduler schedule name Backup_Running
FiberSw01(config-schedule)# time daily 23:00
FiberSw01(config-schedule)# job name backup_running
FiberSw01(config-schedule)# end

Now the schedule(s) can be listed with names, execution times, and status

FiberSw01# show scheduler schedule
Schedule Name       : Backup_Running
------------------------------------
User Name           : user
Schedule Type       : Run every day at 23 Hrs 0 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
     Job Name            Last Execution Status
-----------------------------------------------
    backup_running                        -NA-
==============================================================================

Since the running configuration was backed up previously, we can go in configure the startup configuration backup. All commands are close to above except the tftp file name

FiberSw01# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.
FiberSw01(config)# scheduler job name backup_startup
FiberSw01(config-job)# copy startup-config tftp://172.0.0.22:69/config/FiberSw01_startup
FiberSw01(config-job)# end

FiberSw01# config t
Enter configuration commands, one per line.  End with CNTL/Z.
FiberSw01(config)# scheduler schedule name Backup_Startup
FiberSw01(config-schedule)# time daily 23:05
FiberSw01(config-schedule)# job name backup_startup
FiberSw01(config-schedule)# end

Check the schedule once more and both jobs show up

FiberSw01# show scheduler schedule
Schedule Name       : Backup_Running
------------------------------------
User Name           : user
Schedule Type       : Run every day at 23 Hrs 0 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
     Job Name            Last Execution Status
-----------------------------------------------
    backup_running                        -NA-
==============================================================================
Schedule Name       : Backup_Startup
------------------------------------
User Name           : user
Schedule Type       : Run every day at 23 Hrs 5 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
     Job Name            Last Execution Status
-----------------------------------------------
    backup_startup                        -NA-
==============================================================================
Now that the configuration is done, save the current (running) configuration so the schedules will not be lost on reboot
FiberSw01# copy running-config startup-config
[########################################] 100%
Notes: The above was done on Cisco MDS Switches but should work on most other Ciso IOS versions.  Also, the Execution Status will change after the job is executed.

Create a test file

user01@testsrv:~$ touch testfile

Below, “ls” is used to display the inode number (2009418) of the test file

user01@testsrv:~$ ls -i /home/user01/testfile
2009418 /home/user01/testfile
Check the inode record for the newly created empty file

user01@testsrv:~$ stat /home/user01/testfile
File: `/home/user01/testfile'
Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fe01h/65025d	Inode: 2009418     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-04-10 17:52:59.000000000 -0400
Modify: 2009-04-10 17:52:58.000000000 -0400
Change: 2009-04-10 17:52:58.000000000 -0400

In the following example, a file is being checked that is solely owned by the root user

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 root root 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Review the root owned ISO file inode information. The user and group ownership is referenced as Uid and Gid respectively

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-03-28 13:15:33.000000000 -0400

Change the permissions of the ISO file via the root user (current owner of the file)

root@testsrv:/home/user01# chown user01:user01 /home/user01/w2k3-2.iso

Re-check tje ISO file permission from the initial user (user01) via the ls command

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 user01 user01 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Re-stat the ISO file to view the changes in the inode record. As you can see, the file ownership is now set to user user01 and group user01

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 17:57:35.000000000 -0400

For the next example user01 is kept as the user, but the group is changed to ‘admin’. Since the current user user01 now has full rights to the file, it can be done from that account

user01@testsrv:~$ chown :admin /home/user01/w2k3-2.iso

Re-stat the file once more to see if the group (Gid) has been changed in the inode for this file

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: (  119/   admin)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 18:00:32.000000000 -0400

Now with the basic understanding of finding an retrieving inode numbers and information, we will proceed to removing files using the “rm” command. Specifically, the following will detail how to remove a file by referencing the inode number. All of which is done by using “find” and passing the results to “rm”.

There have only been a few occasions that I can remember where I have needed to know the following, but it is worth knowing! The main benefit is when a file has been created with a control character in the name

Create a test file

user01@testsrv:/tmp/testdir$ touch '`'
user01@testsrv:/tmp/testdir$ ls
`

Try to remove the file without enclosing the “`” filename

user01@testsrv:/tmp/testdir$ rm `
>

Above you can see that the command did not work. Instead it dropped to the “>” sub-prompt

Locate the inode number for the “`” file. Tab based auto-complete does wonders escaping the special character filename on the command line

user01@testsrv:/tmp/testdir$ ls -i /tmp/testdir/`
1163285 /tmp/testdir/`

Now that the inode number is known, the “find” command can be used to retrieve the filename associated with it

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285
/tmp/testdir/`

Putting it all together, we can remove the file passing the “find” results to “rm”. Below uses a command line pipe (“|”) and xargs to accomplish this

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 | xargs rm

Verify the file has been removed

user01@testsrv:/tmp/testdir$ ls /tmp/testdir/

The same can be accomplished via find commands built-in exec feature

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 -exec rm {} ;

In my opinion, the best way to remove a file via the inode number is as follows

user01@testsrv:~$ find /tmp -inum 1163285 -exec rm -i {} ;
rm: remove regular empty file `/tmp/testdir/`'? y

Specifying “-i” after rm makes the user verify the file to be removed.

Notes: There are other ways around removing files with certain characters. In this chase, the “`” file could have been removed with:

user01@testsrv:/tmp/testdir$ rm /tmp/testdir/`

If any information in this post is found to be outdated, incorrect, useful, or needs further detail, please leave a comment or email me.

Check to make sure the licensces are installed
NAS> license
nearstore_option xxxxxZH
a_sis xxxxxCG

Enable sis (deduplication) on the volume
NAS> sis on /vol/testVol
SIS for “/vol/testVol” is enabled.
Already existing data could be processed by running “sis start -s /vol/testVol”.

Initiate deduplication service
NAS> sis start -s /vol/testVol
The file system will be scanned to process existing data in /vol/testVol.
This operation may initialize related existing metafiles.
Are you sure you want to proceed with scan (y/n)? y

Thu Nov 13 10:01:38 EST [wafl.scan.start:info]: Starting SIS volume scan on volume testVol.
The SIS operation for “/vol/testVol” is started.

NAS> Thu Nov 13 10:01:42 EST [wafl.snap.delete:info]: Snapshot copy sis.93057f3c-aed8-11dd-a6d9-00a09804ad86 on volume testVol NetApp was deleted by the Data ONTAP function dense_delete_snapshot. The unique ID for this Snapshot copy is (44, 156).

Sis status will display the state and running status on all volumes
NAS> sis status

View the status on a specific volume
NAS> sis status /vol/testVol

After sis completes, use df to show the amount saved and deduplication percentage
NAS> df -sh /vol/testVol

Verify that sis is scheduled to run frequent enough at the times you want
NAS> sis config

Below is the help output of sis config
NAS> sis help config
sis config [ [ -s schedule ] <path> | <path> … ]
- Sets up, modifies, and retrieves the schedule of SIS volumes.

Reschedule for nightly at 11pm
NAS> sis config -s sun-sat@23 /vol/testVol

Verify the new configuration

NAS> sis config /vol/testVol

Note: ASIS is not real time deduplication. It is ran on a schedule instead of as the data is coming in. Also, when setting the schedule for a volume, you can only specifiy the hour to start running, in this case trying to specify sun-sat@23:45 did not work when testing.