Networking, SAN: Cisco MDS Switch Scheduled Backups

•November 10, 2009 • Leave a Comment

Most people are good about making backup copies of their configuration before changes, but everyone makes mistakes eventually. To me the risk is not worth it, so this will be dedicated to automating Cisco TFTP backups of configurations. Most server administrators have automated tasks using either Cron (Linux/Unix) or Windows Scheduler. Cisco IOS also has the ability to schedule tasks.

I am very picky when it comes to my Cisco devices. A lot of information I read on this had the schedule execute “copy running-config startup” and would only backup one configuration. This is not a good thing, especially when there are multiple device managers. Below will go through setting up two jobs that backup both the running and saved configurations to different files daily.

Note: This assumes that you already have a TFTP server running on the network.

After logging into the switch, move into configuration mode

FiberSw01# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.

Enable the scheduler

FiberSw01(config)# scheduler enable

Crate a job named “backup_running”

FiberSw01(config)# scheduler job name backup_running

Enter the syntax used to copy the running configuration to your TFTP server

FiberSw01(config-job)# copy running-config tftp://172.0.0.22:69/config/FiberSw01_running

Exit configuration mode

FiberSw01(config-job)# end

Now that we have a job defined, make sure it is listed with the scheduler

FiberSw01# show scheduler job
Job Name: backup_running
------------------------
   copy running-config tftp://172.0.0.22:69/config/FiberSw01_running
==============================================================================

With the job defined, we can go back in and set when we want it executed

FiberSw01# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.

Define a new schedule name and set execution time(s)

FiberSw01(config)# scheduler schedule name Backup_Running
FiberSw01(config-schedule)# time daily 23:00
FiberSw01(config-schedule)# job name backup_running
FiberSw01(config-schedule)# end

Now the schedule(s) can be listed with names, execution times, and status

FiberSw01# show scheduler schedule
Schedule Name       : Backup_Running
------------------------------------
User Name           : user
Schedule Type       : Run every day at 23 Hrs 0 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
     Job Name            Last Execution Status
-----------------------------------------------
    backup_running                        -NA-
==============================================================================

Since the running configuration was backed up previously, we can go in configure the startup configuration backup. All commands are close to above except the tftp file name

FiberSw01# config terminal
Enter configuration commands, one per line.  End with CNTL/Z.
FiberSw01(config)# scheduler job name backup_startup
FiberSw01(config-job)# copy startup-config tftp://172.0.0.22:69/config/FiberSw01_startup
FiberSw01(config-job)# end

FiberSw01# config t
Enter configuration commands, one per line.  End with CNTL/Z.
FiberSw01(config)# scheduler schedule name Backup_Startup
FiberSw01(config-schedule)# time daily 23:05
FiberSw01(config-schedule)# job name backup_startup
FiberSw01(config-schedule)# end

Check the schedule once more and both jobs show up

FiberSw01# show scheduler schedule
Schedule Name       : Backup_Running
------------------------------------
User Name           : user
Schedule Type       : Run every day at 23 Hrs 0 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
     Job Name            Last Execution Status
-----------------------------------------------
    backup_running                        -NA-
==============================================================================
Schedule Name       : Backup_Startup
------------------------------------
User Name           : user
Schedule Type       : Run every day at 23 Hrs 5 Mins
Last Execution Time : Yet to be executed
-----------------------------------------------
     Job Name            Last Execution Status
-----------------------------------------------
    backup_startup                        -NA-
==============================================================================
Now that the configuration is done, save the current (running) configuration so the schedules will not be lost on reboot
FiberSw01# copy running-config startup-config
[########################################] 100%

Notes: The above was done on Cisco MDS Switches but should work on most other Ciso IOS versions. Also, the Execution Status will change after the job is executed.

Linux: Generating Strong Passwords Using random/urandom

•January 7, 2009 • 4 Comments

Occasionally, I find myself logged into a system that does not have a random password application installed and do not want to go to the trouble of downloading one.  Below is the easiest processes that I have found to generate a pretty random password from any Linux variant.

To begin, strait from the Linux man page:

/dev/random
When  read,  the /dev/random device will only return random bytes within the estimated number of bits of noise in the entropy pool.  /dev/random should be suitable for uses that need very high quality randomness  such  as  one-time  pad  or key generation.  When the entropy pool is empty, reads from /dev/random will block until additional environmental noise is gathered.

/dev/urandom
A read from the /dev/urandom device will not block waiting for more entropy.  As a result, if there is not sufficient  entropy in the entropy pool, the returned values are theoretically vulnerable to a cryptographic attack on the algorithms used by the driver.  Knowledge of how to do this is not available in  the  current  non-classified literature,  but it is theoretically possible that such an attack may exist.  If this is a concern in your appli‐cation, use /dev/random instead.

So basically, using /dev/random results in the strongest and most random characters.  Only downfall is the wait needed unless you have a lot of noise or specific hardware to accelerate the process.

* I tested a cut and paste from this page and some of the lines did not work correctly due to either the CSS or WordPress doing something weird with the ‘ and ` symbols.  So if one of the strings do not work for you, try deleting the ‘ and adding it back in.

Creating random passwords which contains no special characters, is 10 characters long and displays 4

$ cat /dev/urandom| tr -dc 'a-zA-Z0-9' | fold -w 10| head -n 4
z4w7RENNIn
ZOYg80cuQx
Kgm6IrS5wc
F741uiEXl6

Creating passwords which DO contain special characters, and is 12 characters long.  The grep at the end might seem a little redundant, but depending on how short your character length is (using fold), urandom will result in stings with no special characters.  Grep keeps that from happening here.

$ cat /dev/urandom| tr -dc 'a-zA-Z0-9-_!@#$%^&*()_+{}|:<>?='|fold -w 12| head -n 4| grep -i '[!@#$%^&*()_+{}|:<>?=]'
a(PYY5oid#2Z
>s#e)C5Kl=kc
63r)WBt9Y)^J
2_a5RLJV<CZH

Continue reading ‘Linux: Generating Strong Passwords Using random/urandom’

VMware: ESX Restart A Hung Virtual Machine

•February 9, 2009 • 4 Comments

This is a last case scenario blog. I recommend always start out using the GUI.  If that does’t work, use vmware-cmd to try and stop the VM.  If all else fails, follow below.  First you need to know what ESX server the VM is running on. The easiest way is to click on the VM you are looking for and go to the summary tab. The ESX server it resides on will be in the “General” section under “Host”. The other method is to connect to each of the ESX hosts and run the following:

You have to su to root for this command to work

[root@esxserverroot]# vmware-cmd -l
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/exchangeserv/exchangeserv.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/sql01/sql01.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/domaincontrol01/domaincontrol01.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/tstxp04/tstxp04.vmx
/vmfs/volumes/4844ace-b448s7d4-74e-000055858ed/monitor5/monitor5.vmx

This commands lists all the currently running virtual machines on the individual ESX host. Also displayed is the full path to the vmx file, which is very useful when using vmware-cmd.

Now lets pass -axfww to ps. This will give us the

Continue reading ‘VMware: ESX Restart A Hung Virtual Machine’

Linux/Security: Gathering Filesystem,Device, and Port Process Details With fuser

•March 5, 2009 • Leave a Comment

When troubleshooting a *nix box, working knowledge of file system, network, and process utilities are a necessity. The main ones for me are mount, lsof, dd, ps, fsck, netstat, tcpdump, and fuser. All of these tools are very basic, but most admins seem to not know about or utilise fuser. All of fusers functionality can be accomplished by using any of the above commands together. In the below examples, the same end results can be archived by using kill and lsof together, but why not just use one tool?

To begin, we will be passing fuser the -m switch to specify both a device and file(system).

       -m     name specifies a file on a mounted file system or a block device
              that is mounted. All processes accessing files on that file sys-
              tem  are  listed.  If a directory file is specified, it is auto-
              matically changed to name/. to use any file system that might be
              mounted on that directory

In it’s basic form, fuser will provide only the process id(s) (PID) that are currently utilising the specified file/device

root@testbox:~# fuser -m /media/disk/
/dev/sdb1:            5535c  5589c

The above example showed that process 5355 is currently utilizing something on the /media/disk file system. Below is the same example, but specifying the actual device that is mounted on /media/disk.

root@testbox:~# fuser -m /dev/sdb1
/dev/sdb1:            5535c  5589c
 Continue reading 'Linux/Security: Gathering Filesystem,Device, and Port Process Details With fuser'

Linux/Networking/Security: TFTP Deamon Setup and Cisco Configuration Backup

•March 31, 2009 • 1 Comment

This is just a quick walk-through on setting up TFTP service on a RedHat, Centos, or Fedora system. In general, this process should transfer over to other Linux (not BSD!) derived distributions.

[root@tftpsrv ~]# yum install tftp
Resolving Dependencies
--> Running transaction check
---> Package tftp-server.i386 0:0.42-3.1.el5.centos set to be updated
--> Processing Dependency: xinetd for package: tftp-server
--> Running transaction check
---> Package xinetd.i386 2:2.3.14-10.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

Continue reading ‘Linux/Networking/Security: TFTP Deamon Setup and Cisco Configuration Backup’

Linux, Unix, NAS, File Systems: Inodes (Part 1) – Checking Availability And High Level Overview

•April 10, 2009 • Leave a Comment

Inodes really tell you how many file handles (files) that can be created on a file system. Most people will never exceed the default setting when the file system is created, nor even know that one is set. I will eventually go into more detail concerning this topic here on the blog. The majority (not all) of file systems that are used on Linux and Unix do not support dynamic inode allocation. What this means is that if you exceed the inode limit of a file system before the storage space, the remainder will be un-usable. That is until some of the current files are removed.

So here is the really bad part. The inodes on ext2 and ext3 (Linux default type) are statically set when the file system is formatted. You can not go back and change the max inode settings. The exceptions to this that I know of are as follows:

- Reiser4
- VxFS
- XFS
- JFS
- WAFL (NetApp proprietary)
- XZFS

If you are running one of the above and have max inodes issue, you can correct it.

I have been working with computers for over 15 years and have only ran into this problem once. Luckily, it occurred on a NetApp NAS device that had the ability to increase this value on the live file system. The main killer here are tons of small files. In this case, the file system for that NFS share was 40 gigabytes in size and default was ~1 million inode limit. The quick fix for the issue was to increase this to 3 million.

As far as a ext2 and 3 go, the following shows how to query a file system for relevant inode information
Continue reading ‘Linux, Unix, NAS, File Systems: Inodes (Part 1) – Checking Availability And High Level Overview’

Linux/Unix/File Systems: Inodes (Part 2) – File Level Inode Information And Removal

•April 13, 2009 • Leave a Comment

Inodes “Part 1″ went into locating filesystem level inode information. Here we will move from the main filesystem to the individual file. Besides reviewing how the inode record reflects permissions modification using chown, file removal based on inode number will be covered.

Create a test file

user01@testsrv:~$ touch testfile

Below, “ls” is used to display the inode number (2009418) of the test file

user01@testsrv:~$ ls -i /home/user01/testfile
2009418 /home/user01/testfile
 Continue reading 'Linux/Unix/File Systems: Inodes (Part 2) – File Level Inode Information And Removal'

SAN/Networking/Linux: Using Multipath To Verify and Troubleshoot Connectivity To FC LUNs

•May 11, 2009 • Leave a Comment

A few days ago there was in error on one of our Cisco MDS 9120 fiber switches. The current environment at this datacenter consists of two Cisco MDS 9120 SAN switches with servers redundantly connected between the two. These switches are used to connect the servers to our fiber channel (FC) storage systems. In this case, the servers are generally mapped to an EMC, NetApp, and two RamSans. Below outlines a basic example of what can be expected from multipathing and the Linux environment when there is loss of connectivity to one leg of the fiber network.

Continue reading ‘SAN/Networking/Linux: Using Multipath To Verify and Troubleshoot Connectivity To FC LUNs’

VMware/Linux/ESXi: Running ESX4i From Bootable USB

•June 1, 2009 • 7 Comments

I like running ESXi via booting from USB 2.0 memory sticks.  This makes it that much easier for my home lab.  Especially since I mainly use ISCSI VMFS datastores.  Not to mention that “in a pinch”, having ESXi on memory sticks can aide in disaster recovery (DR) scenarios for small businesses.  Of course the requirement here is that the server MUST be able to boot from USB!  Also, get a big memory stick.  Each time an upgrade is performed to ESXi, the version being upgraded from is still stored on the memory stick in case a “roll back” is needed.  At least this is my understanding.  Larger memory sticks are pretty cheap now.  Below outlines the steps to creating a bootable ESX4i memory stick.  The main reason for me writing this up is that the process turns out to be different from ESX3i.

Continue reading ‘VMware/Linux/ESXi: Running ESX4i From Bootable USB’

SAN / EMC: Clariion CX4 Solid State DAEs (Shelves)

•June 3, 2009 • Leave a Comment

Going over the solid state offerings for the EMC Clariion lines, Texas Memory RamSans came into the conversation.  This was due to the fact that we currently run 2 RamSans in our Environment and consider them the highest tier storage in our datacenters.  One is 128 gigs of solid state DRAM storage and the other is 2 terabyte solid state Flash storage with a 64 gig DRAM cache.

Per the title, this is really about the EMC Clariion, not RamSans.  Since the RamSan 500 was fronted with the DRAM cache, and the EMC CX4 series contains cache as well, I was curious.  I already knew that each Service Processor (SP) in the EMC has 4 gig of cache, and that a LUN can only be active on one SP at a time.  Also, per a previous blog post, each DAE has a theoretical max throughput of 8 gigabit per second, 4 gigabit if a single LUN stripes across the whole shelf.

Continue reading ‘SAN / EMC: Clariion CX4 Solid State DAEs (Shelves)’

Networking / SAN: Cisco MDS 9000 Serial Number (Licensing)

•August 7, 2009 • Leave a Comment

So you need to find the serial number on your Cisco MDS 9000 series fiber switch? This is easy enough, although “show serial number” would have been better.

Quick way to find your serial number.

tstSwitch01# show license host-id
License hostid: VDH=SOZ115568P9

Continue reading ‘Networking / SAN: Cisco MDS 9000 Serial Number (Licensing)’

Linux / Security: Sudo ’sudo su -’ vs ’sudo -s’

•August 18, 2009 • 6 Comments

I always use ’sudo su -’ when I need to get to a root shell. I have seen a few people before, and a new co-worker recently use ’sudo -s’. Since I could not remember off hand the actual differences between the two, I had to check. The following will run through the actual limitations.

The big difference when using ‘-s’ are listed below
Continue reading ‘Linux / Security: Sudo ’sudo su -’ vs ’sudo -s’’

Networking / SAN: Cisco MDS 9000 License Installation

•September 3, 2009 • 1 Comment

This one will be quick and easy! Below is how to install a new port license on a Cisco MDS 9000 switch from the Cisco CLI (Command Line Interface). Doing this does not remove the current license, just adds it to the configuration. As always though, back up your configuration and make sure if there is a current license that you also have a backup copy of it.

First, make sure you put a copy of the license onto a tftp, ftp, or sftp server. The MDS switch supports all of those protocols. Here we will be using tftp.

Continue reading ‘Networking / SAN: Cisco MDS 9000 License Installation’

Linux / Security: Encrypted External Drive Part 1 – Urandom

•September 4, 2009 • 1 Comment

So I am re-doing my external RAID 1 drive enclosure. I love this little thing. It has two 2.5 inch 160gig SATA drives in it. The enclosure is connected via USB 2.0 but it does have an eSATA interface as well. I will be configuring this to have a 10 gig non-encrypted partition. The remaining ~150 gigs will be an encrypted (LUKS) filesystem to be used on my linux machine.

All of this will not be detailed here but will be split up in 3 blogs. Below just shows the time it takes to use Linux to overwrite the disk device using /dev/urandom. This is done to make it just that much harder for a would be hacker to try and brute force the key on the encrypted partition. If this is not done, the un-used space would just show up as empty, allowing for a more targeted attack against the pseudo random filesystem. Being pseudo-random means that it is not truly random. This being the case, with a lot of time and computing power, an attacker might be able to either brute force or find a pattern in the encryption.

So why not use /dev/random? For me, this would take forever! I do not have any special hardware or scripts pulling information from the environment and adding to the entropy pool. The data on this drive not being national security grade, /dev/random will do the job.

I know that the drive is under /dev/sdb. With that information, it is as simple as using “dd” (built in Linux utility) to overwrite all blocks on the drive with pseudo-random data.

root@tstbox:~# dd if=/dev/urandom of=/dev/sdb
dd: writing to `/dev/sdb': No space left on device
312581810+0 records in
312581809+0 records out
160041886208 bytes (160 GB) copied, 40284.5 s, 4.0 MB/s

From above, it tool 40,284.5 seconds to overwrite the drive with urandom data. This equals ~11 hours and 19 minutes. Definitely still a long time, but a lot faster than if /dev/random was used.

This workstation is not an impressive computer. It is a single CPU dual core machine with 2 gigs of ram. Below is the info on one of the cores.

root@tstbox:~# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 CPU          6300  @ 1.86GHz
stepping	: 2
cpu MHz		: 1867.000
cache size	: 2048 KB
physical id	: 0
siblings	: 2

Notes: I wish I could use /dev/random and probably will eventually when I can sit a drive out for a week. Setting up external drives in this fashion is really geared towards data protection. Not only are the drives in a mirrored RAID (one can fail and everything would still run fine), important data is encrypted using a strong key. So who cares is the external enclosure walks away at a conference? I would be out ~190$ but the data will be safe.

Linux / Storage: Memory – Huge Pages Overview

•October 13, 2009 • 2 Comments

A page is really virtual memory which is managed by the Translation Lookaside Buffers(TLB) in the CPU. The TLB controls the mapping of the virtual memory pages to physical memory addresses. In doing so, it bypasses the kernel virtual memory manager.

Per RedHat,

The TLB is a limited hardware resource, so utilising a huge amount of physical memory with the default page size consumes the TLB and adds processing overhead – many pages of size 4096 Bytes equates to many TLB resources consumed.

Continue reading ‘Linux / Storage: Memory – Huge Pages Overview’

Linux / Security: User Account Expiration Management

•October 20, 2009 • 2 Comments

I am a firm believer in regular password rotation/change and Linux has a built in mechanism that makes it easy. The following is a brief overview of password and account ageing for Linux based systems.

The program that enables listing and modification on the expiration parameters is ‘chage’. Each individual user can view their account settings as shown below.
Continue reading ‘Linux / Security: User Account Expiration Management’

Linux / Oracle: IBM WebSphere Partner Gateway Oracle Gotcha

•October 21, 2009 • Leave a Comment

I have been wrestling around with IBM WebSphere Partner Gateway for a few weeks now.  There are so many tiny gotcahs out there that can affect the whole installation process.

The main one that got me was integration with Oracle.   An overview of the installation steps are shown below:
Continue reading ‘Linux / Oracle: IBM WebSphere Partner Gateway Oracle Gotcha’

Linux, IBM: WebSphere WAS and Partner Gateway Version 6.2 FixPack 1

•November 3, 2009 • Leave a Comment

This is just a brief overview. The installation process is pretty easy on these. Same as with most patches, IBM UpdateInstaller “update.sh” was used to install the service “pak” files. These patches must be done in order. Patch the WAS installation before patching WPG.

All WebSphere services must be stopped to install the WAS updates. On a standard installation, bcguser must be used to stop the service

[bcguser@WPGhost ~]$/opt/IBM/bcghub-simple/bin/./bcgStopServer.sh

We do not use ‘/opt’ for our WebSphere location, so change this if yours is different.

Next, use Update Installer to patch the WebSphere Application Server
Continue reading ‘Linux, IBM: WebSphere WAS and Partner Gateway Version 6.2 FixPack 1′