DD 670 Front

We received our new DataDomain 670 last week and it has definitely had a face lift from the 500 series.  First, it is a 2u versus a 3u unit.  Also, the 600 series no longer have the annoying third power supply.  The drives are also horizontal instead of vertical.  Even slots in the back are different.  I wish we had a multi-shelf unit one.  Below are a few pictures from the unpacking and racking.

Notes: Only complaint is the rails.  I really wish they rails on the server were pegged style – drop into extended rack rail types (IBM).  Instead, the rails on the servers are fixed and have to be lined up perfectly with the ones in the rack.  They then slide straight into the rack rails and lock.  I find it a lot hard with this kind of rails to rack heavy equipment alone.

Below are a few videos of the equipment.  Be careful with the sound!  These were taken in the datacenter and you will hear the typical rumbling of servers and AC units.

Video of the NetApp FAS 3020 once removed

Video of the EMC NX4 waiting to go in

Since then I have installed a RamSan for the Oracle environment at this datacenter and have no need for a lot of tiny drives.  So we replaced them with 600gig 10kRPM drives to be used in our VMware environment and RecoverPoint systems.  How about some EMC drive replacement pics until I can get some more content available?

Funny thing is that I forgot to take pictures of the drives once installed.  Might add them next trip/

I removed the faulty switch after disconnecting all cabling. The new 6120 was put on the network and firmware updated to match the cluster. After the upgrade was done, I erased the configuration on the switch and connected the fabric interconnects.

At this point, I ran through the CLI setup utility. Everything looked find and I though the configuration push was successful due to the following information being returned:

Apply and save the configuration (select 'no' if you want to re-enter)? (yes/no): yes
Applying configuration. Please wait.
Configuration file - Ok

When I went into the UCS manager, the switch didn’t look right. Once into the CLI on the new 6120, the following error confirmed there was a problem.

6120-A# scope fabric-interconnect a
Note: SAMCLI process currently not running

That definitely wasn’t a good thing! The 6120-B member did not show 6120-A as working in its “show cluster state”. Below is the needle in the haystack. After connecting to local management on the switch, it threw a configuration error.

6120-A# connect local-mgmt
svcconfig init: Ini file syntax error at line 127.0.0.10

At that time of night ~2am, it didn’t stand out what that IP address was. I called Cisco TAC and waited for an hour. It was getting pretty late so we called off the maintenance. I put the faulty switch back in. It was still functioning and cables moved from the bad ports over to working ones.

I erased to configuration of the replacement switch and brought it back up in standalone mode. The next day I went to check that switch out and all errors were gone. It seemed to be functioning fine when it was not a member of the cluster.

I used Putty client to log all output while I was connected via serial to the switches that nigh. After reviewing them, I came across the syntax error line multiple times. Then the light bulb came on. The IP 127.0.0.10 is the secondary DNS server on our network.

I already had a ticket opened with Cisco TAC on this. Called back in and asked if they had any bugs that were due to DNS issues. Turned out they were two of them. The IDs are as follows:

CSCtg14999
CSCtf95156

Both of the links above require a cisco.com login, but information on them are also listed below:

“CSCtg14999 has been superseded by CSCtf95156 displayed below.

CSCtf95156 Bug Details
secondary DNS entry not properly copied to peer during clustering
Symptom:

If you have 2 DNS servers configured in UCSM and then attempt to peer with a new FI (i.e. a subordinate box is added to the setup or a write erase is done to an existing subordinate) the first DNS entry is copied over properly but the second one is missing the ‘dnsServer’ prefix and ends up in the sam.config file (/opt/db/sam.config) as just an IP address on a line.

As a result the config is not properly read and clustering does not occur.

Workaround:

Deleting the second DNS IP address from the config will allow the FI to cluster (assuming all other criteria are correct and in place).”

So per the above information, ID CSCtg14999 looks have been replaced or is a child bug of CSCtf95156.

The fix: I removed the DNS servers from UCS configuration, and replaced the switch during another maintenance window. This time, it worked great. Was in and out in about 1.5 hours.

Now all that we need to do is get some data over to it.  We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit.  You can not change the order in which each device is racked.  Also, the controllers (SPs) are built into the drive tray.  Best part is that this will be wired up via 10gbE connectivity over fiber.

Storage:

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon.  This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120.  I definitely would love feedback on this.  How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Cisco UCS PALO Details

Per UCS the PALO cards are listed as

Boot times based on firmware version:

PALO Boot Time Firmware 1.2(1b)

PALO Boot Time Firmware 1.3(1c)

So the fix is to upgrade.  Not the easiest process, but not that hard either.  As from the stats above, v1.3(1c) is definitely worth the upgrade!  No more waiting around for 2008 to boot.

Notes: We also run RedHat Linux and VMware ESX in this environment.  They boot without issues or delay with the same PALO adapters. Also, it only took 2 minutes and 16 seconds to get network connectivity up on 2008 running version 1.3(1c) firmware.  Having no other good options I timed the boot using my iPhone.  The above images are from screen-captures on the phone.  UCS is not 3G connected :).

Below are pictures of the full rack that includes:

• Two Cisco UCS Chassis fully populated
• Two Cisco Nexus 5010 switches
• Two Cisco 6120XP switches
• Two horizontal PDUs (Circuit 1)
• Two vertical PDUs (Circuit 2)

Just in case you wanted a pretty view, the last picture is from my mountain trip last weekend

The server itself is running mainly Oracle database VMs.  Blade specs are as follows:

98 gigs of RAMSince the Oracle VMs are running semi-intensive databases, the RAM allocated to the heavy hitters are between 8-10 gigs.

B250m1 Stats From UCS

B250m1 Stats From Virtual Center

Now for the actual utilization statistics.  The following is from Virtual Center showing the physical B250-M1 database server.  You can see that the CPUs are more highly used in this environment.

B250m1 VC Utilization

Again, this is a mix between Windows Remote desktop and Oracle (RedHat Linux) database servers.  The following shows utilization information perm VM.

B250m1_2 VM Utilization

Hopefully this gives you a little more information concerning Oracle, VMware, Cisco UCS, and consolidation ratios for the full-width blades. Again, this server is running 98gigs of RAM.  Next purchase cycle, we will be upgrading the amount of RAM in all of our blades to get better consolidation.

There are a total of 7 blades in our VMware environment, but only 5 of those are dedicated to our main HA/DRS cluster. That gives us ~240 gigs of RAM for the main cluster. Currently, I am seeing a VM consolidation ratio of about 24 VMs (virtual machines) per B200-M1 blade. The limitation here is definitely the RAM. The CPU itself is less than 25% utilized per blade. Below is a screenshot of a B200-M1 host in our cluster.

Cisco UCS B200-M1 VM Ratio

Notes: The VMs running on the above host are

Operating system utilization

Unpacked B200-M1 Blades

Unpacked Blades B200/B250

B250-M1 Blade Pictures

B250-M1 Guide

B250-M1 Open Front

B250-M1 Open Rear

B250-M1 Internal Top

B250-M1 Top Internal 2

B250-M1 Internal Palo

B250-M1 And B200-M1 In Chassis

B250-M1 Bezel

B200-M1 Blade Pictures

B200-M1 Guide

B200-M1 Internal Air Flow

B200-M1 Top Full

B200-M1 Top CPU / RAM

B200-M1 Rear Palo

B200-M1 Top Internal w/vents

B200-M1 Bezel

Below shows the error in VMware Virtual Center.

Here is the profile “iSCSI” before modification. Note the max 32 and available count.

The only way to increase this value is through command line in the active Nexus 1000v.

nexus01# configure t
nexus01(config)# port-profile type vethernet iSCSI
nexus01(config-port-prof)# vmware max-ports 1024
nexus01(config-port-prof)# end

Once the above command is issued, Virtual Center Client should show a reconfiguration task for the distributed virtual switch

Once the tasks is completed, go back into Home -> Inventory -> Networking and select the port profile in the distributed switch.  The “total ports” should now be set to 1024.

Make sure to save your configuration!

nexus01# copy running-config startup-config

Notes:  This should be non-intrussive, but make sure to test it in your environment!

230 VAC, 30 Amp ATS with RS232 access

ATS18A-30 Specifications

The power input (2) on this unit (to the floor) are NEMA L6-30P (twist lock) plugs. We already had APC vertical PDUs in our racks and wanted to be able to integrate them with the ATS solution. This model contains NEMA L6-30R (2 female sockets) that allowed the APC metered PDUs to be plugged directly in.

Testing information
Due to limited resources for testing, only a 6 amp load was placed on the ATS switch. The following devices were used:

Initial login to the ATS system.

ATS-11 Series
(C) 2003 by BayTech

F3.04
Option(s) Installed:
True RMS Current
Internal Temperature
True RMS Voltage

Unit ID: ATS18A-30
Input Power Source:  1
     Average Power:  10 Watts 

True RMS Voltage 1:   209.1 Volts
True RMS Voltage 2:   208.3 Volts
  True RMS Current:     0.0 Amps
  Maximum Detected:     0.1 Amps

Internal Temperature:  29.5 C

Type "Help" for a list of commands

Below lists the commands that can be used while the serial mode is set to console on the ATS unit.

ATS-11>help

Status    --ATS-11 Status
Config    --Enter configuration mode
Source n  --Select the input power source, n=1 or 2
Current   --Display True RMS Current
Clear     --Reset the maximum detected current
Temp      --Read current temperature
Voltage   --Display True RMS Voltage
Logout    --Logoff
Logoff    --Logoff
Exit      --Logoff

Password  --Changes the current user password
Whoami    --Displays the current user name
Unitid    --Displays the unit ID
Help      --This Command

Type "Help" for a list of commands

Information before attaching the devices

ATS-11>status
Input Power Source:  1
     Average Power:       6 Watts
True RMS Voltage 1:   209.1 Volts
True RMS Voltage 2:   208.3 Volts
  True RMS Current:     0.1 Amps
  Maximum Detected:     0.1 Amps

Internal Temperature:  29.5 C

After attaching devices

ATS-11>
Unit ID: ATS18A-30

Input Power Source:  1
     Average Power:    1101 Watts 

True RMS Voltage 1:   208.7 Volts
True RMS Voltage 2:   208.8 Volts
  True RMS Current:     5.9 Amps
  Maximum Detected:     6.0 Amps

Internal Temperature:  30.0 C

Start of testing

ATS-11>current

  True RMS Current:     5.7 Amps
  Maximum Detected:     6.0 Amps

Below shows that we are currently running on power source 1

ATS-11>status

Input Power Source:  1
     Average Power:    1034 Watts
True RMS Voltage 1:   208.8 Volts
True RMS Voltage 2:   208.9 Volts
  True RMS Current:     5.7 Amps
  Maximum Detected:     6.0 Amps

Internal Temperature:  30.0 C

After failover test of source 1, below shows that the unit was failed over to the second power leg. This was completely transparent to ALL devices.

ATS-11>status

Input Power Source:  2
     Average Power:    1014 Watts 

True RMS Voltage 1:     0.4 Volts
True RMS Voltage 2:   208.2 Volts
  True RMS Current:     5.7 Amps
  Maximum Detected:     6.0 Amps

Internal Temperature:  29.5 C

Power source 1 was reconnected. The unit continued to run on source 2. For testing, source 2 was failed. As shown below, the unit automatically failed back over to source 1.

ATS-11>status

Unit ID: ATS18A-30

Input Power Source:    1
     Average Power:    1032 Watts 

True RMS Voltage 1:   208.8 Volts
True RMS Voltage 2:     0.6 Volts
  True RMS Current:     5.7 Amps
  Maximum Detected:     6.0 Amps

Internal Temperature:  30.0 C

Details one more time of current Amps.

ATS-11>current

  True RMS Current:     5.6 Amps
  Maximum Detected:     6.0 Amps

This unit has the ability to manually failover the power source via the console. Below confirms this ability. The input power source is currently 1.

ATS-11 Series

(C) 2003 by BayTech

F3.04
Option(s) Installed:

True RMS Current
Internal Temperature
True RMS Voltage

Unit ID: ATS18A-30

Input Power Source:    1
     Average Power:    1027 Watts 

True RMS Voltage 1:   208.7 Volts
True RMS Voltage 2:   208.9 Volts 

  True RMS Current:     5.6 Amps
  Maximum Detected:     6.0 Amps

Internal Temperature:  30.0 C

Using the “source” command, the unit is manually failed over to the other power source.

ATS-11>source 2

Unit ID: ATS18A-30

Input Power Source:  2
     Average Power:     956 Watts 

True RMS Voltage 1:   209.6 Volts
True RMS Voltage 2:   208.1 Volts 

  True RMS Current:     5.7 Amps
  Maximum Detected:     6.0 Amps

Internal Temperature:  30.0 C

Results:

All testing was completed without error. None of the devices reported any power problems or errors. The following were monitored:
- Each server via the RSA logging interface
- Cisco catalyst logs
- PDU interface and logs
- Visual inspection to check for booting indicators

The ATS was failed over from power source 1 to 2 without issue. Next source 1 was brought back online, and source 2 was failed. The switch failed back to source 1. Source 2 was brought back up and source 1 was failed again. This was performed approximately 5 times for verification. Also, manual failover utilizing the ATS serial console was performed without issue.

Notes: So this really works. I was somewhat sceptical initially, but now I am ready to put it into production. This is a great option for providing redundant power paths for servers or networking equipment that only has one power supply. I know that the single power supply units are still not truly redundantly powered, but at least this gives that single power fed unit a better chance. In this implementation, the power sources are redundant pathed under the floor at the colocation. This would work great also for a small office. Just feed one PDU to commercial building power and the other to a UPS unit.

Multi-bit screwdriver
These make it easy when working in a rack. Just change or flip the head and go from phillips to flat, large diameter to small. Most of these can also attach hex key and nut drivers. I would definitely recommend getting those attachments. I find myself having to use the hex keys on a lot of vendor supplied screws.

Multiple length and width screwdrivers
This one should be easy. You will need short and long screwdrivers. Sometimes absurdly long ones would be your savior. Definitely need a few that are thicker in the ends and even coated tips for strength. I have bent the ends of cheaper screwdrivers a ton of times on screws that are over-tightened on racks. I have also done the same by using to small of width screwdriver in a larger screw.

Needle nose pliers strait and curved
These are definitely a must! Screws fall all the time in places where you cant get your fingers. These work great for that purpose. I have also used these to bend metal and unscrew hex screws that I didn’t have heads for.

Scissors
One of the places not to skimp. Buy industrial scissors! My pairs are electrician scissors. They are extremely solid, made of stainless steal and have a ridged back on one side. The ridged back helps for straitening small cables. Thinking of the small cable pairs inside Ethernet cabling

Flashlight
Should be self explanatory. Go for small size and LED based. Makes it easier to move into position or sit on top of shelves or rails in a rack

Paperclips
Great for miscellaneous things. I have used them for pushing small power buttons, reset switches, and dip switches. They can also be used to hang cables out of the way temporarily.

Battery powered drill/screwdriver
This I can not do without. When racking or unracking one or more racks this is a need! Utilizing a power drill with screw heads will speed up any deployment and save a lot of energy from not having to twist ever screw 20 times. The drill portion is nice as well. I have had to drill a few screws that were stripped or stuck out of the rack. Of course this is a last case scenario.

Power
Multimeter
This is good for troubleshooting any power issue. Mainly used to find faults that aren’t showing on status indicators. Allows you to trace the circuit paths.
power adapters
There are multiple types of power connectors out there. Main ones to be encountered at a colocation are standard wall outlet style, clip in, and twist lock. Not to mention voltage. Depending on the customer, you can wind up with 120 or 208 AC. So make sure whatever you need to plugin can handle the voltage and you have a plug for it

Power tester
This comes in handy in a few ways. This shows if the outlet is live and has power, stability of the power, etc.

DC
When working in a DC environment, there are a few good things to keep on hand. Most telecommunications colocations outlaw zip or twist ties, so get a spool or wax string. Also, I keep fuses, fuse pullers, wire cutters, wire strippers, and termination ends.

Extension cord
This has saved me a few times. I do not always remember to bring my plug adapters or a device will not support 208 power. In this case, most colocations have 120 outlets throughout the perimeter walls. None have ever minded me using an extension cord to run a device or two while in the colocation. Also, I have also worked in a few DC colocations. AC equipment will not work on DC :)

Storage
Having portable storage is a definite need. The can solve the problem of transporting configuration files, ISOs, and pulling reports from servers, SANs, etc. I always keep a few flash drives handy since most Windows and Linux distribution recognize them. This is also good for smaller files. No need to plug in any power cables or USB cables. For all other needs, I have used external hard drives from carrying ISOs around, backup up data, and moving copies of big databases between locations. No need to kill your network connection to move a database 60 miles. External drives are cheap and easy to use. One note though, use encryption. Someone can steal these drives easily if you are not paying attention. Stay secure. Always have blank CD-RWs and DVD-RWs around. There have been a lot of times that I needed an install CD where I had the ISO saved but no media to write it to.

Rack
I always keep on hand a few cage nuts, cage screws in all sizes.

Network
As far as cabling goes, I try to keep the following in sub 1ft and up to 12 ft cables:
- strait through cables
– crossover cables
- patch cables
– fiber cabling
- Ratcheting Ethernet crimper
Used to attach ends onto network cabling.  Most will do RJ-11 (telephony) as well.  The  key here is ratcheting.  This means that you must depress the tool to a specific point before it will release.  Doing so decreases the chance of a poorly terminated cable head.
* digital toner
This is also a definite must! Older can fry out switch ports or network cards. Save yourself the trouble and get an digital toner. I use one from Fluke that cost me about 160$. When working in colocation network sprawl becomes an issue. Even with good change control, it is still worth going through doing a yearly audit of what cables go where. Hostnames change, servers are replaced or removed, cables get re-used. Tracing the cables by hand can be almost impossible when they are bundled and ran between racks. This is where a good toner comes in handy. They will send a pulse down the cable that can be traced with a hand held unit. So plug it in, go to the network rack(s) and wave it around until you find the cable. Saves a lot of time and most will be able to trace Ethernet, and RJ-11(telephone).
- Also keep a few spare serial, mini serial, and USB cables.

Misc
A few miscellaneous things I keep with me are:
- pen and pater
- velcro ties. Because any other are bad to use.
- permanent marker
- labeling machine
- static strap and static pad. Always use these when working on equipment. All it takes is one time and you can burn out anything from a server to a core router.
- digital camera for taking before and after pics.

Below is a list of what was reviewed above:

Tools
Multi-bit screwdriver
with nut driver
multiple length screwdrivers
needle nose pliers strait and curved
flashlight
paperclips for pushing micro power or reset switches
Battery powered drill/screwdriver

Power
multimeter
power adapters
power tester
extension cord

Storage
flash drive
external hard drive
Blank CD-RW and DVD-RW

Rack
cage nuts, cage screws

Network
min length
strait through cable
crossover cable
patch cable
long cables
serial cable
mini serial cable
fiber cable
digital network toner

Misc
pen and pater
velcro ties
permanent marker
labeling machine
static strap
static pad
digital camera

Below are links to examples of the type of products I keep in the toolkit.

http://www.flukenetworks.com/fnet/en-us/products/D-Snips/Overview.htm
http://www.flukenetworks.com/fnet/en-us/products/Cable+Splicing+Knife/Overview.htm
http://www.flukenetworks.com/fnet/en-us/products/IntelliTone+Toner+and+Probe/Features/

Notes:  I am sure that a few things were left off.  If you can think of any addittions to this please email me or leave a comment.  Thanks!

Shared storage
This of course is the lowest cost to the client.  Give them your needs and they will carve out space for the data on their EMCs.  Only requirement is that the servers connecting to them have fiber cards.  Downfalls are as follows:

• shared connectivity to colocation fiber switches
• shared EMC controller between clients
• pooled storage on bigger drives

This all affects the speed that each client recieves.  If a differnt customer is running high throughput, it will reduce the speed the controller can process as well as the read write speed of the shared drives.  The colocation keeps the data secure by using Cisco MDS fiber switches, encryption, and LUN masking.  This offering would be great for a startup company doing file sharing or VMWare servers needing storage.  It would definitely beat the initial price of buying their own fiber switches, EMC, and an employee/contractor to manage them.

Dedicated storage
Price jumps up high here, along with a few more service offerings.  If you already have Cisco fiber SAN switches, the colocation provider will manage them.  If not, they will allow connectivity to their switches using dedicated VSANs for your equipment, as well as encryption.  Dedicated spindles and also a dedicated controller will definitely speed up the read write operations on the unit.  Here, there is no need for LUN masking since it will only be your data on there.  Just with anything else, the costs increases with the number of spindles and or disk space needed.

To end, centralized storage offerings at the colocation is a great service to see coming out.  It will definitely help the smaller guys or startups get going.  Depending on your size, the cost of buying a dedicated system and fiber switches from EMC is a huge undertaking.  Especially with about a 5 year refresh being needed on the EMC.  Who knows, I am sure there are also some colocations providing ISCSI using Ethernet storage.

Note:  Replication of the data can be provided between EMCs within thier nationwide colocations.