Colocation to Virtualization » RamSan

RamSan 500 Pictures / Failure

Kevin Goodman — Mon, 07 Mar 2011 13:00:00 +0000

We had two DRAM cache modules fail in our RamSan 500 recently. The actual error was:

Uncorrected ECC event detected on boards 4,6

The DRAM cache fronts the Flash storage drives in the unit. Basically, loosing these two boards took down the unit completely. To get the unit back up, boards 4 and 6 needed to be removed. RamSan actually had me remove 4 of the boards instead of just the two that were bad. Guessing it has something to do with not being able to have slots not filled between active modules.
Doing so took the DRAM cache from 64 gigs down to 32. All data was lost on the FLASH RAID array, making us have to restore the Oracle database from backup. After the cards were removed, a low level format of the RAID array was done, the unit was operational again. Below are pictures from within the RamSan 500 for all the storage people out there.

Best part of this is we had racked two RamSan 630′s that were being configured as a mirrored pair. The plan was to implement them in two weeks. When the 500 failed, they were implemented that night.

Filed under: Datacenter, Hardware, RamSan, SAN (Storage Area Network)

TMS RamSan 630 Firmware Upgrade

Kevin Goodman — Mon, 17 Jan 2011 18:28:04 +0000

So I have completed the first two of our RamSan firmware updates. Turns out (surprisingly) that the process is very easy! Instead of fluffing this post up a lot with text, I am just posting the pictures in order as they occurred during the upgrade. If you have any questions, just comment or email me.

Filed under: RamSan, SAN (Storage Area Network)

RamSan 630 (SSD) Racking And Flash Module Installation

Kevin Goodman — Tue, 02 Nov 2010 12:49:08 +0000

We are continuing to roll out Texas Memory RamSan SSD devices to support our Oracle environments. This time we went with a RamSan 630 for our disaster recover location to host our primary Oracle databases at this site.

Some specs on the system:

Capacity:	Up to 10 TB
Storage Medium:	SLC Flash
Maximum Bandwidth:	10 GB/s
Read IOPS:	1 M IOPS
Read Latency:	250 microseconds
Write IOPS:	1 M IOPS
Write Latency:	80 microseconds
Expansion Slots:	5 slots
Fibre Ports per Card:	2 ports/card
Fibre Speed per Port:	4 Gb/s
Power Consumption:	500 W
Dimensions:	5.25″ (3U) x 20″
Form Factor:	3U
Weight:	55 lbs

RamSan Box

Turns out that the hot-spare flash module was shipped separately so I had to install that myself. Pictures from the racking and module installation is below.

Filed under: Datacenter, Hardware, RamSan, SAN (Storage Area Network)

EMC CX500 Overview And End of Life’d – In Pictures

Kevin Goodman — Thu, 23 Sep 2010 17:53:54 +0000

Continuing on in my not having much time to blog, here is another “in pictures” post. This time we go from new equipment being installed to the EOL’ing (end of life) of an older EMC CX-500.

Just some technical specs on this model:

- 7 DAEs (drive trays)
- Total of 105 drives / spindles
- 2 gigabit fiber connectivity

Drive Count	Drive Size (GB)	Total Gb
16	300	4800
30	146	5380
45	73	3285
14	36	504
	Total Raw TB	12.969

So approximately 13 terabytes is still a decent amount. Taken into account that we ram mainly RAID 10 on this array, that usable space dwindled dramatically. Of course, this was built at a time where the industry was still throwing disks “spindles” at Oracle databases. Back then there was no “enterprise” solid state in the mainstream market.

Having severed it’s purpose as first a “production” services storage array, it was replaced by a CX3-40. At that time, the CX500 was moved to a disaster recovery site and used there for it’s primary storage. Within about a year the CX3-40 was replaced at production site with a CX4-240. Since then, the CX3-40 was moved to the disaster recover site as primary storage. The CX500 had its data migrated off to either the RamSan 630 there or the EMC CX3-40. We have now almost outgrown our CX4-240 and soon it’s fate might be sealed as well.

Hope you liked the above history lesson and now comes the pictures. Also please note that these DAEs are full length DAEs, not the smaller shelves like the CX3+ series. Hence the reason why Bill (co-worker) came to help!

Notes: Check out the popped (flat) tire on the cart. The Trailblazer was definitely riding low on the way home. Didn’t want the rack rails messing up my trunk, so those were all put in the back seat. Surprisingly, I was able to fit the whole SAN into the back compartment without folding the rear seats down.

Filed under: Datacenter, EMC, Hardware, RamSan, SAN (Storage Area Network)

EMC CX3-40 Drive Replacement In Pictures

Kevin Goodman — Wed, 15 Sep 2010 13:00:32 +0000

So I have been really busy lately and haven’t had time to post almost anything. We had a lot of 73gig drives in our older CX3-40. They were there for spindle count (15krpm x 60 disks). We needed the throughput not space back when this system was built a few years ago.

Since then I have installed a RamSan for the Oracle environment at this datacenter and have no need for a lot of tiny drives. So we replaced them with 600gig 10kRPM drives to be used in our VMware environment and RecoverPoint systems. How about some EMC drive replacement pics until I can get some more content available?

Funny thing is that I forgot to take pictures of the drives once installed. Might add them next trip/

Filed under: Colocation, Datacenter, EMC, RamSan, SAN (Storage Area Network), VMWare

Cisco UCS and Nexus 5000 DataCenter – Our Implementation

Kevin Goodman — Wed, 07 Jul 2010 19:07:07 +0000

Our Cisco 10gbE implementation consists of 2 Chassis fully populated UCS with a mix of full and half width blades. The servers are all boot from SAN with no local disks. “PALO” cards are used in all servers which allow us to do FCOE. 7 of the blades are running VMware ESX 4 (vSphere) and the rest are a mix of RedHat Linux and Windows 2008.

Storage:

All servers boot from SAN over FCOE

Primary storage is via Fibre Channel to an EMC CX4-240

A few servers are also connected via Fibre Channel to RamSan SSD SANs

The CX4-240 is also connected to the UCS environment via quad 10gbE links over fiberNetwork:

Core 10gbE switching is provided via Cisco Nexus 5010s

1 Gigabit Ethernet to legacy servers are provided by dual 48 port C2148 Fabric Extenders

Each C2148 is connected to a Nexus 5010 by dual 10gbE uplinks

Each 5010 is connected via quad 4gig fiber to the CX4-240 to provide FCoE to physical servers outside of the UCS environment

Each 5010 is also connected to the CX4-240 by dual 10gbE fiber ports to provide ISCSI to all of the network

Each UCS switch (6120s) are uplinked to the 5010s by dual 10gbE interfaces

Each UCS chassis is uplinked to the 6120s by 8 qty 10gbE ports (4 connections per IO module)

The fiber switches are a redundant pair of Cisco MDS 9124s10gb ISCSI was added after the initial build. We are running Oracle with ASM in our VMware environment. This allowed for easier management of storage inside of oracle. Using ASM allowed dynamic growth of the database without having to do a lot of resizing of ext3 filesystem when expanding LUNs. We wound up with a ton of VMware RDMs (Raw Device Mappings). To remedy this issue, we have gone with 10gbE ISCSI to the EMC.

Below is a diagram of how we are currently setup.

10gbE DataCenter

Notes: Sounds like Cisco will be able to do multi-hop FCoE soon. This should remove the need of having the CX4 connected via fiber to both the 5010 and the 6120. I definitely would love feedback on this. How is everyone else implementing 10gb? Anyone considering 10gb ISCSI?

Filed under: Colocation, EMC, Hardware, Networking, RamSan, SAN (Storage Area Network)

Storage, Network: What I have Been Doing (EMC,Cisco UCS)

Kevin Goodman — Tue, 02 Mar 2010 19:41:19 +0000

This is more of an informational update of things that I have going on right now. I normally do not publish day-to-day type of things, but here we go.

Storage

We have received both replacement drives for our EMC Clariion CX340 and four new DAEs (disk shelves) for our CX4-240

Clariion CX3

The CX3 was originally bought to speed up our Oracle implementation. This was accomplished by ordering lots of fast disks (spindles) that were small. We wound up with 6 DAEs filled with 73gig 15kRPM disks, totalling 90 dedicated drives for Oracle.

This was great for the original purpose but the unit was replaced a year after initial deployment with a RamSan and EMC CX4. Having been decommissioned from production and moved to the tier 2 site, the need for space over IOPS (speed) drastically increased. Trying to keep performance and space requirements in balance, the decision has been made to go with a smaller RamSan for Oracle at the tier 2 site. This gives us the ability to replace the small 73 gigabyte drives with bigger 600 gigabyte 10kRPM disks. Replacing those disk with the same quantity of 600 gig ones will give us ~8 times as much space.

The RamSan will almost double the IOPS capacity that the CX3 is able to achieve and speed up our data warehouse even more.

Clariion CX4

So last year we went with implementing EMC Recoverpoint SAN based replication. This has been great and served us well! The only downfall was that we were doing “CRR” remote replication only. In a case of a failure and data needed to be recovered, there were no local copies. The snapshot or “point in time” would have to be loaded from the tier 2 site and transferred across the datacenter interconnect. The interconnect being 150 megs slowed this process down.

As planned from the beginning, we are implementing “CLR” local replication as well. This means that there will be a local copy of snapshots saved locally to the CX4. This will give us almost immediate access to the snapshots without being slowed down by the interconnect. The problem with RecoverPoint is that if you have a terrabyte LUN that you want to connect, you must have an extra terrabyte worth of space to save it. This is not really a problem, but a major consideration on the number of drives to buy and the overall expense of the implementation.

In our case, a terrabyte oracle LUN will wind up costing 3 terrabytes in the end. 1 terrabyte for the original data, 1 terrabyte for the local copy (CLR), and 1 terrabyte at the remote tier 2 site (CX3).

Our virtualization effort is continuing and this is another huge factor on the storage expansion. Currently we have 16 LUNs dedicated to the VMware environment. Each is 320 gigs in size. Moving forward, we will be doing a virtual desktop deployment as well. The leftover ~400 gigs will not cut it. So in the new 60 disks, 15 or more will have to be dedicated to VMware.

Cisco UCS

We have begun our UCS voyage. As of last weekend, we did a “rip and replace of our network”. This included rewiring the main network rack and configuring a new network core. Also, the Cisco Nexus 5010, 10 gigabit Ethernet switches are in. Uplinked to them are two 48 port gigabit fabric extenders.

The VMware environment is now connected via dual 10gigE links per server through this infrastructure. Reducing the cable count from 6 to 2 per server. So far verything is stable! A purchase order has been sent out and we should hopefully have two Cisco UCS Blade chassis and switching infrastructure show up within about 30 days.

Filed under: EMC, Networking, RamSan, SAN (Storage Area Network), VMWare

SAN / Storage: Texas Memory RamSan 500

Kevin Goodman — Thu, 04 Jun 2009 04:18:52 +0000

After spending a year using the a RamSan 400 which is a 128 gigabyte solid state DRAM system, I wondered how it could get better. These things are pretty expensive and the only drawback I found to the 400 is the limited storage. There’s not much that you can do with 128 gigs of storage! Granted it served the initial function perfectly. This was used to house our main production database.

About 4 months ago, we were looking to add another RamSan into our environment after deciding EMCs flash DAEs were not for us (at this time). I was initially a little weary of flash based storage, but had faith in RamSan. After reviewing the product literature more and getting some hands on with the unit, those worries are gone. Our RamSan 500 was purchased with 2 terabytes of Flash storage and the DRAM write cache was expanded to 64 gigabytes.

With the write cache expansion, this new units cache was literally 1/2 the size of the current RamSan 400 DRAM capacity. With this in place and the high IO that the flash disks can maintain, there has not been any noticeable speed differences between the two units. The 500 also has an internal battery for power backup. If power is lost, the battery will keep the unit going long enough to copy the cache’s un-written data to disk.

There are two main features that I love. First is tha ability to lock a LUN on the 500 series into the cache, so even-though it is mainly a flash storage system, extreme IO can be accomplished. By locking the lun in cache, the data will reside in RAM and never go to disk unless power is lost. The second is the ability to centrally manage other RamSans on the network (auto discovered) with a specific license key. Almost forgot to mention, the unit has quad 4 gigabit fiber connectivity.

Notes: If you have any questions, please ask! Again, these units are amazing.

Posted in Hardware, RamSan, SAN (Storage Area Network)