Once we got over the fear of FCoE reliability, almost everything is now being ordered with 10gbE. The only reason of worry was during this implementation over a year ago, FCoE was not in the main stream yet. It was almost “bleeding edge” in the market.

Where are we a year later?

One of the big killer of ports in our environment is having to tie the Gigabit Ethernet switches into them. This is wasting 4 ports that could be 10gb pre switch. Hopefully that will be resolved with a future purchase of Nexus 7000 switches.  We also have a pair of fabric extenders connected into the 5010s.  This also takes up two 10gbE ports per 5010.

Below is a port listing from one of our Nexus 5010 switches. This is after we removed 3 ports (per switch) from physical servers that were tied into the 10gbE infrastructure. As you can see from below, we only have 5 ports per switch left (10 total if you count the redundant switch)! That is only 5 more devices at the most.

--------------------------------------------------------------------------------
Ethernet      VLAN   Type Mode   Status  Reason                   Speed     Port
Interface                                                                   Ch #
--------------------------------------------------------------------------------
Eth1/1        x     eth  trunk  up      none                       1000(D) 1
Eth1/2        x     eth  trunk  up      none                       1000(D) 1
Eth1/3        x     eth  trunk  up      none                       1000(D) 1
Eth1/4        x     eth  trunk  up      none                       1000(D) 1
Eth1/5        x     eth  trunk  up      none                        10G(D) --
Eth1/6        x     eth  trunk  up      none                        10G(D) --
Eth1/7        x     eth  access up      none                        10G(D) --
Eth1/8        x     eth  access up      none                        10G(D) --
Eth1/9        x     eth  trunk  down    Link not connected          10G(D) --
Eth1/10       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/11       x     eth  trunk  down    Link not connected          10G(D) --
Eth1/12       x     eth  access down    SFP not inserted            10G(D) --
Eth1/13       x     eth  access up      none                        10G(D) --
Eth1/14       x     eth  access down    SFP not inserted            10G(D) --
Eth1/15       x     eth  trunk  up      none                        10G(D) --
Eth1/16       x     eth  trunk  up      none                        10G(D) --
Eth1/17       x     eth  fabric up      none                        10G(D) --
Eth1/18       x     eth  fabric up      none                        10G(D) --
Eth1/19       x     eth  trunk  up      none                        10G(D) 2
Eth1/20       x     eth  trunk  up      none                        10G(D) 2

At the 6120′s end (UCS switches) we are fine. Currently have 14 ports on each switch available. That should give us the ability to wire in 7 more chassis with 2 – 10gbE uplinks per IOM. A total of 4 links / 40gb throughput per chassis. Our CIFS and NFS access is a lot faster running through the EMC NX4′s 10gbE.

So if you are looking into 10gbE for our network, it would help to have at least a rough draft of where you want your datacenter to be a few years from now. Pretty much all major storage companies are doing some sort of 10gbE connectivity. Using it makes administration a lot easier. FCoE simplifies things even more, including eliminating the need for separate fiber switches. To sum it up, I love 10gbE and would recommend it even if not planning on incorporating FCoE or UCS in your environment.

DD 670 Front

We received our new DataDomain 670 last week and it has definitely had a face lift from the 500 series.  First, it is a 2u versus a 3u unit.  Also, the 600 series no longer have the annoying third power supply.  The drives are also horizontal instead of vertical.  Even slots in the back are different.  I wish we had a multi-shelf unit one.  Below are a few pictures from the unpacking and racking.

Notes: Only complaint is the rails.  I really wish they rails on the server were pegged style – drop into extended rack rail types (IBM).  Instead, the rails on the servers are fixed and have to be lined up perfectly with the ones in the rack.  They then slide straight into the rack rails and lock.  I find it a lot hard with this kind of rails to rack heavy equipment alone.

DD01# filesys show compression /backup/db/xxx/xxx/xx6e_1_1
Total files: 1;  bytes/storage_used: 59.1
       Original Bytes:       16,252,249,684
  Globally Compressed:          954,839,915
   Locally Compressed:          272,055,847
            Meta-data:            3,009,088

bytes/storage_used: 59.1 =	Compression and Deduplication ratio
Globally Compressed Factor = 	Pre-Comp / (Size after de-dupe)
Locally Compressed Factor = 	(Size after de-dupe) / Post-Comp

From above you should be getting an idea of the deduplication, but lets follow DataDomain’s formula

Reduction % = ((Pre-Comp – Post-Comp) / Pre-Comp) * 100

Now put in values from our above example

16,252,249,684 – 272,055,847 / 16,252,249,684 * 100 = 98.326%

That gives us a 98.326% reduction!

Basically what I am trying to say is that if you are not doing some sort of deduplication + compression on your Oracle backups, you should look into it.

Note(s): The above information is from a system running Data Domain OS 4.8.1.0-175623.  Also, remember that deduplication savings grow as the same types of blocks are written.  This system has been in production for over a year.  We have about 1tb raw of Oracle data being written to the DataDomain which helps the dedupe ratio.

Below is an overview of the DataDomain 565 we have at our “Production” datacenter.

Production DataDomain 565
24hr max pre-compressed           =          1869.5
30 day max pre-compressed       =          4278.81
60 day max pre-compressed       =          4387.35
120 day max pre-compressed     =          4922.56
* above is in gigabyte(s)

So with that in mind, we prefer to hit an 8hr backup window.  We currently are not doing that now with our 565 but that is our goal.

Max of 4922.56 / 8hr window       =          ~615.32 gb/hr

615.32 gigabyte = 4 922.56 gigabit
10gbE = 36,000 gigabit an hour

So from above, it would seem that we could theoretically hit our window if using 10gbE

We are using ~4236.5 gig used after deduplication is done I doing so, would be pushing the throughput and maximum capacity limitations of the DD610.  The DD630 would allow ~6tb worth of expansion and we would meet an 8 hour backup window. With the above information, you can see we want to go with 10gbE connectivity.  I currently love our Cisco UCS and 10gb network and plan to hook in all critical infrastructure to it via 10gbE.

This was our main limitation here! * Only the DD670 offers 10gbE connectivity (fiber)

So would the DD630 work for us?  Yes it would, but who wants to trunk a bunch of gigE ports when you have a 10gb network already?  Unless the cost point goes up horrifically, the DD670 would be a better fit in our datacenter.

DataDomain 6xx Overview

Notes: We could expand our DD565 another shelf, but we are getting low on storage at the off-site replication locations for the DataDomains.  Win – win but buying a new unit and shipping the 565 to increase capacity off-site.  DD510/610s can not expand beyond their initial shelf.

On the source, I have already setup a directory tree for “/backup/europe_data”. All files destined for our Euroupe office will be placed here. On the DataDomain devices, replicated folders are added manually. By default none are replicated.
*********************
Be aware that if you are viewing this through a web browser that a lot of the command might be hidden due to formatting of the WordPress theme.  It is best viewed by an RSS reader.  If you would like the full text, email or leave a comment and I can send you a PDF version.
*********************

Below is the help output for “replication add” command.

user01@us-dd565# replication add  ?
Commands matching "replication add":
  replication add source  destination
                 Add a replication pair

I wish the syntax was easy as what was stated above, but below is the actual command used to setup replication on the source DataDomain

user01@us-dd565# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once the source is setup, the destination endpoint/DataDomain also must be configured. The command is identical to what was entered at the source DataDomain

user01@eu-dd01# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once replication is setup on both DataDomains, log back into the source and start the replication.

user01@us-dd565# replication initialize
Incorrect syntax "replication initialize", showing help:
  replication initialize
                 Initialize replication on the source
                 (configure both source and destination
                 first)

Below is the actual syntax to startup replication for the directory that we just configured.

user01@us-dd565# replication initialize dir://eu-dd01.testdomain.com/backup/europe_data
(00:04) Waiting for initialize to start...
(00:06) Initialize started.
Use 'replication watch dir://eu-dd01.testdomain.com/backup/europe_data' to monitor progress.

Since I am impatient and want to know for sure this is working, I use “replication watch”. This shows the current progress of the replication between DataDomains.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
    Use Control-C to stop monitoring.

(00:57) Replication initialize started...
(00:59) initializing 3/3:
(10:35)     : pre-comp:   0%    27 KB/s, network    26 KB/s, 0 of 6 files

When new data is written to the mount point, replication automatically starts. Below is what shows if no data is currently being replicated.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
**** Replication initialize or recover already completed.

“replication show config” is used to display all the configured replication points.

user01@us-dd565# replication show config
CTX   Source                                                    Destination                                                 Connection Host and Port                       Enabled
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------
11    dir://us-dd565.testdomain.com/backup/mysql-rep       	dir://us-t2-dd510.testdomain.com/backup/mysql-rep	us-t2-dd510.testdomain.com   (default)     yes
12    dir://us-dd565.testdomain.com/backup/europe_data        	dir://eu-dd510.testdomain.com/backup/europe_data	eu-dd510.testdomain.com   (default)     yes
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------

Sorry for all of the spacing being off. All the tabs did not come over when copying data from the console.

user01

The following example will show how to query a standard Linux based server running SNMP with version 1 queries allowed. I am passing the “ifPhysAddress” parameter to return only the interface(s) MAC addresses.
snmpwalk basic usage

USAGE: snmpwalk [OPTIONS] AGENT [OID]

Below is specifying version 1, community string “PUBLIC”, server to query “dbserv01.testdomain.com”, and information to return “ifPhysAddress”

root@testbox01:~# snmpwalk -v 1 -c PUBLIC dbserv01.testdomain.com ifPhysAddress
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.3 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.4 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.5 = STRING: 0:0:0:0:36:28

Above shows all the listed MACs that can be referenced by SNMP. As you can see, the same MAC “0:12:39:65:36:28″ is listed 3 times. This typically means that the network interfaces are bonded.

Below shows the actual interface information on the queried Linux server. As noted, the interfaces are bonded (bond0), and the MAC address (HWaddr) listed is the same as what was retrieved by snmpwalk.

[root@dbserv01 ~]# ifconfig
bond0     Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          inet addr:127.1.1.182  Bcast:127.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:1033814138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:833777834 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1051206111974 (979.0 GiB)  TX bytes:602216674869 (560.8 GiB)

eth0      Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1025047842 errors:0 dropped:0 overruns:0 frame:0
          TX packets:833777834 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1050502485293 (978.3 GiB)  TX bytes:602216674869 (560.8 GiB)
          Interrupt:193 Memory:e4000000-e4012100 

eth1      Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:8766296 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:703626681 (671.0 MiB)  TX bytes:0 (0.0 b)
          Interrupt:193 Memory:e2000000-e2012100

Here’s the main “gotcha”. Snmpwalk shows the MAC as “0:12:39:65:36:28″, but ifconfig on the local server shows “00:12:39:65:36:28″. For better comparison, both are listed again below:

0:12:39:65:36:28
00:12:39:65:36:28

As you will probably see, a “0″ is missing. This is not a true issue, but a design in SNMP. Most applications will auto correct/translate this. Of course, some do not.

Here are a few more examples

IF-MIB::ifPhysAddress.2 = STRING: 0:13:99:65:c:80
IF-MIB::ifPhysAddress.3 = STRING: 0:13:99:65:c:81

Each one of the above also has “0″s removed.

0:13:99:65:c:80
is really
00:13:99:65:0c:80

I think the point has been made. “0″s are removed from all fields (“:” separates the fields) if it is the first character. So “00″ is displayed as “0″, “0c” displayed as “c”, etc.

The best information that I have found on this is that each field represents a byte. Logically, 01 is the same as 1, 0d is the same as d, and so on. Also, remember that bonded interfaces will also list the same MAC on all interfaces.

This will be the same on most Linux/Unix based distributions. I was able to successfully retrive MAC information from Linux servers, DataDomain appliances(RedHat based), and NetApp NAS (BSD based).

Same results when running a check on a VMware ESX server

root@testbox01:~# snmpwalk -v 1 -c PUBLIC vmserv01.testdomain.com ifPhysAddress
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 0:1a:54:eb:12:96
IF-MIB::ifPhysAddress.3 = STRING: 0:1a:54:eb:12:94
IF-MIB::ifPhysAddress.4 = STRING: 0:13:26:84:7a:26
IF-MIB::ifPhysAddress.5 = STRING: 0:13:26:84:7a:27
IF-MIB::ifPhysAddress.6 = STRING: 0:13:26:84:7a:24

Notes: This is not an issue/design on all SNMP clients or daemons. Just don’t be freaked out if you stumble across this. Most SNMP clients/MIB browsers automatically add in the 0s that are left out.

DataDomain Web Alert

SSH to the DataDomain and add my Linux systems IP address

admin1@DataDomain01# nfs add /ddvar 172.16.100.6
Mount the DataDomain /ddvar export on the Linux system
root@tstbox02:~# mount DataDomain01:/ddvar /mnt/DataDomain01/

Move into the core sub-directory under the ddvar mount point

root@tstbox02:~# cd /mnt/DataDomain01/core/

Check the directory for files. As you can see below, the core directory contained 47 gig worth of data

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 47G
drwxrwsr-x  2 root group 4.0K 2009-01-14 22:43 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 2.2M 2008-04-10 14:23 cmdb.core.26956.1207851782.gz
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 121K 2008-04-08 07:01 ddfs.core.14547.1207652507.gz
-rw-------  1 root group 352K 2008-04-07 22:56 ddfs.core.15098.1207618361.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:30 ddfs.core.15100.1207747799.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:37 ddfs.core.15125.1207665449.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:38 ddfs.core.15505.1207665497.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:31 ddfs.core.15506.1207747848.gz
-rw-r--r--  1 root group 527M 2008-09-10 16:27 ddfs.core.1560.1221078099.gz
-rw-r--r--  1 root group 166M 2008-04-08 07:32 ddfs.core.15675.1207654304.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:39 ddfs.core.15795.1207665554.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:32 ddfs.core.15813.1207747905.gz
-rw-r--r--  1 root group 636M 2008-07-28 15:20 ddfs.core.15991.1217272516.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:40 ddfs.core.16055.1207665605.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:33 ddfs.core.16095.1207747957.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:32 ddfs.core.18047.1207654349.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:33 ddfs.core.18231.1207654404.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:34 ddfs.core.18591.1207654458.gz
-rw-r--r--  1 root group 548M 2008-09-20 02:56 ddfs.core.22290.1221893385.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22705.1207750958.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22875.1207751006.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:24 ddfs.core.23008.1207751055.gz
-rw-r--r--  1 root group 553M 2008-08-19 15:26 ddfs.core.23183.1219173638.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:25 ddfs.core.23269.1207751113.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:06 ddfs.core.28347.1207746385.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:07 ddfs.core.28507.1207746432.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:08 ddfs.core.28694.1207746490.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:09 ddfs.core.28989.1207746540.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:14 ddfs.core.29883.1207746837.gz
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Since I verified that we do not need the archived (.gz) files, they were removed

root@tstbox02:/mnt/DataDomain01/core# rm *.gz

By only retaining the newer files in this directory, the size is now down to 1.6 gigs.

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 1.6G
drwxrwsr-x  2 root group 4.0K 2009-04-15 14:29 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Once this was completed, the alert in this DataDomain cleared in the web interface.

Data Domain Web Alert Cleared

Notes: Always verify files that are being removed are not needed. If there are core files present, it generally means the system has hit an error. Verify with your DataDomain support before modifying the ddvar filesystem.

From the CLI
# alerts show current
Alert Time         Description
—————-   —————————————————–
Mon Dec  8 08:38   Encl 1 A power supply module has failed.
—————-   —————————————————–
There is 1 active alert

I needed to go over to that colocation anyway so I made sure remote hands didn’t mess anything up.  So this alert is definitely hung.  The new power supply had good status indicators, and the power cable was replaced.  Removing this alert has to be done through the CLI in SE mode.  To do this, I SSHed into the Data Domain and escalated my privileges as follows.

Retrieve the serial number
# system show serialno
Serial number: XXXXXXXXXXX

Escalate to SE privileges
# priv set SE
Enter system password:[enter serial number]

Once in SE mode, the prompt will change to something resembling the following
SE@hostname##

Now, the ‘reg’ command is extended to its full feature set
SE@hostname## reg
Commands matching “reg”:

reg removekey <key>           Remove a key from the configuration
reg set reg-default <key>     Set the specified key to the default value
reg set unset-defaults        Set unset defaults.  Useful for upgrades
reg set <key> = <value>       Set a configuration key to the given value
reg setraw <key> = <value>    Set an encoded reg string
reg show defaults             Show registry defaults
reg show obsolete             Show obsolete keys
reg show stats                Show registry stats
reg showraw [nokey]           Show encoded registry data
reg show [nokey] <key>        Show a value for a configuration key

The autosupport contains the registry key that needs to be removed
dynamic.ems.701.x.1000.x.9.1.5 = Encl 1 A power supply

Cut out the part of the key that is needed
dynamic.ems.701.x.1000.x.9.1.5

The following removes the hung alert from the registry
SE@hostname## reg removekey dynamic.ems.701.x.1000.x.1.5

Note: Unfortunately, the alert does not tell which of the 3 power supplies have failed.  This was not a huge problem, since the remote hands just eyed the power supply that had no status lights on.

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
No NTP servers configured; using multicast mode.

Add a new timeserver to the list
dd# ntp add timeserver pool.ntp.org
Remote Time Servers:
pool.ntp.org

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
#   Server
-   ————
1   pool.ntp.org
-   ————
Showing NTP servers configured manually.
I advise disabling and re-enabling the NTP service
dd# ntp disable
NTP disabled.
dd# ntp enable
NTP enabled.

Check the NTP status.  It may take a while for NTP to query and update this information
# ntp status
Status                                             Enabled
Current Clock Time                         Tue Oct 28 16:07:22.708 2008
Clock Last Synchronized                 Tue Oct 28 15:48:23.398 2008
Clock Last Synchronized With Time Server   208.53.158.34

Removing a timeserver from the list
dd# ntp del timeserver pool.ntp.org
Remote Time Servers:
(multicast)

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
No NTP servers configured; using multicast mode.

Note:  Multiple timeservers can be used in the NTP configuration.  This example only showed one.  To add another, just follow “Add a new timeserver to the list” part again. It may take a while for the NTP service

Check to see what host name mappings are in the Data Domain.  If replacing a unit and using its IP, the old servers information would have to be removed from here.  In this case, dd01 would cause local lookups to have problems, since that’s the one being replaced by the new unit.

newdd# net hosts show
Hostname Mappings:
x.x.x.190 -> dd04.x.com
x.x.x.191 -> dd01.x.com dd01
x.x.x.192 -> dd03.x.com dd03

I decided to just wipe the hosts file and remove all static mappings.
newdd# net hosts reset
Host mappings reset to empty.

Change the IP on a single interface
newdd# net config eth0 x.x.x.191 netmask x.x.x.0
Configuring interface…

Change the IP on a virtual interface if you are doing NIC teaming.  This example will be used in the ping below
newdd# net config veth0 x.x.x.191 netmask x.x.x.0
Configuring interface…

Test connectivity from a server on the network.  It took the DD 565 about 2 minutes to come up on the new IP.
server# ping x.x.x.191
PING x.x.x.191 (x.x.x.191) 56(84) bytes of data.
64 bytes from x.x.x.191: icmp_seq=20 ttl=61 time=217 ms
64 bytes from x.x.x.191: icmp_seq=21 ttl=61 time=1.52 ms
64 bytes from x.x.x.191: icmp_seq=22 ttl=61 time=1.55 ms
64 bytes from x.x.x.191: icmp_seq=23 ttl=61 time=1.52 ms
64 bytes from x.x.x.191: icmp_seq=24 ttl=61 time=1.73 ms

Time to change the hostname of the box
newdd# net set hostname dd01.x.com
The Hostname is: dd01.x.com
CIFS active directory authentication will stop working after hostname change.
Run ‘cifs set authentication’ command again.

As directed, cifs set authentication command had to be ran to to join the new Data Domain back to Active Directory.

Note:  A reboot of the Data Domain had to be done to get the cifs authentication back onto the Acitve Directory domain.  This could have been due to two different issues.  Either the hostname change caused the issue, or the NTP settings.  The system was 5 minutes off from the domain controller before the reboot forced the NTP service to query the timeserver.

Below is information pulled from the NetApp showing two volumes that are being deduplicated

NAS> df -sh
Filesystem                used      saved       %saved
/vol/users/             246GB       68GB          22%
/vol/vmware/           96GB       99GB          51%

Lets see the status of the service on the VMWare volume
NAS> sis status
Path                           State      Status     Progress
/vol/vmware                    Enabled    Active     35 GB Scanned

Current VMWare volume utilization
NAS> df -h vmware
Filesystem               total       used      avail capacity  Mounted on
/vol/vmware/             950GB      147GB      802GB      16%  /vol/vmware/

Here is the sis configuration and status of the VMWare data store
NAS> sis status -l
Path:                    /vol/vmware
State:                   Enabled
Status:                 Active
Progress:             38231884 KB Scanned
Type:                   Regular
Schedule:             sun-sat@0
Last Operation Begin:    Thu Oct 23 00:00:07 EDT 2008
Last Operation End:      Thu Oct 23 00:03:59 EDT 2008
Last Operation Size:     2405 MB
Last Operation Error:    -
As we can see, the last run took almost 4 hours

The first volume is a CIFS (Windows) user shares containing home directories and whatever data they so desire to store there.  Deduplication is low on here due to the majority of the data not being the same thing just stored many times.  The VMWare share has about 9 windows and Linux virtual machines running there.  This volume is getting a lot better compression due to most of the data being the same system files that are needed for the operating systems to run.  Deduplication is great for this cause, meaning that each of those files only need to be stored once.  We will be off loading more of the non critical and development virtual machines to NetApp ASIS (deduplicated) volumes once it is on fiber.  The compression will continue to increase as the number of VMs running within them increase.  From the looks of it, we should easily be able to get 80% or more deduplication rates!  In the long run using this feature will save a lot of money on the number of drive trays needed.

Note:  One downfall, ASIS is not real time deduplication.  It is ran on as schedule instead of as the data is coming in, unlike a Data Domain.

Just goes to show that you shouldnt believe all of the vendors putting down the competitors products or capabilities.  I do not think that the ASIS is the answer to all problems, but it can definitely help smaller companies.  It is also a lot easier than adding on an appliance like EMC.

10/27/2008

Getting better:

NAS> df -sh /vol/vmware/
Filesystem                used      saved       %saved
/vol/vmware/             215GB      306GB          59%

11/15/2008

NAS> df -sh /vol/testVol

Problems

• We are maxed out in capacity (number of disks) on the DD510
• Currently using ~70% of the available storage
• Can not meet a backup window of 8hrs
• We are maxing out the processor during the backup window

Backup window calculations
Using the last 24 hour pre-compression write rate, we are pushing 2810.8 gigs of data to the Data Domain.  Per DD benchmarking and documentation, the max throughput for the DD510 is 290GB an hour.  Using an 8 hour window, the calculations would be as follows:

2810.8/8 = 351.35 GB /hr

This shows that we are already trying to push more data than the DD 510 can handle in the time window we are trying to achieve.  Below is from Data Domains documentation showing a comparison between their models

Future expansion
Plans are in the works for the IT department to implement NetBackup for Exchange and also move their active directory backups to the Data Domain.  Forecasting of the space required for exchange backups would need to be ~1000 gigs.  Currently, the Exchange data store is around 350 gigs.  Added into the current setup that would be

3160.8/8 = 395.1 GB /hr

That would put us over the limit of the DD530 appliance and up to the DD565 in just throughput alone.  The DD565 would give us the ability to scale up to 3 shelves of disks, whereas the DD510 and 530 can not expand beyond one enclosure (shelves).

Conclusion
Best case for achieving a backup window of ~8 hours or below would be to purchase a DD565 for use at the primary site.  This would allow for over double the hourly throughput and up to 6 times as much raw storage capacity.  The costs and compression capabilities of using a Data Domain system in comparison of an EMC or NetApp solution  seems to be the most financially reasonable route.  I do hear that EMC Avamar is coming down in price!  Also, NetApp free deduplication license for the NetApp does reasonable compression as well.  Although with NetApp, that means owning one first and that would cost a lot more than the Data Domain.

Since this initial writing, there is a new Data Domain 565 in the process of being deployed.  Once this is up and running, I will update the information here with the backup window results.

Note:  This is destination based agentless deduplication, not source based.  All the data is pushed from the servers, through the network, and to the datadomain.  The data is then deduped on the Data Domain.  EMC Avamar is agent based, allowing the server to only push the changed blocks or data to the Avamar deduplicaiton appliance.  This will definitely take a load off of a network.  Only problem with that is you have to install agents on all servers, as well as update that software.  You better have at least a true gigabit network when pushing upwards of 5 terrabytes a night to the data domain.  All in all, I am very happy with data domain and recommend them.
Update:  Per a conversation with a Data Domain sales team today, they show in the labs that restore speed (read from the Data Domain) is ~85% of the write speed to the device.

So, if a backup takes 1 hour, retrieval of that data should take approximately 1 hour and 11 minutes (70.588 minutes).  Now, this does not take into account the overhead of the application doing the restore.  Hope I did my math correctly.