Colocation to Virtualization » deduplication

Storage, Networking, EMC: DataDomain Replication

Kevin Goodman — Wed, 26 May 2010 14:49:41 +0000

We have a total of 3 DataDomains currently in production. One at our “Tier 1″ site, “Tier 2″(DR) and one in Europe. All DataDomain appliances have the ability to replicate data among themselves. This will be a general overview of how to setup replication between two DataDomains

On the source, I have already setup a directory tree for “/backup/europe_data”. All files destined for our Euroupe office will be placed here. On the DataDomain devices, replicated folders are added manually. By default none are replicated.
*********************
Be aware that if you are viewing this through a web browser that a lot of the command might be hidden due to formatting of the WordPress theme. It is best viewed by an RSS reader. If you would like the full text, email or leave a comment and I can send you a PDF version.
*********************

Below is the help output for “replication add” command.

user01@us-dd565# replication add  ?
Commands matching "replication add":
  replication add source  destination
                 Add a replication pair

I wish the syntax was easy as what was stated above, but below is the actual command used to setup replication on the source DataDomain

user01@us-dd565# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once the source is setup, the destination endpoint/DataDomain also must be configured. The command is identical to what was entered at the source DataDomain

user01@eu-dd01# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once replication is setup on both DataDomains, log back into the source and start the replication.

user01@us-dd565# replication initialize
Incorrect syntax "replication initialize", showing help:
  replication initialize
                 Initialize replication on the source
                 (configure both source and destination
                 first)

Below is the actual syntax to startup replication for the directory that we just configured.

user01@us-dd565# replication initialize dir://eu-dd01.testdomain.com/backup/europe_data
(00:04) Waiting for initialize to start...
(00:06) Initialize started.
Use 'replication watch dir://eu-dd01.testdomain.com/backup/europe_data' to monitor progress.

Since I am impatient and want to know for sure this is working, I use “replication watch”. This shows the current progress of the replication between DataDomains.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
    Use Control-C to stop monitoring.

(00:57) Replication initialize started...
(00:59) initializing 3/3:
(10:35)     : pre-comp:   0%    27 KB/s, network    26 KB/s, 0 of 6 files

When new data is written to the mount point, replication automatically starts. Below is what shows if no data is currently being replicated.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
**** Replication initialize or recover already completed.

“replication show config” is used to display all the configured replication points.

user01@us-dd565# replication show config
CTX   Source                                                    Destination                                                 Connection Host and Port                       Enabled
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------
11    dir://us-dd565.testdomain.com/backup/mysql-rep       	dir://us-t2-dd510.testdomain.com/backup/mysql-rep	us-t2-dd510.testdomain.com   (default)     yes
12    dir://us-dd565.testdomain.com/backup/europe_data        	dir://eu-dd510.testdomain.com/backup/europe_data	eu-dd510.testdomain.com   (default)     yes
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------

Sorry for all of the spacing being off. All the tabs did not come over when copying data from the console.

user01

Filed under: Data Domain, deduplication, NAS, Networking

Linux/Networking/VMware: Snmpwalk MAC Address Missing 0s

Kevin Goodman — Tue, 05 May 2009 09:26:59 +0000

A few days ago I was looking for a quick way to retrieve MAC address information from servers to map ports in a switch cluster. I figured I would share a few gotchas that might not be well known to people that do not have a background working with SNMP. Snmpwalk is a command line open-source utility for Linux. I am sure there is a port for Windows, but for the scope of this writing I am sticking to the Linux version.

The following example will show how to query a standard Linux based server running SNMP with version 1 queries allowed. I am passing the “ifPhysAddress” parameter to return only the interface(s) MAC addresses.
snmpwalk basic usage

USAGE: snmpwalk [OPTIONS] AGENT [OID]

Below is specifying version 1, community string “PUBLIC”, server to query “dbserv01.testdomain.com”, and information to return “ifPhysAddress”

root@testbox01:~# snmpwalk -v 1 -c PUBLIC dbserv01.testdomain.com ifPhysAddress
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.3 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.4 = STRING: 0:12:39:65:36:28
IF-MIB::ifPhysAddress.5 = STRING: 0:0:0:0:36:28

Above shows all the listed MACs that can be referenced by SNMP. As you can see, the same MAC “0:12:39:65:36:28″ is listed 3 times. This typically means that the network interfaces are bonded.

Below shows the actual interface information on the queried Linux server. As noted, the interfaces are bonded (bond0), and the MAC address (HWaddr) listed is the same as what was retrieved by snmpwalk.

[root@dbserv01 ~]# ifconfig
bond0     Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          inet addr:127.1.1.182  Bcast:127.1.1.255  Mask:255.255.255.0
          inet6 addr: fe80::200:ff:fe00:0/64 Scope:Link
          UP BROADCAST RUNNING MASTER MULTICAST  MTU:1500  Metric:1
          RX packets:1033814138 errors:0 dropped:0 overruns:0 frame:0
          TX packets:833777834 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:0
          RX bytes:1051206111974 (979.0 GiB)  TX bytes:602216674869 (560.8 GiB)

eth0      Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:1025047842 errors:0 dropped:0 overruns:0 frame:0
          TX packets:833777834 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:1050502485293 (978.3 GiB)  TX bytes:602216674869 (560.8 GiB)
          Interrupt:193 Memory:e4000000-e4012100 

eth1      Link encap:Ethernet  HWaddr 00:12:39:65:36:28
          UP BROADCAST RUNNING SLAVE MULTICAST  MTU:1500  Metric:1
          RX packets:8766296 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000
          RX bytes:703626681 (671.0 MiB)  TX bytes:0 (0.0 b)
          Interrupt:193 Memory:e2000000-e2012100

Here’s the main “gotcha”. Snmpwalk shows the MAC as “0:12:39:65:36:28″, but ifconfig on the local server shows “00:12:39:65:36:28″. For better comparison, both are listed again below:

0:12:39:65:36:28
00:12:39:65:36:28

As you will probably see, a “0″ is missing. This is not a true issue, but a design in SNMP. Most applications will auto correct/translate this. Of course, some do not.

Here are a few more examples

IF-MIB::ifPhysAddress.2 = STRING: 0:13:99:65:c:80
IF-MIB::ifPhysAddress.3 = STRING: 0:13:99:65:c:81

Each one of the above also has “0″s removed.

0:13:99:65:c:80
is really
00:13:99:65:0c:80

I think the point has been made. “0″s are removed from all fields (“:” separates the fields) if it is the first character. So “00″ is displayed as “0″, “0c” displayed as “c”, etc.

The best information that I have found on this is that each field represents a byte. Logically, 01 is the same as 1, 0d is the same as d, and so on. Also, remember that bonded interfaces will also list the same MAC on all interfaces.

This will be the same on most Linux/Unix based distributions. I was able to successfully retrive MAC information from Linux servers, DataDomain appliances(RedHat based), and NetApp NAS (BSD based).

Same results when running a check on a VMware ESX server

root@testbox01:~# snmpwalk -v 1 -c PUBLIC vmserv01.testdomain.com ifPhysAddress
IF-MIB::ifPhysAddress.1 = STRING:
IF-MIB::ifPhysAddress.2 = STRING: 0:1a:54:eb:12:96
IF-MIB::ifPhysAddress.3 = STRING: 0:1a:54:eb:12:94
IF-MIB::ifPhysAddress.4 = STRING: 0:13:26:84:7a:26
IF-MIB::ifPhysAddress.5 = STRING: 0:13:26:84:7a:27
IF-MIB::ifPhysAddress.6 = STRING: 0:13:26:84:7a:24

Notes: This is not an issue/design on all SNMP clients or daemons. Just don’t be freaked out if you stumble across this. Most SNMP clients/MIB browsers automatically add in the 0s that are left out.

Posted in Data Domain, Linux, Networking, VMWare

DataDomain/NAS/Filesystems/Linux: Remove Files From A DataDomain’s /ddvar

Kevin Goodman — Thu, 16 Apr 2009 15:19:59 +0000

A few days ago we received alerts for the /ddvar fileystem from one of our Data Domain units. Normally, I would not manually remove any files from this filesystem due to it being mainly used by the underlining OS and not for NAS storage. In this case, the problem was the “core” subdirectory. I tried to remove the files from a Windows machine, but due to permission issues, I was unable to do so. Even through Windows (CIFS) to the DataDomain, I was able to modify the permissions of the file(s) and still would receive a permission denied issue when trying to remove. The quick solution here was to add my Linux box to the NFS access list, mount “/ddvar”, and remove the files as the root user. Below details the process that worked for me.

DataDomain Web Alert

SSH to the DataDomain and add my Linux systems IP address

admin1@DataDomain01# nfs add /ddvar 172.16.100.6
Mount the DataDomain /ddvar export on the Linux system
root@tstbox02:~# mount DataDomain01:/ddvar /mnt/DataDomain01/

Move into the core sub-directory under the ddvar mount point

root@tstbox02:~# cd /mnt/DataDomain01/core/

Check the directory for files. As you can see below, the core directory contained 47 gig worth of data

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 47G
drwxrwsr-x  2 root group 4.0K 2009-01-14 22:43 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 2.2M 2008-04-10 14:23 cmdb.core.26956.1207851782.gz
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 121K 2008-04-08 07:01 ddfs.core.14547.1207652507.gz
-rw-------  1 root group 352K 2008-04-07 22:56 ddfs.core.15098.1207618361.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:30 ddfs.core.15100.1207747799.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:37 ddfs.core.15125.1207665449.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:38 ddfs.core.15505.1207665497.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:31 ddfs.core.15506.1207747848.gz
-rw-r--r--  1 root group 527M 2008-09-10 16:27 ddfs.core.1560.1221078099.gz
-rw-r--r--  1 root group 166M 2008-04-08 07:32 ddfs.core.15675.1207654304.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:39 ddfs.core.15795.1207665554.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:32 ddfs.core.15813.1207747905.gz
-rw-r--r--  1 root group 636M 2008-07-28 15:20 ddfs.core.15991.1217272516.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:40 ddfs.core.16055.1207665605.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:33 ddfs.core.16095.1207747957.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:32 ddfs.core.18047.1207654349.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:33 ddfs.core.18231.1207654404.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:34 ddfs.core.18591.1207654458.gz
-rw-r--r--  1 root group 548M 2008-09-20 02:56 ddfs.core.22290.1221893385.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22705.1207750958.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22875.1207751006.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:24 ddfs.core.23008.1207751055.gz
-rw-r--r--  1 root group 553M 2008-08-19 15:26 ddfs.core.23183.1219173638.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:25 ddfs.core.23269.1207751113.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:06 ddfs.core.28347.1207746385.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:07 ddfs.core.28507.1207746432.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:08 ddfs.core.28694.1207746490.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:09 ddfs.core.28989.1207746540.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:14 ddfs.core.29883.1207746837.gz
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Since I verified that we do not need the archived (.gz) files, they were removed

root@tstbox02:/mnt/DataDomain01/core# rm *.gz

By only retaining the newer files in this directory, the size is now down to 1.6 gigs.

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 1.6G
drwxrwsr-x  2 root group 4.0K 2009-04-15 14:29 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Once this was completed, the alert in this DataDomain cleared in the web interface.

Data Domain Web Alert Cleared

Notes: Always verify files that are being removed are not needed. If there are core files present, it generally means the system has hit an error. Verify with your DataDomain support before modifying the ddvar filesystem.

Posted in Data Domain, Filesystems, Linux, NAS

NAS / Data Domain: Resetting Hung Alert

Kevin Goodman — Mon, 29 Dec 2008 19:40:23 +0000

We had a power supply go out on one of our Data Domain 510 appliances about 2 weeks ago. They shipped out a replacement power supply and it was put in place. After the power supply was replaced, the alarm was still showing via the web page interface and CLI (Command Line Interface). Since this is a standby backup unit, I did do also tried a reboot. This also did not clear the alert. Remember, I am in now way responsible for anyone ruining their system or voiding their warranty.

Snippet from the autosupport(email) log
Power Supply
Status
——–
DEGRADED
——–

From the CLI
# alerts show current
Alert Time         Description
—————-   —————————————————–
Mon Dec 8 08:38   Encl 1 A power supply module has failed.
—————-   —————————————————–
There is 1 active alert

I needed to go over to that colocation anyway so I made sure remote hands didn’t mess anything up. So this alert is definitely hung. The new power supply had good status indicators, and the power cable was replaced. Removing this alert has to be done through the CLI in SE mode. To do this, I SSHed into the Data Domain and escalated my privileges as follows.

Retrieve the serial number
# system show serialno
Serial number: XXXXXXXXXXX

Escalate to SE privileges
# priv set SE
Enter system password:[enter serial number]

Once in SE mode, the prompt will change to something resembling the following
SE@hostname##

Now, the ‘reg’ command is extended to its full feature set
SE@hostname## reg
Commands matching “reg”:

reg removekey            Remove a key from the configuration
reg set reg-default      Set the specified key to the default value
reg set unset-defaults        Set unset defaults. Useful for upgrades
reg set =        Set a configuration key to the given value
reg setraw =     Set an encoded reg string
reg show defaults             Show registry defaults
reg show obsolete             Show obsolete keys
reg show stats                Show registry stats
reg showraw [nokey]           Show encoded registry data
reg show [nokey]         Show a value for a configuration key

The autosupport contains the registry key that needs to be removed
dynamic.ems.701.x.1000.x.9.1.5 = Encl 1 A power supply

Cut out the part of the key that is needed
dynamic.ems.701.x.1000.x.9.1.5

The following removes the hung alert from the registry
SE@hostname## reg removekey dynamic.ems.701.x.1000.x.1.5

Note: Unfortunately, the alert does not tell which of the 3 power supplies have failed. This was not a huge problem, since the remote hands just eyed the power supply that had no status lights on.

Posted in Data Domain, deduplication, Hardware, NAS

Data Domain NTP Configuration

Kevin Goodman — Tue, 28 Oct 2008 22:10:24 +0000

Below is information on how to setup a Data Domain appliance to work with the Network Time Protocol (NTP)

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
No NTP servers configured; using multicast mode.

Add a new timeserver to the list
dd# ntp add timeserver pool.ntp.org
Remote Time Servers:
pool.ntp.org

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
#   Server
-   ————
1   pool.ntp.org
-   ————
Showing NTP servers configured manually.
I advise disabling and re-enabling the NTP service
dd# ntp disable
NTP disabled.
dd# ntp enable
NTP enabled.

Check the NTP status. It may take a while for NTP to query and update this information
# ntp status
Status                                             Enabled
Current Clock Time                         Tue Oct 28 16:07:22.708 2008
Clock Last Synchronized                 Tue Oct 28 15:48:23.398 2008
Clock Last Synchronized With Time Server   208.53.158.34

Removing a timeserver from the list
dd# ntp del timeserver pool.ntp.org
Remote Time Servers:
(multicast)

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
No NTP servers configured; using multicast mode.

Note: Multiple timeservers can be used in the NTP configuration. This example only showed one. To add another, just follow “Add a new timeserver to the list” part again. It may take a while for the NTP service

Posted in Data Domain, NAS

Data Domain Hostname / IP Change

Kevin Goodman — Tue, 28 Oct 2008 19:46:36 +0000

The following is information on how to re-ip and change the hostname on a Data Domain appliance. This was done on both a Data Domain 510 and 565.

Check to see what host name mappings are in the Data Domain. If replacing a unit and using its IP, the old servers information would have to be removed from here. In this case, dd01 would cause local lookups to have problems, since that’s the one being replaced by the new unit.

newdd# net hosts show
Hostname Mappings:
x.x.x.190 -> dd04.x.com
x.x.x.191 -> dd01.x.com dd01
x.x.x.192 -> dd03.x.com dd03

I decided to just wipe the hosts file and remove all static mappings.
newdd# net hosts reset
Host mappings reset to empty.

Change the IP on a single interface
newdd# net config eth0 x.x.x.191 netmask x.x.x.0
Configuring interface…

Change the IP on a virtual interface if you are doing NIC teaming. This example will be used in the ping below
newdd# net config veth0 x.x.x.191 netmask x.x.x.0
Configuring interface…

Test connectivity from a server on the network. It took the DD 565 about 2 minutes to come up on the new IP.
server# ping x.x.x.191
PING x.x.x.191 (x.x.x.191) 56(84) bytes of data.
64 bytes from x.x.x.191: icmp_seq=20 ttl=61 time=217 ms
64 bytes from x.x.x.191: icmp_seq=21 ttl=61 time=1.52 ms
64 bytes from x.x.x.191: icmp_seq=22 ttl=61 time=1.55 ms
64 bytes from x.x.x.191: icmp_seq=23 ttl=61 time=1.52 ms
64 bytes from x.x.x.191: icmp_seq=24 ttl=61 time=1.73 ms

Time to change the hostname of the box
newdd# net set hostname dd01.x.com
The Hostname is: dd01.x.com
CIFS active directory authentication will stop working after hostname change.
Run ‘cifs set authentication’ command again.

As directed, cifs set authentication command had to be ran to to join the new Data Domain back to Active Directory.

Note: A reboot of the Data Domain had to be done to get the cifs authentication back onto the Acitve Directory domain. This could have been due to two different issues. Either the hostname change caused the issue, or the NTP settings. The system was 5 minutes off from the domain controller before the reboot forced the NTP service to query the timeserver.

Posted in Data Domain, NAS

NetApp: Deduplication ASIS and VMWare

Kevin Goodman — Thu, 23 Oct 2008 23:59:38 +0000

Man, where did this come from? I used to rule out NetApp deduplication due to the costs of the licenses, but now it is free (if you have one of their NAS). Also, the max volume size that NetApp can dedupe is 1 terabyte for the model we are on. A coworker of mine enabled deduplication on a VMWare ISCSI volume recently. Turns out this option could be great for small to mid sized companies that can not afford to have both an EMC and NetApp. Also as a cost effective way to save money on storage by offloading lower priority servers or VMs to the ASIS volumes

Below is information pulled from the NetApp showing two volumes that are being deduplicated

NAS> df -sh
Filesystem                used      saved       %saved
/vol/users/             246GB       68GB          22%
/vol/vmware/           96GB       99GB          51%

Lets see the status of the service on the VMWare volume
NAS> sis status
Path State Status Progress
/vol/vmware Enabled Active 35 GB Scanned

Current VMWare volume utilization
NAS> df -h vmware
Filesystem total used avail capacity Mounted on
/vol/vmware/ 950GB 147GB 802GB 16% /vol/vmware/

Here is the sis configuration and status of the VMWare data store
NAS> sis status -l
Path:                    /vol/vmware
State:                   Enabled
Status:                 Active
Progress:             38231884 KB Scanned
Type:                   Regular
Schedule:             sun-sat@0
Last Operation Begin:    Thu Oct 23 00:00:07 EDT 2008
Last Operation End:      Thu Oct 23 00:03:59 EDT 2008
Last Operation Size:     2405 MB
Last Operation Error:    -
As we can see, the last run took almost 4 hours

The first volume is a CIFS (Windows) user shares containing home directories and whatever data they so desire to store there. Deduplication is low on here due to the majority of the data not being the same thing just stored many times. The VMWare share has about 9 windows and Linux virtual machines running there. This volume is getting a lot better compression due to most of the data being the same system files that are needed for the operating systems to run. Deduplication is great for this cause, meaning that each of those files only need to be stored once. We will be off loading more of the non critical and development virtual machines to NetApp ASIS (deduplicated) volumes once it is on fiber. The compression will continue to increase as the number of VMs running within them increase. From the looks of it, we should easily be able to get 80% or more deduplication rates! In the long run using this feature will save a lot of money on the number of drive trays needed.

Note: One downfall, ASIS is not real time deduplication. It is ran on as schedule instead of as the data is coming in, unlike a Data Domain.

Just goes to show that you shouldnt believe all of the vendors putting down the competitors products or capabilities. I do not think that the ASIS is the answer to all problems, but it can definitely help smaller companies. It is also a lot easier than adding on an appliance like EMC.

10/27/2008

Getting better:

NAS> df -sh /vol/vmware/
Filesystem used saved %saved
/vol/vmware/ 215GB 306GB 59%

11/15/2008

NAS> df -sh /vol/testVol

Filesystem	used	saved	%saved
/vol/testVol/	519GB	754GB	59%

Posted in deduplication, NAS, SAN (Storage Area Network), VMWare

Data Domain

Kevin Goodman — Thu, 23 Oct 2008 03:44:51 +0000

I have been managing Data Domain units for about a year now. Currently, there are 3 running on the network. Initially, some issues came up that made me a little “weary” of them. First was that we had a few occurrences of one unit disabling its file system. The biggest problem was when it hit a “less than 1 percent chance” error. The hash function used in the RAID software to isolate corruption had a problem slip through. This rendered the unit worthless for about 3 days worth of debugging from Data Domain. Since then, there have been no major problem with either 3 of the units. Also, the compression that we are getting out of the units is amazing. The following information is coming from a report I did when we were only running Data Domain 510s. Backups started to surpass the designated backup window and this is the argument for the upgrade.

3.75 TB Raw

1.95 TB Usable

290 GB/hr throughput

Problems

We are maxed out in capacity (number of disks) on the DD510
Currently using ~70% of the available storage
Can not meet a backup window of 8hrs
We are maxing out the processor during the backup window

Backup window calculations
Using the last 24 hour pre-compression write rate, we are pushing 2810.8 gigs of data to the Data Domain. Per DD benchmarking and documentation, the max throughput for the DD510 is 290GB an hour. Using an 8 hour window, the calculations would be as follows:

2810.8/8 = 351.35 GB /hr

This shows that we are already trying to push more data than the DD 510 can handle in the time window we are trying to achieve. Below is from Data Domains documentation showing a comparison between their models

Future expansion
Plans are in the works for the IT department to implement NetBackup for Exchange and also move their active directory backups to the Data Domain. Forecasting of the space required for exchange backups would need to be ~1000 gigs. Currently, the Exchange data store is around 350 gigs. Added into the current setup that would be

3160.8/8 = 395.1 GB /hr

That would put us over the limit of the DD530 appliance and up to the DD565 in just throughput alone. The DD565 would give us the ability to scale up to 3 shelves of disks, whereas the DD510 and 530 can not expand beyond one enclosure (shelves).

Conclusion
Best case for achieving a backup window of ~8 hours or below would be to purchase a DD565 for use at the primary site. This would allow for over double the hourly throughput and up to 6 times as much raw storage capacity. The costs and compression capabilities of using a Data Domain system in comparison of an EMC or NetApp solution seems to be the most financially reasonable route. I do hear that EMC Avamar is coming down in price! Also, NetApp free deduplication license for the NetApp does reasonable compression as well. Although with NetApp, that means owning one first and that would cost a lot more than the Data Domain.

Since this initial writing, there is a new Data Domain 565 in the process of being deployed. Once this is up and running, I will update the information here with the backup window results.

Note: This is destination based agentless deduplication, not source based. All the data is pushed from the servers, through the network, and to the datadomain. The data is then deduped on the Data Domain. EMC Avamar is agent based, allowing the server to only push the changed blocks or data to the Avamar deduplicaiton appliance. This will definitely take a load off of a network. Only problem with that is you have to install agents on all servers, as well as update that software. You better have at least a true gigabit network when pushing upwards of 5 terrabytes a night to the data domain. All in all, I am very happy with data domain and recommend them.
Update: Per a conversation with a Data Domain sales team today, they show in the labs that restore speed (read from the Data Domain) is ~85% of the write speed to the device.

So, if a backup takes 1 hour, retrieval of that data should take approximately 1 hour and 11 minutes (70.588 minutes). Now, this does not take into account the overhead of the application doing the restore. Hope I did my math correctly.

Posted in Data Domain, Miscellaneous, NAS, SAN (Storage Area Network)