Colocation to Virtualization » NAS

EMC Celerra NX4 NAS Install In Pictures

Kevin Goodman — Tue, 27 Jul 2010 19:34:07 +0000

So we received our 12tb raw EMC Celerra NX4 system(s) about 3 weeks ago. Eager to get going, I went ahead and racked the units. Below are pictures showing the different pieces that make up the NX4

Now all that we need to do is get some data over to it. We have a matching unit a few racks down that will be moved to our Tier 2 site once replication has been completed.

Notes: One thing to note is that the rails are an all in one unit. You can not change the order in which each device is racked. Also, the controllers (SPs) are built into the drive tray. Best part is that this will be wired up via 10gbE connectivity over fiber.

Filed under: Colocation, EMC, Hardware, NAS, SAN (Storage Area Network)

Storage, Networking, EMC: DataDomain Replication

Kevin Goodman — Wed, 26 May 2010 14:49:41 +0000

We have a total of 3 DataDomains currently in production. One at our “Tier 1″ site, “Tier 2″(DR) and one in Europe. All DataDomain appliances have the ability to replicate data among themselves. This will be a general overview of how to setup replication between two DataDomains

On the source, I have already setup a directory tree for “/backup/europe_data”. All files destined for our Euroupe office will be placed here. On the DataDomain devices, replicated folders are added manually. By default none are replicated.
*********************
Be aware that if you are viewing this through a web browser that a lot of the command might be hidden due to formatting of the WordPress theme. It is best viewed by an RSS reader. If you would like the full text, email or leave a comment and I can send you a PDF version.
*********************

Below is the help output for “replication add” command.

user01@us-dd565# replication add  ?
Commands matching "replication add":
  replication add source  destination
                 Add a replication pair

I wish the syntax was easy as what was stated above, but below is the actual command used to setup replication on the source DataDomain

user01@us-dd565# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once the source is setup, the destination endpoint/DataDomain also must be configured. The command is identical to what was entered at the source DataDomain

user01@eu-dd01# replication add source dir://us-dd565.testdomain.com/backup/europe_data destination dir://eu-dd01.testdomain.com/backup/europe_data

Once replication is setup on both DataDomains, log back into the source and start the replication.

user01@us-dd565# replication initialize
Incorrect syntax "replication initialize", showing help:
  replication initialize
                 Initialize replication on the source
                 (configure both source and destination
                 first)

Below is the actual syntax to startup replication for the directory that we just configured.

user01@us-dd565# replication initialize dir://eu-dd01.testdomain.com/backup/europe_data
(00:04) Waiting for initialize to start...
(00:06) Initialize started.
Use 'replication watch dir://eu-dd01.testdomain.com/backup/europe_data' to monitor progress.

Since I am impatient and want to know for sure this is working, I use “replication watch”. This shows the current progress of the replication between DataDomains.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
    Use Control-C to stop monitoring.

(00:57) Replication initialize started...
(00:59) initializing 3/3:
(10:35)     : pre-comp:   0%    27 KB/s, network    26 KB/s, 0 of 6 files

When new data is written to the mount point, replication automatically starts. Below is what shows if no data is currently being replicated.

user01@us-dd565# replication watch dir://eu-dd01.testdomain.com/backup/europe_data
**** Replication initialize or recover already completed.

“replication show config” is used to display all the configured replication points.

user01@us-dd565# replication show config
CTX   Source                                                    Destination                                                 Connection Host and Port                       Enabled
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------
11    dir://us-dd565.testdomain.com/backup/mysql-rep       	dir://us-t2-dd510.testdomain.com/backup/mysql-rep	us-t2-dd510.testdomain.com   (default)     yes
12    dir://us-dd565.testdomain.com/backup/europe_data        	dir://eu-dd510.testdomain.com/backup/europe_data	eu-dd510.testdomain.com   (default)     yes
---   -------------------------------------------------------   ---------------------------------------------------------   --------------------------------------------   -------

Sorry for all of the spacing being off. All the tabs did not come over when copying data from the console.

user01

Filed under: Data Domain, deduplication, NAS, Networking

DataDomain/NAS/Filesystems/Linux: Remove Files From A DataDomain’s /ddvar

Kevin Goodman — Thu, 16 Apr 2009 15:19:59 +0000

A few days ago we received alerts for the /ddvar fileystem from one of our Data Domain units. Normally, I would not manually remove any files from this filesystem due to it being mainly used by the underlining OS and not for NAS storage. In this case, the problem was the “core” subdirectory. I tried to remove the files from a Windows machine, but due to permission issues, I was unable to do so. Even through Windows (CIFS) to the DataDomain, I was able to modify the permissions of the file(s) and still would receive a permission denied issue when trying to remove. The quick solution here was to add my Linux box to the NFS access list, mount “/ddvar”, and remove the files as the root user. Below details the process that worked for me.

DataDomain Web Alert

SSH to the DataDomain and add my Linux systems IP address

admin1@DataDomain01# nfs add /ddvar 172.16.100.6
Mount the DataDomain /ddvar export on the Linux system
root@tstbox02:~# mount DataDomain01:/ddvar /mnt/DataDomain01/

Move into the core sub-directory under the ddvar mount point

root@tstbox02:~# cd /mnt/DataDomain01/core/

Check the directory for files. As you can see below, the core directory contained 47 gig worth of data

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 47G
drwxrwsr-x  2 root group 4.0K 2009-01-14 22:43 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 2.2M 2008-04-10 14:23 cmdb.core.26956.1207851782.gz
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 121K 2008-04-08 07:01 ddfs.core.14547.1207652507.gz
-rw-------  1 root group 352K 2008-04-07 22:56 ddfs.core.15098.1207618361.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:30 ddfs.core.15100.1207747799.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:37 ddfs.core.15125.1207665449.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:38 ddfs.core.15505.1207665497.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:31 ddfs.core.15506.1207747848.gz
-rw-r--r--  1 root group 527M 2008-09-10 16:27 ddfs.core.1560.1221078099.gz
-rw-r--r--  1 root group 166M 2008-04-08 07:32 ddfs.core.15675.1207654304.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:39 ddfs.core.15795.1207665554.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:32 ddfs.core.15813.1207747905.gz
-rw-r--r--  1 root group 636M 2008-07-28 15:20 ddfs.core.15991.1217272516.gz
-rw-r--r--  1 root group 167M 2008-04-08 10:40 ddfs.core.16055.1207665605.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:33 ddfs.core.16095.1207747957.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:32 ddfs.core.18047.1207654349.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:33 ddfs.core.18231.1207654404.gz
-rw-r--r--  1 root group 167M 2008-04-08 07:34 ddfs.core.18591.1207654458.gz
-rw-r--r--  1 root group 548M 2008-09-20 02:56 ddfs.core.22290.1221893385.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22705.1207750958.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:23 ddfs.core.22875.1207751006.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:24 ddfs.core.23008.1207751055.gz
-rw-r--r--  1 root group 553M 2008-08-19 15:26 ddfs.core.23183.1219173638.gz
-rw-r--r--  1 root group 167M 2008-04-09 10:25 ddfs.core.23269.1207751113.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:06 ddfs.core.28347.1207746385.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:07 ddfs.core.28507.1207746432.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:08 ddfs.core.28694.1207746490.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:09 ddfs.core.28989.1207746540.gz
-rw-r--r--  1 root group 167M 2008-04-09 09:14 ddfs.core.29883.1207746837.gz
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Since I verified that we do not need the archived (.gz) files, they were removed

root@tstbox02:/mnt/DataDomain01/core# rm *.gz

By only retaining the newer files in this directory, the size is now down to 1.6 gigs.

root@tstbox02:/mnt/DataDomain01/core# ls -lah
total 1.6G
drwxrwsr-x  2 root group 4.0K 2009-04-15 14:29 .
drwxr-xr-x 12 root group 4.0K 2009-04-02 10:28 ..
-rw-r--r--  1 root group 3.8M 2008-04-10 15:06 core.13124
-rw-r--r--  1 root group 213M 2008-04-07 22:05 ddfs.gdb.15098.1207618361

Once this was completed, the alert in this DataDomain cleared in the web interface.

Data Domain Web Alert Cleared

Notes: Always verify files that are being removed are not needed. If there are core files present, it generally means the system has hit an error. Verify with your DataDomain support before modifying the ddvar filesystem.

Posted in Data Domain, Filesystems, Linux, NAS

Linux, Unix, NAS, File Systems: Inodes (Part 1) – Checking Availability And High Level Overview

Kevin Goodman — Fri, 10 Apr 2009 19:37:40 +0000

Inodes really tell you how many file handles (files) that can be created on a file system. Most people will never exceed the default setting when the file system is created, nor even know that one is set. I will eventually go into more detail concerning this topic here on the blog. The majority (not all) of file systems that are used on Linux and Unix do not support dynamic inode allocation. What this means is that if you exceed the inode limit of a file system before the storage space, the remainder will be un-usable. That is until some of the current files are removed.

So here is the really bad part. The inodes on ext2 and ext3 (Linux default type) are statically set when the file system is formatted. You can not go back and change the max inode settings. The exceptions to this that I know of are as follows:

- Reiser4
- VxFS
- XFS
- JFS
- WAFL (NetApp proprietary)
- XZFS

If you are running one of the above and have max inodes issue, you can correct it.

I have been working with computers for over 15 years and have only ran into this problem once. Luckily, it occurred on a NetApp NAS device that had the ability to increase this value on the live file system. The main killer here are tons of small files. In this case, the file system for that NFS share was 40 gigabytes in size and default was ~1 million inode limit. The quick fix for the issue was to increase this to 3 million.

As far as a ext2 and 3 go, the following shows how to query a file system for relevant inode information

root@testbox:~# tune2fs -l /dev/sda1
tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   
Last mounted on:          
Filesystem UUID:          56161dd8-9d1d-4c54-851d-938bb88ce6d4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4685824
Block count:              18731782
Reserved block count:     936589
Free blocks:              15534374
Free inodes:              4459463
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1019
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   256
Filesystem created:       Mon Sep 29 16:25:20 2008
Last mount time:          Fri Jan 23 14:27:02 2009
Last write time:          Fri Jan 23 14:27:02 2009
Mount count:              4
Maximum mount count:      33
Last checked:             Thu Jan 15 09:00:37 2009
Check interval:           15552000 (6 months)
Next check after:         Tue Jul 14 10:00:37 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          128
Journal inode:            8
First orphan inode:       2908742
Default directory hash:   tea
Directory Hash Seed:      a6544c5xxxxxxxxxxxxxxxxx
Journal backup:           inode blocks

The above is good to know, especially to check an un-mounted file system. The command below shows a friendlier formatted output

root@testbox:~# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            4685824  226361 4459463    5% /
tmpfs                 222201       4  222197    1% /lib/init/rw
varrun                222201      64  222137    1% /var/run
varlock               222201       5  222196    1% /var/lock
udev                  222201    5142  217059    3% /dev
tmpfs                 222201       5  222196    1% /dev/shm
/dev/sdb1            61063168    1116 61062052    1% /media/disk

As you can see, there are no issues to be worried about on this test computer. Most systems administrators perform centralized monitoring of disk usage at a disk space level (capacity). On highly used servers that utilize locally stored and/or direct attached storage, it is a good idea to have a script check and report on the available inodes.

PDF Version

Notes: This information is provided for a high level overview concerning inodes. More in-depth information will be provided in up-coming posts.

Posted in Filesystems, Linux, Monitoring, NAS

VMWare/NAS/SAN: How To DOS Your Old NetApp With Snapshots

Kevin Goodman — Mon, 09 Mar 2009 07:24:24 +0000

I feel that I have a pretty good foundation of knowledge for the ESX product line. Well, I found out that I did not know everything. It is now 2am and the last server is being restored from a snapshot on the NetApp. Earlier I was auditing the VCB backup logs and found about 12 servers that were failing due to “open snapshots on disk”. This normally occurs when the previous VCB operation fails. Trying to be a good administrator, I decided to go ahead and delete them before leaving the office, as to get a good run tonight. Here is where I made the mistake. The netapp is an older FAS3020 that is single headed. The NetApp performs well under normal conditions and only houses VMs for development.

I was outside hooking the kid bike trailer to the 12 speed when my wife came out holding my cell phone

Wife> Eric just called you twice
Me> Startup my laptop for me
... Call Eric
Eric> Sorry to bother you but  .... .

Luckily he noticed a couple of VMs dropped off the face of the earth. To be exact, we lost 4 virtual machines. When told to power on, they would error with “Can not open disk: PathToVmdk.. The parent virtual disk has been modified since the child was created“.

I never trust the GUI, so I went strait to the ESX console. Sure enough, things failed there also.

Per /var/log/vmkernel

Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.271 cpu6:1215)<6>Debug scsi underrun
Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.342 cpu6:1183)<6>Debug scsi underrun
Mar  5 23:02:59 vmsrv06 vmkernel: 28:09:39:57.413 cpu6:1098)<6>Debug scsi underrun
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.898 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1497
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.898 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1501
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.899 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1499
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.899 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1504
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.900 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1496
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.901 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1494
Mar  5 23:06:49 vmsrv06 vmkernel: 28:09:43:46.901 cpu15:1074)<6>qla24xx_abort_command(1): handle to abort=1502
Mar  5 23:10:06 vmsrv06 vmkernel: 28:09:47:04.375 cpu2:1051)WARNING: SCSI: 119: Failing I/O due to too many reservation conflicts
Mar  5 23:10:06 vmsrv06 vmkernel: 28:09:47:04.375 cpu2:1051)WARNING: FS3: 4785: Reservation error: SCSI reservation conflict

Next we will look at the NetApp. Below is from a console session. As you can see, the CPUs are hit hard.
From the NetApp

NAS>stats show system
system:system:nfs_ops:536/s
system:system:cifs_ops:0/s
system:system:http_ops:0/s
system:system:dafs_ops:0/s
system:system:fcp_ops:481/s
system:system:iscsi_ops:97/s
system:system:net_data_recv:136KB/s
system:system:net_data_sent:2323KB/s
system:system:disk_data_read:63752KB/s
system:system:disk_data_written:51640KB/s
system:system:cpu_busy:50%
system:system:avg_processor_busy:50%
system:system:total_processor_busy:100%
system:system:num_processors:2
system:system:time:1236313212s
system:system:uptime:9164385s

The conclusion is as follows: When VMware creates a snapshot, a new vmdk is created that references the new snapshot. The main vmx files will then be modified to link to the snapshot as well. When a snapshot is deleted, VMware is not just deleting a file from the file system. VMware gathers all the snapshots for that VM, collapses all their contained data into the main vmdk file, modifies the vmx files, then deletes the selected snapshots files. So in essence, when going down the list of all VMs that had snapshots(9-12) and deleting them, this caused the NetApp to overload. During this, the ESX hosts intermittently lost connectivity to the VMFS LUNS. This happened at such a fast rate, that no alerts were generated, but caused the VMs that were currently writing the snapshot to the main vmdk files to corrupt their configurations.

Moral of the story: Rapidly deleting snapshots can DOS older/slower storage systems.

Posted in NAS, SAN (Storage Area Network), VMWare

NAS / Data Domain: Resetting Hung Alert

Kevin Goodman — Mon, 29 Dec 2008 19:40:23 +0000

We had a power supply go out on one of our Data Domain 510 appliances about 2 weeks ago. They shipped out a replacement power supply and it was put in place. After the power supply was replaced, the alarm was still showing via the web page interface and CLI (Command Line Interface). Since this is a standby backup unit, I did do also tried a reboot. This also did not clear the alert. Remember, I am in now way responsible for anyone ruining their system or voiding their warranty.

Snippet from the autosupport(email) log
Power Supply
Status
——–
DEGRADED
——–

From the CLI
# alerts show current
Alert Time         Description
—————-   —————————————————–
Mon Dec 8 08:38   Encl 1 A power supply module has failed.
—————-   —————————————————–
There is 1 active alert

I needed to go over to that colocation anyway so I made sure remote hands didn’t mess anything up. So this alert is definitely hung. The new power supply had good status indicators, and the power cable was replaced. Removing this alert has to be done through the CLI in SE mode. To do this, I SSHed into the Data Domain and escalated my privileges as follows.

Retrieve the serial number
# system show serialno
Serial number: XXXXXXXXXXX

Escalate to SE privileges
# priv set SE
Enter system password:[enter serial number]

Once in SE mode, the prompt will change to something resembling the following
SE@hostname##

Now, the ‘reg’ command is extended to its full feature set
SE@hostname## reg
Commands matching “reg”:

reg removekey            Remove a key from the configuration
reg set reg-default      Set the specified key to the default value
reg set unset-defaults        Set unset defaults. Useful for upgrades
reg set =        Set a configuration key to the given value
reg setraw =     Set an encoded reg string
reg show defaults             Show registry defaults
reg show obsolete             Show obsolete keys
reg show stats                Show registry stats
reg showraw [nokey]           Show encoded registry data
reg show [nokey]         Show a value for a configuration key

The autosupport contains the registry key that needs to be removed
dynamic.ems.701.x.1000.x.9.1.5 = Encl 1 A power supply

Cut out the part of the key that is needed
dynamic.ems.701.x.1000.x.9.1.5

The following removes the hung alert from the registry
SE@hostname## reg removekey dynamic.ems.701.x.1000.x.1.5

Note: Unfortunately, the alert does not tell which of the 3 power supplies have failed. This was not a huge problem, since the remote hands just eyed the power supply that had no status lights on.

Posted in Data Domain, deduplication, Hardware, NAS

VMware: What VMkernel Ports means to you

Kevin Goodman — Wed, 03 Dec 2008 19:05:45 +0000

Just wanted to put this information out there. Where I work, our ESX cluster will have 10 NICS (6 currently on 2 VLANS) and 3 different VLANs segmenting off different traffic. The VMkernel Port is needed to utilize the following

ISCSI
NFS/NAS
VMotion

This information is provided when going through “Add Networking” under the configuration tab in ESX3.x.

VMkernel Add Networking

In ESX 3.x VMware has the ability to create a VMkernel Port on each vSwitch. So far, we have had no luck utilizing NFS or software ISCSI through two different VMkernel Ports. Example:

There is a NAS on both a private VLAN and the normal LAN network. Each goes through a different vSwitch. After adding a VMkernel Port on both vSwitches, NFS and ISCSI worked fine to the first defined connection. After VMware first connects to a server on either subnet, in this example 172.0.1.2, the next connection to 172.0.2.1 will fail. It will fail no matter if trying to connect through the other vSwitch using the same protocol (NFS) or ISCSI. Whichever connection is established first, VMWare seems not to be able to route through the other VMkernel Port

Dual NFS Mount Errors

Below shows a vSwitch configuration with a VMkernel Port

vSwitch With VMkernel

Note: I am unsure if this would be an issue when using an hardware ISCSI (ISCSI card). As far as NFS is concerned, I do not have a work around for this currently. If you will need to mount NFS shares from two different sources, through different vSwitches, better find the answer to this problem first. I hope that ESX 4 solves this issue.

Posted in NAS, VMWare

NetApp: When NetApp Lies (Web GUI Gateway)

Kevin Goodman — Mon, 24 Nov 2008 05:32:58 +0000

I was changing the VIF interface configuration on a NetApp a few days ago. I was doing so via command line and outside of an intrusive maintenance window. The modification was being done on the LAN interface, but the unit also had a VIF that was running production ISCSI data stores. During this maintenance, only thing that was going to be affected was the snap mirroring through the vif0 interface. I removed the vif by using ‘vif destroy’. In doing so, of course the link to the rest of the locations dropped. After the interface was re-created, I was getting alerts from the console that the remote NetApp connection failed. The NetApp was accessible locally, was able to ping the gateway, but could not ping outside of the subnet or be reached from other locations. Coming from a networking background, I knew immediate that this looked to be a gateway issue. The web interface showed the correct gateway, but ‘route -s’ showed no default gateway.

nas01> route -s
Routing tables

Internet:

Destination	Gateway	Flags	Refs	Use	Interface
localhost	localhost	UH	2	639	lo
172.16.100	link#8	UC	0	0	vif1

Example local ping
nas> ping 172.16.100.1
172.16.100.1 is alive

Pinging the other locations gateway would return with no route to host
ping 172.16.0.1

Add the default gateway in
nas> route add default 172.16.100.1 1

Access restored
nas> ping 172.16.0.1
172.16.0.1 is alive

Verify the routing table
nas01> route -s
Routing tables

Internet:

Destination	Gateway	Flags	Refs	Use	Interface
default	172.16.100.1	UGS	7	820733	vif0
localhost	localhost	UH	2	639	lo
172.16.100	link#8	UC	0	0	vif1

Note: So it seems that the web GUI just reads the /etc/dgateways from the filesystem. When I destroyed the vif that acts as the path from the default gateway, it looks to have removed the gateway from the active routing table. Normally I reboot the whole filer to be safe, but again this was to be transparent to the ISCSI data stores. Everything worked fine after I manually re-entered the route. Also, ‘netstat -r’ will also display the routing table.

Model:	FAS3020
Version:	Data ONTAP Release 7.2.5.1

Posted in NAS, Networking

NetApp: Enabling Deduplication (ASIS) On A Volume

Kevin Goodman — Fri, 14 Nov 2008 09:31:05 +0000

Below shows how to setup NetApp deduplication (formerly ASIS) on an existing volume. The following example is running NetApp Release 7.2.5.1. The license are now free, and were already installed.

Check to make sure the licensces are installed
NAS> license
nearstore_option xxxxxZH
a_sis xxxxxCG

Enable sis (deduplication) on the volume
NAS> sis on /vol/testVol
SIS for “/vol/testVol” is enabled.
Already existing data could be processed by running “sis start -s /vol/testVol”.

Initiate deduplication service
NAS> sis start -s /vol/testVol
The file system will be scanned to process existing data in /vol/testVol.
This operation may initialize related existing metafiles.
Are you sure you want to proceed with scan (y/n)? y

Thu Nov 13 10:01:38 EST [wafl.scan.start:info]: Starting SIS volume scan on volume testVol.
The SIS operation for “/vol/testVol” is started.

NAS> Thu Nov 13 10:01:42 EST [wafl.snap.delete:info]: Snapshot copy sis.93057f3c-aed8-11dd-a6d9-00a09804ad86 on volume testVol NetApp was deleted by the Data ONTAP function dense_delete_snapshot. The unique ID for this Snapshot copy is (44, 156).

Sis status will display the state and running status on all volumes
NAS> sis status

Path	State	Status	Progress
/vol/Shares	Enabled	Idle	Idle for 10:05:52
/vol/testVol	Enabled	Idle	Idle for 08:48:40
/vol/luns	Disabled	Idle	Idle for 2330:43:02
/vol/vm	Enabled	Idle	Idle for 00:09:35

View the status on a specific volume
NAS> sis status /vol/testVol

Path	State	Status	Progress
/vol/testVol	Enabled	Idle	Idle for 00:11:23

After sis completes, use df to show the amount saved and deduplication percentage
NAS> df -sh /vol/testVol

Filesystem	used	saved	%saved
/vol/testVol/	519GB	754GB	59%

Verify that sis is scheduled to run frequent enough at the times you want
NAS> sis config

Path	Schedule
/vol/shares	sun-sat@0
/vol/testVol	sun-sat@0
/vol/luns	sun-sat@0
/vol/vm	sun-sat@0

Below is the help output of sis config
NAS> sis help config
sis config [ [ -s schedule ] | … ]
- Sets up, modifies, and retrieves the schedule of SIS volumes.

Reschedule for nightly at 11pm
NAS> sis config -s sun-sat@23 /vol/testVol

Verify the new configuration

NAS> sis config /vol/testVol

Path	Schedule
/vol/testVol	sun-sat@23

Note: ASIS is not real time deduplication. It is ran on a schedule instead of as the data is coming in. Also, when setting the schedule for a volume, you can only specifiy the hour to start running, in this case trying to specify sun-sat@23:45 did not work when testing.

Posted in NAS, Uncategorized

VMware: VCB Troubleshooting With vcbmounter

Kevin Goodman — Thu, 13 Nov 2008 09:48:21 +0000

Lately there has been problems with the VCB backup software that I am using. Unfortunately, troubleshooting it is not “cut and dry”. There is the main application that has to be installed, its proprietary VCB plugun, on top of VCB plugin from VMware. Below will go through using vcbmounter to test VCB through VMwares interface. First, you have to be running Virtual Center, license VCB, and and install the VCB plugin from VMware for this to work. The path to vcbmounter.exe is the default install location.

Command structure utilizing the VM name as source
vcbmounter -h -u -p -a name: -r -t fullvm -m san

Use vcbmounter to pull down a full backup of a VM.
C:Program FilesVMwareVMware Consolidated Backup Framework>vcbmounter -h viServer.x.com -u vcb -p vcbuserPass -a name:TestVM -r Z:vcprepotestTestVM -t fullvm -m san

Copying “[NetApp] TestVM//vmware-938.log”:
0%=====================50%=====================100%
**************************************************

Copying “[NetApp] TestVM//vmware-940.log”:
0%=====================50%=====================100%
**************************************************

Copying “[NetApp] TestVM//vmware-942.log”:
0%=====================50%=====================100%
**************************************************

Copying “[NetApp] TestVM//vmware.log”:
0%=====================50%=====================100%
**************************************************

[2008-11-12 15:53:34.093 'BaseLibs' 3100 warning] SSLVerifyCertAgainstSystemStore: Subject mismatch: VMware vs viServer.x.com
[2008-11-12 15:53:41.730 'BaseLibs' 3100 warning] SSLVerifyIsEnabled: failed to read registry value. Assuming verification is disabled. LastError = 0
[2008-11-12 15:53:41.730 'BaseLibs' 3100 warning] SSLVerifyCertAgainstSystemStore: Certificate verification is disabled, so connection will proceed de
spite the error
Converting “Z:vcprepotestTestVMscsi0-0-0-TestVM.vmdk” (compact file):
0%=====================50%=====================100%
**************************************************

Above is the truncated output of the command. It rambled on for about two pages. In this example, VCB is pulling down a full backup of the VM. TestVM storage is connected via ISCSI on a NetApp. Vcbmounter works fine on our VMs whose storage is on an EMC. Z: data store is actually a mapped drive connected to an ISCSI point.

Below is an image showing the files that were created from vcbmounter.exe

vcbmounter files

Note: Pasting in the information above messed up the alignments of the *s, the transfers were actually 100% complete.

Posted in NAS, VMWare

Linux/NetApp: NFS (NetApp) Fstab Mount

Kevin Goodman — Wed, 12 Nov 2008 11:13:43 +0000

The following details how to map an NFS point, this case to a Netapp, using fstab

Edit the fstab file. In this example, 192.168.1.100 is the NFS, the NFS export to be mounted is /vol/dir1, and local directory to be mapped to is /dir1. Next is the mount point type. For details on the rest of the options, use ‘man nfs’ from the command line.
# vi /etc/fstab
192.1.1.100:/vol/dir1 /dir1 nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,nfsvers=3,timeo=600,actimeo=0 0 0

Create the local directory
# mkdir /dir1/

Mount always searches /etc/fstab if just a label (/dir1) is passed to it. Here, it found the entry in fstab and mounted it.
# mount /dir1/

Using df to verify that mount did connect the NFS point
# df -h | grep dir
192.1.1.100:/vol/dir1
32G 96K 32G 1% /dir1

Move into the mount point
# cd /dir1

In this example, the local group luser needs to have full access to the share. Below shows the changing of ownership from root to luser.
# chgrp luser .

Switch to the user luser1, who is also in the luser group
# su – luser1

Move into the new NFS mount point
[luser1@x ~]$ cd /dir1/

Create an empty file to test write access
[luser1@x dir1]$ touch asdf

Use ls to make sure the file exists.
[luser1@x dir1]$ ls
asdf

Logout of luser since testing is done.
[luser1@x /]$ exit
logout

Note: The title of this is NFS (NetApp) Fstab Mount because technically this should work for any NFS mount point, not just a NetApp one.

Posted in Linux, NAS, Security

Data Domain NTP Configuration

Kevin Goodman — Tue, 28 Oct 2008 22:10:24 +0000

Below is information on how to setup a Data Domain appliance to work with the Network Time Protocol (NTP)

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
No NTP servers configured; using multicast mode.

Add a new timeserver to the list
dd# ntp add timeserver pool.ntp.org
Remote Time Servers:
pool.ntp.org

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
#   Server
-   ————
1   pool.ntp.org
-   ————
Showing NTP servers configured manually.
I advise disabling and re-enabling the NTP service
dd# ntp disable
NTP disabled.
dd# ntp enable
NTP enabled.

Check the NTP status. It may take a while for NTP to query and update this information
# ntp status
Status                                             Enabled
Current Clock Time                         Tue Oct 28 16:07:22.708 2008
Clock Last Synchronized                 Tue Oct 28 15:48:23.398 2008
Clock Last Synchronized With Time Server   208.53.158.34

Removing a timeserver from the list
dd# ntp del timeserver pool.ntp.org
Remote Time Servers:
(multicast)

Check the NTP configuration
dd# ntp show config
NTP is currently enabled.
No NTP servers configured; using multicast mode.

Note: Multiple timeservers can be used in the NTP configuration. This example only showed one. To add another, just follow “Add a new timeserver to the list” part again. It may take a while for the NTP service

Posted in Data Domain, NAS

Data Domain Hostname / IP Change

Kevin Goodman — Tue, 28 Oct 2008 19:46:36 +0000

The following is information on how to re-ip and change the hostname on a Data Domain appliance. This was done on both a Data Domain 510 and 565.

Check to see what host name mappings are in the Data Domain. If replacing a unit and using its IP, the old servers information would have to be removed from here. In this case, dd01 would cause local lookups to have problems, since that’s the one being replaced by the new unit.

newdd# net hosts show
Hostname Mappings:
x.x.x.190 -> dd04.x.com
x.x.x.191 -> dd01.x.com dd01
x.x.x.192 -> dd03.x.com dd03

I decided to just wipe the hosts file and remove all static mappings.
newdd# net hosts reset
Host mappings reset to empty.

Change the IP on a single interface
newdd# net config eth0 x.x.x.191 netmask x.x.x.0
Configuring interface…

Change the IP on a virtual interface if you are doing NIC teaming. This example will be used in the ping below
newdd# net config veth0 x.x.x.191 netmask x.x.x.0
Configuring interface…

Test connectivity from a server on the network. It took the DD 565 about 2 minutes to come up on the new IP.
server# ping x.x.x.191
PING x.x.x.191 (x.x.x.191) 56(84) bytes of data.
64 bytes from x.x.x.191: icmp_seq=20 ttl=61 time=217 ms
64 bytes from x.x.x.191: icmp_seq=21 ttl=61 time=1.52 ms
64 bytes from x.x.x.191: icmp_seq=22 ttl=61 time=1.55 ms
64 bytes from x.x.x.191: icmp_seq=23 ttl=61 time=1.52 ms
64 bytes from x.x.x.191: icmp_seq=24 ttl=61 time=1.73 ms

Time to change the hostname of the box
newdd# net set hostname dd01.x.com
The Hostname is: dd01.x.com
CIFS active directory authentication will stop working after hostname change.
Run ‘cifs set authentication’ command again.

As directed, cifs set authentication command had to be ran to to join the new Data Domain back to Active Directory.

Note: A reboot of the Data Domain had to be done to get the cifs authentication back onto the Acitve Directory domain. This could have been due to two different issues. Either the hostname change caused the issue, or the NTP settings. The system was 5 minutes off from the domain controller before the reboot forced the NTP service to query the timeserver.

Posted in Data Domain, NAS

NetApp: Deduplication ASIS and VMWare

Kevin Goodman — Thu, 23 Oct 2008 23:59:38 +0000

Man, where did this come from? I used to rule out NetApp deduplication due to the costs of the licenses, but now it is free (if you have one of their NAS). Also, the max volume size that NetApp can dedupe is 1 terabyte for the model we are on. A coworker of mine enabled deduplication on a VMWare ISCSI volume recently. Turns out this option could be great for small to mid sized companies that can not afford to have both an EMC and NetApp. Also as a cost effective way to save money on storage by offloading lower priority servers or VMs to the ASIS volumes

Below is information pulled from the NetApp showing two volumes that are being deduplicated

NAS> df -sh
Filesystem                used      saved       %saved
/vol/users/             246GB       68GB          22%
/vol/vmware/           96GB       99GB          51%

Lets see the status of the service on the VMWare volume
NAS> sis status
Path State Status Progress
/vol/vmware Enabled Active 35 GB Scanned

Current VMWare volume utilization
NAS> df -h vmware
Filesystem total used avail capacity Mounted on
/vol/vmware/ 950GB 147GB 802GB 16% /vol/vmware/

Here is the sis configuration and status of the VMWare data store
NAS> sis status -l
Path:                    /vol/vmware
State:                   Enabled
Status:                 Active
Progress:             38231884 KB Scanned
Type:                   Regular
Schedule:             sun-sat@0
Last Operation Begin:    Thu Oct 23 00:00:07 EDT 2008
Last Operation End:      Thu Oct 23 00:03:59 EDT 2008
Last Operation Size:     2405 MB
Last Operation Error:    -
As we can see, the last run took almost 4 hours

The first volume is a CIFS (Windows) user shares containing home directories and whatever data they so desire to store there. Deduplication is low on here due to the majority of the data not being the same thing just stored many times. The VMWare share has about 9 windows and Linux virtual machines running there. This volume is getting a lot better compression due to most of the data being the same system files that are needed for the operating systems to run. Deduplication is great for this cause, meaning that each of those files only need to be stored once. We will be off loading more of the non critical and development virtual machines to NetApp ASIS (deduplicated) volumes once it is on fiber. The compression will continue to increase as the number of VMs running within them increase. From the looks of it, we should easily be able to get 80% or more deduplication rates! In the long run using this feature will save a lot of money on the number of drive trays needed.

Note: One downfall, ASIS is not real time deduplication. It is ran on as schedule instead of as the data is coming in, unlike a Data Domain.

Just goes to show that you shouldnt believe all of the vendors putting down the competitors products or capabilities. I do not think that the ASIS is the answer to all problems, but it can definitely help smaller companies. It is also a lot easier than adding on an appliance like EMC.

10/27/2008

Getting better:

NAS> df -sh /vol/vmware/
Filesystem used saved %saved
/vol/vmware/ 215GB 306GB 59%

11/15/2008

NAS> df -sh /vol/testVol

Filesystem	used	saved	%saved
/vol/testVol/	519GB	754GB	59%

Posted in deduplication, NAS, SAN (Storage Area Network), VMWare

NetApp Active Directory DNS issues. dns.update.enable

Kevin Goodman — Thu, 23 Oct 2008 17:15:40 +0000

There are 2 IPs assigned to each of the NetApp units here. One for the regular network VLAN and one on the ISCSI network. When querying the DNS for the IP information, both IPs would come back. Turns out that under options in the NetApp command line, dns.update.enable was the issue. This option told the NetApp to send DNS updates to the DNS servers, thus auto updating the records with both its public IP and ISCSI VLAN IP. This was causing an intermittent problem for all servers and clients that were trying to access NFS or CIFS (windows) shares on the NetApp.

Incorrect DNS information
$ nslookup netapphn

Name: netapphn.x.com
Address: 127.x.1.85
Name: netapphn.x.com
Address: 127.x.200.85

Our other NetApp off site was not having this issue. It only resolved to one IP

From the Netapp CLI, it shows that DNS updating is turned on
NAS> options dns.update.enable
dns.update.enable on

Trun it off
NAS> options dns.update.enable off

Now, verify that the change took
NAS> options dns.update.enable
dns.update.enable off

The change did take. Now time to check the DNS
$ nslookup netapphn
Name: netapphn.x.com
Address: 127.x.1.85

Great, DNS now resolves correctly. No more hading out two IPs.

Note: Will need to flush the DNS cache on the server or delete the incorrect “A” record out of the DNS server to see immediate results.

Posted in Miscellaneous, NAS, Networking