First, copy the RPM installation package over to the server. Below shows the package to be installed.

[root@test_server01 user01]# ll
total 7036
-rw-r--r-- 1 user01 user01 7191661 Apr 27 09:24 EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm

Install the package via “rpm -i”.

[root@test_server01 user01]# rpm -i EMCpower.LINUX-5.3.1.00.00-111.rhel5.x86_64.rpm
All trademarks used herein are the property of their respective owners.
NOTE:License registration is not required to manage the CLARiiON AX series array.

Before powerpath can be used, a license key must be installed.

[root@test_server01 user01]# emcpreg -list
unable to open license key file: No such file or directory

Overview of the “emcpreg -add” syntax.

[root@test_server01 user01]# emcpreg -add
Missing option parameter.
Usage:
    emcpreg [opts] -add key [key ...]
    emcpreg [opts] -remove key [key ...]
    emcpreg [opts] -check key [key ...]
    emcpreg [opts] -list
    emcpreg [opts] -edit
    emcpreg [opts] -install
Options:
    -f file     license file

Now we add the license key to powerpath. The following key is fake! You must obtain yours from EMC.

[root@test_server01 user01]# emcpreg -add AGE4-DFD3-89842-DSAF-JIJ0-WKG50
1 key(s) successfully added.

Make sure the license was installed correctly.
[root@test_server01 user01]# emcpreg -list

Key AGE4-DFD3-89842-DSAF-JIJ0-WKG50
  Product: PowerPath
  Capabilities: All
[root@test_server01 user01]#

Next, start the Power Path service.

[root@test_server01 user01]# /etc/init.d/PowerPath start
Starting PowerPath:  done

Display the current paths to storage via “powermt”. Since this server is booting from SAN and just being installed, there is currently only one path to storage.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=Unknown, current=SP A    Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sda       SP A0     active  alive      0      0

Now that powerpath is installed, we need to edit fstab to boot off of the Power Path device.

Origional fstab using labels for “/boot”.

/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Below is the edited fstab with “LABEL=/boot” commented out and /boot changed to use “/dev/emcpowera1″

[root@test_server01 ~]# vi /etc/fstab
/dev/lvm/root           /                       ext3    defaults        1 1
/dev/lvm/usr            /usr                    ext3    defaults        1 2
/dev/lvm/app            /app                    ext3    defaults        1 2
/dev/lvm/home           /home                   ext3    defaults        1 2
/dev/lvm/var            /var                    ext3    defaults        1 2
/dev/lvm/vartmp         /var/tmp                ext3    defaults        1 2
/dev/lvm/UsrLocal       /usr/local              ext3    defaults        1 2
/dev/emcpowera1         /boot                   ext3    defaults        0 0
#LABEL=/boot             /boot                   ext3    defaults        1 2
tmpfs                   /dev/shm                tmpfs   defaults        0 0
devpts                  /dev/pts                devpts  gid=5,mode=620  0 0
sysfs                   /sys                    sysfs   defaults        0 0
proc                    /proc                   proc    defaults        0 0
LABEL=SWAP-sda3         swap                    swap    defaults        0 0

Filesystem options were changed to “0 0″ on emcpowera due to RedHat trying to do filesystem scans before the Power Path driver is started.

All paths now need to be zoned in the fiber switch, initiators registered in Navisphere, and paths added to the host in it’s storage group. This will not be covered here.

After zoning both paths on one switch, “powermt” now shows a path to both Clariion SPA and SPB. If not, try either stopping and restartaring “/etc/init.d/PowerPath” or restarting the server.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP A       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0

Configuration has now been completed on fiber switch 2 and both SPA and SPB in the Clariion. Reboot the server. Again, “powermt” is used to list the paths.

[root@test_server01 ~]# powermt display dev=all
Pseudo name=emcpowera
CLARiiON ID=AXE00515480482 [test_server01_ucs]
Logical device ID=15618646804648SDSDFW84FW4894949 [test_server01_ucs_boot]
state=alive; policy=CLAROpt; priority=0; queued-IOs=0
Owner: default=SP B, current=SP B       Array failover mode: 1
==============================================================================
---------------- Host ---------------   - Stor -   -- I/O Path -  -- Stats ---
###  HW Path                I/O Paths    Interf.   Mode    State  Q-IOs Errors
==============================================================================
   0 fnic                      sdc       SP B1     active  alive      0      0
   0 fnic                      sdd       SP A0     active  alive      0      0
   1 fnic                      sde       SP B0     active  alive      0      0
   1 fnic                      sdf       SP A1     active  alive      0      0

From above, you can see that we now have 4 paths definied. Both fnic interfaces can see SPA and SPB. Each fnic is attached to a seperage fiber switch, so we have redundant paths to both Clariion heads (SP’s). Once rebooted, the server should load fine with no issues and see all paths via powermt.

Notes:  “/boot” is the storage label used in this example.  If your mount point is different, modify it’s entry instead.  “/dev/emcpowera1″ is used since there is only one LUN mapped to this host.  Like anything else, if there are more than one, each would have it’s own device.

[root@tsthost01 log]# tail -50 /var/log/messages | more
Apr 21 14:27:58 t1ps-db01 snmpd[2936]: Received SNMP packet(s) from UDP: [127.0.10.77]
Apr 21 14:28:13 127.0.10.99 last message repeated 9 times
Apr 21 14:28:13 127.0.10.99 snmpd[2291]: Connection from UDP: [127.0.10.77]
Apr 21 14:28:13 127.0.10.99 snmpd[2291]: Received SNMP packet(s) from UDP: [127.0.10.77]
Apr 21 14:28:13 127.0.10.99 snmpd[2291]: Connection from UDP: [127.0.10.77]
Apr 21 14:28:13 127.0.10.99 snmpd[2291]: Received SNMP packet(s) from UDP: [127.0.10.77]
Apr 21 14:28:13 127.0.10.99 snmpd[2291]: Connection from UDP: [127.0.10.77]
Apr 21 14:28:19 127.0.10.81 last message repeated 18 times
Apr 21 14:28:25 127.0.10.26 MultiModemiSMS last message repeated 2 time(s)
... truncated
--More--

Then I scroll through the results and how what I am looking for is in those 50 linues. Unfortunately the information nornally is not and I re-run the command adding “-100″ or “-200″ to replace the “-50″. That is definitely not the best way to do it.

The better way for searching from the end of large files is to use “tac” instead of “cat” or tail. It might be obvious, but “tac” is just “cat” reversed. Bellow are from the man pages of each command

NAME       cat - concatenate files and print on the standard output

NAME       tail - output the last part of files

NAME       tac - concatenate and print files in reverse

Below is an example of cat used to read a file.

root@kdesk-l:~# cat GreenEggs
Do you like
green eggs and ham?
I do not like them, Sam-I-am.
I do not like
green eggs and ham.

Would you like them
here or there?

I would not like them
here or there.
I would not like them anywhere.

Next is tac reading the same file. Notice the content has been read in reverse.

root@kdesk-l:~# tac GreenEggs
I would not like them anywhere.
here or there.
I would not like them

here or there?
Would you like them

green eggs and ham.
I do not like
I do not like them, Sam-I-am.
green eggs and ham?
Do you like

Hopefully you can see some benefit to using this, especially in conjunction with the “more” command.

[root@tsthost01 log]# tac messages | more

Below are some schenarios that tac would be good for
- Going through log files from newest events to old
- Reviewing Java error log files (normally waaayyyyy to much information in those)
- Checking mailserver or DNS server logs
- When needing to go through a file without knowing exactly what you need to use “grep” to search for

Notes:  The “tac” command is pretty much standard issue on Linux based systems. I tested this on RedHat, CentOS, and Ubuntu

First, go into the VMware console and right-click on the VM (Virtual Machine) that you are going to install VMware tools on.  Select “Install/Upgrade VMware Tools” option from the list.  Below is a screen shot of the menu.

VMware Tools Menu

By default, most CDROM devices are symbolically linked to /dev/cdrom by the operating system.

Just in case, you can search the messages file to see the actual device. This is needed only if /dev/cdrom is not automatically linked or you have setup multiple cdrom devices on the VM (Virtual Machine).

[root@RHserver01 media]# cat /var/log/messages | grep CDROM
Jan 10 10:59:03 RHserver01 kernel: hda: VMware Virtual IDE CDROM Drive, ATAPI CD/DVD-ROM drive

From above, you can see that the actual device is hda, specifically /dev/hda. If you are just curious you can do an “ll” on the /dev/cdrom device to see where is it linked to. In this case again, it’s going to hda.

[root@RHserver01 ~]# ll /dev/cdrom
lrwxrwxrwx 1 root root 3 Jan 12 13:18 /dev/cdrom -> hda

Mount the cdrom device to an empty or non-mounted point on the filesystem. Here I use the defaultly present /media location.

[root@RHserver01 /]# mount /dev/cdrom /media/
mount: block device /dev/cdrom is write-protected, mounting read-only

Below we move into the /media location using “cd”.

[root@RHserver01 /]# cd /media/

“ls” is used to display what files are present. Here we see both an RPM (native RedHat based OS package) and a gzip archive. If you were installing VMware Tools on a non-RedHat derived distribution, you would use the .gz package.

[root@RHserver01 media]# ls
VMwareTools-3.5.0-143128.i386.rpm  VMwareTools-3.5.0-143128.tar.gz

Since we are on RedHat, this is simple. Pass “-i” to the rpm command then the package name to be installed.

[root@RHserver01 media]# rpm -i VMwareTools-3.5.0-143128.i386.rpm

Immediately after installing the RPM, you might see the following errors to your console, or in /var/log/messages.

Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda
Jan 12 13:15:07 RHserver01 kernel: VFS: busy inodes on changed media or resized disk hda

If you are getting these to the console, it makes it hard to continue working form the command line. This is easy to stop. First, cd out of the /media/ mount point

[root@RHserver01 ~]# cd ..

Next, unmount the cdrom device. After doing so, the messages will stop

[root@RHserver01 ~]# umount /dev/cdrom

The “vmware-config-tools.pl” command must be ran from the VMware console. Below is the output you would get if it was tried through a remote session (SSH).

[root@RHserver01 ~]# vmware-config-tools.pl

It looks like you are trying to run this program in a remote session. This
program will temporarily shut down your network connection, so you should only
run it from a local console session. Are you SURE you want to continue?
[no]
Please re-run this program from a local console shell.
Execution aborted.

There is a good reason for this. vmware-config-tools.pl drops networking on the server to install the VMware network drives. In doing so, you loose remote connectivity.

Below shows the actual output from vmware-config-tools.pl on the console

[root@RHserver01 ~]# vmware-config-tools.pl
Shutting down interface eth0:                              [  OK  ]
Shutting down loopback interface:                          [  OK  ]
Stopping VMware Tools services in the virtual machine:
   Guest operating system daemon:                          [  OK  ]
   Unmounting HGFS shares:                                 [  OK  ]
   Guest filesystem driver:                                [  OK  ]
   Guest memory manager:                                   [  OK  ]
Trying to find a suitable vmmemctl module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmhgfs module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmxnet module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

Trying to find a suitable vmblock module for your running kernel.

The module bld-2.6.18-8.el5-i686smp-RHEL5 loads perfectly in the running
kernel.

No X install found.

Starting VMware Tools services in the virtual machine:
   Switching to guest configuration:                       [  OK  ]
   Guest memory manager:                                   [  OK  ]
   Guest vmxnet fast network device:                       [  OK  ]
   DMA setup:                                              [  OK  ]
   Guest operating system daemon:                          [  OK  ]

The configuration of VMware Tools 3.5.0 build-143128 for Linux for this running
kernel completed successfully.

You must restart your X session before any mouse or graphics changes take
effect.

You can now run VMware Tools by invoking the following command:
"/usr/bin/vmware-toolbox" during an X server session.

To use the vmxnet driver, restart networking using the following commands:
/etc/rc.d/init.d/network stop
rmmod pcnet32
rmmod vmxnet
depmod -a
modprobe vmxnet
/etc/rc.d/init.d/network start

If you wish to configure any experimental features, please run the following
command: "vmware-config-tools.pl --experimental".

Enjoy,

--the VMware team

Notes: From my experience, restart of networking via init.d scripts or rebooting the server is always needed. I personally always reboot the server to be safe.

testuser@workstation4-l:~$ ssh test123@ldapclientsrv
Connection closed by 172.16.0.192

To begin, lets look at a typical error that you would get on the system if LDAP communication was down.

Dec 13 12:52:58 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:52:58 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 4 seconds)...
Dec 13 12:53:02 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:53:02 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 8 seconds)...
Dec 13 12:53:10 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:53:10 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 16 seconds)...

As noted before, I was unable to login with a local account. Turns out that the problem was with the default “bind_policy” in /etc/ldap.conf.  Per the document:

# Reconnect policy: hard (default) will retry connecting to
# the software with exponential backoff, soft will fail
# immediately.
#bind_policy hard

This was changed to:

bind_policy soft

Once this was changed, I brought up the firewall on the LDAP server and refused connections. Ability to login via LDAP was gone, but the server did fail back to local system authentication

Note(s): When failing back to local authentication, there is no error sent back to the client trying to login, only errors go to /var/log/secure file. The server will just keep rejecting the users login until LDAP is back up. At least this gives you the ability to get in with a local system account in an emergency.

Example error to /var/log/secure when LDAP server is down and local authentication is rejecting the LDAP user received from the client:

Dec 13 12:59:59 ldapServer sshd[2588]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=172.16.0.22

All WebSphere services must be stopped to install the WAS updates. On a standard installation, bcguser must be used to stop the service

[bcguser@WPGhost ~]$/opt/IBM/bcghub-simple/bin/./bcgStopServer.sh

We do not use ‘/opt’ for our WebSphere location, so change this if yours is different.

Next, use Update Installer to patch the WebSphere Application Server
[user@WPGhost ~]$ sudo /opt/IBM/WebSphere/UpdateInstaller/./update.sh

There is a gotcha here that had me “chasing my tail” for about 10 minutes. When going to install the WebSphere Partner Gateway fix pack, the Partner Gateway and WAS server must be started. Installation of the update will fail with error “user input validation”.

So before installing the WPG update, re-launch the Application Server and Partner Gateway

[bcguser@WPGhost ~]$ /opt/IBM/bcghub-simple/bin/./bcgStartServer.sh

Once done, launch IBM Update Installer again, passing the customized responce file for your environment. This needs to be executed as the root user, so sudo was used to allow xforwarding from a non-root account

[user@WPGhost ~]$ sudo /opt/IBM/WebSphere/UpdateInstaller/./update.sh -options /opt/IBM/bcghub-simple/responsefiles/bcgupdate_en_US.txt

Those are my miscellaneous notes about the update installation. Everything went fine here and I hope this fixes some of the SFTP issues we have been having.
This brings the WebSphere Partner Gateway Console form version 6.2.0.0.273 to 6.2.0.1.333

Notes: Here is the link to IBM’s website that lists the fixes that are provided in the update.

The main one that got me was integration with Oracle.   An overview of the installation steps are shown below:

So the problem came down to the ‘Database owner name’ and ‘Schema owner login’ being the same.  This typically is not an issue.  The worst part is that the WAS (WebSphere Application Server) and WPG (WebSphere Partner Gateway) installation would both complete successfully. Not only that, they system would run with no errors.

That being said, once I started the patching process, it would always fail.   So as a last resort, I tried changing the ‘Database user name’ and ‘Schema owner login’ to be different.   Thanks to DBA Eric’s recommendation.  This worked!

I decided to put this blog up because I could not find any useful information for this when searching.  The patching process is a pain and I might go into more details on it in more blogs later.  Anyone else ran into this issue?

The program that enables listing and modification on the expiration parameters is ‘chage’. Each individual user can view their account settings as shown below.
testuser@testServer:~$ chage -l testuser

Last password change					: Aug 07, 2009
Password expires					: Nov 05, 2009
Password inactive					: never
Account expires						: Aug 05, 1992
Minimum number of days between password change		: 90
Maximum number of days between password change		: 90
Number of days of warning before password expires	: 7

As you can see above, the last password change date is listed, as well as the expiration date for the current password. When executed from a non-privileged account, the user can only view their own account.

testuser@testServer:~$ chage -l root
chage: Permission denied.

Also, the non-privileged account can not change their settings either.

testuser@testServer:~$ chage -M 99 testuser
chage: Permission denied.

From the root account, you have to ability to modify all the settings for individual users.

root@testServer:~# chage
Usage: chage [options] [LOGIN]

Options:
  -d, --lastday LAST_DAY        set last password change to LAST_DAY
  -E, --expiredate EXPIRE_DATE  set account expiration date to EXPIRE_DATE
  -h, --help                    display this help message and exit
  -I, --inactive INACTIVE       set password inactive after expiration
                                to INACTIVE
  -l, --list                    show account aging information
  -m, --mindays MIN_DAYS        set minimum number of days before password
                                change to MIN_DAYS
  -M, --maxdays MAX_DAYS        set maximim number of days before password
                                change to MAX_DAYS
  -W, --warndays WARN_DAYS      set expiration warning days to WARN_DAYS

Before modification, I am going to turn off all expiration settings on the ‘testuser’ account. This is disabling password expiration on that individual account.

root@testServer:~# chage -E -1 -I -1 -m 0 -M 99999 testuser

No lets configure password aging for the test user. The first example below runs change in interactive mode.

root@testServer:~# chage testuser
Changing the aging information for testuser
Enter the new value, or press ENTER for the default

	Minimum Password Age [0]:
	Maximum Password Age [99999]: 90
	Last Password Change (YYYY-MM-DD) [2009-10-16]:
	Password Expiration Warning [7]:
	Password Inactive [-1]:
	Account Expiration Date (YYYY-MM-DD) [1969-12-31]: 2012-12-31

Verify that the settings took.

root@testServer:~# chage -l testuser
Last password change					: Oct 16, 2009
Password expires					: Jan 14, 2010
Password inactive					: never
Account expires						: Dec 31, 2012
Minimum number of days between password change		: 0
Maximum number of days between password change		: 90
Number of days of warning before password expires	: 7

The same can be accomplished using the command line, non-interactively.

root@testServer:~# chage -E 2012-12-31 -I -1 -m 0 -M 90 -W 7 testuser

With the above settings in place, the user should be warned 7 days before the password expires on their account. If the password is not changed before expiration day, on the next login the user will be forced to change their password.

Per RedHat,

The TLB is a limited hardware resource, so utilising a huge amount of physical memory with the default page size consumes the TLB and adds processing overhead – many pages of size 4096 Bytes equates to many TLB resources consumed.

This is where Huge Pages come in. Pages are created at a larger size than the default 4096 bytes, and each page will consume only one TLB resource. So you can see this is a huge benefit. Using Huge Pages decrease the number of TLB resources required.

Side Affect
This is great, depending on what you are trying to accomplish. Once the physical memory is mapped to a Huge Page, it can no longer be used for “normal” memory allocation. This is because the memory is no longer mapped by the kernel virtual memory manager. The applications that you want to dedicate the Huge Pages to have to have support for them.

Benefit
So here is the best part of Huge Pages. It is dedicated memory to be used by only applications that request them. This dedicated memory is stored in physical RAM and will NEVER be swapped out! Thus, guaranteeing a level of performance. When memory is swapped to disk, it’s a lot slower than RAM and decreases the performance of the process(s)/program(s) gets pushed there.

Now knowing that Huge Pages are stored in RAM, this also means that the allocated RAM is dedicated. This is a little bit redundant to the above, but I want to make sure this point is clear.

Example: If a server has 8gigs of RAM and 5gigs are allocated to Huge Pages, that only leaves 3gigs for all other processes, programs, and underlining operating system to use.

Below shows my Linux desktop that has the default page size of 4096 set

user@workstation:~$ cat /proc/meminfo | grep Huge
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       4096 kB

So as you can see, I have no Huge Pages reserved or in use. The next example is from a production Oracle database server

[root@OracleServer1 ~]# cat /proc/meminfo | grep Huge
HugePages_Total: 12200
HugePages_Free:     85
Hugepagesize:     2048 kB

So to calculate the space dedicated to Huge Pages from above, it is 12,200 x 2048 kB which gives us

24 985 600 kilobytes = 23.828125 gigabytes

In the 2.6x Linux kenel, Huge Pages are enabled using the CONFIG_HUGETLB_PAGE feature when compiling the kernel. Most “Enterprise” Linux OSs by default have this enabled. The ones that I know of are RedHat, CentOS, and possibly Fedora from version 4+.

Notes: Again, applications that you want to dedicate Huge Pages to must have support for them. Most memory intensive ones do, but check for this first.

All of this will not be detailed here but will be split up in 3 blogs. Below just shows the time it takes to use Linux to overwrite the disk device using /dev/urandom. This is done to make it just that much harder for a would be hacker to try and brute force the key on the encrypted partition. If this is not done, the un-used space would just show up as empty, allowing for a more targeted attack against the pseudo random filesystem. Being pseudo-random means that it is not truly random. This being the case, with a lot of time and computing power, an attacker might be able to either brute force or find a pattern in the encryption.

So why not use /dev/random? For me, this would take forever! I do not have any special hardware or scripts pulling information from the environment and adding to the entropy pool. The data on this drive not being national security grade, /dev/random will do the job.

I know that the drive is under /dev/sdb. With that information, it is as simple as using “dd” (built in Linux utility) to overwrite all blocks on the drive with pseudo-random data.

root@tstbox:~# dd if=/dev/urandom of=/dev/sdb
dd: writing to `/dev/sdb': No space left on device
312581810+0 records in
312581809+0 records out
160041886208 bytes (160 GB) copied, 40284.5 s, 4.0 MB/s

From above, it tool 40,284.5 seconds to overwrite the drive with urandom data. This equals ~11 hours and 19 minutes. Definitely still a long time, but a lot faster than if /dev/random was used.

This workstation is not an impressive computer. It is a single CPU dual core machine with 2 gigs of ram. Below is the info on one of the cores.

root@tstbox:~# cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 15
model name	: Intel(R) Core(TM)2 CPU          6300  @ 1.86GHz
stepping	: 2
cpu MHz		: 1867.000
cache size	: 2048 KB
physical id	: 0
siblings	: 2

Notes: I wish I could use /dev/random and probably will eventually when I can sit a drive out for a week. Setting up external drives in this fashion is really geared towards data protection. Not only are the drives in a mirrored RAID (one can fail and everything would still run fine), important data is encrypted using a strong key. So who cares is the external enclosure walks away at a conference? I would be out ~190$ but the data will be safe.

Create a test file

user01@testsrv:~$ touch testfile

Below, “ls” is used to display the inode number (2009418) of the test file

user01@testsrv:~$ ls -i /home/user01/testfile
2009418 /home/user01/testfile
Check the inode record for the newly created empty file

user01@testsrv:~$ stat /home/user01/testfile
File: `/home/user01/testfile'
Size: 0         	Blocks: 0          IO Block: 4096   regular empty file
Device: fe01h/65025d	Inode: 2009418     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-04-10 17:52:59.000000000 -0400
Modify: 2009-04-10 17:52:58.000000000 -0400
Change: 2009-04-10 17:52:58.000000000 -0400

In the following example, a file is being checked that is solely owned by the root user

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 root root 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Review the root owned ISO file inode information. The user and group ownership is referenced as Uid and Gid respectively

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-03-28 13:15:33.000000000 -0400

Change the permissions of the ISO file via the root user (current owner of the file)

root@testsrv:/home/user01# chown user01:user01 /home/user01/w2k3-2.iso

Re-check tje ISO file permission from the initial user (user01) via the ls command

user01@testsrv:~$ ls -la /home/user01/w2k3-2.iso
-rw-r--r-- 1 user01 user01 171769856 2009-03-28 13:15 /home/user01/w2k3-2.iso

Re-stat the ISO file to view the changes in the inode record. As you can see, the file ownership is now set to user user01 and group user01

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: ( 1000/user01)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 17:57:35.000000000 -0400

For the next example user01 is kept as the user, but the group is changed to ‘admin’. Since the current user user01 now has full rights to the file, it can be done from that account

user01@testsrv:~$ chown :admin /home/user01/w2k3-2.iso

Re-stat the file once more to see if the group (Gid) has been changed in the inode for this file

user01@testsrv:~$ stat /home/user01/w2k3-2.iso
File: `/home/user01/w2k3-2.iso'
Size: 171769856 	Blocks: 335824     IO Block: 4096   regular file
Device: fe01h/65025d	Inode: 2009538     Links: 1
Access: (0644/-rw-r--r--)  Uid: ( 1000/user01)   Gid: (  119/   admin)
Access: 2009-03-28 13:11:44.000000000 -0400
Modify: 2009-03-28 13:15:33.000000000 -0400
Change: 2009-04-10 18:00:32.000000000 -0400

Now with the basic understanding of finding an retrieving inode numbers and information, we will proceed to removing files using the “rm” command. Specifically, the following will detail how to remove a file by referencing the inode number. All of which is done by using “find” and passing the results to “rm”.

There have only been a few occasions that I can remember where I have needed to know the following, but it is worth knowing! The main benefit is when a file has been created with a control character in the name

Create a test file

user01@testsrv:/tmp/testdir$ touch '`'
user01@testsrv:/tmp/testdir$ ls
`

Try to remove the file without enclosing the “`” filename

user01@testsrv:/tmp/testdir$ rm `
>

Above you can see that the command did not work. Instead it dropped to the “>” sub-prompt

Locate the inode number for the “`” file. Tab based auto-complete does wonders escaping the special character filename on the command line

user01@testsrv:/tmp/testdir$ ls -i /tmp/testdir/`
1163285 /tmp/testdir/`

Now that the inode number is known, the “find” command can be used to retrieve the filename associated with it

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285
/tmp/testdir/`

Putting it all together, we can remove the file passing the “find” results to “rm”. Below uses a command line pipe (“|”) and xargs to accomplish this

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 | xargs rm

Verify the file has been removed

user01@testsrv:/tmp/testdir$ ls /tmp/testdir/

The same can be accomplished via find commands built-in exec feature

user01@testsrv:/tmp/testdir$ find /tmp -inum 1163285 -exec rm {} ;

In my opinion, the best way to remove a file via the inode number is as follows

user01@testsrv:~$ find /tmp -inum 1163285 -exec rm -i {} ;
rm: remove regular empty file `/tmp/testdir/`'? y

Specifying “-i” after rm makes the user verify the file to be removed.

Notes: There are other ways around removing files with certain characters. In this chase, the “`” file could have been removed with:

user01@testsrv:/tmp/testdir$ rm /tmp/testdir/`

If any information in this post is found to be outdated, incorrect, useful, or needs further detail, please leave a comment or email me.

So here is the really bad part. The inodes on ext2 and ext3 (Linux default type) are statically set when the file system is formatted. You can not go back and change the max inode settings. The exceptions to this that I know of are as follows:

- Reiser4
- VxFS
- XFS
- JFS
- WAFL (NetApp proprietary)
- XZFS

If you are running one of the above and have max inodes issue, you can correct it.

I have been working with computers for over 15 years and have only ran into this problem once. Luckily, it occurred on a NetApp NAS device that had the ability to increase this value on the live file system. The main killer here are tons of small files. In this case, the file system for that NFS share was 40 gigabytes in size and default was ~1 million inode limit. The quick fix for the issue was to increase this to 3 million.

As far as a ext2 and 3 go, the following shows how to query a file system for relevant inode information

root@testbox:~# tune2fs -l /dev/sda1
tune2fs 1.41.3 (12-Oct-2008)
Filesystem volume name:   <none>
Last mounted on:          <not available>
Filesystem UUID:          56161dd8-9d1d-4c54-851d-938bb88ce6d4
Filesystem magic number:  0xEF53
Filesystem revision #:    1 (dynamic)
Filesystem features:      has_journal ext_attr resize_inode dir_index filetype needs_recovery sparse_super large_file
Filesystem flags:         signed_directory_hash
Default mount options:    (none)
Filesystem state:         clean
Errors behavior:          Continue
Filesystem OS type:       Linux
Inode count:              4685824
Block count:              18731782
Reserved block count:     936589
Free blocks:              15534374
Free inodes:              4459463
First block:              0
Block size:               4096
Fragment size:            4096
Reserved GDT blocks:      1019
Blocks per group:         32768
Fragments per group:      32768
Inodes per group:         8192
Inode blocks per group:   256
Filesystem created:       Mon Sep 29 16:25:20 2008
Last mount time:          Fri Jan 23 14:27:02 2009
Last write time:          Fri Jan 23 14:27:02 2009
Mount count:              4
Maximum mount count:      33
Last checked:             Thu Jan 15 09:00:37 2009
Check interval:           15552000 (6 months)
Next check after:         Tue Jul 14 10:00:37 2009
Reserved blocks uid:      0 (user root)
Reserved blocks gid:      0 (group root)
First inode:              11
Inode size:	          128
Journal inode:            8
First orphan inode:       2908742
Default directory hash:   tea
Directory Hash Seed:      a6544c5xxxxxxxxxxxxxxxxx
Journal backup:           inode blocks

The above is good to know, especially to check an un-mounted file system. The command below shows a friendlier formatted output

root@testbox:~# df -i
Filesystem            Inodes   IUsed   IFree IUse% Mounted on
/dev/sda1            4685824  226361 4459463    5% /
tmpfs                 222201       4  222197    1% /lib/init/rw
varrun                222201      64  222137    1% /var/run
varlock               222201       5  222196    1% /var/lock
udev                  222201    5142  217059    3% /dev
tmpfs                 222201       5  222196    1% /dev/shm
/dev/sdb1            61063168    1116 61062052    1% /media/disk

As you can see, there are no issues to be worried about on this test computer. Most systems administrators perform centralized monitoring of disk usage at a disk space level (capacity). On highly used servers that utilize locally stored and/or direct attached storage, it is a good idea to have a script check and report on the available inodes.

PDF Version

Notes: This information is provided for a high level overview concerning inodes. More in-depth information will be provided in up-coming posts.

[root@tftpsrv ~]# yum install tftp
Resolving Dependencies
--> Running transaction check
---> Package tftp-server.i386 0:0.42-3.1.el5.centos set to be updated
--> Processing Dependency: xinetd for package: tftp-server
--> Running transaction check
---> Package xinetd.i386 2:2.3.14-10.el5 set to be updated
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================
 Package                 Arch       Version          Repository        Size
=============================================================================
Installing:
 tftp-server             i386       0.42-3.1.el5.centos  base               27 k
Installing for dependencies:
 xinetd                  i386       2:2.3.14-10.el5  base              124 k

Transaction Summary
=============================================================================
Install      2 Package(s)
Update       0 Package(s)
Remove       0 Package(s)         

Total download size: 151 k
Is this ok [y/N]: y
Downloading Packages:
(1/2): tftp-server-0.42-3 100% |=========================|  27 kB    00:00
(2/2): xinetd-2.3.14-10.e 100% |=========================| 124 kB    00:00
Running rpm_check_debug
Running Transaction Test
Finished Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing: xinetd                       ######################### [1/2]
  Installing: tftp-server                  ######################### [2/2]

Installed: tftp-server.i386 0:0.42-3.1.el5.centos
Dependency Installed: xinetd.i386 2:2.3.14-10.el5
Complete!

Edit configuration to enable tftp

[root@tftpsrv ~]# vi /etc/xinetd.d/tftp
# default: off
# description: The tftp server serves files using the trivial file transfer \
#       protocol.  The tftp protocol is often used to boot diskless \
#       workstations, download configuration files to network-aware printers, \
#       and to start the installation process for some operating systems.
service tftp
{
        socket_type             = dgram
        protocol                = udp
        wait                    = yes
        user                    = root
        server                  = /usr/sbin/in.tftpd
        server_args             = -s /tftpboot        -> some directory (/tftpfiles)
        disable                 = yes            -> no
        per_source              = 11
        cps                     = 100 2
        flags                   = IPv4
}

Create directory specified in tftp configuration file

[root@tftpsrv xinetd.d]# mkdir /tftpfiles

Start up xinetd. This is used to call tftp

[root@tftpsrv ~]# /etc/init.d/xinetd start
Starting xinetd:                                           [  OK  ]

[root@tftpsrv xinetd.d]# iptables-save > /etc/init.d/iptables
[root@tftpsrv xinetd.d]# iptables -F

Below, the tftp put will fail. This is due to the file needing to be created on the TFTP server before the client can write to it. This is the only real security there is to TFTP. You at least need to know the filename before the file can be written or read.

C9124SW5# copy running-config tftp:CISCSCOCFG1
Enter hostname for the tftp server: 172.16.100.6
Trying to connect to tftp server......

TFTP put operation failed:Undefined error code (2)

Create the file to be saved from switch and change the permissions

[root@tftpsrv ~]# touch /tftpfiles/CISCSCOCFG1
[root@tftpsrv ~]# chmod 777 /tftpfiles/CISCSCOCFG1

Tell the switch to save the file

C9124SW5# copy running-config tftp:CISCSCOCFG1
Enter hostname for the tftp server: 172.16.100.6
Trying to connect to tftp server......
|
TFTP put operation was successful

Check the services file to find the TFTP port and protocol information

[root@tftpsrv]# cat /etc/services | grep tftp
tftp        69/tcp
tftp        69/udp

Bring the firewall back up so we can insert rules to allow TFTP in

[root@tftpsrv]# /etc/init.d/iptables restart

On my test server, the firewall chain is “RH-Firewall-1-INPUT”. I always prefer inserting new firewall rules as the first rule. Most servers keep a few custom reject rules and most are explicit allow with the default deny at the end. Inserting the new rule as the first will normally bypass those that might reject before it ever gets to the tftp rule.

[root@tftpsrv]# iptables -I RH-Firewall-1-INPUT 1 -s 172.16.100.98 -p tcp --dport 69 -j ACCEPT
[root@tftpsrv]# iptables -I RH-Firewall-1-INPUT 1 -s 172.16.100.98 -p udp --dport 69 -j ACCEPT

The above statements tell iptables to insert “-I” the new rule into the chain “RH-Firewall-1-INPUT” as rule number “1″. The -s is specifying the source, -p the protocol –dport the destination port and -j allows the connection to establish by jumping over to ACCEPT.

Verify the rules are there

[root@tftpsrv]# iptables -L
Chain RH-Firewall-1-INPUT
target     prot opt source               destination
ACCEPT     udp  --  172.16.100.98        anywhere            udp dpt:tftp
ACCEPT     tcp  --  172.16.100.98        anywhere            tcp dpt:tftp

Save the rules in sysconfig so they will be persistent through reboots

[root@tftpsrv]# iptables-save > /etc/sysconfig/iptables

Notes: Never flush your iptables rules “iptables -F” on production systems that are not protected by a firewall or are on are public IP. Always be sure to backup/save your iptables configuration when testing. Also, if you are not familiar with security, or there is someone else responsible for security in the company, as them before or have them modify the local iptables rules. Another good rule for servers running TFTP, FTP, Telnet, DNS, and mail is to have servers dedicated for each. These are some of the most exploited servers out there.

Using scponly “make jail” command to setup the initial user. I removed most of the generic output from the command.

[root@testserver01 scponly-4.8]# make jail
/usr/bin/install -c -d /usr/local/bin
/usr/bin/install -c -d /usr/local/man/man8
/usr/bin/install -c -d /usr/local/etc/scponly
/usr/bin/install -c -o 0 -g 0 scponly /usr/local/bin/scponly
/usr/bin/install -c -o 0 -g 0 -m 0644 scponly.8 /usr/local/man/man8/scponly.8
/usr/bin/install -c -o 0 -g 0 -m 0644 debuglevel /usr/local/etc/scponly/debuglevel
if test "xscponlyc" != "x"; then
		/usr/bin/install -c -d /usr/local/sbin;
		rm -f /usr/local/sbin/scponlyc;
		cp scponly scponlyc;
		/usr/bin/install -c -o 0 -g 0 -m 4755 scponlyc /usr/local/sbin/scponlyc;
	fi
chmod u+x ./setup_chroot.sh
./setup_chroot.sh

Username to install [scponly]newact
home directory you wish to set for this user [/home/newact]
name of the writeable subdirectory [incoming]

creating  /home/newact/incoming directory for uploading files

Your platform (Linux) does not have a platform specific setup script.
This install script will attempt a best guess.
If you perform customizations, please consider sending me your changes.
Look to the templates in build_extras/arch.
 - joe at sublimation dot org

please set the password for newact:
Changing password for user newact.
New UNIX password:
Retype new UNIX password:
passwd: all authentication tokens updated successfully.

Now that the user is created, lets test the sftp session from a different system

user1@workstation03:~/.ssh$ sftp newact@10.1.3.43
Connecting to 10.1.3.43...
newact@10.1.3.43's password:
Connection closed

So we see that the connection failed. The reason here is that there is no /dev/null device within the users chrooted home (jail). Scponly does not auto-create this needed device.

[root@testserver01 scponly-4.8]# mkdir -p /home/newact/dev/
[root@testserver01 scponly-4.8]# cp -a /dev/null /home/newact/dev/

After re-testing the connection, you can see that everything is now functioning fine. I was able to push a file to the incoming folder in the working chrooted (jailed) environment.

user1@workstation03:~/.ssh$ sftp newact@10.1.3.43
Connecting to 10.1.3.43...
newact@10.1.3.43's password:
sftp> ls
dev       etc       incoming  lib       usr
sftp> cd incoming
sftp> put testfile
Uploading testfile to /incoming/testfile
testfile                                                                                                100%  885     0.9KB/s   00:00
sftp> exit

Notes: For each user account that you create with scponly chrooting scripts, you will need to create the dev directory, as well as the null device under the users home directory.  This is definitely something that can be manually added to the setup_chroot.sh easily.

Original hostname:	newhn.testdomain.com
Original IP:		10.1.1.65

Make sure to edit the host file! If not, this can real havoc on daemons that bind to network ports.

[root@orighn ssh]# vi /etc/hosts
# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
10.1.1.65            orighn.testdomain.com orighn

Change to

# Do not remove the following line, or various programs
# that require network functionality will fail.
127.0.0.1               localhost.localdomain localhost
::1             localhost6.localdomain6 localhost6
10.1.1.40            newhn.testdomain.com newhn

If you want the changes to be persistent through a reboot, the stored configuration files under /etc/sysconfig also need to be modified.

Change the hostname to newhn. If the server was being moved to a different subnet, or needed to route though another gateway, GATEWAY would need to be changed here as well.

[root@orighn ssh]# vi /etc/sysconfig/network
NETWORKING=yes
NETWORKING_IPV6=yes
HOSTNAME=orighn.testdomain.com	-> newhn.testdomain.com
GATEWAY=10.1.1.1

Each network port has a configuration script prefixed with ifcfg-*. Depending on your setup, the server could have a different IP per port or have a bonded port setup. In those cases, you will have to determine which file needed to be modified by viewing each one. This example changes the ip for the eth0 port(1 on the back of the server).

[root@orighn ssh]# vi /etc/sysconfig/network-scripts/ifcfg-eth0
# Advanced Micro Devices [AMD] 79c970 [PCnet32 LANCE]
DEVICE=eth0
BOOTPROTO=static
DHCPCLASS=
HWADDR=00:50:56:A0:02:B2
IPADDR=10.1.1.65			-> 10.1.1.40
NETMASK=255.255.255.0
ONBOOT=yes

Changes are done, last thing to do is restart the system to make sure changes are persistent.

[root@orighn ssh]# reboot -f
Ok, reboot and watch pings
kgoodman@kdesk:~$ ping 10.1.1.40
PING 10.1.1.40 (10.1.1.40) 56(84) bytes of data.
64 bytes from 10.1.1.40: icmp_seq=35 ttl=61 time=8.09 ms
64 bytes from 10.1.1.40: icmp_seq=36 ttl=61 time=1.69 ms
64 bytes from 10.1.1.40: icmp_seq=37 ttl=61 time=1.93 ms
64 bytes from 10.1.1.40: icmp_seq=38 ttl=61 time=1.74 ms
64 bytes from 10.1.1.40: icmp_seq=39 ttl=61 time=3.19 ms
^C
--- 10.1.1.40 ping statistics ---
39 packets transmitted, 5 received, 87% packet loss, time 38240ms
rtt min/avg/max/mdev = 1.696/3.333/8.095/2.443 ms

It came up! SSH to the server and check to make sure the hostname change to effect as well.

[root@newhn ~]# hostname
newhn.testdomain.com

Verify the ip information as well.

[root@newhn ~]# ifconfig eth0
eth0      Link encap:Ethernet  HWaddr 00:50:56:A0:02:B2
inet addr:10.1.1.40  Bcast:192.168.0.255  Mask:255.255.255.0
inet6 addr: fe80::250:56ff:fea0:2b2/64 Scope:Link
UP BROADCAST RUNNING MULTICAST  MTU:1500  Metric:1
RX packets:407 errors:0 dropped:0 overruns:0 frame:0
TX packets:433 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:34117 (33.3 KiB)  TX bytes:48091 (46.9 KiB)
Interrupt:177 Base address:0x1400

Notes: Depending on what you call Linux, some systems will not have the /etc/sysconfig/ directory structure. This is due to some flavors being toted as a Linux distribution actually being derived from Unix/BSD systems.