Colocation to Virtualization

I have been setting up and integrating an LDAP authentication system into our infrastructure over the past few days. This is just one small “got-cha” that I ran into. The default setting in the OpenLDAP configuration (/etc/ldap.conf) is to continuously try reconnecting to the LDAP server on failure. This is definitely not what I want to happen if we loose LDAP. In this scenario, when connecting to the server via SSH, the session will hang and eventually timeout. This even removes the ability to login with a local system account.
Example of the timeout when LDAP server is down:

testuser@workstation4-l:~$ ssh test123@ldapclientsrv
Connection closed by 172.16.0.192

To begin, lets look at a typical error that you would get on the system if LDAP communication was down.

Dec 13 12:52:58 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:52:58 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 4 seconds)...
Dec 13 12:53:02 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:53:02 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 8 seconds)...
Dec 13 12:53:10 ldapServer sshd[15965]: nss_ldap: failed to bind to LDAP server ldap://127.0.0.16: Can't contact LDAP server
Dec 13 12:53:10 ldapServer sshd[15965]: nss_ldap: reconnecting to LDAP server (sleeping 16 seconds)...

As noted before, I was unable to login with a local account. Turns out that the problem was with the default “bind_policy” in /etc/ldap.conf.  Per the document:

# Reconnect policy: hard (default) will retry connecting to
# the software with exponential backoff, soft will fail
# immediately.
#bind_policy hard

This was changed to:

bind_policy soft

Once this was changed, I brought up the firewall on the LDAP server and refused connections. Ability to login via LDAP was gone, but the server did fail back to local system authentication

Note(s): When failing back to local authentication, there is no error sent back to the client trying to login, only errors go to /var/log/secure file. The server will just keep rejecting the users login until LDAP is back up. At least this gives you the ability to get in with a local system account in an emergency.

Example error to /var/log/secure when LDAP server is down and local authentication is rejecting the LDAP user received from the client:

Dec 13 12:59:59 ldapServer sshd[2588]: pam_unix(sshd:auth): authentication failure; logname= uid=0 euid=0 tty=ssh ruser= rhost=172.16.0.22

~ by Kevin Goodman on December 16, 2009.

Posted in Linux, Networking, Security
Tags: authentication, centos, failed to bind to LDA, fallback, fedora, ldap, ldap.conf, LinkedIn, Linux, local authentication, nss_ldap, openldap, redhat, rh