Colocation to Virtualization

After installing a new license key for Splunk, the next step is to restart the Splunk service.  I did so from the web interface for Splunk and it eventually timed out.

I tried to stop the service via splunk command line, but that failed.
[user@splunk ~]# /opt/splunk/bin/splunk stop
splunkweb is not running.                                 [FAILED]
Stopping splunkd. This can take a few minutes.  Please be patient…
… still shutting down …
… still shutting down …

I used ctrl+c to cancel the “splunk stop” since it was not getting anywhere.  A ps reveals that splunk is still currently running
[user@splunk ~]# ps -auxc | grep splunk
user      9409 75.9 30.0 386256 154812 ?       Sl   10:09   0:55 splunkd
user      9410  0.3  0.2  15172  1464 ?        Ss   10:09   0:00 splunkd
user      9429  0.0  0.1  15172   576 ?        S    10:09   0:00 splunkd
user      9604 13.2  3.1  63188 16184 ?        RNs  10:09   0:06 splunk-optimize
user      9873 11.0  4.9  30552 25668 ?        RNs  10:10   0:03 splunk-optimize
user      9963  9.3  5.9  56124 30804 ?        RNs  10:10   0:00 splunk-optimize

Killed the process using -9
[user@splunk ~]# kill -9 9409

Went to restart Splunk and it failed
[user@splunk ~]# /opt/splunk/bin/splunk start
splunkd 9409 was not running.
Stopping splunk helpers. This can take a few minutes.  Plea[  OK  ]tient…
Stopped helpers.
Removing stale pid file… done.
Checking prerequisites…
Checking http port [8000]: already bound
ERROR: The http port [8000] is already bound.  Splunk needs to use this port.
Would you like to change ports? [y/n]:

Ok, so something still currently has the port bound.  Lets find out.  I used lsof to do so
[user@splunk ~]# lsof -i TCP:8000
COMMAND  PID USER   FD   TYPE  DEVICE SIZE NODE NAME
python  9521 user   15u  IPv4 1867658       TCP *:irdmi (LISTEN)

So it looks like splunk stats a python listener to grab the incoming HTTP requests.  Time to kill that also
[user@splunk ~]# kill -9 9521

This time the restart works and the system is back online with the new license.
[user@splunk ~]# /opt/splunk/bin/splunk start
Checking prerequisites…
Checking http port [8000]: open
Checking mgmt port [8089]: open
Verifying configuration.  This may take a while…
Finished verifying configuration.
Checking index directory…
Verifying databases…
Verified databases: _audit, _blocksignature, _internal, _thefishbucket, history, mail, main, sampledata, splunklogger, summary

Checking for SELinux.
Checking index files
All index checks passed.
All preliminary checks passed.
Starting splunkd…                                        [  OK  ]
PID 9521 was not running.  removing stale pid file… done.
Starting splunkweb…                                      [  OK  ]
Splunk Server started.

The Splunk web interface is at https://splunk.x.com:8000
If you get stuck, we’re here to help.  Feel free to email us at ’support@splunk.com’.

Note:  I love lsof to locate what process is bound to a port.  I will be doing a blog just on troubleshooting with lsof soon.