Thursday, December 1, 2016

Exploring Splunk Light




Use case:


I have a product deployment which spans multiple machines. And I am running some test suites. When there is an error in the test suite, its easy to debug and fix when the issue is directly related to the tests.
But sometimes the root cause is the test environment. Some of the dependent services might go down for a short duration. And the test could fail due to this.

The test suite bundle all the logs from all the boxes into a zip file. The usual approach is to unzip this bundle and grep(recursive) the log files. But this approach is time consuming and cumbersome.

I have tried Upload zip file option in Splunk light and found it very helpful.

Splunk allows to upload zip files less than 500MB. And I can narrow down the search to a specific time window. For example, consider log entries within ten minutes window.
Here is the link to documentation:

http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Uploaddata


Install Splunk



# rpm -ivh splunklight-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm

warning: splunklight-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 653fb112: NOKEY
Preparing...                ########################################### [100%]
   1:splunk                 ########################################### [100%]
complete

[# ps -ef | grep splunk
root      9652  8618  0 22:31 pts/11   00:00:00 grep splunk


# cd  /opt/splunk/bin
# sudo su splunk



$ ./splunk start --accept-license




 This appears to be your first time running this version of Splunk.
 Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
 Generating RSA private key, 1024 bit long modulus
 ..++++++
 .............++++++
 e is 65537 (0x10001)
 writing RSA key

 Generating RSA private key, 1024 bit long modulus
 ...............++++++
 ..........................++++++
 e is 65537 (0x10001)
 writing RSA key

 Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.

 Splunk> Be an IT superhero. Go home early.

 Checking prerequisites...
         Checking http port [8000]: already bound
 ERROR: The http port [8000] is already bound.  Splunk needs to use this port.
 Would you like to change ports? [y/n]: y
 Enter a new http port: 1111
 Setting http to port: 1111
 The server's web port has been changed.
 You need to restart the Splunk Web Server (splunkweb) for your changes to take effect.
         Checking http port [1111]: open
         Checking mgmt port [8089]: open
         Checking appserver port [127.0.0.1:8065]: open
         Checking kvstore port [8191]: open
         Checking configuration...  Done.
                 Creating: /opt/splunk/var/run/splunk/appserver/i18n
                 Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
                 Creating: /opt/splunk/var/run/splunk/upload
                 Creating: /opt/splunk/var/spool/splunk
                 Creating: /opt/splunk/var/spool/dirmoncache
                 Creating: /opt/splunk/var/lib/splunk/authDb
                 Creating: /opt/splunk/var/lib/splunk/hashDb
         Checking critical directories...        Done
         Checking indexes...
                 Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
         Done
 New certs have been generated in '/opt/splunk/etc/auth'.
         Checking filesystem compatibility...  Done
         Checking conf files for problems...
         Done
         Checking default conf files for edits...
         Validating installed files against hashes from '/opt/splunk/splunklight-6.5.0-59c8927def0f-linux-2.6-x86_64-manifest'
         All installed files intact.
         Done
 All preliminary checks passed.

 Starting splunk server daemon (splunkd)...
 Generating a 1024 bit RSA private key
 ...................................................................++++++
 ............++++++
 writing new private key to 'privKeySecure.pem'



 -----
 Signature ok
 subject=/CN=slc02oxu/O=SplunkUser
 Getting CA Private Key
 writing RSA key
 Done
                                                            [  OK  ]

 Waiting for web server at http://127.0.0.1:1111 to be available.... Done


 If you get stuck, we're here to help.
 Look for answers here: http://docs.splunk.com

 The Splunk web interface is at http://:1111



 Login to Splunk :  http://hostname:1111

credentials:  admin/changeme

Add Data


Go to "Add Data" link







Select File


Click on "Select  File" Button and select the zip file.



Input Settings


I left the input settings to default





Review Page











Upload Zip and Index


Now the indexing will start and it will take few minutes based on the size of the zip. And Splunk supports upto 500MB zip file.





Search



Once indexing is over, go to Search Tab.






Click on the hostname entry created while uploading the zip

Now this filter gets added to the Search bar.







There could be millions of entries in the logs, so first thing is to narrow down to the time window that you are interested in. I have looked at the time window of the error in my test file and selected the time
window.





Now the search results will show relevant entries from all the logs for the selected time window. This is a coll feature as it helps to focus on the errors for the specific 10 minute window or so. Splunk converts the time in log files to UTC. So you need to select the time window accordingly.





Even now we could have large number of logs to look at. The Pattern tab helps to deduplicate the errors and it shows the unique error messages based on the number of occurrence of the message in the logs.