Use case:
I have a product deployment which spans multiple machines. And I am running some test suites. When there is an error in the test suite, its easy to debug and fix when the issue is directly related to the tests.
But sometimes the root cause is the test environment. Some of the dependent services might go down for a short duration. And the test could fail due to this.
The test suite bundle all the logs from all the boxes into a zip file. The usual approach is to unzip this bundle and grep(recursive) the log files. But this approach is time consuming and cumbersome.
I have tried Upload zip file option in Splunk light and found it very helpful.
Splunk allows to upload zip files less than 500MB. And I can narrow down the search to a specific time window. For example, consider log entries within ten minutes window.
Here is the link to documentation:
http://docs.splunk.com/Documentation/Splunk/6.5.1/Data/Uploaddata
Install Splunk
# rpm -ivh splunklight-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm
warning: splunklight-6.5.0-59c8927def0f-linux-2.6-x86_64.rpm: Header V4 DSA/SHA1 Signature, key ID 653fb112: NOKEY
Preparing... ########################################### [100%]
1:splunk ########################################### [100%]
complete
[# ps -ef | grep splunk
root 9652 8618 0 22:31 pts/11 00:00:00 grep splunk
# cd /opt/splunk/bin
# sudo su splunk
$ ./splunk start --accept-license
This appears to be your first time running this version of Splunk.
Copying '/opt/splunk/etc/openldap/ldap.conf.default' to '/opt/splunk/etc/openldap/ldap.conf'.
Generating RSA private key, 1024 bit long modulus
..++++++
.............++++++
e is 65537 (0x10001)
writing RSA key
Generating RSA private key, 1024 bit long modulus
...............++++++
..........................++++++
e is 65537 (0x10001)
writing RSA key
Moving '/opt/splunk/share/splunk/search_mrsparkle/modules.new' to '/opt/splunk/share/splunk/search_mrsparkle/modules'.
Splunk> Be an IT superhero. Go home early.
Checking prerequisites...
Checking http port [8000]: already bound
ERROR: The http port [8000] is already bound. Splunk needs to use this port.
Would you like to change ports? [y/n]: y
Enter a new http port: 1111
Setting http to port: 1111
The server's web port has been changed.
You need to restart the Splunk Web Server (splunkweb) for your changes to take effect.
Checking http port [1111]: open
Checking mgmt port [8089]: open
Checking appserver port [127.0.0.1:8065]: open
Checking kvstore port [8191]: open
Checking configuration... Done.
Creating: /opt/splunk/var/run/splunk/appserver/i18n
Creating: /opt/splunk/var/run/splunk/appserver/modules/static/css
Creating: /opt/splunk/var/run/splunk/upload
Creating: /opt/splunk/var/spool/splunk
Creating: /opt/splunk/var/spool/dirmoncache
Creating: /opt/splunk/var/lib/splunk/authDb
Creating: /opt/splunk/var/lib/splunk/hashDb
Checking critical directories... Done
Checking indexes...
Validated: _audit _internal _introspection _telemetry _thefishbucket history main summary
Done
New certs have been generated in '/opt/splunk/etc/auth'.
Checking filesystem compatibility... Done
Checking conf files for problems...
Done
Checking default conf files for edits...
Validating installed files against hashes from '/opt/splunk/splunklight-6.5.0-59c8927def0f-linux-2.6-x86_64-manifest'
All installed files intact.
Done
All preliminary checks passed.
Starting splunk server daemon (splunkd)...
Generating a 1024 bit RSA private key
...................................................................++++++
............++++++
writing new private key to 'privKeySecure.pem'
-----
Signature ok
subject=/CN=slc02oxu/O=SplunkUser
Getting CA Private Key
writing RSA key
Done
[ OK ]
Waiting for web server at http://127.0.0.1:1111 to be available.... Done
If you get stuck, we're here to help.
Look for answers here: http://docs.splunk.com
The Splunk web interface is at http://:1111
Login to Splunk : http://hostname
credentials: admin/changeme
Add Data
Go to "Add Data" link
Select File
Click on "Select File" Button and select the zip file.
Input Settings
I left the input settings to default
Review Page
Upload Zip and Index
Now the indexing will start and it will take few minutes based on the size of the zip. And Splunk supports upto 500MB zip file.
Search
Once indexing is over, go to Search Tab.
Click on the hostname entry created while uploading the zip
Now this filter gets added to the Search bar.
There could be millions of entries in the logs, so first thing is to narrow down to the time window that you are interested in. I have looked at the time window of the error in my test file and selected the time
window.
Now the search results will show relevant entries from all the logs for the selected time window. This is a coll feature as it helps to focus on the errors for the specific 10 minute window or so. Splunk converts the time in log files to UTC. So you need to select the time window accordingly.
Even now we could have large number of logs to look at. The Pattern tab helps to deduplicate the errors and it shows the unique error messages based on the number of occurrence of the message in the logs.
Posts
