Monday, September 25, 2017

Install ELK on Oracle Enterprise Linux (from tar.gz archives)

Here is a primer on The ELK stack https://rajiv-kuriakose.blogspot.in/2017/09/elk-stack-basics.html



Install JDK

 

 

Get the latest jdk version from 

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Accept the license and download the tar.gz relevant to your box(32 bit vs 64 bit) and operating system

bash
export http_proxy=:80
export https_proxy=:80


wget --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie;" --no-check-certificate  http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz

 

Install Elasticsearch


Go to https://www.elastic.co/downloads
Get the latest elasitcsearch version from https://www.elastic.co/downloads/elasticsearch

bash
export http_proxy=:80
export https_proxy=:80
export JAVA_HOME=`pwd`/
download and extract  latest jdk tar.gz


 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.1.tar.gz

tar -xvzf  elasticsearch-5.6.1.tar.gz

export JAVA_HOME=`pwd`/jdk1.8.0_144
bash-4.1$ echo $JAVA_HOME
/scratch/rkuriako/softwares/elk/jdk1.8.0_144

cd elasticsearch-5.6.1



Start Elasticsearch 


./bin/elasticsearch






-------------------------------------


bash-4.1$ ./bin/elasticsearch
[2017-09-22T02:40:56,277][INFO ][o.e.n.Node               ] [] initializing ...
[2017-09-22T02:40:56,413][INFO ][o.e.e.NodeEnvironment    ] [VSYSxM9] using [1] data paths, mounts [[/ (/dev/xvda2)]], net usable_space [192.1gb], net total_space [380.2gb], spins? [no], types [ext4]
[2017-09-22T02:40:56,414][INFO ][o.e.e.NodeEnvironment    ] [VSYSxM9] heap size [1.9gb], compressed ordinary object pointers [true]
[2017-09-22T02:40:56,416][INFO ][o.e.n.Node               ] node name [VSYSxM9] derived from node ID [VSYSxM9PTxmzlVRKe6BGDw]; set [node.name] to override
[2017-09-22T02:40:56,416][INFO ][o.e.n.Node               ] version[5.6.1], pid[9656], build[667b497/2017-09-14T19:22:05.189Z], OS[Linux/3.8.13-16.2.1.el6uek.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01]
[2017-09-22T02:40:56,416][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/scratch/rkuriako/softwares/elk/elasticsearch-5.6.1]
[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [aggs-matrix-stats]
[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [ingest-common]
[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-expression]
[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-groovy]
[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-mustache]
[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-painless]
[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [parent-join]
[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [percolator]
[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [reindex]
[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [transport-netty3]
[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [transport-netty4]
[2017-09-22T02:40:57,440][INFO ][o.e.p.PluginsService     ] [VSYSxM9] no plugins loaded
[2017-09-22T02:40:59,481][INFO ][o.e.d.DiscoveryModule    ] [VSYSxM9] using discovery type [zen]
[2017-09-22T02:41:00,187][INFO ][o.e.n.Node               ] initialized
[2017-09-22T02:41:00,187][INFO ][o.e.n.Node               ] [VSYSxM9] starting ...
[2017-09-22T02:41:00,455][INFO ][o.e.t.TransportService   ] [VSYSxM9] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}
[2017-09-22T02:41:00,475][WARN ][o.e.b.BootstrapChecks    ] [VSYSxM9] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]
[2017-09-22T02:41:03,543][INFO ][o.e.c.s.ClusterService   ] [VSYSxM9] new_master {VSYSxM9}{VSYSxM9PTxmzlVRKe6BGDw}{UBKHbMC2Rw-ayoMVpS8n0w}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)
[2017-09-22T02:41:03,569][INFO ][o.e.h.n.Netty4HttpServerTransport] [VSYSxM9] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}
[2017-09-22T02:41:03,570][INFO ][o.e.n.Node               ] [VSYSxM9] started
[2017-09-22T02:41:03,595][INFO ][o.e.g.GatewayService     ] [VSYSxM9] recovered [0] indices into cluster_state
[2017-09-22T02:48:42,011][INFO ][o.e.c.m.MetaDataCreateIndexService] [VSYSxM9] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_default_, index-pattern, server, visualization, search, timelion-sheet, config, dashboard, url]
[2017-09-22T03:02:55,388][INFO ][o.e.c.m.MetaDataCreateIndexService] [VSYSxM9] [logstash-2017.09.22] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]
[2017-09-22T03:02:55,481][INFO ][o.e.c.m.MetaDataMappingService] [VSYSxM9] [logstash-2017.09.22/P4anPGb6SMSxzRLq0aVoTQ] create_mapping [logs]

------------------------------

Verify that Elasticsearch is up and running


http://localhost:9200/

----------------------------------------------------------------------------------------------------------------------


Install Kibana



bash
export http_proxy=:80
export https_proxy=:80

export JAVA_HOME=`pwd`/jdk1.8.0_144


Get latest version from https://www.elastic.co/downloads/kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.1-linux-x86_64.tar.gz

cd kibana-5.6.1-linux-x86_64/


vim  config/kibana.yml

uncomment below line
elasticsearch.url: "http://localhost:9200"


Start Kibana



./bin/kibana

-------------------------------------------------
bash-4.1$ ./bin/kibana
  log   [10:02:31.777] [info][status][plugin:kibana@5.6.1] Status changed from uninitialized to green - Ready
  log   [10:02:31.867] [info][status][plugin:elasticsearch@5.6.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch
  log   [10:02:31.906] [info][status][plugin:console@5.6.1] Status changed from uninitialized to green - Ready
  log   [10:02:31.951] [info][status][plugin:metrics@5.6.1] Status changed from uninitialized to green - Ready
  log   [10:02:32.194] [info][status][plugin:elasticsearch@5.6.1] Status changed from yellow to green - Kibana index ready
  log   [10:02:32.195] [info][status][plugin:timelion@5.6.1] Status changed from uninitialized to green - Ready
  log   [10:02:32.201] [info][listening] Server running at http://localhost:5601
  log   [10:02:32.203] [info][status][ui settings] Status changed from uninitialized to green - Ready

-------------------------------------------------



Install Logstash



bash
export http_proxy=:80
export https_proxy=:80



Create sample config file


Create a simple configuration file like below.

vim logstash-simple.conf

input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}



-----------------------------

Start Logstash

bash-4.1$ bin/logstash -f logstash-simple.conf
Sending Logstash's logs to /scratch/rkuriako/softwares/elk/logstash-5.6.1/logs which is now configured via log4j2.properties
[2017-09-22T02:55:22,044][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/modules/fb_apache/configuration"}
[2017-09-22T02:55:22,050][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/modules/netflow/configuration"}
[2017-09-22T02:55:22,058][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/data/queue"}
[2017-09-22T02:55:22,059][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/data/dead_letter_queue"}
[2017-09-22T02:55:22,090][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"5493515e-08b7-44d2-a6b5-136fc1ff7f9f", :path=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/data/uuid"}
[2017-09-22T02:55:22,650][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}
[2017-09-22T02:55:22,652][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}
[2017-09-22T02:55:22,779][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}
[2017-09-22T02:55:22,780][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}
[2017-09-22T02:55:22,833][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}
[2017-09-22T02:55:22,841][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash
[2017-09-22T02:55:22,920][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}
[2017-09-22T02:55:22,922][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1250}
[2017-09-22T02:55:22,938][INFO ][logstash.pipeline        ] Pipeline main started
The stdin plugin is now waiting for input:
[2017-09-22T02:55:22,986][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

-----------------------------------------------------

Verify Logstash is up and running


To verify Logstash is configured properly, type "Hello World" on the command line and you will see output like below.

Basically logstash reads stdin and parse it and send it to stdout in the below format.










You can view the message in Kibana UI also.

Go to  http://localhost:5601


Change the time picker to "This week"







And then you can see log messages like below(scroll to bottom of the page).





 

 

 

 

 

 

 

 

 

 

 

 

 

Install Filebeat



bash
export http_proxy=:80
export https_proxy=:80

export JAVA_HOME=`pwd`/jdk1.8.0_144

Get latest version from here

https://www.elastic.co/downloads/beats/filebeat

 wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-linux-x86_64.tar.gz

tar -xvzf filebeat-5.6.1-linux-x86_64.tar.gz 

cd filebeat-5.6.1-linux-x86_64

----------------------

bash-4.1$ sudo ./filebeat -e -c filebeat.yml
Password:
filebeat2017/09/22 10:09:06.664574 beat.go:346: CRIT Exiting: error loading config file: config file ("filebeat.yml") must be owned by the beat user (uid=0) or root
Exiting: error loading config file: config file ("filebeat.yml") must be owned by the beat user (uid=0) or root


--------------------------------

sudo su

chown root filebeat.yml

ls -l filebeat.yml 
-rw------- 1 root g900 4196 Sep 14 12:13 filebeat.yml


---------
[root@den00nyk filebeat-5.6.1-linux-x86_64]# sudo ./filebeat -e -c filebeat.yml
2017/09/22 10:11:25.759761 beat.go:297: INFO Home path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64] Config path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64] Data path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/data] Logs path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/logs]
2017/09/22 10:11:25.759884 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.1
2017/09/22 10:11:25.760060 metrics.go:23: INFO Metrics logging every 30s
2017/09/22 10:11:25.760174 output.go:258: INFO Loading template enabled. Reading template file: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/filebeat.template.json
2017/09/22 10:11:25.761183 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/filebeat.template-es2x.json
2017/09/22 10:11:25.762126 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/filebeat.template-es6x.json
2017/09/22 10:11:25.763067 client.go:128: INFO Elasticsearch url: http://localhost:9200
2017/09/22 10:11:25.763145 outputs.go:108: INFO Activated elasticsearch as output plugin.
2017/09/22 10:11:25.763288 publish.go:300: INFO Publisher name: den00nyk.us.oracle.com
2017/09/22 10:11:25.763535 async.go:63: INFO Flush Interval set to: 1s
2017/09/22 10:11:25.763577 async.go:64: INFO Max Bulk Size set to: 50
2017/09/22 10:11:25.763933 beat.go:233: INFO filebeat start running.
2017/09/22 10:11:25.764012 registrar.go:68: INFO No registry file found under: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/data/registry. Creating a new registry file.
2017/09/22 10:11:25.765021 registrar.go:106: INFO Loading registrar data from /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/data/registry
2017/09/22 10:11:25.765099 registrar.go:123: INFO States Loaded from registrar: 0
2017/09/22 10:11:25.765198 crawler.go:38: INFO Loading Prospectors: 1
2017/09/22 10:11:25.765332 registrar.go:236: INFO Starting Registrar
2017/09/22 10:11:25.765449 sync.go:41: INFO Start sending events to output
2017/09/22 10:11:25.765402 prospector_log.go:65: INFO Prospector with previous states loaded: 0
2017/09/22 10:11:25.765795 prospector.go:124: INFO Starting prospector of type: log; id: 17005676086519951868 
2017/09/22 10:11:25.765819 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1
2017/09/22 10:11:25.765428 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s
2017/09/22 10:11:25.766752 log.go:91: INFO Harvester started for file: /var/log/anaconda.log
2017/09/22 10:11:25.766921 log.go:91: INFO Harvester started for file: /var/log/emcsboot.log
2017/09/22 10:11:25.766938 log.go:91: INFO Harvester started for file: /var/log/anaconda.yum.log
2017/09/22 10:11:25.767145 log.go:91: INFO Harvester started for file: /var/log/yum.log
2017/09/22 10:11:25.767233 log.go:91: INFO Harvester started for file: /var/log/emagent_deployment.log
2017/09/22 10:11:25.767294 log.go:91: INFO Harvester started for file: /var/log/systemtap.log
2017/09/22 10:11:25.767491 log.go:91: INFO Harvester started for file: /var/log/pm-powersave.log
2017/09/22 10:11:25.767593 log.go:91: INFO Harvester started for file: /var/log/anaconda.storage.log
2017/09/22 10:11:25.767699 log.go:91: INFO Harvester started for file: /var/log/anaconda.ifcfg.log
2017/09/22 10:11:25.767799 log.go:91: INFO Harvester started for file: /var/log/spice-vdagent.log
2017/09/22 10:11:25.767939 log.go:91: INFO Harvester started for file: /var/log/ibacm.log
2017/09/22 10:11:25.768312 log.go:91: INFO Harvester started for file: /var/log/trace-cmd.log
2017/09/22 10:11:25.771856 log.go:91: INFO Harvester started for file: /var/log/dracut.log
2017/09/22 10:11:25.771933 log.go:91: INFO Harvester started for file: /var/log/wpa_supplicant.log
2017/09/22 10:11:25.771989 log.go:91: INFO Harvester started for file: /var/log/anaconda.program.log
2017/09/22 10:11:25.772766 log.go:91: INFO Harvester started for file: /var/log/boot.log
2017/09/22 10:11:25.775174 log.go:91: INFO Harvester started for file: /var/log/sa-update.log
2017/09/22 10:11:25.775276 log.go:91: INFO Harvester started for file: /var/log/mysqld.log
2017/09/22 10:11:25.775385 log.go:91: INFO Harvester started for file: /var/log/sudo.log
2017/09/22 10:11:25.849578 client.go:667: INFO Connected to Elasticsearch version 5.6.1
2017/09/22 10:11:25.849650 output.go:317: INFO Trying to load template for client: http://localhost:9200
2017/09/22 10:11:25.878450 client.go:592: INFO Elasticsearch template with name 'filebeat' loaded

---------------------------------------

ELK Stack Basics



ELK Primer



The ELK Stack started off as a Log Analytics solution. And it has evolved into an enterprise search and analytics platform. ELK has three major components:
  • Logstash
  • Elasticsearch
  • Kibana





 

Elasticsearch provides multi-tenant and horizontal scaling capabilities around Lucene's indexing and searching capabilities. And exposes RESTful API for all CRUD operations. Then Kibana was introduced as a visualization layer on top of Elasticsearch. Later Logstash got integrated with Elasticsearch. And these three tools together called The ELK Stack.


Logstash


Logstash is developed by Jordan Sissel. Logstash is the entrypoint into ELK Stack. Logstash is used to collect, parse and send the processed data to a persistent store. Its an agent which can process structured or unstructured data. Logstash has evolved to a data pipeline which can collect,parse and send data for further analysis/visualization to external destination. LogStash is written in JRuby and runs in a Java Virtual Machine(JVM). 



Elasticsearch


Shay Banon is the creator of Elasticsearch. Elasticsearch is a tool to index and search text data. Elasticsearch is built on top of Apache Lucene. Lucene is an open source java library for indexing and searching. Elasticsearch has REST API support(JSON) for all operations. For example, you can query the data present in Elasticsearch using REST API. Similarly you can persist a document(say log file entry) in Elasticsearch by posting the json text using REST API. Elasticsearch is a document oriented(JSON) NoSQL Database.

Elasticsearch is used by many big companies. For example, Wikipedia uses Elasticsearch for full text search. GitHub uses it to provide search capability over billions of lines of source code.

Elasticsearch is schema-free. And can scale  and horizontally.  Elasticsearch is used for real time analytics also.





Kibana


Kibana is a data visualizaton platform. It a visualization layer on top of Elasticsearch. Kibana invokes the RESTful APIs exposed by Elasticsearch to fetch data and allows to visualize and analyze data. User can create and share Dashboards using Kibana. User can customize the Kibana Dashboard by adding removing widgets. Kibana is written in HTML/JavaScript.