Install ELK on Oracle Enterprise Linux (from tar.gz archives)

Here is a primer on The ELK stack https://rajiv-kuriakose.blogspot.in/2017/09/elk-stack-basics.html

Install JDK

 

 

Get the latest jdk version from 

http://www.oracle.com/technetwork/java/javase/downloads/jdk8-downloads-2133151.html

Accept the license and download the tar.gz relevant to your box(32 bit vs 64 bit) and operating system

bash

export http_proxy=:80

export https_proxy=:80

wget --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie;" --no-check-certificate  http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz

 

Install Elasticsearch

Go to https://www.elastic.co/downloads

Get the latest elasitcsearch version from https://www.elastic.co/downloads/elasticsearch

bash

export http_proxy=:80

export https_proxy=:80

export JAVA_HOME=`pwd`/

download and extract  latest jdk tar.gz

 wget https://artifacts.elastic.co/downloads/elasticsearch/elasticsearch-5.6.1.tar.gz

tar -xvzf  elasticsearch-5.6.1.tar.gz

export JAVA_HOME=`pwd`/jdk1.8.0_144

bash-4.1$ echo $JAVA_HOME

/scratch/rkuriako/softwares/elk/jdk1.8.0_144

cd elasticsearch-5.6.1

Start Elasticsearch 


./bin/elasticsearch

-------------------------------------

bash-4.1$ ./bin/elasticsearch

[2017-09-22T02:40:56,277][INFO ][o.e.n.Node               ] [] initializing ...

[2017-09-22T02:40:56,413][INFO ][o.e.e.NodeEnvironment    ] [VSYSxM9] using [1] data paths, mounts [[/ (/dev/xvda2)]], net usable_space [192.1gb], net total_space [380.2gb], spins? [no], types [ext4]

[2017-09-22T02:40:56,414][INFO ][o.e.e.NodeEnvironment    ] [VSYSxM9] heap size [1.9gb], compressed ordinary object pointers [true]

[2017-09-22T02:40:56,416][INFO ][o.e.n.Node               ] node name [VSYSxM9] derived from node ID [VSYSxM9PTxmzlVRKe6BGDw]; set [node.name] to override

[2017-09-22T02:40:56,416][INFO ][o.e.n.Node               ] version[5.6.1], pid[9656], build[667b497/2017-09-14T19:22:05.189Z], OS[Linux/3.8.13-16.2.1.el6uek.x86_64/amd64], JVM[Oracle Corporation/Java HotSpot(TM) 64-Bit Server VM/1.8.0_144/25.144-b01]

[2017-09-22T02:40:56,416][INFO ][o.e.n.Node               ] JVM arguments [-Xms2g, -Xmx2g, -XX:+UseConcMarkSweepGC, -XX:CMSInitiatingOccupancyFraction=75, -XX:+UseCMSInitiatingOccupancyOnly, -XX:+AlwaysPreTouch, -Xss1m, -Djava.awt.headless=true, -Dfile.encoding=UTF-8, -Djna.nosys=true, -Djdk.io.permissionsUseCanonicalPath=true, -Dio.netty.noUnsafe=true, -Dio.netty.noKeySetOptimization=true, -Dio.netty.recycler.maxCapacityPerThread=0, -Dlog4j.shutdownHookEnabled=false, -Dlog4j2.disable.jmx=true, -Dlog4j.skipJansi=true, -XX:+HeapDumpOnOutOfMemoryError, -Des.path.home=/scratch/rkuriako/softwares/elk/elasticsearch-5.6.1]

[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [aggs-matrix-stats]

[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [ingest-common]

[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-expression]

[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-groovy]

[2017-09-22T02:40:57,438][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-mustache]

[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [lang-painless]

[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [parent-join]

[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [percolator]

[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [reindex]

[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [transport-netty3]

[2017-09-22T02:40:57,439][INFO ][o.e.p.PluginsService     ] [VSYSxM9] loaded module [transport-netty4]

[2017-09-22T02:40:57,440][INFO ][o.e.p.PluginsService     ] [VSYSxM9] no plugins loaded

[2017-09-22T02:40:59,481][INFO ][o.e.d.DiscoveryModule    ] [VSYSxM9] using discovery type [zen]

[2017-09-22T02:41:00,187][INFO ][o.e.n.Node               ] initialized

[2017-09-22T02:41:00,187][INFO ][o.e.n.Node               ] [VSYSxM9] starting ...

[2017-09-22T02:41:00,455][INFO ][o.e.t.TransportService   ] [VSYSxM9] publish_address {127.0.0.1:9300}, bound_addresses {127.0.0.1:9300}

[2017-09-22T02:41:00,475][WARN ][o.e.b.BootstrapChecks    ] [VSYSxM9] max virtual memory areas vm.max_map_count [65530] is too low, increase to at least [262144]

[2017-09-22T02:41:03,543][INFO ][o.e.c.s.ClusterService   ] [VSYSxM9] new_master {VSYSxM9}{VSYSxM9PTxmzlVRKe6BGDw}{UBKHbMC2Rw-ayoMVpS8n0w}{127.0.0.1}{127.0.0.1:9300}, reason: zen-disco-elected-as-master ([0] nodes joined)

[2017-09-22T02:41:03,569][INFO ][o.e.h.n.Netty4HttpServerTransport] [VSYSxM9] publish_address {127.0.0.1:9200}, bound_addresses {127.0.0.1:9200}

[2017-09-22T02:41:03,570][INFO ][o.e.n.Node               ] [VSYSxM9] started

[2017-09-22T02:41:03,595][INFO ][o.e.g.GatewayService     ] [VSYSxM9] recovered [0] indices into cluster_state

[2017-09-22T02:48:42,011][INFO ][o.e.c.m.MetaDataCreateIndexService] [VSYSxM9] [.kibana] creating index, cause [api], templates [], shards [1]/[1], mappings [_default_, index-pattern, server, visualization, search, timelion-sheet, config, dashboard, url]

[2017-09-22T03:02:55,388][INFO ][o.e.c.m.MetaDataCreateIndexService] [VSYSxM9] [logstash-2017.09.22] creating index, cause [auto(bulk api)], templates [logstash], shards [5]/[1], mappings [_default_]

[2017-09-22T03:02:55,481][INFO ][o.e.c.m.MetaDataMappingService] [VSYSxM9] [logstash-2017.09.22/P4anPGb6SMSxzRLq0aVoTQ] create_mapping [logs]

------------------------------

Verify that Elasticsearch is up and running

http://localhost:9200/

----------------------------------------------------------------------------------------------------------------------

Install Kibana

bash

export http_proxy=:80

export https_proxy=:80

export JAVA_HOME=`pwd`/jdk1.8.0_144

Get latest version from https://www.elastic.co/downloads/kibana

wget https://artifacts.elastic.co/downloads/kibana/kibana-5.6.1-linux-x86_64.tar.gz

cd kibana-5.6.1-linux-x86_64/

vim  config/kibana.yml

uncomment below line

elasticsearch.url: "http://localhost:9200"

Start Kibana

./bin/kibana

-------------------------------------------------

bash-4.1$ ./bin/kibana

  log   [10:02:31.777] [info][status][plugin:kibana@5.6.1] Status changed from uninitialized to green - Ready

  log   [10:02:31.867] [info][status][plugin:elasticsearch@5.6.1] Status changed from uninitialized to yellow - Waiting for Elasticsearch

  log   [10:02:31.906] [info][status][plugin:console@5.6.1] Status changed from uninitialized to green - Ready

  log   [10:02:31.951] [info][status][plugin:metrics@5.6.1] Status changed from uninitialized to green - Ready

  log   [10:02:32.194] [info][status][plugin:elasticsearch@5.6.1] Status changed from yellow to green - Kibana index ready

  log   [10:02:32.195] [info][status][plugin:timelion@5.6.1] Status changed from uninitialized to green - Ready

  log   [10:02:32.201] [info][listening] Server running at http://localhost:5601

  log   [10:02:32.203] [info][status][ui settings] Status changed from uninitialized to green - Ready

-------------------------------------------------

Install Logstash

bash

export http_proxy=:80

export https_proxy=:80

Create sample config file


Create a simple configuration file like below.

vim logstash-simple.conf

input { stdin { } }
output {
  elasticsearch { hosts => ["localhost:9200"] }
  stdout { codec => rubydebug }
}

-----------------------------

Start Logstash

bash-4.1$ bin/logstash -f logstash-simple.conf

Sending Logstash's logs to /scratch/rkuriako/softwares/elk/logstash-5.6.1/logs which is now configured via log4j2.properties

[2017-09-22T02:55:22,044][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"fb_apache", :directory=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/modules/fb_apache/configuration"}

[2017-09-22T02:55:22,050][INFO ][logstash.modules.scaffold] Initializing module {:module_name=>"netflow", :directory=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/modules/netflow/configuration"}

[2017-09-22T02:55:22,058][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.queue", :path=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/data/queue"}

[2017-09-22T02:55:22,059][INFO ][logstash.setting.writabledirectory] Creating directory {:setting=>"path.dead_letter_queue", :path=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/data/dead_letter_queue"}

[2017-09-22T02:55:22,090][INFO ][logstash.agent           ] No persistent UUID file found. Generating new UUID {:uuid=>"5493515e-08b7-44d2-a6b5-136fc1ff7f9f", :path=>"/scratch/rkuriako/softwares/elk/logstash-5.6.1/data/uuid"}

[2017-09-22T02:55:22,650][INFO ][logstash.outputs.elasticsearch] Elasticsearch pool URLs updated {:changes=>{:removed=>[], :added=>[http://localhost:9200/]}}

[2017-09-22T02:55:22,652][INFO ][logstash.outputs.elasticsearch] Running health check to see if an Elasticsearch connection is working {:healthcheck_url=>http://localhost:9200/, :path=>"/"}

[2017-09-22T02:55:22,779][WARN ][logstash.outputs.elasticsearch] Restored connection to ES instance {:url=>"http://localhost:9200/"}

[2017-09-22T02:55:22,780][INFO ][logstash.outputs.elasticsearch] Using mapping template from {:path=>nil}

[2017-09-22T02:55:22,833][INFO ][logstash.outputs.elasticsearch] Attempting to install template {:manage_template=>{"template"=>"logstash-*", "version"=>50001, "settings"=>{"index.refresh_interval"=>"5s"}, "mappings"=>{"_default_"=>{"_all"=>{"enabled"=>true, "norms"=>false}, "dynamic_templates"=>[{"message_field"=>{"path_match"=>"message", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false}}}, {"string_fields"=>{"match"=>"*", "match_mapping_type"=>"string", "mapping"=>{"type"=>"text", "norms"=>false, "fields"=>{"keyword"=>{"type"=>"keyword", "ignore_above"=>256}}}}}], "properties"=>{"@timestamp"=>{"type"=>"date", "include_in_all"=>false}, "@version"=>{"type"=>"keyword", "include_in_all"=>false}, "geoip"=>{"dynamic"=>true, "properties"=>{"ip"=>{"type"=>"ip"}, "location"=>{"type"=>"geo_point"}, "latitude"=>{"type"=>"half_float"}, "longitude"=>{"type"=>"half_float"}}}}}}}}

[2017-09-22T02:55:22,841][INFO ][logstash.outputs.elasticsearch] Installing elasticsearch template to _template/logstash

[2017-09-22T02:55:22,920][INFO ][logstash.outputs.elasticsearch] New Elasticsearch output {:class=>"LogStash::Outputs::ElasticSearch", :hosts=>["//localhost:9200"]}

[2017-09-22T02:55:22,922][INFO ][logstash.pipeline        ] Starting pipeline {"id"=>"main", "pipeline.workers"=>10, "pipeline.batch.size"=>125, "pipeline.batch.delay"=>5, "pipeline.max_inflight"=>1250}

[2017-09-22T02:55:22,938][INFO ][logstash.pipeline        ] Pipeline main started

The stdin plugin is now waiting for input:

[2017-09-22T02:55:22,986][INFO ][logstash.agent           ] Successfully started Logstash API endpoint {:port=>9600}

-----------------------------------------------------

Verify Logstash is up and running

To verify Logstash is configured properly, type "Hello World" on the command line and you will see output like below.

Basically logstash reads stdin and parse it and send it to stdout in the below format.

You can view the message in Kibana UI also.

Go to  http://localhost:5601

Change the time picker to "This week"

And then you can see log messages like below(scroll to bottom of the page).

 

 

 

 

 

 

 

 

 

 

 

 

 

Install Filebeat

bash

export http_proxy=:80

export https_proxy=:80

export JAVA_HOME=`pwd`/jdk1.8.0_144

Get latest version from here

https://www.elastic.co/downloads/beats/filebeat

 wget https://artifacts.elastic.co/downloads/beats/filebeat/filebeat-5.6.1-linux-x86_64.tar.gz

tar -xvzf filebeat-5.6.1-linux-x86_64.tar.gz 

cd filebeat-5.6.1-linux-x86_64

----------------------

bash-4.1$ sudo ./filebeat -e -c filebeat.yml

Password:

filebeat2017/09/22 10:09:06.664574 beat.go:346: CRIT Exiting: error loading config file: config file ("filebeat.yml") must be owned by the beat user (uid=0) or root

Exiting: error loading config file: config file ("filebeat.yml") must be owned by the beat user (uid=0) or root

--------------------------------

sudo su

chown root filebeat.yml

ls -l filebeat.yml 

-rw------- 1 root g900 4196 Sep 14 12:13 filebeat.yml

---------

[root@den00nyk filebeat-5.6.1-linux-x86_64]# sudo ./filebeat -e -c filebeat.yml

2017/09/22 10:11:25.759761 beat.go:297: INFO Home path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64] Config path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64] Data path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/data] Logs path: [/scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/logs]

2017/09/22 10:11:25.759884 beat.go:192: INFO Setup Beat: filebeat; Version: 5.6.1

2017/09/22 10:11:25.760060 metrics.go:23: INFO Metrics logging every 30s

2017/09/22 10:11:25.760174 output.go:258: INFO Loading template enabled. Reading template file: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/filebeat.template.json

2017/09/22 10:11:25.761183 output.go:269: INFO Loading template enabled for Elasticsearch 2.x. Reading template file: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/filebeat.template-es2x.json

2017/09/22 10:11:25.762126 output.go:281: INFO Loading template enabled for Elasticsearch 6.x. Reading template file: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/filebeat.template-es6x.json

2017/09/22 10:11:25.763067 client.go:128: INFO Elasticsearch url: http://localhost:9200

2017/09/22 10:11:25.763145 outputs.go:108: INFO Activated elasticsearch as output plugin.

2017/09/22 10:11:25.763288 publish.go:300: INFO Publisher name: den00nyk.us.oracle.com

2017/09/22 10:11:25.763535 async.go:63: INFO Flush Interval set to: 1s

2017/09/22 10:11:25.763577 async.go:64: INFO Max Bulk Size set to: 50

2017/09/22 10:11:25.763933 beat.go:233: INFO filebeat start running.

2017/09/22 10:11:25.764012 registrar.go:68: INFO No registry file found under: /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/data/registry. Creating a new registry file.

2017/09/22 10:11:25.765021 registrar.go:106: INFO Loading registrar data from /scratch/rkuriako/softwares/elk/filebeat-5.6.1-linux-x86_64/data/registry

2017/09/22 10:11:25.765099 registrar.go:123: INFO States Loaded from registrar: 0

2017/09/22 10:11:25.765198 crawler.go:38: INFO Loading Prospectors: 1

2017/09/22 10:11:25.765332 registrar.go:236: INFO Starting Registrar

2017/09/22 10:11:25.765449 sync.go:41: INFO Start sending events to output

2017/09/22 10:11:25.765402 prospector_log.go:65: INFO Prospector with previous states loaded: 0

2017/09/22 10:11:25.765795 prospector.go:124: INFO Starting prospector of type: log; id: 17005676086519951868 

2017/09/22 10:11:25.765819 crawler.go:58: INFO Loading and starting Prospectors completed. Enabled prospectors: 1

2017/09/22 10:11:25.765428 spooler.go:63: INFO Starting spooler: spool_size: 2048; idle_timeout: 5s

2017/09/22 10:11:25.766752 log.go:91: INFO Harvester started for file: /var/log/anaconda.log

2017/09/22 10:11:25.766921 log.go:91: INFO Harvester started for file: /var/log/emcsboot.log

2017/09/22 10:11:25.766938 log.go:91: INFO Harvester started for file: /var/log/anaconda.yum.log

2017/09/22 10:11:25.767145 log.go:91: INFO Harvester started for file: /var/log/yum.log

2017/09/22 10:11:25.767233 log.go:91: INFO Harvester started for file: /var/log/emagent_deployment.log

2017/09/22 10:11:25.767294 log.go:91: INFO Harvester started for file: /var/log/systemtap.log

2017/09/22 10:11:25.767491 log.go:91: INFO Harvester started for file: /var/log/pm-powersave.log

2017/09/22 10:11:25.767593 log.go:91: INFO Harvester started for file: /var/log/anaconda.storage.log

2017/09/22 10:11:25.767699 log.go:91: INFO Harvester started for file: /var/log/anaconda.ifcfg.log

2017/09/22 10:11:25.767799 log.go:91: INFO Harvester started for file: /var/log/spice-vdagent.log

2017/09/22 10:11:25.767939 log.go:91: INFO Harvester started for file: /var/log/ibacm.log

2017/09/22 10:11:25.768312 log.go:91: INFO Harvester started for file: /var/log/trace-cmd.log

2017/09/22 10:11:25.771856 log.go:91: INFO Harvester started for file: /var/log/dracut.log

2017/09/22 10:11:25.771933 log.go:91: INFO Harvester started for file: /var/log/wpa_supplicant.log

2017/09/22 10:11:25.771989 log.go:91: INFO Harvester started for file: /var/log/anaconda.program.log

2017/09/22 10:11:25.772766 log.go:91: INFO Harvester started for file: /var/log/boot.log

2017/09/22 10:11:25.775174 log.go:91: INFO Harvester started for file: /var/log/sa-update.log

2017/09/22 10:11:25.775276 log.go:91: INFO Harvester started for file: /var/log/mysqld.log

2017/09/22 10:11:25.775385 log.go:91: INFO Harvester started for file: /var/log/sudo.log

2017/09/22 10:11:25.849578 client.go:667: INFO Connected to Elasticsearch version 5.6.1

2017/09/22 10:11:25.849650 output.go:317: INFO Trying to load template for client: http://localhost:9200

2017/09/22 10:11:25.878450 client.go:592: INFO Elasticsearch template with name 'filebeat' loaded

---------------------------------------

How to install ELK 5.0 (Elastic Stack) on Oracle Enterprise Linux

 

This document explains how to install Elasticsearch, Logstash and Kibana on same host. I am using the latest stable version (version 5.0) for all three. There are multiple options to install ELK. I have downloaded rpm for all three components and installed it using yum.

ELK Stack aka Elastic Stack

The ELK Stack

source

Logstash agents running on remote hosts collect logs and forward them to a central server for further processing using Filters. Filters are used to convert  unstructured log data to structured format. And the output is saved into Elasticsearch. Elasticsearch is a text indexer and search engine. Searches are very fast as they are performed on the index. Elastic search will index the data coming in from Logstash and later make it available for search using the powerful query language.  Kibana is the web interface for Elastic stack. It enables data visualization, search and dash-boarding. Kibana supports add-on functionalities via plugin model. Kibana supports search using the Apache Lucene query syntax.

Download / Install latest Oracle jdk rpm

Install jdk

# yum -y localinstall jdk-8u111-linux-x64.rpm



Sample output

Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock
Loading support for kernel ABI
Setting up Local Package Process
Examining jdk-8u111-linux-x64.rpm: 2000:jdk1.8.0_111-1.8.0_111-fcs.x86_64
Marking jdk-8u111-linux-x64.rpm to be installed
oel6_addons                                                                                                           | 1.2 kB     00:00
oel6_addons/primary                                                                                                   | 128 kB     00:00
oel6_addons                                                                                                                          410/410
ol6_UEK_latest                                                                                                        | 1.2 kB     00:00
ol6_UEK_latest/primary                                                                                                |  28 MB     00:09
ol6_UEK_latest                                                                                                                       603/603
ol6_latest                                                                                                            | 1.4 kB     00:00
ol6_latest/primary                                                                                                    |  60 MB     00:20
ol6_latest                                                                                                                       36584/36584
Resolving Dependencies
--> Running transaction check
---> Package jdk1.8.0_111.x86_64 2000:1.8.0_111-fcs will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================
 Package                        Arch                     Version                                Repository                              Size
=============================================================================================================================================
Installing:
 jdk1.8.0_111                   x86_64                   2000:1.8.0_111-fcs                     /jdk-8u111-linux-x64                   258 M

Transaction Summary
=============================================================================================================================================
Install       1 Package(s)

Total size: 258 M
Installed size: 258 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Warning: RPMDB altered outside of yum.
** Found 1 pre-existing rpmdb problem(s), 'yum check' output follows:
openssl098e-0.9.8e-18.0.1.el6_5.2.x86_64 is a duplicate with openssl098e-0.9.8e-17.0.1.el6_2.2.i686
  Installing : 2000:jdk1.8.0_111-1.8.0_111-fcs.x86_64                                                                                    1/1
Unpacking JAR files...
        tools.jar...
        plugin.jar...
        javaws.jar...
        deploy.jar...
        rt.jar...
        jsse.jar...
        charsets.jar...
        localedata.jar...
  Verifying  : 2000:jdk1.8.0_111-1.8.0_111-fcs.x86_64                                                                                    1/1

Installed:
  jdk1.8.0_111.x86_64 2000:1.8.0_111-fcs

Complete!

Check installation location:

By default jdk is installed under /usr/java/jdk1.8.0_111

We need this location to update Logstash config file.

Install ElasticSearch

Download latest version of elastic search from https://www.elastic.co/downloads/elasticsearch

I have downloaded version 5.0.0 (elasticsearch-5.0.0.rpm)

#  yum -y localinstall elasticsearch-5.0.0.rpm


Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock
Loading support for kernel ABI
Setting up Local Package Process
Examining elasticsearch-5.0.0.rpm: elasticsearch-5.0.0-1.noarch
Marking elasticsearch-5.0.0.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package elasticsearch.noarch 0:5.0.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================
 Package                            Arch                        Version                      Repository                                 Size
=============================================================================================================================================
Installing:
 elasticsearch                      noarch                      5.0.0-1                      /elasticsearch-5.0.0                       35 M

Transaction Summary
=============================================================================================================================================
Install       1 Package(s)

Total size: 35 M
Installed size: 35 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : elasticsearch-5.0.0-1.noarch                                                                                              1/1
Creating elasticsearch group... OK
Creating elasticsearch user... OK
### NOT starting on installation, please execute the following statements to configure elasticsearch service to start automatically using chkconfig
 sudo chkconfig --add elasticsearch
### You can start elasticsearch service by executing
 sudo service elasticsearch start
  Verifying  : elasticsearch-5.0.0-1.noarch                                                                                              1/1

Installed:
  elasticsearch.noarch 0:5.0.0-1

Complete!

Start elastic search

Elasticsearch config file is at /etc/elasticsearch/elasticsearch.yml



cd /etc/init.d

[root@host init.d]# ./elasticsearch start
Starting elasticsearch:                                    [  OK  ]

[root@host init.d]# ./elasticsearch status
elasticsearch (pid  21985) is running...

Check status of elasticsearch

ElasticSearch runs on port 9200 by default.

Connect to http://localhost:9200/

Sample output:

{
  "name" : "arB5Gic",
  "cluster_name" : "elasticsearch",
  "cluster_uuid" : "zjTZpHcUTRqBFQj6ZpFR6g",
  "version" : {
    "number" : "5.0.0",
    "build_hash" : "253032b",
    "build_date" : "2016-10-26T04:37:51.531Z",
    "build_snapshot" : false,
    "lucene_version" : "6.2.0"
  },
  "tagline" : "You Know, for Search"
}

Install Kibana

# yum -y localinstall kibana-5.0.0-x86_64.rpm

Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock
Loading support for kernel ABI
Setting up Local Package Process
Examining kibana-5.0.0-x86_64.rpm: kibana-5.0.0-1.x86_64
Marking kibana-5.0.0-x86_64.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package kibana.x86_64 0:5.0.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================
 Package                      Arch                         Version                          Repository                                  Size
=============================================================================================================================================
Installing:
 kibana                       x86_64                       5.0.0-1                          /kibana-5.0.0-x86_64                       140 M

Transaction Summary
=============================================================================================================================================
Install       1 Package(s)

Total size: 140 M
Installed size: 140 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : kibana-5.0.0-1.x86_64                                                                                                     1/1
  Verifying  : kibana-5.0.0-1.x86_64                                                                                                     1/1

Installed:
  kibana.x86_64 0:5.0.0-1

Complete!

Start Kibana

By default Kibana connects to the Elasticsearch instance running on same host.
So we can just start Kibana.
Kibana configuration file is at  /etc/kibana/kibana.yml

cd /etc/init.d

./kibana start
kibana started

[root@host init.d]# ./kibana status
kibana is running

Check status of Kibana

Access localhost:5601 from browser. You should see below screen.

 Install Logstash

I have downloaded logstash-5.0.0.rpm and installed it using yum.


 yum -y localinstall logstash-5.0.0.rpm

Loaded plugins: aliases, changelog, downloadonly, kabi, presto, refresh-packagekit, security, tmprepo, verify, versionlock
Loading support for kernel ABI
Setting up Local Package Process
Examining logstash-5.0.0.rpm: 1:logstash-5.0.0-1.noarch
Marking logstash-5.0.0.rpm to be installed
Resolving Dependencies
--> Running transaction check
---> Package logstash.noarch 1:5.0.0-1 will be installed
--> Finished Dependency Resolution

Dependencies Resolved

=============================================================================================================================================
 Package                         Arch                          Version                          Repository                              Size
=============================================================================================================================================
Installing:
 logstash                        noarch                        1:5.0.0-1                        /logstash-5.0.0                        189 M

Transaction Summary
=============================================================================================================================================
Install       1 Package(s)

Total size: 189 M
Installed size: 189 M
Downloading Packages:
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : 1:logstash-5.0.0-1.noarch                                                                                                 1/1
Using provided startup.options file: /etc/logstash/startup.options
Successfully created system startup script for Logstash
  Verifying  : 1:logstash-5.0.0-1.noarch                                                                                                 1/1

Installed:
  logstash.noarch 1:5.0.0-1

Complete!

Start logstash

Somehow the /etc/init.d/logstash was not present after install. Looks like system-install script didn't get executed at the end of the rpm installation. So I ran it manually.

Check /etc/logstash/startup.options to see where its installed. 
In my host LS_HOME in /etc/logstash/startup.option points to /usr/share/logstash

Run system-install script manually

I have installed it manually using system-install script. Before that modified /etc/logstash/startup.options to set the correct path to JDK 1.8

I have changed jdk location to below
# Override Java location
#JAVACMD=/usr/bin/java
JAVACMD=/usr/java/jdk1.8.0_111/bin/java

 /usr/share/logstash/bin/system-install /etc/logstash/startup.options
Using provided startup.options file: /etc/logstash/startup.options
Sending all logs to /tmp/tmp.ADeV3QdriK
Successfully created system startup script for Logstash


#cd /etc/init.d

# ./logstash start
logstash started

[root@host init.d]# ./logstash status
logstash is running

Stop all three

./logstash status
./kibana stop
./elasticsearch stop

  Install x-pack plugin

 
 Refer https://www.elastic.co/start

Download  x-pack-5.0.0.zip and use the offline installation option.
I have downloaded latest version from https://artifacts.elastic.co/downloads/packs/x-pack/x-pack-5.0.0.zip

This zip file contains Elasticsearch and Kibana plugin.

# cd /usr/share/elasticsearch/bin
 # ./elasticsearch-plugin install file:///scratch/softwares/elk/x-pack-5.0.0.zip

-> Downloading file:///scratch/softwares/elk/x-pack-5.0.0.zip
[=================================================] 100% Â
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@     WARNING: plugin requires additional permissions     @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
* java.lang.RuntimePermission accessClassInPackage.com.sun.activation.registries
* java.lang.RuntimePermission getClassLoader
* java.lang.RuntimePermission setContextClassLoader
* java.lang.RuntimePermission setFactory
* java.security.SecurityPermission createPolicy.JavaPolicy
* java.security.SecurityPermission getPolicy
* java.security.SecurityPermission putProviderProperty.BC
* java.security.SecurityPermission setPolicy
* java.util.PropertyPermission * read,write
* java.util.PropertyPermission sun.nio.ch.bugLevel write
* javax.net.ssl.SSLPermission setHostnameVerifier
See http://docs.oracle.com/javase/8/docs/technotes/guides/security/permissions.html
for descriptions of what these permissions allow and the associated risks.

Continue with installation? [y/N]y
-> Installed x-pack

Now install Kibana plugin

# cd /usr/share/kibana/bin
# ./kibana-plugin install file:///scratch/rkuriako/softwares/elk/x-pack-5.0.0.zip

Attempting to transfer from file:///scratch/rkuriako/softwares/elk/x-pack-5.0.0.zip
Transferring 72364732 bytes....................
Transfer complete
Retrieving metadata from plugin archive
Extracting plugin archive
Extraction complete
Optimizing and caching browser bundles...
Plugin installation complete

Refer xpack docs for more deails.


X-Pack security provides a built-in elastic superuser.
This elastic user has full access to the cluster, including all indices and data.
The default password for the elastic user is changeme.

Start Elasticsearch and Kibana

[root@host bin]# cd /etc/init.d/
[root@host init.d]# ./elasticsearch start
Starting elasticsearch:                                    [  OK  ]
[root@host init.d]# ./kibana start
kibana started

Navigate to http://localhost:5601/