ELK Stack Basics

ELK Primer

The ELK Stack started off as a Log Analytics solution. And it has evolved into an enterprise search and analytics platform. ELK has three major components:

• Logstash
• Elasticsearch
• Kibana

 

Elasticsearch provides multi-tenant and horizontal scaling capabilities around Lucene's indexing and searching capabilities. And exposes RESTful API for all CRUD operations. Then Kibana was introduced as a visualization layer on top of Elasticsearch. Later Logstash got integrated with Elasticsearch. And these three tools together called The ELK Stack.

Logstash

Logstash is developed by Jordan Sissel. Logstash is the entrypoint into ELK Stack. Logstash is used to collect, parse and send the processed data to a persistent store. Its an agent which can process structured or unstructured data. Logstash has evolved to a data pipeline which can collect,parse and send data for further analysis/visualization to external destination. LogStash is written in JRuby and runs in a Java Virtual Machine(JVM). 

Elasticsearch

Shay Banon is the creator of Elasticsearch. Elasticsearch is a tool to index and search text data. Elasticsearch is built on top of Apache Lucene. Lucene is an open source java library for indexing and searching. Elasticsearch has REST API support(JSON) for all operations. For example, you can query the data present in Elasticsearch using REST API. Similarly you can persist a document(say log file entry) in Elasticsearch by posting the json text using REST API. Elasticsearch is a document oriented(JSON) NoSQL Database.

Elasticsearch is used by many big companies. For example, Wikipedia uses Elasticsearch for full text search. GitHub uses it to provide search capability over billions of lines of source code.

Elasticsearch is schema-free. And can scale  and horizontally.  Elasticsearch is used for real time analytics also.

Kibana

Kibana is a data visualizaton platform. It a visualization layer on top of Elasticsearch. Kibana invokes the RESTful APIs exposed by Elasticsearch to fetch data and allows to visualize and analyze data. User can create and share Dashboards using Kibana. User can customize the Kibana Dashboard by adding removing widgets. Kibana is written in HTML/JavaScript.