Setting up ELK Stack in few steps !

Self hosted Elastic Setup is not a trivial process. It requires good knowledge of every stack you use and how to architect based on needs. I will walk through these below four stack in this blog and will continue adding more stack to make it production ready.

ELK: Elastic Search, Logstash, Kibana

Let’s start backwards to understand these one by one. In Simple terms:

We collect data from source (Beat) and send it to a pipeline(Logstash) to process the data and send it to database(Elastic Search) to store it. We use a visualization tools (Kibana) to analyse and understand the data well and be inform about system all the time.

This Elastic Stack setup need docker ( ≥ 18.06.0) and docker-compose(≥ 1.26.0). You can install docker from digital ocean blog, here.

>> git clone https://github.com/deviantony/docker-elk.git
>> cd docker-elk
>> docker-compose up -d

You must understand, What is happening with the code?

Here we have four containers running:

1. Setup: This container create docker volume for Elasticsearch and create elastic role based access for Logstash/Kibana. Then it stopped after doing its job.
2. Logstash: This container keeps on listening on a port 5044 and start a pipeline and once it receive data on that port , it send to elastic search.
3. Elastic Search: Create Elastic Database, limits storage and start database server at port 9200.
4. Kibana: It is a web-based visualization tools for Elastic Search and its server start at port 5600.

In this setup, we are keeping one port open as tcp so any data on that port will be forwarded to elastic search. You can use telnet to check it.

Let’s read the definition from their official documentation:

Beats

Beats is a free and open platform for single-purpose data shippers. They send data from hundreds or thousands of machines and systems to Logstash or Elasticsearch.

Logstash

Logstash is a free and open server-side data processing pipeline that ingests data from a multitude of sources, transforms it, and then sends it to your favorite “stash.” (like Collections in Database)

ElasticSearch

Elasticsearch is a distributed, free and open search and analytics engine for all types of data, including textual, numerical, geospatial, structured, and unstructured.

Kibana

Kibana is a free and open user interface that lets you visualize your Elasticsearch data and navigate the Elastic Stack. Do anything from tracking query load to understanding the way requests flow through your apps.

I will be sharing more on this setup and extend its current functionality from local to production on cloud (AWS). More to come with its use cases…

Meanwhile, I will also extend this to use in MLOps Application as well. Quick look: https://github.com/sauravsolanki/fastapi-docker-elk

If you like this article, please give it clap and follow me to get the updates:

I am open to opportunity and you can find me here: LinkedIn