Building a centralised logging and monitoring system using ELK Stack | elasticsearch, logstash, filebeat, kibana, elk

Logs are a critical part of any system as they provide vital information about what’s happening with your service and if anything requires your attention. While log files are very useful while debugging any issue, you can also build lot of analytics using log files.

Its easier to access and analyse logs when your logic is hosted on a single or just a couple of machines. However once your system grows to multiple hosts, monitoring logs can become a huge nightmare. The most common approach to solve this problem is to have a centralised logging system and pushing all your log files to a central location, that way all your logs will be at a single place for you to analyse.

Basic Components

This system is not that difficult to build once you break it down into components required to build it. Pause a bit here and give a thought about what all things you should be requiring if you want to build this kind of System.Below are the components which have to be present in any centralised logging system, irrespective of the stack we use

  1. Log Streaming Service
  2. Log Parsing and Transformation Service
  3. Common Persistent Storage System
  4. Data Processing Engine
  5. Data Exploration and visualisation tool

You can build a full fledged logging system with these 5 components. Although there are various solutions available to achieve this, we will be using ELK stack along with filebeat in this article. ELK stands for ElasticSearch Logstash Kibana, all three being products offered by

Filebeat being the log streaming service, logstash being the parsing and transformation service, elastic search being the storage & processing engine both and kibana being the data exploration & visualisation tool

Lets say you have 10 application servers, you need to setup on more server as your central log server. And you need to install logstash, elastic search and kibana on this central server while filebeat should be installed on each of the application servers.


As mentioned earlier, the very first thing we need is a service that can stream our log files to a central location. Filebeat is that log shipping component launched by Elastic as part of the Beats tool set and its pretty light weight.

Filebeat process reads and  forwards log lines from all files configured in filebeat. Its very robust and there is no chance of missing any log lines as filebeat remembers the location of where it left off once everything is back online.

The other interesting feature is the back pressure sensitive protocol when sending data to elastic search or logstash. What that means is that if logstash is busy processing data, it lets Filebeat know to slow down the data streaming. Once the congestion is over, filebeat picks up the streaming speed to catch up with its original pace and keeps on shipping logs.

Its pretty easy to setup, you can read more about filebeat on their official website link 



Now that we have setup the log streaming service, we need a service to receive this data, transform it and then persist it somewhere. Logstash acts as an aggregator and does all this for you. It is a server side data processing pipeline that ingests data from multiple sources simultaneously, transforms the data and sends it to a particular output. Logstash has a pluggable framework featuring over 200 plugins. You can mix, match, and orchestrate different inputs, filters, and outputs to work in pipeline harmony.


Logstash is capable of ingesting data from variety of sources like files, elastic search, Kafka, mysql, rabbitmq etc.


Logstash provides functionality to parse each event, identify named fields to build structure, and transform them to a common format for easier analysis and building business metrics on top of it. You can decipher geo coordinates from IP addresses, exclude sensitive information, do aggregation etc among library of filters available.


Finally you can output your transformed logs to any of the outputs supported by logstash such as elasticsearch, file, mongoDb etc. This gives us the flexibility to choose any of the output depending upon the use case.



ElasticSearch is a NoSQL database that is based on the Lucene search engine with a strong focus on search capabilities and is built with RESTful APIs. Its really fast and easily scalable by just adding nodes to the cluster. I am not going into much detail about ElasticSearch in this article, in a nut shell though, Elasticsearch forms the core of our centralised logging system, serving as the storage as well as the data processing engine. You can read more about ElasticSearch on their official website



Now that we are able to store the logs centrally in ElasticSearch, streaming it near realtime. The only thing missing is a tool which can enable us to query and visualise that data. Kibana is built precisely for that use case. Its a data visualisation tool having support for features like geo data, time series, graph analysis etc. Some of the features of Kibana are listed below

Free Text Search

Free text search works in all fields, if you don’t specify any field, then search is done across all analysed fields. The searches are case-insensitive and also supports wildcard characters like *,? etc. Just go to the discover tab and try your hands on full text search.

Logical Statements and Special Characters

You can also use logical statements like AND, OR etc while searching. You can also use parentheses to define complex statements. Some examples being

– node AND result

– (node AND result) OR cluster

You can also escape special characters listed below

+ – && || ! ( ) { } [ ] ^ ” ~ * ? : \

Proximity Searches

Proximity Search is an advanced feature of Kibana. It enables you to search for data in proximity to your query. Example being [elasticsearge~2] means a search for all the terms which differ elasticsearge by at max 2 changes. This means that elasticsearch will be matched as well.

This feature uses a lot of system resources, so be absolutely sure when you use this.

Visualisation Support

You can easily create lot of visualisations including histograms, pie charts, data tables, area charts, geo map etc. Apart from creating visualisations, you can save it for later use and can also share it with others.



The combination of ElasticSearch, LogStash, Filebeat and Kibana is sufficient for you to explore and visualise data in near realtime. It is also easily scalable, so you don’t need to worry about what will happen if you grow big.

If you just want to try out the ELK stack first without going through installation process, you can use elk docker image and play around with the features.

PS: If you are wondering what a docker is, you have got lot of catching up to do 😛


Thought of the day

The next Bill Gates will not build an operating system, the next Larry Page or Sergey Brin won’t make a search engine. If you are copying these guys, you aren’t learning from them. A company’s most important strength comes from new thinking, so keep innovating fellas and let the new ideas flow 🙂

– Quote from Zero To One by Peter Thiel

Come on, I know you want to say it