This blog will look into the entire Elastic Stack (Elasticsearch, Logstash, Kibana).
What is Kibana?
Kibana is the visualization layer of Elastic stack. With Kibana, you can create different types of charts, tables, histograms, and maps to understand and analyze your data better. You can also use Kibana to secure, monitor, and manage your elastic stack. For a more in-depth explanation, check out the official demo for Kibana here.
What is Logstash?
Logstash is an open-source, server-side data processing pipeline that simultaneously ingests data from multiple sources, transforms it, and then sends it to Elasticsearch. Since log data lives in different repositories, Logstash allows you to take data from various places as input (ingestion). Then you can apply multiple filters to parse and process your data. These filters help to transform and prepare your data dynamically, preparing the data to migrate to Elasticsearch.
- For a full list of output, click here
- For a full list of input sources, click here
- For a full list of available filters, click here
Elasticsearch use cases
If you are unfamiliar with Elasticsearch, read my last article: Searching with Elasticsearch
Due to Elasticsearch’s flexibility, it has multiple use cases. You can check how various companies use elasticsearch here. Here is a list of the most exciting:
- Log storage: storing log data that originates from multiple locations
- Analyzing data: searching through data and analyzing data for errors
- Data visualization: understand your data better by creating bar charts, pie charts, tables, histograms, and maps
These use cases are less common but very interesting:
- NLP: In my last article (link), I explained how elasticsearch stores data in an inverted index. This means term-by-term co-occurrences are incredibly fast to extract on the fly. Essential terms can also be identified via the standard data science ‘tf-idf’ procedure. We can use this to have elasticsearch as a pre-trained model for extracting keywords (for details read this article).
- Elasticsearch as a backend: Though Elasticsearch should not be used as db in the long term (as delete in Elasticsearch works differently and not always delete data from memory). But in the short term for people who are learning frontend can use Elasticsearch and API provided by Elasticsearch out of the box to store/update/delete the data. In the next article, I will create a sample angular project which uses Elasticsearch as a backend.
Conclusion
The Elastic Stack is a distributed, scalable, real-time search, and analytics engine. Each tool, Kibana, Logstash, and Elasticsearch, work together to make searching and processing possible. In the next article, we will create a sample angular project which uses Elasticsearch as a backend.
References:
https://www.elastic.co/what-is/kibana
https://www.elastic.co/logstash
Did you enjoy this content? Follow our linkedin page!
