Best practices for Logstash
Most of us, working with elastic stack has come to the point where we have to make optimizations in a matter of getting better indexing throughput or lower query response time.
The Elastic stack is quite easy to set up and get running. On the other hand, it is a pretty complex science to get the most of it and keep it in good condition. In this article, we will take a look at some basic rules and best practices to keep your stack healthy and robust.
As we are talking about Elastic stack, we will cover Logstash in this article and Beats and Elasticsearch in the upcoming sections. Although Logstash is not as complicated as Elasticsearch itself, it is crucial for some use cases and also needs the same level of focus for its proper configuration.
Logstash work modus is quite simple, it ingests data, process them, and then it outputs them somewhere. Each of this phase requires different tuning and has different requirements. Processing phase heavily relies on raw processor power, while outputs rely on the speed of the system to which Logstash outputs data.
To be able to solve a problem, you need to know where it is, so If you are able to use Monitoring UI (part of X-Pack/Features) in Kibana, you have all information served in an easy-to-understand graphical way
If you are not that lucky, you can still get the information about running logstash instance by calling its API — which in default listens on 9600.
For example to get statistics about your pipelines, call:
curl -XGET http://localhost:9600/_node/stats/pipelines?pretty
and you will get all info in json format: