I recently needed to investigate an issue on a development environment that had been highlighted by way of visualisations in Kibana based on application specific logs. The logs themselves were structured as JSON and contained some important metrics about the application's performance. In order to investigate this issue locally, I needed to run the application under similar conditions to the real environment and analyse the logs with a similar visualisation to production.

Unfortunately, although running the application locally was reasonably easy, replicating the log ingestion pipeline was less-so. We run load tests in our CI pipeline using f1, so I could use our local test environment to run load through the application. However, our log processing pipeline involves integration between AWS, Fluentd and Logz. When I ran the application locally, all of the log output was just dumped to my terminal's stdout, which made it really hard to analyse. All I really wanted to do was pickup the logs from the stdout, parse their JSON content, and ingest them into an Elasticsearch database. This would mimic what we do in real life, and would allow me to analyse them in Kibana.

In other words, I just wanted to run a local ELK stack.

It turns out, this was quite easy to achieve, and - whilst there are plenty of examples out there on the internet - this post ties together my learnings in a simple way. If you just want to jump to the implementation, you can clone https://github.com/andykuszyk/local-elk and run docker-compose up. Don't forget to checkout the README.md.

It's pretty easy to get a local ELK stack up and running using docker-compose. The following docker-compose.yml file demonstrates this:

The /usr/share/elasticsearch/data directory is mounted into a named volume here (see the end of the docker-compose.yml file) so that the data stored in Elasticsearch is persisted between instances of the container. This is useful if you're starting up and tearing down the compose file regularly and don't want to re-create things like Kibana configuration. It also preserves all your previous logs.

No additional config is required for Kibana, the vanilla Docker image is fine.

A custom configuration for Logstash is useful here, so build: logstash instructs docker-compose to use the Dockerfile in the ./logstash directory. See later for details.

As with Logstash, a custom configuration for Filebeat is useful here. Furthermore, we're giving Filebeat access to the Docker daemon on your local host so that it can interrogate information about containers directly and retrieve their logs.

Finally, this is the named volume in use by the elasticsearch service.

Its useful to do two things to configure Logstash for your local ELK setup:

1. Provide a custom Logstash pipeline definition for any specific log parsing you might want to do;
2. Override the default Logstash Docker entrypoint to reduce the amount of noise in your logs.

This is achieved through three files in a ./logstash directory.

This file uses the base Logstash Docker image and copies in the two other files mentioned here, overriding the entrypoint.

This file simply re-directs the Logstash output to /dev/null. By default, Logstash outputs information for every message that it parses which adds a lot of noise to the logs ingested into Elasticsearch.

For now, this pipeline definition does nothing more than pass on your log messages from Filebeat to Elasticsearch, however it can be useful for more advanced processing of your log messages. See later for details.

Filebeat needs some basic configuration to allow it to automatically read information from Docker about containers and their logs as well as to work with Logstash to send the log messages to Elasticsearch. This is achieved through two files in the ./filebeat directory.

This Dockerfile simply uses the base Docker image and copies in the configuration file in this directory.

The guts of this file are in the filebeat.autodiscover directive, which instructs Filebeat to source its logs from Docker. The output directive simply tells Filebeat to send its logs to Logstash, rather than directly to Elasticsearch.

If you're logs are structured - for example, as JSON - this configuration can be extended to parse them. See later for details.

If your logs need parsing, this can be achieved in the ./filebeat/filebeat.yml config or in the ./logstash/pipeline.conf depending on which approach you'd like to take (Filebeat vs. Logstash).

If your logs are structured as JSON, the simplest thing to do is get Filebeat to parse them. An example filebeat.yml is as follows:

In this example, replace YOUR_CONTAINER_NAME with part of your container's image name. This will instruct Filebeat to only try parsing structured logs for your particular container (and avoid it trying to parse unstructured logs).

Often, Filebeat does an alright job of parsing your logs, but might get things like datatypes wrong. Parsing the correct datatypes (or anything else more complicated) cannot be done in Filebeat, but a simple pipeline in Logstash can be used. An example pipeline.conf demonstrates this:

In this example, the field YOUR_NUMERIC_FIELD in your JSON log message has been converted to an integer by Logstash.

That's it - with the above config and Docker files, its pretty easy to get a local ELK stack running with docker-compose up. All of the config files reference in this post can be found at https://github.com/andykuszyk/local-elk, which you can also clone and use to run docker-compose up directly.