[How2Tips] How to test Fluentd config
KNPLabs develops apps and websites with Symfony and React, for client projects, and also contributing to open source projects.
Albin came to KNPLabs as a Symfony Developer and discovered his passion for devops during a project with an international client. He loves to work on our open source bundles and also to share his best practices and hints with the community.
He wants to share with you his experience with Fluentd and how to configure it.
How to automatically test your Fluentd config?
If you’ve already used Fluentd or any other log forwarder you might know how painful and complex testing their configuration is. In this little post, I’d like to show you how we implemented automatic testing for our Fluentd config too, at least partially, tackle this complexity. If you don’t know yet what Fluentd is and you are reading this post out of curiosity, you can find a great overview here.
- The initial problem
- Get rid of Elasticsearch
- The test helper
- Our first test scenarios
- The test script
- What’s next?
The initial problem
In our current project, my team runs a bunch of services on a Swarm cluster. Each node has its own instance of Fluentd, each doing some relabelling, pattern matching, etc… to format logs as we want and to finally store them in Elasticsearch.
Some of these softwares have been written by us so we can enforce taxonomy and field types out there, but there’s still a great deal of OS softwares running in our stack. For the most modern of them, we can at least configure them to log in JSON format in order to offload a bit Fluentd. But for the others, we obviously have to parse their plain text logs and enforce the global taxonomy and the field types from Fluentd.
Obviously, testing the whole thing on our local computer is tedious: we have to start Fluentd, then Elasticsearch and finally the service(s) emitting the logs we’re working on. When everything has started and we triggered whatever code path on our service(s) to generate some logs, we still have to wait for the Fluentd output buffers to be flushed to Elasticsearch. So boring…
To ensure the important parts of our Fluentd config works as expected, and to improve the overall confidence of the development team in its ability to manage Fluentd configuration, we decided to find a way to automate the testing process.
And here’s our current solution!
Get rid of Elasticsearch
First, we’re not really interested in testing the Elasticsearch output plugin nor Elasticsearch itself. Moreoever, running Elasticsearch on local machines is one of the pain point of the manual testing process. So in order to reduce the scope and remove some painfulness from the testing process, we’ve to get rid of Elasticsearch.
To do so, we used the `out_file` plugin. This is the simplest output plugin we can use to write our tests later.
Unfortunately, this plugin is a buffered one (Fluentd output plugins can be either unbuffered, buffered or async buffered). Thus we’ve to set it up with a buffer size of 1 so any log written to the buffer will fill it and it will be flushed to disk as soon as possible.
Here’s how we did that:
As you can see, we’re using some templating to enable the store we want. This is done through dockerize. In that way, we can switch to test mode with a single env var change.
The test helper
Now we get rid of Elasticsearch, what about the tests itself? Well, Fluentd has been written in Ruby. Let alone Bash which would be a terrible scripting language to maintain a test suite, Ruby is the only scripting language available in the container provided by Fluentd team. Thus we opted for Ruby.
After some failed trials, we decided to use `test-unit` which seems to be a popular unit testing framework in the Ruby community. In order to provide some helper methods to more easily write tests, we created a dedicated Ruby module:
Our first test scenarios
Then, we created our two first scenarios. These ones are quite simple but they demonstrate well the capabilities of our new tool:
The test script
Now we wrote our first test scenarios, we still have to write a little script to run them.
We put both files above in: `/fluentd/tests/` folder. And we put along with the Ruby files a little script named `fluent-test-config`. Here’s what it looks like:
And here we are! We now just need to type the following command to run the test suite:
And here’s the output:
So, what’s next?
We’re using this only for some weeks now. But even if it works well, the Fluentd configuration templating is still quite a hack. It’d be actually gold to be able to run such tests directly using Fluentd tools. But that’s for another day :)