[How2Tips] How to test Fluentd config

KNPLabs develops apps and websites with Symfony and React, for client projects, and also contributing to open source projects. Albin came to KNPLabs as a Symfony Developer and discovered his passion for devops during a project with an international client. He loves to work on our  open source bundles and also to share his best practices and hints with the community.

He wants to share with you his experience with Fluentd and how to configure it.

How to automatically test your Fluentd config?

If you’ve already used Fluentd or any other log forwarder you might know how painful and complex testing their configuration is. In this little post, I'd like to show you how we implemented automatic testing for our Fluentd config too, at least partially, tackle this complexity. If you don't know yet what Fluentd is and you are reading this post out of curiosity, you can find a great overview here.

Summary

1. The initial problem
2. Get rid of Elasticsearch
3. The test helper
4. Our first test scenarios
5. The test script
6. What's next?

The initial problem

In our current project, my team runs a bunch of services on a Swarm cluster. Each node has its own instance of Fluentd, each doing some relabelling, pattern matching, etc... to format logs as we want and to finally store them in Elasticsearch.

Some of these softwares have been written by us so we can enforce taxonomy and field types out there, but there's still a great deal of OS softwares running in our stack. For the most modern of them, we can at least configure them to log in JSON format in order to offload a bit Fluentd. But for the others, we obviously have to parse their plain text logs and enforce the global taxonomy and the field types from Fluentd.

Obviously, testing the whole thing on our local computer is tedious: we have to start Fluentd, then Elasticsearch and finally the service(s) emitting the logs we're working on. When everything has started and we triggered whatever code path on our service(s) to generate some logs, we still have to wait for the Fluentd output buffers to be flushed to Elasticsearch. So boring...

To ensure the important parts of our Fluentd config works as expected, and to improve the overall confidence of the development team in its ability to manage Fluentd configuration, we decided to find a way to automate the testing process.

And here's our current solution!

Get rid of Elasticsearch

First, we're not really interested in testing the Elasticsearch output plugin nor Elasticsearch itself. Moreoever, running Elasticsearch on local machines is one of the pain point of the manual testing process. So in order to reduce the scope and remove some painfulness from the testing process, we've to get rid of Elasticsearch.

To do so, we used the `out_file` plugin. This is the simplest output plugin we can use to write our tests later.

Unfortunately, this plugin is a buffered one (Fluentd output plugins can be either unbuffered, buffered or async buffered). Thus we've to set it up with a buffer size of 1 so any log written to the buffer will fill it and it will be flushed to disk as soon as possible.

Here's how we did that:

<match>
    @type copy
{{if .Env.TEST }}
    <store>
        @type file
        path /tmp/fluentd-test-output.log
        append true
        add_path_suffix false
        <format>
            @type json
        </format>
        <inject>
            tag_key tag
            time_key date
            time_type string
            time_format %F:%T
        </inject>
        <buffer []>
            @type memory
            chunk_limit_records 1
            retry_max_times 0
        </buffer>
    </store>
{{else}}
    <store>
        @type elasticsearch
        # ES plugin parameters here...
    </store>
{{end}}
</match>

As you can see, we're using some templating to enable the store we want. This is done through dockerize. In that way, we can switch to test mode with a single env var change.

The test helper

Now we get rid of Elasticsearch, what about the tests itself? Well, Fluentd has been written in Ruby. Let alone Bash which would be a terrible scripting language to maintain a test suite, Ruby is the only scripting language available in the container provided by Fluentd team. Thus we opted for Ruby.

After some failed trials, we decided to use `test-unit` which seems to be a popular unit testing framework in the Ruby community. In order to provide some helper methods to more easily write tests, we created a dedicated Ruby module:

require 'json'
module Helper
  private
  def run_case(emitted, expected)
    emit_log(emitted)
    wait = wait_for_output(500)
    if wait == false && expected != nil
      raise "Nothing has been written to the log file."
    elsif wait == false && expected == nil
      return
    end
    actual = fetch_output().pop
    actual = yield actual if block_given?
    assert_equal(expected, actual)
  end
  def emit_log(log)
    tag = if log.key?('tag') then log['tag'] else 'docker.CONTAINER-NAME' end
    formatted = JSON.generate(log).gsub("'", "\\\\'")
    `echo '#{formatted}' | fluent-cat #{tag}`
  end
  def fetch_output()
    @outfile.readlines.map do |line|
      JSON.parse! line
    end
  end
  def wait_for_output(delay)
    i = 0
    imax = delay / 100
    while i < imax do
      return true unless outfile_empty?
      sleep 0.1
      i += 1
    end
    return false
  end
  def outfile_empty?()
    File.stat(@outpath).size == 0
  end
  def cleanup_record(record, fields)
    fields.each { |field| record.delete(field) }
    return record
  end
end

Our first test scenarios

Then, we created our two first scenarios. These ones are quite simple but they demonstrate well the capabilities of our new tool:

require 'test/unit'
require './helper'
class TestCase < Test::Unit::TestCase
  include Helper
  def setup
    @outpath = "/tmp/fluentd-test-output.log"
    @outfile = File::open(@outpath, File::RDWR|File::TRUNC|File::CREAT)
    File.chmod(0666, @outpath)
  end
  def teardown
    @outfile.close
  end
  def test_junk_logs_sent_to_fluentd_for_health_checking_are_dropped
    emitted = {
      "log" => "junk log",
      "tag" => "fluentd.healthcheck"
    }
    # We just test nothing has been emitted
    run_case(emitted, nil)
  end
  def test_records_without_fluentd_tag_label_are_tagged_with_notag_and_raw_log_messages_are_preserved
    emitted = {
      "log" => "some random log",
    }
    expected = {
      "raw_log" => "some random log",
      "tag" => "notag",
    }
    # As there're no pattern to extract a date, the value of the date field 
    # is set by default by Fluentd to the datetime of the log intake. We don't
    # want to test that.
    run_case(emitted, expected) do |record|
      cleanup_record(record, ["date"])
    end
  end
end

The test script

Now we wrote our first test scenarios, we still have to write a little script to run them.

We put both files above in: `/fluentd/tests/` folder. And we put along with the Ruby files a little script named `fluent-test-config`. Here's what it looks like:

#!/usr/bin/env ruby
# -*- encoding: utf-8 -*-
require './tests'

And here we are! We now just need to type the following command to run the test suite:

docker exec -it $(docker ps -q --filter=name=monit_fluentd) sh -c "cd /fluentd/tests/ && ./fluent-test-config`

And here's the output:

$ make test-fluentd-config
docker exec -it 6a10c40ace61 \
        sh -c 'cd /fluentd/tests && ./fluent-test-config'
Loaded suite ./fluent-test-config
Started
..
Finished in 0.915340192 seconds.
-------------------------------------------------------------------------------
2 tests, 1 assertions, 0 failures, 0 errors, 0 pendings, 0 omissions, 0 notifications
100% passed
-------------------------------------------------------------------------------
2.18 tests/s, 1.09 assertions/s

:tada:

So, what's next?

We're using this only for some weeks now. But even if it works well, the Fluentd configuration templating is still quite a hack. It'd be actually gold to be able to run such tests directly using Fluentd tools. But that's for another day :)

Any questions ? The KNPTeam is on twitter

Copyright 2019 KNPLabs

Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.