Recently, Travis CI announces “Multi-OS feature” on their blog.

Multi-OS Feature Available

If this feature is enabled, our project can be tested on both Linux and Mac OS X.
Multi OS feature has been enabled for Fluentd repository. See following build:

fluent/fluentd on Travis CI

This is very useful for checking commit and PR. Thanks Travis CI and Facebook team!

Cool.io was revived after I became a maintainer. This article describes the current stauts and future work of Cool.io.

Cool.io is a core of Fluentd so I am focusing Fluentd related features.

Windows support

Cool.io is the one of blocker for Fluentd Windows support. Since version 1.2, Cool.io works on Windows and I release mingw based cross-compiling gem for Windows environment.

You can now use Cool.io via gem install cool.io on Windows and we started to implement Windows support of Fluentd.

The limitation on Windows

Cool.io uses select on Windows, not IOCP. In this result, the performance isn’t better than other environments. This limitation is of little concern in Fluentd forward use-case.

JRuby support

Fluentd works on CRuby and Rubinius. If support JRuby, Fluentd will work on almost popular Ruby environemnts.

Since May 2014, @taichi started to implement JRuby support. @taichi has experience with Java and evented IO frameworks so he is a good person for this task ;)

Remove Ruby 1.8 support

Ruby 1.8 is dead language so Cool.io removes 1.8 support since version 1.2.

Remove HttpClient

AFAIK, there is no HttpClient users including Fluentd. This class depends on Ragel to parse HTTP. It increases the maintenance cost and hard to support on cross platform.

So HttpClient will not be maintained in the future.

Conclusion

Cool.io isn’t dead and Cool.io is still improved. We are now working on more better environment support and fixing the bugs.

If you have a bug of Cool.io, then please file it on github :)

Fluentd is a flexible and robust event log collector, but Fluentd doesn’t have own data-store and Web UI. If you want to analyze the event logs collected by Fluentd, then you can use Elasticsearch and Kibana :)

Elasticsearch is an easy to use Distributed Search Engine and Kibana is an awesome Web front-end for Elasticsearch.

Setup

I tested on Mac OS X.

Pre requirements

• Java for Elasticsearch

Use Mac OS X’s Java.

1
2
3
4

% java -version
java version "1.6.0_51"
Java(TM) SE Runtime Environment (build 1.6.0_51-b11-457-11M4509)
Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-457, mixed mode)

• Ruby for Fluentd

In this article, I use my rbenv’s Ruby and Fluentd gem directly.

1
2

% ruby --version
ruby 1.9.3p484 (2013-11-22 revision 43786) [x86_64-darwin12.4.0]

Note that Fluentd doesn’t work on Ruby 1.8.

Treasure Agent (td-agent)

Treasure Agent, as known as td-agent, is a stable distribution which consists of Fluentd, popular plugins and own Ruby processor. Many users use td-agent instead of Fluentd gem. See this FAQ.

Elasticsearch

Downlod and extract the latest package.

1
2
3

% curl -O https://download.elasticsearch.org/elasticsearch/elasticsearch/elasticsearch-0.90.11.tar.gz
% tar zxvf elasticsearch-0.90.11.tar.gz
% cd elasticsearch-0.90.11/

I use version v0.90.11 but v1.0 will be released soon!

Start Elasticsearch:

1
2
3
4
5
6
7
8
9
10
11
12

% ./bin/elasticsearch -f
[2014-02-12 17:51:32,645][INFO ][node                     ] [Powerhouse] version[0.90.11], pid[79737], build[11da1ba/2014-02-03T15:27:39Z]
[2014-02-12 17:51:32,646][INFO ][node                     ] [Powerhouse] initializing ...
[2014-02-12 17:51:32,651][INFO ][plugins                  ] [Powerhouse] loaded [], sites []
[2014-02-12 17:51:34,319][INFO ][node                     ] [Powerhouse] initialized
[2014-02-12 17:51:34,320][INFO ][node                     ] [Powerhouse] starting ...
[2014-02-12 17:51:34,395][INFO ][transport                ] [Powerhouse] bound_address { inet[/0:0:0:0:0:0:0:0%0:9300]}, publish_address { inet[/192.168.1.112:9300]}
[2014-02-12 17:51:37,448][INFO ][cluster.service          ] [Powerhouse] new_master [Powerhouse][tL1IC8xHSCudeVsFt4JFsQ][inet[/192.168.1.112:9300]], reason: zen-disco-join (elected_as_master)
[2014-02-12 17:51:37,481][INFO ][discovery                ] [Powerhouse] elasticsearch/tL1IC8xHSCudeVsFt4JFsQ
[2014-02-12 17:51:37,492][INFO ][http                     ] [Powerhouse] bound_address { inet[/0:0:0:0:0:0:0:0%0:9200]}, publish_address { inet[/192.168.1.112:9200]}
[2014-02-12 17:51:37,493][INFO ][node                     ] [Powerhouse] started
[2014-02-12 17:51:37,505][INFO ][gateway                  ] [Powerhouse] recovered [0] indices into cluster_state

Kibana

Latest Kibana, called Kibana 3, consists of only HTML and JavaScript, so setup is very easy:

1
2
3

% curl -O https://download.elasticsearch.org/kibana/kibana/kibana-3.0.0milestone5.tar.gz
% tar zxvf kibana-3.0.0milestone5.tar.gz
% cd kibana-3.0.0milestone5/

Open index.html in the kibana directory:

1

% open index.html

If you want to change Kibana configuration, please edit config.js, e.g. change elasticsearch URL.

Fluentd

Install Fluentd gem and Elasticsearch plugin:

1

% gem install fluentd fluent-plugin-elasticsearch

Fluentd configuration is below:

1
2
3
4
5
6
7
8
9
10

# es.conf
<source>
  type forward
</source>

<match es.**>
  type elasticsearch
  logstash_format true
  flush_interval 5s # 5s for testing. On production environment, 60s or higher is better
</match>

fluent-plugin-elasticsearch provides logstash_format option. It enables Kibana to analyze the event logs with day based indexes.

Start Fluentd:

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18

% fluentd -c es.conf
2014-02-12 18:43:31 +0900 [info]: starting fluentd-0.10.43
2014-02-12 18:43:31 +0900 [info]: reading config file path = "es.conf"
2014-02-12 18:43:32 +0900 [info]: gem 'fluent-plugin-elasticsearch' version '0.2.0'
2014-02-12 18:43:32 +0900 [info]: gem 'fluentd' version '0.10.43'
2014-02-12 18:43:32 +0900 [info]: using configuration file: <ROOT>
  <source>
    type forward
  </source>
  <match es.**>
    type elasticsearch
    logstash_format true
    flush_interval 5s
  </match>
</ROOT>
2014-02-12 18:43:32 +0900 [info]: adding source type = "forward"
2014-02-12 18:43:32 +0900 [info]: adding match pattern = "es.**" type = "elasticsearch"
2014-02-12 18:43:32 +0900 [info]: listening fluent socket on 0.0.0.0:24224

Analyze event logs

Send some events to Fluentd. fluent-cat is an utility command to send json text to Fluentd’s in_forward plugin.

1
2
3
4
5

% echo '{"message":"D"}' | fluent-cat es.event # es.event is a tag. es.event matches es.** of <match>
% echo '{"message":"Ruby"}' | fluent-cat es.event
% echo '{"message":"Elasticsearch"}' | fluent-cat es.event
% echo '{"message":"msgpack"}' | fluent-cat es.event
% ...

After Fluentd flushed received events to Elasticsearch, you can analyze the event logs via Kibana! Following image is one panel example:

Kibana has some built-in panels, so you can create own dashboard easily. See Kibana demo

Advanced tips

If your service has a high traffic, then fluent-plugin-elasticsearch sometimes get stucked. In this case, built-in out_roundrobin plugin is useful. You can distribute a write request to elasticsearch nodes for load balancing.

Of course, putting Queue with multiple fluentd nodes is an another approach.

Conclusion

This article introduced Fluentd, Elasticsearch and Kibana combination to analyze the event logs. These components are easy to setup and work fine, so you can try this framework soon! I heard many companies have already tried / deployed this setup on production :)

Enjoy Fluentd and Elasticsearch!

In this week, Fluentd v0.10.43 has been released. Since this version, Fluentd introduced log_level parameter in Input / Output plugin. It enables you can set different log level separated from global log level, e.g. -v, -q command line option.

This article shows “How to support log_level option in your plugin.”

log_level option use cases

Disable in_tail warning

in_tail prints “pattern no match” warning when receives invalid log. It is useful information for almost users, but some users want to ignore this log for other important plugin warning.

In this case, you can set “log_level error” in in_tail configuration to disable “pattern no match”.

1
2
3
4
5

<source>
  type tail
  ...
  log_level error
</source>

Debugging

Without log_level, we get many verbose logs using -vv command line option for one plugin. With log_level, you can set verbose configuration in only one plugin.

It is useful for debugging a plugin on acutual environment.

1
2
3
4
5

<match foo.**>
  type unstable_plugin
  ...
  log_level trace
</match>

Support log_level option in your plugin

This section is for plugin developers.

First of all, Fluentd provides $log object as heretofore. So all plugin should work without changing on Fluentd v0.10.43 or later.

To suppot log_level is very easy. Replace $log with log. Following example is fluent-plugin-td’s diff:

1
2

-        $log.debug "checking whether table '#{key}' exists on Treasure Data"
+        log.debug "checking whether table '#{key}' exists on Treasure Data"

Support older Fluentd versions

After replaced $log with log, your plugin only works on Fluentd v0.10.43 or later. If you want to support older Fluentd versions, you can use following code in your plugin.

1
2
3
4
5
6
7
8
9

module Fluent
  module FooPluginOutput < Output
    # Define `log` method for v0.10.42 or earlier
    unless method_defined?(:log)
      define_method(:log) { $log }
    end
    ...
  end
end

This code defines log method using $log when log method is not defined, so log.error is same as $log.error on older Fluentd.

fluent-plugin-td uses same approach.

Conculusion

log_level feature is very useful for Fluentd users. So if you have a time, please apply above changes and release new version plugin ;)

I will release new version of several plugins soon, S3, TD, Mongo and etc.

Got Walter and Andrei autograph in our TDPL japanese edition :)