Fluentd is a flexible and robust event log collector, but Fluentd doesn’t have own data-store and Web UI.
If you want to analyze the event logs collected by Fluentd, then you can use Elasticsearch and Kibana :)
Elasticsearch is an easy to use Distributed Search Engine and Kibana is an awesome Web front-end for Elasticsearch.
I tested on Mac OS X.
Java for Elasticsearch
Use Mac OS X’s Java.
% java -version
java version "1.6.0_51"Java(TM) SE Runtime Environment (build 1.6.0_51-b11-457-11M4509)Java HotSpot(TM) 64-Bit Server VM (build 20.51-b01-457, mixed mode)
Ruby for Fluentd
In this article, I use my rbenv’s Ruby and Fluentd gem directly.
After Fluentd flushed received events to Elasticsearch, you can analyze the event logs via Kibana!
Following image is one panel example:
Kibana has some built-in panels, so you can create own dashboard easily. See Kibana demo
If your service has a high traffic, then fluent-plugin-elasticsearch sometimes get stucked.
In this case, built-in out_roundrobin plugin is useful.
You can distribute a write request to elasticsearch nodes for load balancing.
Of course, putting Queue with multiple fluentd nodes is an another approach.
This article introduced Fluentd, Elasticsearch and Kibana combination to analyze the event logs.
These components are easy to setup and work fine, so you can try this framework soon!
I heard many companies have already tried / deployed this setup on production :)