With the advent of Logstash 1.4.1, I wanted to make sure everyone knows about the new collectd codec.

In Logstash 1.3.x, we introduced the collectd input plugin.  It was awesome!  We could process metrics in Logstash, store them in Elasticsearch and view them with Kibana.  The only downside was that you could only get around 3100 events per second through the plugin.  With Logstash 1.4.0 we introduced a newly revamped UDP input plugin which was multi-threaded and had a queue.  I refactored the collectd input plugin to be a codec (with some help from my co-workers and the community) to take advantage of this huge performance increase.  Now with only 3 threads on my dual-core Macbook Air I can get over 45,000 events per second through the collectd codec!

So, I wanted to provide some quick examples you could use to change your plugin configuration to use the codec instead.

The old way:

input {
  collectd {}
}

The new way:

input {
  udp {
    port => 25826         # Must be specified. 25826 is the default for collectd
    buffer_size => 1452   # Should be specified. 1452 is the default for recent versions of collectd
    codec => collectd { } # This will invoke the default options for the codec
    type => "collectd"
  }
}

This new configuration will use 2 threads and a queue size of 2000 by default for the UDP input plugin. With this you should easily be able to break 30,000 events per second!

I have provided a gist with some other configuration examples. For more information, please check out the Logstash documentation for the collectd codec.

Happy Logstashing!