Using rsyslog to send pre-formatted JSON to logstash

My last post was about sending pre-formatted JSON to logstash to avoid unnecessary grok parsing. In this post I will show how to do the same thing from rsyslog.

And again, this comes with a disclaimer. My exact model here depends on a version of logstash recent enough to have the udp input. You could do tcp here, but that’s not my example.

Prerequsites: rsyslog version 6+ (I used version 6.4.2, with more recent json patching). UPDATE: I just tested with 7.2.5 with success. The problem with earlier versions in the v7 branch were addressed in 7.2.2

bugfix: garbled message if field name was used with jsonf property option
The length for the field name was invalidly computed, resulting in either
truncated field names or including extra random data. If the random data
contained NULs, the rest of the message became unreadable.

For the record: 6.4.2 works, 6.6 does not, 7.2.5 does. These are the limits of my testing, so far.

rsyslog does what apache does (if you tell it to): escapes quotes and other characters so you can send legitimate JSON. What I did was create a template (including an @message field, to mimic what is normally logged), and then send everything to a local logstash agent over a UDP port.

## rsyslogd.conf
$ModLoad immark.so
$ModLoad imuxsock.so
$ModLoad imklog.so
$ModLoad imudp
# You only need $UDPServerRun if you want your syslog to be a centralized server.
$UDPServerRun 514
$AllowedSender UDP, 127.0.0.1, 172.19.42.0/24, [::1]/128

$template ls_json,"{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,"@source":"syslog://%fromhost-ip:::json%","@message":"%timestamp% %app-name%:%msg:::json%","@fields":{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}"

*.*                                             @localhost:55514;ls_json

This sends out everything (every level/severity) to localhost:55514 (UDP) and formats with the ls_json format as defined above.

Here’s the logstash agent config, which is listening on 55514:

## logstash.conf

input {
   udp {
      port => 55514
      type => "syslog"
      buffer_size => 8192
      format => "json_event"
   }
}

I don’t need any date magic here as the @timestamp works (still don’t know why it’s flaky for apache). These events are ready for consumption, no filters necessary!

You could send the JSON out to file. This is what an example line looks like:


## JSON output

{"@source":"syslog://127.0.0.1","@type":"syslog","@tags":[],"@fields":{"facility":"cron","severity":"info","program":"","processid":"10522"},"@timestamp":"2012-09-29T17:30:00.975141-05:00","@source_host":"blackbox","@message":"Sep 29 17:30:00 : (root) CMD (/usr/libexec/atrun)"}

If you have an existing, native syslog config, you can keep it as-is, and just add the lines above to it (and re-name it to rsyslogd.conf or something). rsyslogd will continue to write out to your same files in /var/log/*whatever* and also send in json format to port 55514. Again, the idea here is minimal invasiveness: Allow the logging to continue in the way it has been, but also forward it along to a centralized server.

30 thoughts on “Using rsyslog to send pre-formatted JSON to logstash”

  1. Hi, thanks for this great tip.

    There may be some problems with rsyslog 7 (I’m on 7.2) where it truncates the message right after %syslogfacility-text:::jsonf:facility%.

    To work around this, I had to remove the custom jsonf field name from the property. I.e. the “fields” property becomes:

    \"@fields\":{%syslogfacility-text:::jsonf%,%syslogseverity-text:::jsonf%,%app-name:::jsonf:program%,%procid:::jsonf%}

    Otherwise rsyslog will truncate the message after the first field which is strange.

    Hope this helps other people out until rsyslog gets fixed!

    Reply
    • Andrew Stangl says:

      Much appreciated! I was experiencing the same issue on CentOS7, with rsyslog 7.4.7 – amending the fields in the template, as suggested, sorted it out

      Reply

  3. Very helpful, Aaron. Have you been able to figure out a rsyslog (in my case 5.8.10) template to translate Apache’s (2.2) error logs to JSON?

    The main error log has no set format whatsoever, so I will send it to the central logging facility as is, no template. But an individual vhost’s error log, AFAIK, always follows the format specified in the apache man:

    [Timestamp] [Severity] [RemoteIP] msg

    It would be nice to have at least part of the error logs also in JSON…

    Reply
  4. Marcelo says:

    Thank you for this !
    I modified it to remove the date and program name from the msg field since we are already sending that to logstash but it seems that with this template rsyslog is not able to pass full paths as %app-name.
    For instance, the following gets logged from a remote machine using the default standard rsyslog template.

    Apr 1 14:40:30 somemachine.domain.com /project/gtvol0/libexec/sudo.d/Linux/i686/2.6/bin/sudo: someuser : TTY=pts/0 ; PWD=/login/someuser ; USER=jboss ; COMMAND=/usr/bin/crontab -l

    And this is what we get in logstash:

    @data={“@source”=>”syslog://x.x.x.x”, “@tags”=>[], “@fields”=>{“facility”=>”auth”, “severity”=>”notice”, “program”=>””, “processid”=>”-“}, “@timestamp”=>”2013-04-01T15:29:20.669758-05:00”, “@source_host”=>”somemachine.domain.com”, “@message”=>” marcelo : TTY=pts/0 ; PWD=/login/marcelo ; USER=root ; COMMAND=/bin/su -“, “@type”=>”syslog”}>,

    For completeness here’s the actual json output template:

    $template ls_json,”{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,\”@source\”:\”syslog://%fromhost-ip:::json%\”,\”@message\”:\”%msg:::json%\”,\”@fields\”:{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}”

    Reply
  5. Howie says:

    Your ‘smart-quotes’ plugin for your blogging software makes these long examples impossible to cut/paste… just thought you should know!

    (very useful discussion, although I haven’t successfully got rsyslog to connect to logstash just yet. The JSON validates, but logstash ignores it, even in log files)

    Reply
    • Aaron says:

      Really? I just tested it out and it cut/paste straight into vim on my Mac perfectly, and maintained the spacing and everything. What OS and browser are you using?

      Reply
      • Howie says:

        Interesting. Chrome on Windows. I get dots where the quotes were. I often see similar things with sites that inadvertently convert double-dash gnu-style options in em-dashes…

  6. Sam says:

    Great article, I am using logback json encoder in my app. I get the following line
    {“@timestamp”:”2013-10-08T17:14:21.870-07:00″,”@message”:”THisis the new JSON format test 1000″,”@fields”:{“logger_name”:”com.test.foo.”,”thread_name”:”main”,”level”:”DEBUG”,”level_value”:20000,”HOSTNAME”:”newhost”},”@tags”:null},

    The above format is not properly parsed when I use format as json_event, Instead the whole line taken as a message. Instead of tokenizing on the different json elements. Any help?

    Reply
    • Aaron says:

      What version of Logstash are you using? That logback encoder sends in the format used by Logstash 1.1.x. If you’re using Logstash 1.2.x you’ll need to use a slightly different approach.

      Reply
    • Aaron says:

      I’m not sure, because none of my rsyslog messages are multi-line. I would presume, however, that a “multi-line” message in rsyslog still only has one timestamp. If that’s so, it should send that big message as a single event still.

      I would hope that rsyslog is smarter than to send multi-line messages as separate lines with separate timestamps.

      Reply
  8. Hemant says:

    Hey I am using the templet format that you mentioned above but its not sending my syslog.

    $template ls_json,”{%timestamp:::date-rfc3339,jsonf:@timestamp%,%source:::jsonf:@source_host%,\”@source\”:\”syslog://%fromhost-ip:::json%\”,\”@message\”:\”%timestamp% %app-name%:%msg:::json%\”,\”@fields\”:{%syslogfacility-text:::jsonf:facility%,%syslogseverity-text:::jsonf:severity%,%app-name:::jsonf:program%,%procid:::jsonf:processid%}}”

    Reply
    • Aaron says:

      This is kind of outdated now, especially if you look at the date.

      Try my current configuration at this gist, which works with more current releases of Rsyslog.

      Reply
  9. Hemant says:

    Hey I am using syslog server, now i want write template that convert syslog into json format as you have done above in rsyslog server. can anybody tell me how i can convert system log into json ????

    Thanks in advance

    Reply
    • Aaron says:

      Two ways to do that. One is inside Rsyslog, the other is in Logstash.

      Underneath:
      constant(value="\",\"@version\":\"1")
      You could add:
      constant(value="\",\"custom_tag\":\"tag_value")

      In Logstash, you’d add that in the input block:
      add_field => { "custom_tag" => "tag_value" }

      Does that help?

      Reply
      • Aaron says:

        tag_value from a file? As in, something from the file name? Or some content from within the file?

        Creative application of mutate filters could get the job done with the file name, but there is no clever way to do it with file content unless each line of the file has the desired tag.

Leave a reply to Aaron Cancel reply