Update 2015-08-31: My most recent template/mapping can be found here.

2013-11-07: Another year, and things in the Logstash and Elasticsearch have grown and changed considerably.  I am now employed by Elasticsearch to work on Logstash.  This was one of the first things they wanted me to work on.  So I am announcing that a new and improved, Logstash v1.2+ compatible mapping template is coming.  It will not be on my personal site, however.  It will be on http://www.elasticsearch.org in the main documentation there.  I will paste the link here as soon as it’s available.  In the interim, you can find a Github gist version here.

Expect this post to get updated from time to time. You can come back here to check out what I’m using and why.

2012-11-05: I now map IP addresses (clientip field) as type IP to allow for range searches. I also map the fields in the geoip filter output to allow for non-analyzed terms facet output (allows full city names with spaces; proper capitalization, etc.)

DO NOT USE THIS with Logstash v1.2+.  This is deprecated and remains here as an archived example!

curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
    "template" : "logstash*",
    "settings" : {
        "number_of_shards" : 4,
        "index.cache.field.type" : "soft",
        "index.refresh_interval" : "5s",
        "index.store.compress.stored" : true,
        "index.query.default_field" : "@message",
        "index.routing.allocation.total_shards_per_node" : 4
    },
    "mappings" : {
        "_default_" : {
            "_all" : {"enabled" : false},
            "properties" : {
               "@fields" : {
                    "type" : "object",
                    "dynamic": true,
                    "path": "full",
                    "properties" : {
                        "clientip" : { "type": "ip" },
                        "geoip" : {
                            "type" : "object",
                            "dynamic": true,
                            "path": "full",
                            "properties" : {
                                    "area_code" : { "type": "string", "index": "not_analyzed" },
                                    "city_name" : { "type": "string", "index": "not_analyzed" },
                                    "continent_code" : { "type": "string", "index": "not_analyzed" },
                                    "country_code2" : { "type": "string", "index": "not_analyzed" },
                                    "country_code3" : { "type": "string", "index": "not_analyzed" },
                                    "country_name" : { "type": "string", "index": "not_analyzed" },
                                    "dma_code" : { "type": "string", "index": "not_analyzed" },
                                    "ip" : { "type": "string", "index": "not_analyzed" },
                                    "latitude" : { "type": "float", "index": "not_analyzed" },
                                    "longitude" : { "type": "float", "index": "not_analyzed" },
                                    "metro_code" : { "type": "float", "index": "not_analyzed" },
                                    "postal_code" : { "type": "string", "index": "not_analyzed" },
                                    "region" : { "type": "string", "index": "not_analyzed" },
                                    "region_name" : { "type": "string", "index": "not_analyzed" },
                                    "timezone" : { "type": "string", "index": "not_analyzed" }
                            }
                        }
                    }
               },
               "@message": { "type": "string", "index": "analyzed" },
               "@source": { "type": "string", "index": "not_analyzed" },
               "@source_host": { "type": "string", "index": "not_analyzed" },
               "@source_path": { "type": "string", "index": "not_analyzed" },
               "@tags": { "type": "string", "index": "not_analyzed" },
               "@timestamp": { "type": "date", "index": "not_analyzed" },
               "@type": { "type": "string", "index": "not_analyzed" }
            }
        }
    }

}
'
Tagged with:
 

7 Responses to DEPRECATED! My current template/mapping

  1. Aaron says:

    Apologies for the potential unreadability of the code, here. It should still cut/paste properly.

  2. Please add the following css to your blog:


    pre {
    overflow: scroll;
    background-color: rgb(240, 240, 240);
    }

    Will make it much more readable.

  3. Aaron says:

    Cool! That worked.

  4. […] Home > Geek > Using elasticsearch mappings appropriately to map as type IP, int, float, etc. Using rsyslog to send pre-formatted JSON to logstash My current template/mapping […]

  5. […] My Logstash Notes – Jan-Piet Mens     Untergeek – Current Template/Mapping    08-2012   Modern Log Management and Monitoring 08-2011   Write Logs for Machines, use […]

  6. […] a reply My Logstash Notes – Jan-Piet Mens Untergeek – Current Template/Mapping Modern Log Management and Monitoring 08-2011 Write Logs for Machines, use JSON 12-2011 Centralized […]

Leave a Reply

Your email address will not be published. Required fields are marked *