Update 2015-08-31: My most recent template/mapping can be found here.
2013-11-07: Another year, and things in the Logstash and Elasticsearch have grown and changed considerably. I am now employed by Elasticsearch to work on Logstash. This was one of the first things they wanted me to work on. So I am announcing that a new and improved, Logstash v1.2+ compatible mapping template is coming. It will not be on my personal site, however. It will be on http://www.elasticsearch.org in the main documentation there. I will paste the link here as soon as it’s available. In the interim, you can find a Github gist version here.
Expect this post to get updated from time to time. You can come back here to check out what I’m using and why.
2012-11-05: I now map IP addresses (clientip field) as type IP to allow for range searches. I also map the fields in the geoip filter output to allow for non-analyzed terms facet output (allows full city names with spaces; proper capitalization, etc.)
DO NOT USE THIS with Logstash v1.2+. This is deprecated and remains here as an archived example!
curl -XPUT http://localhost:9200/_template/logstash_per_index -d '
{
"template" : "logstash*",
"settings" : {
"number_of_shards" : 4,
"index.cache.field.type" : "soft",
"index.refresh_interval" : "5s",
"index.store.compress.stored" : true,
"index.query.default_field" : "@message",
"index.routing.allocation.total_shards_per_node" : 4
},
"mappings" : {
"_default_" : {
"_all" : {"enabled" : false},
"properties" : {
"@fields" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"clientip" : { "type": "ip" },
"geoip" : {
"type" : "object",
"dynamic": true,
"path": "full",
"properties" : {
"area_code" : { "type": "string", "index": "not_analyzed" },
"city_name" : { "type": "string", "index": "not_analyzed" },
"continent_code" : { "type": "string", "index": "not_analyzed" },
"country_code2" : { "type": "string", "index": "not_analyzed" },
"country_code3" : { "type": "string", "index": "not_analyzed" },
"country_name" : { "type": "string", "index": "not_analyzed" },
"dma_code" : { "type": "string", "index": "not_analyzed" },
"ip" : { "type": "string", "index": "not_analyzed" },
"latitude" : { "type": "float", "index": "not_analyzed" },
"longitude" : { "type": "float", "index": "not_analyzed" },
"metro_code" : { "type": "float", "index": "not_analyzed" },
"postal_code" : { "type": "string", "index": "not_analyzed" },
"region" : { "type": "string", "index": "not_analyzed" },
"region_name" : { "type": "string", "index": "not_analyzed" },
"timezone" : { "type": "string", "index": "not_analyzed" }
}
}
}
},
"@message": { "type": "string", "index": "analyzed" },
"@source": { "type": "string", "index": "not_analyzed" },
"@source_host": { "type": "string", "index": "not_analyzed" },
"@source_path": { "type": "string", "index": "not_analyzed" },
"@tags": { "type": "string", "index": "not_analyzed" },
"@timestamp": { "type": "date", "index": "not_analyzed" },
"@type": { "type": "string", "index": "not_analyzed" }
}
}
}
}
'
If you enjoyed this article, please consider sharing it!
Apologies for the potential unreadability of the code, here. It should still cut/paste properly.
Please add the following css to your blog:
pre {
overflow: scroll;
background-color: rgb(240, 240, 240);
}
Will make it much more readable.
I will try. The template and CSS hierarchy may not let me.
Cool! That worked.
[…] Home > Geek > Using elasticsearch mappings appropriately to map as type IP, int, float, etc. Using rsyslog to send pre-formatted JSON to logstash My current template/mapping […]
[…] My Logstash Notes – Jan-Piet Mens Untergeek – Current Template/Mapping 08-2012 Modern Log Management and Monitoring 08-2011 Write Logs for Machines, use […]
[…] a reply My Logstash Notes – Jan-Piet Mens Untergeek – Current Template/Mapping Modern Log Management and Monitoring 08-2011 Write Logs for Machines, use JSON 12-2011 Centralized […]