Logstash: custome index template

1) create template json

vim elasticsearch-template_v00005.json
{
        "index_patterns": "logstash_v00005-*",
        "settings": {
                "index" : {
                        "refresh_interval": "10s" ,
                        "number_of_shards" : 1,
                        "number_of_replicas" : 0
                }
       }
} 

2) import template and check

$ curl -XPUT -H 'Content-Type: application/json' 'http://localhost:9200/_template/logstash_v00005?pretty' -d@elasticsearch-template_v00005.json
{
  "acknowledged" : true
}

and check..

$ curl -XGET 'http://localhost:9200/_template/logstash_v00005?pretty'
{
  "logstash_v00005" : {
    "order" : 0,
    "index_patterns" : [
      "logstash_v00005-*"
    ],
    "settings" : {
      "index" : {
        "number_of_shards" : "1",
        "number_of_replicas" : "0",
        "refresh_interval" : "10s"
      }
    },
    "mappings" : { },
    "aliases" : { }
  }
}

3) set logstash output config

cat /etc/logstash/conf.d/98-elasticsearch-output.conf
output {
  elasticsearch {
    hosts => ["localhost:9200"]
    manage_template => true
    template_overwrite => true

    template => '/etc/logstash/elasticsearch-template_v00005.json'
    template_name => 'logstash_v00005'
    index => 'logstash_v00005-%{+YYYY.MM.dd}'
  }
  stdout { codec => rubydebug }
}

last, dont forget to restart your logstash

ref  https://github.com/elastic/logstash/issues/10687

filebeat -> logstash -> rabbitmq - > logstash -> elastic

1) filebeat - logstash
  normal case, you can google it

2) logstash - rabbitmq
  https://stackoverflow.com/questions/23207812/logstash-rabbitmq-output-never-posts-to-exchange

output { 
   rabbitmq {
      codec => plain
      host => localhost
      exchange => yomtvraps
      exchange_type => direct
      key => yomtvraps

      # these are defaults but you never know...
      durable => true
      port => 5672
      user => "guest"
      password => "guest"
   }
}

3) rabbitmq -logstash
  https://discuss.elastic.co/t/rabbitmq-as-logstash-input/95756
input { rabbitmq { host => "localhost" port => 15672 heartbeat => 30 durable => true exchange => "logging_queue" exchange_type => "logging_queue" } } output { elasticsearch { hosts => "localhost:9200" } stdout {} }




4) logstash - elastic
   normal case, please google

elastic roles privileges

To write/ingest indices,  user must have role with:
- cluster: "manage_index_templates", "monitor", "manage_ilm"
- indices privileges: "write","create","delete","create_index","manage","manage_ilm"



To read the indices, minumum priveleges:
- "read","view_index_metadata"

ref: https://www.elastic.co/guide/en/logstash/current/ls-security.html