elastic: how many shard should I have

Aim for shard sizes between 10GB and 50GBedit

ref:    https://www.elastic.co/guide/en/elasticsearch/reference/current/size-your-shards.html

To check shard size:

GET _cat/shards?v=true&h=index,prirep,shard,store&s=prirep,store&bytes=gb&index=index_name*

#the store field should be in GB unit

-----------------------old-------------------------

settings": {
    "index": {
      "number_of_shards": "1",
      "number_of_replicas": "1",


number _of_shard: how many shard per index
number_of_replicas: how many clone for each shard



#notes to have different index for each day, might be too much(max shard per ES node is 1000.)
You should try separate the index per month, (and increase number_of_shards).

How much is too many shard? Basically number of shard is ~ number of total CPU core across your cluster.(or you might one to double it if you have Hyper-Threading). More than this, it might not help you to go any quicker in searching data.

#some say it would be ok if your shard size between 2GB - 8GB


ref:  https://qbox.io/blog/optimizing-elasticsearch-how-many-shards-per-index

filebeat -> logstash -> rabbitmq - > logstash -> elastic

1) filebeat - logstash
  normal case, you can google it

2) logstash - rabbitmq
  https://stackoverflow.com/questions/23207812/logstash-rabbitmq-output-never-posts-to-exchange

output { 
   rabbitmq {
      codec => plain
      host => localhost
      exchange => yomtvraps
      exchange_type => direct
      key => yomtvraps

      # these are defaults but you never know...
      durable => true
      port => 5672
      user => "guest"
      password => "guest"
   }
}

3) rabbitmq -logstash
  https://discuss.elastic.co/t/rabbitmq-as-logstash-input/95756
input { rabbitmq { host => "localhost" port => 15672 heartbeat => 30 durable => true exchange => "logging_queue" exchange_type => "logging_queue" } } output { elasticsearch { hosts => "localhost:9200" } stdout {} }




4) logstash - elastic
   normal case, please google

good tmux tutorial for beginner

https://www.hamvocke.com/blog/a-quick-and-easy-guide-to-tmux/


https://linuxize.com/post/getting-started-with-tmux/

Below are some most common commands for managing Tmux windows and panes:

• Ctrl+b c Create a new window (with shell)
• Ctrl+b w Choose window from a list
• Ctrl+b 0 Switch to window 0 (by number )
• Ctrl+b , Rename the current window
• Ctrl+b % Split current pane horizontally into two panes
• Ctrl+b " Split current pane vertically into two panes
• Ctrl+b o Go to the next pane
• Ctrl+b ; Toggle between the current and previous pane
• Ctrl+b x Close the current pane

elastic roles privileges

To write/ingest indices,  user must have role with:
- cluster: "manage_index_templates", "monitor", "manage_ilm"
- indices privileges: "write","create","delete","create_index","manage","manage_ilm"



To read the indices, minumum priveleges:
- "read","view_index_metadata"

ref: https://www.elastic.co/guide/en/logstash/current/ls-security.html

setup Proxmox cluster, and ceph storage to achieve hyperconvergenc

https://www.youtube.com/watch?v=GgliWaOfvsA

combining multiple condition fo refine your searching in elasticsearch

a good example to learn

Bool Query fields:
- must    (and)
- must_not
- should  (or)
- filter


example 1 :(field_1 = "mana" AND field_2 = "mari")
{ "query" : { "bool" : { "must": [{ "match": { "field_1": "mana" } }, { "match": { "field_2": "mari" } }] } } }



example 2 :(field_1 != "mana"  AND field_2 != "mari")
{ "query" : { "bool" : { "must_not": [{ "match": { "field_1": "mana" } }, { "match": { "field_2": "mari" } }] } } }



example 3 : (field_1 = "mana" OR field_2 = "mari")
{ "query" : { "bool" : { "should": [
{ "match": { "field_1": "mana" } },
{
"match": { "field_2": "mari" } }] } } }


example 4 : (field_1 = "mana")

{

  "query": {

    "bool" : {

      "filter" : {          "term": {          "field_1": "mana"         }       }

    }

  }

}

### filter is much less expensive, as it will NOT have scoring

example 5: to combine filter and others:

(( field_1 = "mana" and field_2 = "mari") and field_3 = "arah")
{ "query" : { "bool" : { "must": [{ "match": { "field_1": "mana" } }, { "match": { "field_2": "mari" } }],
"filter" : { "term": { "field_3": "arah" } } }
} }
### again, 'filter' is less expensive compare to 'match', as it will NOT have scoring

ref:
https://www.elastic.co/blog/lost-in-translation-boolean-operations-and-filters-in-the-bool-query

painless (elasticsearch)

To check is field exist:
if (doc['alert.metadata.direction.keyword'].size()!=0){
  // do something
}

or
if (!doc['alert.metadata.direction.keyword'].empty){
  // do something
}