keyword mapping on elasticsearch

Issue:
For example there is field category. 
If fitler using event.category , will get result. 
But if using event.category.keyword, will not get result.



 1) Check the mapping

GET logs-*/_mapping

"event": {

"properties": {

"action": {

"type": "keyword",

"ignore_above": 1024

},

"agent_id_status": {

"type": "keyword",

"ignore_above": 1024

},

"category": {

"type": "keyword",

"ignore_above": 1024

},

"code": {

"type": "keyword",

"ignore_above": 1024

},

"created": {

"type": "date"

}

}

Look for category:

• If you see "type": "keyword" → the field is already keyword. Use category (not category.keyword).

• If you see "type": "text" and no "fields": { "keyword": ... } → there is no .keyword subfield.

• If you see a subfield:
  

  "category": {
    "type": "text",
    "fields": {
      "keyword": { "type": "keyword", "ignore_above": 256 }
    }
  }
  
  Be aware of ignore_above: strings longer than that number are not indexed on .keyword, which can yield 0 hits on category.keyword while category (text) still matches.

Escape character in Elasticsearch - 3 Layers

Quick Tips:
- semak dulu bagaimana data disimpan dalam elastic (lihat json, bukan table view)
- kemudian letak escape character utk setiap character yang perlu
Lihat Contoh 3

Contoh 1)
Data Display in Kibana:
  process.command_line : wc -l

Data Display in Json(Actual Data in Elastic)
process.command_line : wc -l
 (sama sebab tiada special character) 

Filter kql
 process.command_line :  * -l*

DSL query:

{ "wildcard": { "process.command_line": { "value": "* -l*" } } }

Contoh 2)
Data Display in Kibana:

process.command_line: "certutil -encode fee449ee0e3965a5246f000e87fde2a065fd89d4.crt temp.cer"

Data Display in Json(Actual Data in Elastic)

"command_line": "certutil -encode cdd4eeae6000ac7f40c3802c171e30148030c072.crt temp.cer "

(sama sebab tiada special character) 

Filter KQL

 process.command_line :  * -encode *

DSL Query
Kql above will translate by kibana to following DSL

{ "wildcard": { "process.command_line": {"value": "* \\-encode *" } } }

Contoh 3)
Data Display in Kibana:
"C:\Users\Asus\AppData\Local\Programs\Opera GX\autoupdate\opera_autoupdate.exe" --scheduledtask --bypasslauncher $(Arg0)

Data Display in Json(Actual String in Elastic):

"command_line": "\"C:\\Users\\Asus\\AppData\\Local\\Programs\\Opera GX\\autoupdate\\opera_autoupdate.exe\" --scheduledtask --bypasslauncher $(Arg0)"

 (tak sama sebab ada special character) 

Filter KQL

process.command_line: *\\Users\\* 

DSL Query

{ "wildcard": { "process.command_line": { "value": "*\\\\Users\\\\*" } } }

Elastic Alert Aliases

 GET /_cat/aliases?v

ada byk index yg alias nama lain.

Aliases                                     Index
.alerts-security.alerts-default     .internal.alerts-security.alerts-default-000024
.siem-signals-default                 .internal.alerts-security.alerts-default-000024

Permission, kena bagi secara berasingan. 

Di page 'Alerts', kena bagi permission index:

        .alerts-security.alerts-default

Di 'Discover', allow

        .siem-signales-default

Ahli Sunnah Wal Jamaah

who is ahlu Sunnah Wal Jamaah

ref: https://www.youtube.com/watch?v=oSYYd4RjKE8

elasticsearch- unassigned shard issue

unassigned shards: 

Shard masih tidak di assign pada mana mana node. maka data dalam shards tak available bila di query.

Sebab:

1- node down
2- free space tiada
3- dalam proses di pindahkan ke node yg lain.(ini utk pastikan bilangan node seimbang dalam semua node)

Diagnos:
- _cat/shards

- _cluster/health

- _cluster/reroute

$ curl -XPOST 'localhost:9200/_cluster/reroute' -H 'Content-Type: application/json' -d '{
    "commands": [{
        "allocate_empty_primary": {
            "index": "test-index",
            "shard": 1,
            "node": "node-1",
            "accept_data_loss": true
        }
    }]
}'
{"acknowledged":true,"state":{"cluster_uuid":"0W8o4rxdSniXsf_grVvxvQ","version":299,"state_uuid":"rHRyobYiSZ2VIOlxep0jxw","master_node":"nIzM4TPDQuS0WDHkSjEN1w","blocks":{}...

allocate_empty_primary  - utk shard primary

allocate_replica - utk shard replika 

 

- _cluster/settings

    cluster.routing.allocation.enable 

    cluster.routing.allocation.node_concurrent


 

 ref: https://www.baeldung.com/ops/elasticsearch-unassigned-shards

Salin MFT file

 Salin guna rawcopy

RawCopy.exe /FileNamePath:C:0 /OutputPath:C:\Audit /OutputName:MFT_C.bin

ref: 

https://www.jaiminton.com/cheatsheet/DFIR/#master-file-table

https://github.com/jschicht/RawCopy

Kibana KQL escape character

Need to escape these characters:
    \():<>"*

with quotes, does not need to escape:
    http.request.referrer: "https://example.com"

 
without quotes, must escape:

    http.request.referrer: https\://example.com

 

This not work as expected because * is interprate as literal *:
    http.request.referrer: "https://example.com*"

Instead use this:
    http.request.referrer: https\://example.com*

File path issue in windows:
Kibana display value as:
    C:\WINDOWS\system32\MRT.exe

But if you check in json, actual value stored in elastic is:
    "c:\\windows\\system32\\mrt.exe"

Thus, to find all files in folder system32(and sub folder) you need to escape the backslash character:
    file.path.caseless : c\:\\\\windows\\\\system32\\\\*

    

To find all files in folder system32(exclude sub folder):
    file.path.caseless : c\:\\\\windows\\\\system32\\\\* and not file.path.caseless : c\:\\\\windows\\\\system32\\\\*\\\\*

ref: https://www.elastic.co/guide/en/kibana/current/kuery-query.html