python elasticsearch

To read a documents:
  res = es.get(index='rules-test',
          doc_type='_doc',
          id='cJG_UnEB1e0J32pjx0pG')



To update a documents:
   doc = {
    "doc": {           #      < - - cannot use other than "doc"
        "new_or_existing_field": {
        "sub_field": {
        "other_name" : "cadang1",
        "other_class": "cadang2"
        }
        }
    }
}

   es.update(index='rules-test',
          doc_type='_doc',
          id='cJG_UnEB1e0J32pjx0pG',
          body=doc)




To filter result:
  query = {
                "query": {
                    "bool": {
                        "must": [
                            {   "term": {
                                    "feed.keyword": args.feed
                                }
                            },
                            {   "term": {
                                    "version.keyword": args.version
                                }
                            }
                        ]
                    }
                },
                "aggs": {
                    "uniq_f_name": {
                        "terms": {
                            "field": "file_name.keyword",
                            "size": 200
                        }
                    }
                },
                "size": 0
            }
    result = es.search(index=args.es_index, body= query)
    # print(json.dumps(query), "\n",  result)

    for bucket in result['aggregations']['uniq_f_name']['buckets']:
        print( bucket)

To count: 
- add 'size=0' parameter when call search() method
res = client.search(index = "indexname", doc_type = "doc_type", body = q, size=0)

# if res['hits']['total']['value'] is greater than 10K, means total record is larger, and you need to use scroll to retrieve all the documents.

To use scroll:

result = es.search(index="indexname", body= query, scroll = '1m')

while result['hits']['hits'] > 0:

      result = es.scroll(scroll_id= scroll_id, scroll = '1m')

      print(result['hits']['hits'])

Elasticsearch query API

List all indices:
  GET /_cat/indices


List all field in index:

  GET /rules-test

List all documents(records) in index:

   GET /rules-test/_search/

Get document by ID:

   GET /rules-test/_doc/cJG_UnEB1e0J32pjx0pG



Count documents:
   GET /rules-test/_count?q=user:kimchy

Timezone for mft, plaso, elasticsearch

MFT store timestone in UTC, user(window explorer) will convert to chosen timezone when display to user.

log2timeline will parse the disk partition and put data in dot.plaso file, using UTC timezone.

When psort.py run againts dot.plaso, it will produce dot.csv file, using UTC timezone.


When data in the csv is push into elasticsearch, elasticsearch will always assume the timezone is UTC,

Then, when kibana display the data through browser, it will convert the timezone base on browser timezone(which is same as user desktop timezone).


But bare in mind, if you query directly to elasticsearch using your own script/tools, the timezone is in UTC.

Missleading psort.py (plaso) timezone paremeter

psort.py -z Singapore  -o l2tcsv -w result.csv  input_data.plaso



psort.py has '-z' option which is for timezone parameter.

But it will not convert the time to the preferred timezone, instead it just put the label.


For example if we run the command with '-z UTC' , it will result as:
     04/15/2020,23:59:26,UTC, ........

And if we run the psort command with  -z Singopore, the result is:
   04/15/2020,23:59:26,Singapore, .....


The timestamp is exactly same.

Kibana substring in scripted fields

Let say we have field datetime with type date in elasticsearch (eg: "2018-09-15T17:16:47") .
We want to get value 17 (hours),

We can achieve this by create Scripted field either  (1) in kibana scripted field, or (2)put it directly in your query.
Name it to sc-hour_myt.


(1) kibana scripted field.

String masa = String.valueOf(doc['datetime'].value); 

// since my timezone is +0800

int hour = Integer.parseInt(masa.substring(11,13)) + 8 ; 

if (hour >= 24) {

hour = hour - 24;  

}

String minute = masa.substring(14,16);

String second = masa.substring(17,19);

String utc = String.valueOf(hour) + minute + second;

return hour;

(2) put directly in your query:
{
  "query": {
    "match_all": {}
  },
  "_source": ["datetime"],
  "script_fields": {
    "sc-hour_myt": {
      "script": {
        "lang": "painless",
        "source": """
              String masa = String.valueOf(doc['datetime'].value);
              int hour = Integer.parseInt(masa.substring(11,13)) + 8 ;
              if (hour >= 24) {
              hour = hour - 24; 
              }
              String minute = masa.substring(14,16);
              String second = masa.substring(17,19);
             
              String utc = String.valueOf(hour) + minute + second;
              return hour;
        """
      }
    }
  }
}


--------------------------------------------------------------

This will result:
{ "_id": "CqSU6nEBOMr994UREJt1", "datetime": "2018-09-15T07:16:47", "sc-hour_myt": [ 15 ] }, { "_id": "C6SU6nEBOMr994UREJt1", "datetime": "2018-09-15T07:16:47", "sc-hour_myt": [ 15 ] },

Kawal masa anak guna komputer(ubuntu)

Penat je buat sendiri timecontrol https://github.com/cperdana/limit/blob/master/README.md . Dah siap baru terjumpa available apps 

https://www.linuxuprising.com/2019/11/timekpr-next-is-linux-parental-control.html

haih...

Analisa MFT

1) Extract MFT using icat


2) parse using MFTAnalyzer
     analyzeMFT.py -f mft.raw -o mftanalyzed.csv


ref: https://www.andreafortuna.org/2017/07/18/how-to-extract-data-and-timeline-from-master-file-table-on-ntfs-filesystem/



notes: can use https://github.com/jschicht/Mft2Csv/wiki/Mft2Csv to parse from live system