python3 encode - bytes - hex -fromhex - decode

[string] encode() >>  [bytes] 

[bytes] hex() >>  [hex_string] 

[hex_string] fromhex() >> [bytes] 

[bytes] decode() >> [string]

1) bytes dan encode()

k = bytes("4d", "utf-8")

setara dengan

j = "4d".encode()

>> k = j = b"4d"

2) hex()

k = bytes("a4d", "utf-8")
>  k = b'a4d'

m = k.hex()
> m = '613464'

3) fromhex() dan decode()

n = bytes.fromhex(m)
> n = b'a4d'

p = n.decode("utf-8")
> p = 'a4d'

UTF Notes:

 encoded

j = "4d".encode("utf-8")        

>> j = b'4d'

j = "4d".encode("utf-16")

>> j = b'\xff\xfe4\x00d\x00'

pip install package on previous version

 1- to list available package:

   pip install thepackage==randomwords

(attemp to install unexist package, will return error, and pip will display available package)

2- install certain version of package

   pip install libvirt-python==5.10.0

python elasticsearch

To read a documents:
  res = es.get(index='rules-test',
          doc_type='_doc',
          id='cJG_UnEB1e0J32pjx0pG')



To update a documents:
   doc = {
    "doc": {           #      < - - cannot use other than "doc"
        "new_or_existing_field": {
        "sub_field": {
        "other_name" : "cadang1",
        "other_class": "cadang2"
        }
        }
    }
}

   es.update(index='rules-test',
          doc_type='_doc',
          id='cJG_UnEB1e0J32pjx0pG',
          body=doc)




To filter result:
  query = {
                "query": {
                    "bool": {
                        "must": [
                            {   "term": {
                                    "feed.keyword": args.feed
                                }
                            },
                            {   "term": {
                                    "version.keyword": args.version
                                }
                            }
                        ]
                    }
                },
                "aggs": {
                    "uniq_f_name": {
                        "terms": {
                            "field": "file_name.keyword",
                            "size": 200
                        }
                    }
                },
                "size": 0
            }
    result = es.search(index=args.es_index, body= query)
    # print(json.dumps(query), "\n",  result)

    for bucket in result['aggregations']['uniq_f_name']['buckets']:
        print( bucket)

To count: 
- add 'size=0' parameter when call search() method
res = client.search(index = "indexname", doc_type = "doc_type", body = q, size=0)

# if res['hits']['total']['value'] is greater than 10K, means total record is larger, and you need to use scroll to retrieve all the documents.

To use scroll:

result = es.search(index="indexname", body= query, scroll = '1m')

while result['hits']['hits'] > 0:

      result = es.scroll(scroll_id= scroll_id, scroll = '1m')

      print(result['hits']['hits'])

virtualenv guna python3

1- link python ke python3

update-alternatives --remove python /usr/bin/python2
update-alternatives --install /usr/bin/python python /usr/bin/python3 1

pastikan python --version paparkan python3

2-  link pip ke pip3

  -install

    apt-get install python3-pip

  which pip3

  - link kan

  ln -s /usr/bin/pip3 /usr/bin/pip

  pip -V

3- install virtualenv

  pip install virtualenv

  - setup virtual env for flask(eg)

    virtualenv --python=python3 flask_env

4- enter into flask_env

  source/flask_env/bin/activate

  - prompt will be change

   # (flask_env) mycert@my_dir:

5- nak keluar virt env

   deactivate

ref:

1- https://www.howtoforge.com/tutorial/how-to-install-django-1-10-on-ubuntu-16-04/

2- https://www.howtoforge.com/tutorial/how-to-install-django-on-ubuntu/

Python virtual environment

mkvirtualenv [-a project_path] [-i package] [-r requirements_file] [virtualenv options] ENVNAME

lsvirtualenv [-b] [-l] [-h]

showvirtualenv [env]

rmvirtualenv ENVNAME

workon [(-c|--cd)|(-n|--no-cd)] [environment_name|"."]

deactivate

lssitepackages

virtualenvwrapper

ref: http://virtualenvwrapper.readthedocs.io/en/latest/command_ref.html

gevent not so parallel

2 issue here.

1) gevent.pool is not so perfect. It would not start new thread untill all running thread exit.
Perfect implementation should start a thread immediately when there is a vacant in the pool.

2) the thread seem not execute parallel. In below example, function do_work_function clearly not running in parallel.

Really feel been bluffing all this while..... thanx 0xpcode :)



from gevent.pool import Pool
import time
import logging

rows = range(10)
CONCURRENCY = 4
pool = Pool(CONCURRENCY)

def do_work_function(param1):
print "start" , str(param1 )
    time.sleep(4)
    print "--finish" , str(param1 )
 
for row in rows:
  #logging.info(count)
  pool.spawn(do_work_function,row) # blocks here when pool size == CONCURRENCY
  print "spawn thread %s" % str(row)

print "start join"
pool.join(timeout=1) #blocks here until the last 10 are complete
print "end join" 

python package

install location (ubuntu 14.04)
/usr/local/lib/python2.7/dist-packages/secfeed-0.1-py2.7.egg/secfeed/

python virtual thread

gevent

greenlet -> virtual thread instance



gevent.pool.Pool(3)                       # limit concurrent thread to 3,
                                                 #habis satu baru run satu lagi
pool.spawn(function_name)

ConfigParser python multiple line

kalau nak handle multiple line, mesti ada space atau tab selepas CRLF

ref:http://www.velocityreviews.com/forums/t516084-reading-multiline-values-using-configparser.html

python list, dict, tuple

>>> list = ['satu', 'dua', 333, 456546, 'infinite']
>>> print list[1]
dua
>>> list
['satu', 'dua', 333, 456546, 'infinite']
ref >> http://www.tutorialspoint.com/python/python_lists.htm



>>> my_tuples = 'satu', 'dua', 333, 'infinite'
>>> my_tuples[1]
dua
>>> my_tuples
('satu', 'dua', 333, 'infinite')
ref >>> http://www.tutorialspoint.com/python/python_tuples.htm




>>> dict = {'Alice': '2341', 'Beth': '9102', 'Cecil': '3258'}
>>> dict['Beth']
9102
>>> dict
{'Alice': '2341', 'Beth': '9102', 'Cecil': '3258'}

ref >>> http://www.tutorialspoint.com/python/python_dictionary.htm

python pgp

$ sudo apt-get install python-pip $ sudo pip install python-gnupg

ref >>> http://www.gefira.pl/blog/2011/04/11/securing-sec-walls-secrets-with-gnupg-and-python-gnupg/

import gnupg

gpg = gnupg.GPG(gnupghome='/home/testgpguser/gpghome')
with open('my-encrypted.txt.gpg', 'rb') as f:
    status = gpg.decrypt_file(f, passphrase='my passphrase', output='my-decrypted.txt')

print 'ok: ', status.ok
print 'status: ', status.status
print 'stderr: ', status.stderr

ref >> http://www.saltycrane.com/blog/2011/10/python-gnupg-gpg-example/

python senarai method

dir(nama_module)

help(nama_module)

jejak ralat libemu

1) run python dlm gdb(maksudnya, debug mode)

# gdb /usr/bin/python

2) Start python

(gdb) run

3) Jika nama file libemu_test.py, maka

>> import libemu_test
0x401019L hex: A83D asm: test al,0x3d

Program received signal SIGSEGV, Segmentation fault.
0x0000000000000000 in ?? ()

4) Back trace

(gdb) bt
#0  0x0000000000000000 in ?? ()
#1  0x00007ffff58650f6 in emu_cpu_step (c=0xc3fb40) at emu_cpu.c:839
#2  0x00007ffff5b51fbb in __pyx_pf_8pylibemu_8Emulator_25cpu_step (__pyx_v_self=, unused=)
    at src/pylibemu.c:6735
#3  0x000000000042afec in PyEval_EvalFrameEx ()
#4  0x00000000004317f2 in PyEval_EvalCodeEx ()
#5  0x000000000054a078 in PyImport_ExecCodeModuleEx ()
#6  0x000000000050d091 in ?? ()
#7  0x000000000050da8b in ?? ()

5) tukar kepada frame 1  (frame ni refering to stack) (rasanya lah:))

(gdb) frame 1
#1  0x00007ffff58650f6 in emu_cpu_step (c=0xc3fb40) at emu_cpu.c:839
839                     ret = c->cpu_instr_info->function(c, &c->instr.cpu);

6)

(gdb) ptype c
type = struct emu_cpu {
    struct emu *emu;
    struct emu_memory *mem;
    uint32_t debugflags;
    uint32_t eip;
    uint32_t eflags;
    uint32_t reg[8];
    uint16_t *reg16[8];
    uint8_t *reg8[8];
    struct emu_instruction instr;
    struct emu_cpu_instruction_info *cpu_instr_info;
    uint32_t last_fpu_instr[2];
    char *instr_string;
    _Bool repeat_current_instr;
    struct emu_track_and_source *tracking;
} *

7)

(gdb) print c
$1 = (struct emu_cpu *) 0xc3fb40

8)

(gdb) print c->cpu_instr_info
$2 = (struct emu_cpu_instruction_info *) 0x7ffff5aedbc0

9)

(gdb) print c->cpu_instr_info->function
$3 = (int32_t (*)(struct emu_cpu *, struct emu_cpu_instruction *)) 0

sah yang problem adalah kat sini, sebab value adalah 0 (kosong)(yg paling kanan). Maksudnya pointer fungsi 'function' point to null.

Maka kena semak code yang initialize cpu_instr_info-> function

10)

(gdb) ptype c->cpu_instr_info
type = struct emu_cpu_instruction_info {
    int32_t (*function)(struct emu_cpu *, struct emu_cpu_instruction *);
    const char *name;
    struct {
        uint8_t s_bit : 1;
        uint8_t w_bit : 1;
        uint8_t modrm_byte : 4;
        uint8_t imm_data : 3;
        uint8_t disp_data : 3;
        uint8_t level : 2;
        uint8_t type : 2;
        uint8_t fpu_info : 1;
    } format;
} *

python binary

2 baris berikut, sama:
packed_data = binascii.unhexlify('0100000061620000cdcc2c40')
packed_data = '0100000061620000cdcc2c40'.decode('hex')

pack/unpack

Apa maksud pack(python)


Compare to C, in Python, the data like list may not be stored as contiguous chunk of bytes.

C variable
     +-----+         +--------+
     |array|-------> |10|9|2|3|
     +-----+         +--------+
                     contiguous bytes

Python variable

     +-----+         +-------+
     |array|-------> |.|.|.|.|
     +-----+         +-------+
                      | | | |
                      | | | |
                     10 9 2 3

To handle them, it is important to convert python values to "C" structs, i.e pack them as contiguous bytes of data or dissemble a contiguous chunk of bytes to Python objects.



ref>> http://pyfunc.blogspot.com/2010/09/python-and-binary-data-part-3.html

python split string by length

def split_len(seq, length):

    return [seq[i:i+length] for i in range(0, len(seq), length)]

ref(http://code.activestate.com/recipes/496784-split-string-into-n-size-pieces/)