decode unfragmented .log files:
- break leveldb_log.go into leveldb_log_blocks.go and leveldb_log.go;
the former is used by both .MANIFEST (descriptor) and .LOG.
- in leveldb_log, introduce readBatch that decodes further
fix UTF8 decoding:
- introduce fieldUTF8ReturnBytes and stringify to handle multi-byte
UTF8-encodings correctly.
MSDOS time/date was read in wrong order and also did not take into account
that the bit ranges in the shortis are in little-endian.
Remodel modification_time/date to be one struct with fat_time, fat_date LE shorts
and then synthetic values for day, hours, minute etc and also a unix field with the
timestamp as unix time.
Also refactor and clenaup extra fields/extended code a bit.
Fixes#792
This is used for >1gb files. Disable decode will speed up
deocde a lot but will probably also produce some gaps as same
part of the movi chunks will not be reference by the indx index.
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.
Update docs and add additional example for non-byte-aligned field.
Also fixes issue showing zero bit ranges as start-NA.
Remove -0 short arg, now it means the expression "-0".
This is to be in sync with jq 1.7 https://github.com/jqlang/jq/pull/2684
Correct and clarify that NUL and new lines are outputted after and not between each output.
Still missing is to abort on output containing zero when using --raw-output0
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.
Add lazyre package to help delay compile of RE and make it concurrency safe.
Skips gap fields in struct and arrays.
Gaps fields are bit ranges that a decoder did not add any fields for.
Note that skipping gaps in arrays will affect indexes.
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.
What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet
Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.
Will be base for decoding http2 and other TLS based on protocols.
Fixes#587
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.
Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.
This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
Feels less cluttered, easier to read and more consistent.
Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.
Also fixes lots of spelling errors while reading thru.
