As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.
Add lazyre package to help delay compile of RE and make it concurrency safe.
I think this is closer to what one would expect.
With this fq -d bytes 'scan("...")' will match as a binary and also return binaries,
before as raw decode value one would have to use tobytes to get the binary.
So raw binay is only outputted if stdout is not a terminal and "... | d" is not used..
fq -n '[1,2,3] | tobytes' | cat > binary
fq -n '[1,2,3] | tobytes | d' | cat > hexdump
Skips gap fields in struct and arrays.
Gaps fields are bit ranges that a decoder did not add any fields for.
Note that skipping gaps in arrays will affect indexes.
Now ex "tovalue | .some.thing" on a decode value will make some.thing be jq value
instead of a decode value which woud be displayed as a decode treee, seems confusing.
I think this is more intuetive and make more sense.
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.
This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
Before it failed when passing a decode value, like an integer, string etc, as a object value
to an internal function (written in go), ex some_function({name: <decode value>}).
A future improvement could be to allow an internal function to say that it wants a "raw"
decode value etc and not one converted to primitive type.
This preserves the callstack on non-recoverable panics so that using
a debugger and fuzzing is much easier.
Add vscode debug config.
Remove fuzz stacktrace log workaround.
readline Config was used to pass completer function per readline call,
was changed in #612 and caused regression. Now use our own member in
stdOS to pass it instead.
Add a test but test script completer is implemented differently.
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.
What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet
Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.
Will be base for decoding http2 and other TLS based on protocols.
Fixes#587
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.
Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.
This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
Feels less cluttered, easier to read and more consistent.
Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.
Also fixes lots of spelling errors while reading thru.
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
Refactor mp4 decoder to be simpler and have fallback for unknown box type
Cleanup some old ilst hacks
Add generic string reader to decode API that takes an encoding parameters
Fix issue with JQValue:s in assig/update paths.
Rebase on upstream master, commits since last rebase:
c1a3ebd (tag: v0.12.10, origin/main) bump up version to 0.12.10
5294c9e update CHANGELOG.md for v0.12.10
319a797 update dependencies
20ca575 add notes on precedence of unary operators against variable binding
62ffdf7 fix break in try catch syntax
35c123b improve the order of test cases
fe0e20a improve performance of ascii_downcase, ascii_upcase functions
76648c7 improve match/2 to reduce syntax tree nodes just like splits/2
6a683bb improve the install targets in Makefile
6631dd3 escape the null character in csv, tsv, sh formats (close#200)
5253817 fix path value validation on getpath function (fix#198)
I think this is more intuitive but might in some case cause very large JSON output
but maybe that less common or expected. In does cases i think you either want to
use some other bits_format (md5, truncate, etc) or you delete/transform the jq value
before turn it into JSON.
Strings in gojq are binary safe so you can use to hold raw bytes. But note that
convert the binary into JSON is lossy, same as the JSON standard.
Add bits_format option documentation.
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
Make more sense to use a more general name when there is both info and xing
and possibly vbri in the future. Also both xing and info seems to have extension
fields
Remove some old tests that are not really needed anymore.
