As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.
Add lazyre package to help delay compile of RE and make it concurrency safe.
Now ex "tovalue | .some.thing" on a decode value will make some.thing be jq value
instead of a decode value which woud be displayed as a decode treee, seems confusing.
I think this is more intuetive and make more sense.
Before it failed when passing a decode value, like an integer, string etc, as a object value
to an internal function (written in go), ex some_function({name: <decode value>}).
A future improvement could be to allow an internal function to say that it wants a "raw"
decode value etc and not one converted to primitive type.
This preserves the callstack on non-recoverable panics so that using
a debugger and fuzzing is much easier.
Add vscode debug config.
Remove fuzz stacktrace log workaround.
Make sure there is a top cancel function before calling it.
Fixes panic caused when interrupting decode before interp context has been pushed.
Also cleanup confusing naming a bit.
Thanks @pldin601 for reporting
This removes bar column drawing responsility from already complicated dump code.
Start of dump code refactor that will enable configurable columns and proper column truncate/wrap.
Feels like what one would expect when doing to_entries on a decode value struct etc,
that is get the pairs in same order as decoder wants.
Quite sure this used to the case but was change to sorted key order when
to_entries was move from bein implemented in jq to be internal for performance
reasons. Now internal gojq to_entries has a special case for JQValue objects.
Also use new gojq.JQType* constants for type strings and switch binary type
to look more like strings. Can still use _exttype to get "binary".
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).
Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.
Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.
Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.
Refactor out some mpeg nal things to mpeg format.
Refactor interp jq code into display.q and init.jq.
Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
Expose number parse validate functions, will be used later by text format decoders:
gojq.ValidNumber(s string)
gojq.NormalizeNumber(v json.Number)
gojq.NormalizeNumbers(v interface{})
From upstream:
Fixes: Regression since dace191 (fix fromjson to keep integer precision (close#172)) (https://github.com/itchyny/gojq/issues/178)
otherwise mostly cosmetic and cleanups since last rebase
Some encoding fuctions accept binary used string as input type, should be any.
Add cast helper functions, hopefully can be useful in future for even nicer
function bindings api.
Use smallest int type for int64, uint6 and *big.Int
Fixes integer serialization for yaml and toml for small integers, othweise
they will end up as strings.
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth
Add tojson, same as in jq but also has {indent:number} options
Add toyaml/fromyaml for YAML
Add totoml/fromtoml for TOML
Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)
Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character
Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)
Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex
Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512
Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.
Add test with playread and widevine pssh boxes
Related to #282
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
Upstream changes:
Many performance improvements
Error message improvments
Lots of refactoring
Most of the JQValue interface changes in gojq fork had to be reworked but
resultet in a much nicer and cleaner changeset.
fq changes:
Assignment to JQValue (like a decode value) now shallowly converts the value into
a jq value before assigning. Was a bit hacky as it was and this makes JQValue
behave more like real jq values. This also fixes some advanced indexing issues.
Actual custom path updates will be something for the future.
Padding could end up double.
Remove to*range($pad) as it probably just confusing to be able
to pad an existing range, what to show in hexdump etc? zero bits
that do not actuall exist at that range?
Add tests and binary stdout support to tests
repl now rewrites query to do implicit display inside the sub eval.
This makes it possible to interrupt eval and output in a better and faster way.
Make JSON encoder fail early on errors.
Add more interrupt tests.
Add explode for binary that explode into an array of the binary unit tobits
Remove scan_toend as it feels a bit weird and one can use tobytes | .[match(...):] instead
Some general cleanup also
Fixes missing function errors for fq builtins
Add includes to fix some other missing function errors
Add COMPLETION_TIMEOUT env to increase comepletion timeout during test using -race
