1
1
mirror of https://github.com/wader/fq.git synced 2024-11-28 11:42:50 +03:00
Commit Graph

30 Commits

Author SHA1 Message Date
Mattias Wadman
fa20c74c22 xml,csv,interp: Handle JQValue when string normalizing
Otherwise nested array/object JQValue:s could end up being JSON marshalled
2022-07-26 17:43:59 +02:00
Mattias Wadman
af35b2848d gojq: Preserve keys order for to_entries when used with JQValue
Feels like what one would expect when doing to_entries on a decode value struct etc,
that is get the pairs in same order as decoder wants.

Quite sure this used to the case but was change to sorted key order when
to_entries was move from bein implemented in jq to be internal for performance
reasons. Now internal gojq to_entries has a special case for JQValue objects.

Also use new gojq.JQType* constants for type strings and switch binary type
to look more like strings. Can still use _exttype to get "binary".
2022-07-26 15:55:58 +02:00
Mattias Wadman
9b2e474ea8 gojqextra: Simplify function type helpers
No need for env to be a function
2022-07-16 21:55:47 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
c57dc17d17 gojq: Rebase fq fork
Expose number parse validate functions, will be used later by text format decoders:
gojq.ValidNumber(s string)
gojq.NormalizeNumber(v json.Number)
gojq.NormalizeNumbers(v interface{})

From upstream:
Fixes: Regression since dace191 (fix fromjson to keep integer precision (close #172)) (https://github.com/itchyny/gojq/issues/178)
otherwise mostly cosmetic and cleanups since last rebase
2022-07-03 11:55:28 +02:00
Mattias Wadman
c7701851b0 interp: Extract to/from map/struct to own package 2022-06-20 21:24:59 +02:00
Mattias Wadman
32361deefe gojqextra: Cleanup gojq type cast code 2022-06-20 18:02:35 +02:00
Mattias Wadman
6b0880002d interp: Cast jq value to go value properly for encoding functions
Some encoding fuctions accept binary used string as input type, should be any.
Add cast helper functions, hopefully can be useful in future for even nicer
function bindings api.
2022-05-30 18:27:12 +02:00
Mattias Wadman
735c443b37 interp: Improve type normalization and use it for toyaml and totoml
Use smallest int type for int64, uint6 and *big.Int

Fixes integer serialization for yaml and toml for small integers, othweise
they will end up as strings.
2022-05-28 20:33:37 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
9a7ce14886 gojq: Update rebased fq fork
Use new gojq.TypeOf and gojq.Preview
2022-05-23 21:33:12 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
518f6af4a8 gojq: Rebase fq fork
Upstream changes:
Many performance improvements
Error message improvments
Lots of refactoring

Most of the JQValue interface changes in gojq fork had to be reworked but
resultet in a much nicer and cleaner changeset.

fq changes:
Assignment to JQValue (like a decode value) now shallowly converts the value into
a jq value before assigning. Was a bit hacky as it was and this makes JQValue
behave more like real jq values. This also fixes some advanced indexing issues.

Actual custom path updates will be something for the future.
2022-04-11 19:23:15 +02:00
Mattias Wadman
9dc59e5d39 interp: Move _is_decode_value to jq
Make _extype work like type
2022-02-15 22:20:32 +01:00
Mattias Wadman
548a065388 decode,interp: Finish up nil value support
Update msgpack, bson and cbor to use it.
2022-01-17 10:49:46 +01:00
Mattias Wadman
be0ef80c8d interp,fq: Make bit operators normal functions
Uses same rule as jq math functions, 1 arg uses input, more than one all passed as args.
So "bnot 1" -> "1 | bnot", "1 bsl 1" -> "bsl(1; 1)"

Don't think it's worth changing the jq syntax for this and also it could make fq
scripts no compatiblr with other jq tools. Non-10 base number literals are still allowed
but should probably not be used in scripts, only in repl and with cli.
2022-01-09 01:06:36 +01:00
Mattias Wadman
9f2dddc6e1 interp,gojqextra: Make buffers values even more lazy and error early on non-scalar calls 2021-11-29 14:51:00 +01:00
Mattias Wadman
fdb811ec54 gojqextra,decode: Add generic lazy JQValue 2021-11-22 01:23:59 +01:00
Mattias Wadman
aab32cf2db gojqextra,interp: Add lazy string to speed usage of decode value buffer where string is not used 2021-11-21 21:27:44 +01:00
Mattias Wadman
6a15625587 interp,decode: Refactor out Scalar from Value and merge Array/Struct into Compound
Also add tosym, toactual
2021-11-05 17:29:22 +01:00
Mattias Wadman
4af5739712 interp: Rework string/buffer for decode values 2021-10-20 01:31:59 +02:00
Mattias Wadman
7f36f703dd Add to/bytes/bits[range] 2021-10-17 14:43:37 +02:00
Mattias Wadman
e86b45bd1a Add *grep/1/2 and find/1/2 2021-10-13 01:27:04 +02:00
Mattias Wadman
9035278e81 gojqextra: Move errors to own file 2021-09-28 01:45:32 +02:00
Mattias Wadman
f1507f7f65 mod: Use proper path and dont use replace 2021-09-12 13:08:50 +02:00
Mattias Wadman
0c7fa090b0 gojq: Initial update support
decode value give no updateable error
JQValue json value tries to replicate behaveior
2021-09-12 13:08:50 +02:00
Mattias Wadman
ad850d69e3 lint: want explicit types in gojqextra.go 2021-09-12 13:08:49 +02:00
Mattias Wadman
f33b31049d Refactor decode.Value gojq bindings 2021-09-12 13:08:48 +02:00
Mattias Wadman
970465996c Init 2021-09-12 13:08:42 +02:00