1
1
mirror of https://github.com/wader/fq.git synced 2024-11-05 02:34:02 +03:00
Commit Graph

102 Commits

Author SHA1 Message Date
loselarry
208b3e6bf2 chore: fix function name in comment
Signed-off-by: loselarry <bikangning@yeah.net>
2024-04-10 16:16:51 +08:00
Mattias Wadman
bf7fa07c41 fq: Use go 1.20 and cleanup
Also rename *ex packages to *x
2024-04-01 19:14:10 +02:00
Mattias Wadman
aaa43dfb81 test: Replace pmezard/go-difflib with go's internal diff 2024-02-26 11:39:03 +01:00
Mattias Wadman
a07ce11715 fq: Relicense float80 code to MIT
Related to #835
2023-12-15 00:41:16 +01:00
Mattias Wadman
29e75411ed interp: Fix infinite recursion when casting synthetic raw value into a jq value 2023-10-29 16:11:05 +01:00
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
Mattias Wadman
0d0147643a gomod: Update x/exp and gomarkdown
Add cmp package from go 1.21 to have cmp.Compare to make sort easier
2023-08-09 14:36:51 +02:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex
interp: Add hex bits format
2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df interp: Add hex bits format
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d

Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Kian-Meng Ang
dd4fa26867 doc: fix typos
Found via `codespell -S format -L bu,ue,trys,nd,tbe,te,trun,actuall`
2023-05-14 21:45:15 +08:00
Mattias Wadman
e2eb667091 html: Add to probe group
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.

Add lazyre package to help delay compile of RE and make it concurrency safe.
2023-05-11 19:07:18 +02:00
Mattias Wadman
684a083813 interp,decode: Support decode group argument
Will be used for probing via -d <string> and filename
2023-05-03 10:01:13 +02:00
Mattias Wadman
ee66fece18 interp: Make tovalue output behave as jq value
Now ex "tovalue | .some.thing" on a decode value will make some.thing be jq value
instead of a decode value which woud be displayed as a decode treee, seems confusing.

I think this is more intuetive and make more sense.
2023-04-29 22:31:53 +02:00
Mattias Wadman
9a1ef84cdd interp: Allow and convert JQValues:s (ex decode value) in function arg objects
Before it failed when passing a decode value, like an integer, string etc, as a object value
to an internal function (written in go), ex some_function({name: <decode value>}).

A future improvement could be to allow an internal function to say that it wants a "raw"
decode value etc and not one converted to primitive type.
2023-04-12 12:17:46 +02:00
Mattias Wadman
c5f6809b02 decode,fuzz,dev: Move recoverable error check to recoverfn.Run
This preserves the callstack on non-recoverable panics so that using
a debugger and fuzzing is much easier.

Add vscode debug config.
Remove fuzz stacktrace log workaround.
2023-03-31 12:47:04 +02:00
Mattias Wadman
0b6ef2a9d8 golangci-lint: Disable revive unused-parameter and update for new default config
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
2023-03-18 22:56:58 +01:00
Mattias Wadman
dc4a82eeed aiff: Add basic decoder 2023-03-09 15:16:52 +01:00
Mattias Wadman
a1bb630a2a doc,fq: Improve cli help and some cleanup 2023-02-15 11:04:39 +01:00
Mattias Wadman
50d26ec759 colorjson: Handle encoding error value
Can happen if ValueFn is used and it returns an error, reader is canceled etc
2023-02-07 16:45:29 +01:00
Mattias Wadman
5c8e115106 colorjson: Refactor to option struct 2023-02-07 16:18:32 +01:00
Mattias Wadman
ba88a68466 interp: mimic jq: if expr arg is given read stdin even if tty 2022-11-29 18:39:57 +01:00
Mattias Wadman
568afff3f0 interp: Fix panic when trigger before any context has been pushed
Make sure there is a top cancel function before calling it.
Fixes panic caused when interrupting decode before interp context has been pushed.

Also cleanup confusing naming a bit.

Thanks @pldin601 for reporting
2022-11-29 17:38:58 +01:00
Mattias Wadman
376f0ebb24 gojq: Update rebased fq fork
Also sync colorjson encoder with gojq encoder
2022-09-28 15:44:16 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
913f5780f4 columnwriter,dump: Add Column interface and refactor into BarColumn and MultiLineColumn
This removes bar column drawing responsility from already complicated dump code.

Start of dump code refactor that will enable configurable columns and proper column truncate/wrap.
2022-08-24 21:48:56 +02:00
Mattias Wadman
dc32ac08e2 script: Use strings.Builder to collect output
String concat Caused exponential slowdown
2022-08-23 20:08:44 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
fa20c74c22 xml,csv,interp: Handle JQValue when string normalizing
Otherwise nested array/object JQValue:s could end up being JSON marshalled
2022-07-26 17:43:59 +02:00
Mattias Wadman
af35b2848d gojq: Preserve keys order for to_entries when used with JQValue
Feels like what one would expect when doing to_entries on a decode value struct etc,
that is get the pairs in same order as decoder wants.

Quite sure this used to the case but was change to sorted key order when
to_entries was move from bein implemented in jq to be internal for performance
reasons. Now internal gojq to_entries has a special case for JQValue objects.

Also use new gojq.JQType* constants for type strings and switch binary type
to look more like strings. Can still use _exttype to get "binary".
2022-07-26 15:55:58 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
9b2e474ea8 gojqextra: Simplify function type helpers
No need for env to be a function
2022-07-16 21:55:47 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
767143491f mapstruct: Handle nested values when converting to camel case 2022-07-10 10:50:47 +02:00
Mattias Wadman
c57dc17d17 gojq: Rebase fq fork
Expose number parse validate functions, will be used later by text format decoders:
gojq.ValidNumber(s string)
gojq.NormalizeNumber(v json.Number)
gojq.NormalizeNumbers(v interface{})

From upstream:
Fixes: Regression since dace191 (fix fromjson to keep integer precision (close #172)) (https://github.com/itchyny/gojq/issues/178)
otherwise mostly cosmetic and cleanups since last rebase
2022-07-03 11:55:28 +02:00
Mattias Wadman
c7701851b0 interp: Extract to/from map/struct to own package 2022-06-20 21:24:59 +02:00
Mattias Wadman
32361deefe gojqextra: Cleanup gojq type cast code 2022-06-20 18:02:35 +02:00
Mattias Wadman
6b0880002d interp: Cast jq value to go value properly for encoding functions
Some encoding fuctions accept binary used string as input type, should be any.
Add cast helper functions, hopefully can be useful in future for even nicer
function bindings api.
2022-05-30 18:27:12 +02:00
Mattias Wadman
735c443b37 interp: Improve type normalization and use it for toyaml and totoml
Use smallest int type for int64, uint6 and *big.Int

Fixes integer serialization for yaml and toml for small integers, othweise
they will end up as strings.
2022-05-28 20:33:37 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
7b27e506f1 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.

Add test with playread and widevine pssh boxes

Related to #282
2022-05-28 13:42:38 +02:00
Mattias Wadman
9a7ce14886 gojq: Update rebased fq fork
Use new gojq.TypeOf and gojq.Preview
2022-05-23 21:33:12 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage
Make cwd for a test script the directory where the script is.
Use relative paths
2022-05-21 20:03:25 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
8dde3ef5b1 interp: Fix crash when including relatve path when no search paths are set 2022-05-17 18:05:10 +02:00
Mattias Wadman
dff3cc1199 interp: dump: Fix column truncate issue with unicode bars 2022-05-11 00:37:52 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
607202bb67 interp: Don't truncate last display column
Wrapping lines might be annoying but not seeing things even worse i think.
Hopefully once column config is in jq it could be an option.
2022-04-19 18:19:42 +02:00