1
1
mirror of https://github.com/wader/fq.git synced 2024-11-23 00:57:15 +03:00
Commit Graph

417 Commits

Author SHA1 Message Date
Mattias Wadman
fc74f6857c intepr: Revert binary value each
Is a bad idea, behaves confusingly with .. etc. Maybe can be
done in the future via some option to make all raw bits behave
as arrays are instroduced.
2024-08-21 16:33:20 +02:00
Mattias Wadman
2a9dc432c2
Merge pull request #995 from wader/binary-each
intepr: Support each for binary values
2024-08-21 16:23:47 +02:00
Mattias Wadman
3f3183a5c5 intepr: Support each for binary values
Makes it possible to output all bytes or bits as integers in a binary value.
Ex: [1,2,3] | [tobytes[]] -> [1,2,3]
2024-08-21 16:17:19 +02:00
Mattias Wadman
550bcc27ff cli: Add go version to version string 2024-08-20 13:31:51 +02:00
Mattias Wadman
7bf112917b interp: Add bits_format=byte_array support
Represent binary values into an array of bytes when turn values into JSON
2024-08-20 08:43:22 +02:00
Mattias Wadman
824e51ec58 gojq: Update fq fork
Rebase fq changes on latester gojq main.

Most notable change visiable to users is that gojq now allows queries
as object literal values. For example this now works: {a: 1+2+3}

From upstream:
083fb39 refactor code using built-in min and max functions
470db58 bump up to Go 1.23, drop support for Go 1.20
0607aa5 bump up version to 0.12.16
0709341 update CHANGELOG.md for v0.12.16
1324e6e update dependencies
01355e9 improve parser to allow binary operators as object values
a41a5f8 fix debug/1 function to be available only when debug/0 is defined
f694c1b fix a benchmark test BenchmarkCompile
f2559f6 remove private compare function
0cd3a66 improve compiler to abort with error if query is missing
422cc9d refactor stringify function declarations of query
1130c4e refactor program body, rename rules, remove empty actions in parser
8d7ccc9 add tests for immutability of arrays
375e90d remove useless receivers
1b5ce7f set correct offset for multibyte tokens
8874f53 fix tests of exp10 and atan2 failing on some platforms
2024-08-19 14:15:02 +02:00
bump
2c38cf1324 Update github-golangci-lint to 1.60.1 from 1.59.1
Release notes https://github.com/golangci/golangci-lint/releases/tag/v1.60.1

Require go 1.21 and fix various lint warnings and use new features
2024-08-14 20:05:40 +02:00
Mattias Wadman
ebf063d1c0 doc: Cleanup and improve texts a bit 2024-04-30 14:16:13 +02:00
Mattias Wadman
6059b9ee1b fq: Use trim from gojq 2024-04-19 18:09:00 +02:00
Mattias Wadman
c8f9cdc9d7 interp: Add string_truncate option
Was hardcoded to 50, now confiugurable also used affected by dd, dv etc
2024-04-08 17:45:10 +02:00
Mattias Wadman
bf7fa07c41 fq: Use go 1.20 and cleanup
Also rename *ex packages to *x
2024-04-01 19:14:10 +02:00
Mattias Wadman
b542ff1d6c lint: More linters and some fixes 2024-04-01 18:24:41 +02:00
Mattias Wadman
f5fd587350 gojq: Update fq fork
Fixes issue using ascii_downcase and ascii_upcase with decode value.

From upstream https://github.com/itchyny/gojq/blob/main/CHANGELOG.md:
* implement `ltrim`, `rtrim`, and `trim` functions
* implement `gojq.ParseError` for getting the offset and token of query parsing error
* implement `gojq.HaltError` for detecting halt errors and stopping outer iteration
* fix object construction with duplicate keys (`{x:0,y:1} | {a:.x,a:.y}`)
* fix `halt` and `halt_error` functions to stop the command execution immediately
* fix variable scope of binding syntax (`"a" as $v | def f: $v; "b" as $v | f`)
* fix pre-defined variables to be available in initial modules (`$ARGS` in `~/.jq`)
* fix `ltrimstr` and `rtrimstr` functions to emit error on non-string input
* fix `nearbyint` and `rint` functions to round ties to even
* improve parser to allow `reduce`, `foreach`, `if`, `try`-`catch` syntax as object values
* remove `pow10` in favor of `exp10`, define `scalbn` and `scalbln` by `ldexp`
2024-04-01 15:15:26 +02:00
Mattias Wadman
e26704049f gojq: Update fq fork
From upstream:
fix halt and halt_error to stop command immediately
2024-03-24 14:57:42 +01:00
Mattias Wadman
0db671f626 interp: Add todo test for eval error in path 2024-03-10 09:59:18 +01:00
bump
888b0be069 Update github-golangci-lint to 1.56.0 from 1.55.2
Release notes https://github.com/golangci/golangci-lint/releases/tag/v1.56.0

Remove some nolint:errorlint not needed anymore
2024-02-07 17:31:49 +01:00
Mattias Wadman
69ae76593c interp: Fix crash when using line_bytes=0 2024-02-06 16:33:50 +01:00
Mattias Wadman
38b4412ad4 png: Type flags were off-by-one bit 2024-01-05 16:53:05 +01:00
Mattias Wadman
7906f33d51 test: Support to more common -update flag 2023-12-06 11:47:38 +01:00
Mattias Wadman
dc376f3491 gojq: Update rebased fq fork
From upstream:
    implement pick function
    fix comment continuation with newline in dos format
    implement comment continuation with backslash
2023-11-05 18:04:09 +01:00
Mattias Wadman
29e75411ed interp: Fix infinite recursion when casting synthetic raw value into a jq value 2023-10-29 16:11:05 +01:00
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
Mattias Wadman
c28163f8f4 interp: Improve colors when using light background
Related to #729
2023-10-10 11:08:38 +02:00
Mattias Wadman
23ae4d978b decode,interp: Make synthetic values more visible and not act as decode values 2023-10-07 17:40:16 +02:00
Mattias Wadman
a614c9dfee interp: split: Correctly split binary
The thing splitted on should not be included.
2023-09-24 19:52:22 +02:00
Mattias Wadman
0699c80bac mp3_frame_xing: Detect lame ext more similar to ffmpeg and mediainfo 2023-09-08 12:09:35 +02:00
Mattias Wadman
c0352f2fd1 decode,interp: Don't shadow _key and error on missing _key
For fromjson and other "value" decode values fq make then behave both like
a normal jq value and decode value. This is to make tobytes, format etc work.
Before all _* would be treated as special keys. Now they are first looked up in
the wrapped value and then as decode values.

Also now ._key that don't exist reutrn null instead of throw error.

$ fq -n '`{"_format": 123}` | fromjson | ._format'
Now:
123
Before:
"json"

$ fq -n '`{}` | fromjson | ._missing'
Now:
null
Before
error
2023-09-03 10:47:59 +02:00
Mattias Wadman
0d0147643a gomod: Update x/exp and gomarkdown
Add cmp package from go 1.21 to have cmp.Compare to make sort easier
2023-08-09 14:36:51 +02:00
Mattias Wadman
d5345f0b8a cli: Rename --null/nul-output to --raw-output0
Remove -0 short arg, now it means the expression "-0".
This is to be in sync with jq 1.7 https://github.com/jqlang/jq/pull/2684

Correct and clarify that NUL and new lines are outputted after and not between each output.

Still missing is to abort on output containing zero when using --raw-output0
2023-08-08 16:33:39 +02:00
Mattias Wadman
fa3dba104b gojq: Update fq fork
Changes from upstream:
8fcc90e implement abs function
cbbd4bb fix default module paths assuming gojq is located in a bin directory
45d4c5b change behavior of walk with multiple outputs
846cf99 fix stderr to output strings in raw format
c352d50 change zero division to produce an error when dividend is zero
d579009 fix empty string repeating with the maximum integer
d4de65c fix string multiplication by zero to emit empty string
04b48fd remove deprecated leaf_paths function
c882861 bump up version to 0.12.13
c310d5d update CHANGELOG.md for v0.12.13
8fb5cc3 update dependencies
fb00b66 improve error message of indices, setpath, delpaths functions
2023-08-08 12:22:29 +02:00
Mattias Wadman
44f0060205 dev,jq: Reformat jq code to look more the same 2023-07-27 13:21:04 +02:00
Mattias Wadman
41f40b7f94 interp: Add to binary fast path for arrays with only 0-255 numbers and strings 2023-06-14 19:09:15 +02:00
Mattias Wadman
38cb82929a gojq: Update rebased fq fork
From upstream:
e940c43 update GitHub action dependencies
2d97ddb update jq links to the new jqlang organization
2a6cb00 improve error message of various functions
5e26bef add a note on differences of supported functions missing in jq
96c4843 fix functions returning arrays not to emit nil slices
4a681c8 improve .dockerignore
850177b Merge pull request #213 from movableink/uri-decode
dbba5e7 add urid to decode URI values, mirroring uri
2023-05-29 17:47:01 +02:00
Mattias Wadman
b2c0e5fcd8 interp: Make binary also respect bits_format 2023-05-17 15:10:32 +02:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex
interp: Add hex bits format
2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df interp: Add hex bits format
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d

Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Kian-Meng Ang
dd4fa26867 doc: fix typos
Found via `codespell -S format -L bu,ue,trys,nd,tbe,te,trun,actuall`
2023-05-14 21:45:15 +08:00
Mattias Wadman
d5ae11659f interp: trim: Add multi-line support 2023-05-12 00:22:04 +02:00
Mattias Wadman
222cd88b77 bits,bytes: Behave as binary instead of raw decode value
I think this is closer to what one would expect.
With this fq -d bytes 'scan("...")' will match as a binary and also return binaries,
before as raw decode value one would have to use tobytes to get the binary.
2023-05-11 18:45:16 +02:00
Mattias Wadman
033498b289 interp: Don't output raw binary if display is called explicitly
So raw binay is only outputted if stdout is not a terminal and "... | d" is not used..
fq -n '[1,2,3] | tobytes' | cat > binary
fq -n '[1,2,3] | tobytes | d' | cat > hexdump
2023-05-11 14:15:00 +02:00
Mattias Wadman
684a083813 interp,decode: Support decode group argument
Will be used for probing via -d <string> and filename
2023-05-03 10:01:13 +02:00
Mattias Wadman
c51271399c interp: Add skip_gaps option for tovalue/-V
Skips gap fields in struct and arrays.

Gaps fields are bit ranges that a decoder did not add any fields for.
Note that skipping gaps in arrays will affect indexes.
2023-04-30 13:04:39 +02:00
Mattias Wadman
ee66fece18 interp: Make tovalue output behave as jq value
Now ex "tovalue | .some.thing" on a decode value will make some.thing be jq value
instead of a decode value which woud be displayed as a decode treee, seems confusing.

I think this is more intuetive and make more sense.
2023-04-29 22:31:53 +02:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
427ce78df6 interp: Add --value-output/-V option to do tovalue before output
Idea is to skip display so that JSON is outputted instead of showing tree and hexdump etc.
2023-04-13 11:29:16 +02:00
Mattias Wadman
9a1ef84cdd interp: Allow and convert JQValues:s (ex decode value) in function arg objects
Before it failed when passing a decode value, like an integer, string etc, as a object value
to an internal function (written in go), ex some_function({name: <decode value>}).

A future improvement could be to allow an internal function to say that it wants a "raw"
decode value etc and not one converted to primitive type.
2023-04-12 12:17:46 +02:00
Mattias Wadman
3dd2c61d3c interp: Fix input completion regression in sub-REPLs
readline Config was used to pass completer function per readline call,
was changed in #612 and caused regression. Now use our own member in
stdOS to pass it instead.

Add a test but test script completer is implemented differently.
2023-03-10 13:57:21 +01:00
Mattias Wadman
cc52a4419d id3v2: Decode subframes for CTOC and add struct for headers 2023-03-05 19:29:22 +01:00
Mattias Wadman
c75a83c829 help: Show default option value as JSON 2023-03-04 09:53:33 +01:00
Mattias Wadman
dec433fc53 help,markdown: Fix double line breaks when converting to text
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00