1
1
mirror of https://github.com/wader/fq.git synced 2024-11-23 09:56:07 +03:00
Commit Graph

541 Commits

Author SHA1 Message Date
Mattias Wadman
ebf063d1c0 doc: Cleanup and improve texts a bit 2024-04-30 14:16:13 +02:00
Mattias Wadman
6059b9ee1b fq: Use trim from gojq 2024-04-19 18:09:00 +02:00
Mattias Wadman
ee2ee24dbb decode: Cleanup io panic a bit 2024-04-12 17:23:03 +02:00
Mattias Wadman
c8f9cdc9d7 interp: Add string_truncate option
Was hardcoded to 50, now confiugurable also used affected by dd, dv etc
2024-04-08 17:45:10 +02:00
Mattias Wadman
bf7fa07c41 fq: Use go 1.20 and cleanup
Also rename *ex packages to *x
2024-04-01 19:14:10 +02:00
Mattias Wadman
48351028c5
Merge pull request #909 from wader/lint-more-and-fixes
lint: More linters and some fixes
2024-04-01 18:32:41 +02:00
Mattias Wadman
b542ff1d6c lint: More linters and some fixes 2024-04-01 18:24:41 +02:00
Mattias Wadman
621d7f2c22 decode: Align some heavily used structs to save space
From betteralign:
Compound: 8 bytes saved: struct with 40 pointer bytes could be 32
Value: 40 bytes saved: struct with 128 pointer bytes could be 88
2024-04-01 18:12:31 +02:00
Mattias Wadman
f5fd587350 gojq: Update fq fork
Fixes issue using ascii_downcase and ascii_upcase with decode value.

From upstream https://github.com/itchyny/gojq/blob/main/CHANGELOG.md:
* implement `ltrim`, `rtrim`, and `trim` functions
* implement `gojq.ParseError` for getting the offset and token of query parsing error
* implement `gojq.HaltError` for detecting halt errors and stopping outer iteration
* fix object construction with duplicate keys (`{x:0,y:1} | {a:.x,a:.y}`)
* fix `halt` and `halt_error` functions to stop the command execution immediately
* fix variable scope of binding syntax (`"a" as $v | def f: $v; "b" as $v | f`)
* fix pre-defined variables to be available in initial modules (`$ARGS` in `~/.jq`)
* fix `ltrimstr` and `rtrimstr` functions to emit error on non-string input
* fix `nearbyint` and `rint` functions to round ties to even
* improve parser to allow `reduce`, `foreach`, `if`, `try`-`catch` syntax as object values
* remove `pow10` in favor of `exp10`, define `scalbn` and `scalbln` by `ldexp`
2024-04-01 15:15:26 +02:00
Mattias Wadman
e26704049f gojq: Update fq fork
From upstream:
fix halt and halt_error to stop command immediately
2024-03-24 14:57:42 +01:00
Mattias Wadman
0db671f626 interp: Add todo test for eval error in path 2024-03-10 09:59:18 +01:00
Sam James
08df7f45c8
pkg/cli/test_exp.sh: use command -v
`which` is non-portable and not part of POSIX, but we can use `command -v`
for the same effect.

Debian [0] and Gentoo [1] are both trying to drop which from their base system.

[0] https://lwn.net/Articles/874049/
[1] https://bugs.gentoo.org/646588

Signed-off-by: Sam James <sam@gentoo.org>
2024-02-27 23:04:10 +00:00
Mattias Wadman
217660005e interp: Support ~/.config/fq as fallback on macOS
Got fed up with ~/Library/Application Support/fq
2024-02-09 16:10:38 +01:00
bump
888b0be069 Update github-golangci-lint to 1.56.0 from 1.55.2
Release notes https://github.com/golangci/golangci-lint/releases/tag/v1.56.0

Remove some nolint:errorlint not needed anymore
2024-02-07 17:31:49 +01:00
Mattias Wadman
69ae76593c interp: Fix crash when using line_bytes=0 2024-02-06 16:33:50 +01:00
Mattias Wadman
38b4412ad4 png: Type flags were off-by-one bit 2024-01-05 16:53:05 +01:00
Mattias Wadman
41226f48d8 readline: Switch to ergochat/readline
All fq changes merged.
2023-12-24 20:35:15 +01:00
Mattias Wadman
b05c7ec54f
Merge pull request #824 from mikez/leveldb
leveldb: Add LevelDB support
2023-12-09 16:47:43 +01:00
Mattias Wadman
7906f33d51 test: Support to more common -update flag 2023-12-06 11:47:38 +01:00
Michael B.
b05aa997ab leveldb: address PR comments 2023-12-05 11:31:30 +01:00
Mattias Wadman
dc376f3491 gojq: Update rebased fq fork
From upstream:
    implement pick function
    fix comment continuation with newline in dos format
    implement comment continuation with backslash
2023-11-05 18:04:09 +01:00
Mattias Wadman
29e75411ed interp: Fix infinite recursion when casting synthetic raw value into a jq value 2023-10-29 16:11:05 +01:00
Mattias Wadman
ac276ee119 gzip: Correctly handle multiple members
A gzip file consists of one or more "members" that are concatenated on uncompress.

Introduce a members array with each member and uncompressed is now the concatenation.

Fixes #794
2023-10-24 10:45:32 +02:00
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
Mattias Wadman
1d14ea51cd matroska: Decode ebml date type
Also some refactor of shared date decode functions to allow unit
2023-10-17 11:10:48 +02:00
Mattias Wadman
c28163f8f4 interp: Improve colors when using light background
Related to #729
2023-10-10 11:08:38 +02:00
Mattias Wadman
23ae4d978b decode,interp: Make synthetic values more visible and not act as decode values 2023-10-07 17:40:16 +02:00
Mattias Wadman
a614c9dfee interp: split: Correctly split binary
The thing splitted on should not be included.
2023-09-24 19:52:22 +02:00
Mattias Wadman
0699c80bac mp3_frame_xing: Detect lame ext more similar to ffmpeg and mediainfo 2023-09-08 12:09:35 +02:00
Mattias Wadman
c0352f2fd1 decode,interp: Don't shadow _key and error on missing _key
For fromjson and other "value" decode values fq make then behave both like
a normal jq value and decode value. This is to make tobytes, format etc work.
Before all _* would be treated as special keys. Now they are first looked up in
the wrapped value and then as decode values.

Also now ._key that don't exist reutrn null instead of throw error.

$ fq -n '`{"_format": 123}` | fromjson | ._format'
Now:
123
Before:
"json"

$ fq -n '`{}` | fromjson | ._missing'
Now:
null
Before
error
2023-09-03 10:47:59 +02:00
Mattias Wadman
0d0147643a gomod: Update x/exp and gomarkdown
Add cmp package from go 1.21 to have cmp.Compare to make sort easier
2023-08-09 14:36:51 +02:00
Mattias Wadman
d5345f0b8a cli: Rename --null/nul-output to --raw-output0
Remove -0 short arg, now it means the expression "-0".
This is to be in sync with jq 1.7 https://github.com/jqlang/jq/pull/2684

Correct and clarify that NUL and new lines are outputted after and not between each output.

Still missing is to abort on output containing zero when using --raw-output0
2023-08-08 16:33:39 +02:00
Mattias Wadman
fa3dba104b gojq: Update fq fork
Changes from upstream:
8fcc90e implement abs function
cbbd4bb fix default module paths assuming gojq is located in a bin directory
45d4c5b change behavior of walk with multiple outputs
846cf99 fix stderr to output strings in raw format
c352d50 change zero division to produce an error when dividend is zero
d579009 fix empty string repeating with the maximum integer
d4de65c fix string multiplication by zero to emit empty string
04b48fd remove deprecated leaf_paths function
c882861 bump up version to 0.12.13
c310d5d update CHANGELOG.md for v0.12.13
8fb5cc3 update dependencies
fb00b66 improve error message of indices, setpath, delpaths functions
2023-08-08 12:22:29 +02:00
Mattias Wadman
44f0060205 dev,jq: Reformat jq code to look more the same 2023-07-27 13:21:04 +02:00
Mattias Wadman
97194ad8bc mp4: Nicer major brand and handle some qt brand short strings better 2023-07-26 12:15:51 +02:00
Mattias Wadman
41f40b7f94 interp: Add to binary fast path for arrays with only 0-255 numbers and strings 2023-06-14 19:09:15 +02:00
Mattias Wadman
38cb82929a gojq: Update rebased fq fork
From upstream:
e940c43 update GitHub action dependencies
2d97ddb update jq links to the new jqlang organization
2a6cb00 improve error message of various functions
5e26bef add a note on differences of supported functions missing in jq
96c4843 fix functions returning arrays not to emit nil slices
4a681c8 improve .dockerignore
850177b Merge pull request #213 from movableink/uri-decode
dbba5e7 add urid to decode URI values, mirroring uri
2023-05-29 17:47:01 +02:00
Mattias Wadman
b2c0e5fcd8 interp: Make binary also respect bits_format 2023-05-17 15:10:32 +02:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex
interp: Add hex bits format
2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df interp: Add hex bits format
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d

Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Kian-Meng Ang
dd4fa26867 doc: fix typos
Found via `codespell -S format -L bu,ue,trys,nd,tbe,te,trun,actuall`
2023-05-14 21:45:15 +08:00
Mattias Wadman
070d5f43ab
Merge pull request #668 from wader/trim-multiline
interp: trim: Add multi-line support
2023-05-12 00:35:04 +02:00
Mattias Wadman
d5ae11659f interp: trim: Add multi-line support 2023-05-12 00:22:04 +02:00
Mattias Wadman
e2eb667091 html: Add to probe group
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.

Add lazyre package to help delay compile of RE and make it concurrency safe.
2023-05-11 19:07:18 +02:00
Mattias Wadman
222cd88b77 bits,bytes: Behave as binary instead of raw decode value
I think this is closer to what one would expect.
With this fq -d bytes 'scan("...")' will match as a binary and also return binaries,
before as raw decode value one would have to use tobytes to get the binary.
2023-05-11 18:45:16 +02:00
Mattias Wadman
033498b289 interp: Don't output raw binary if display is called explicitly
So raw binay is only outputted if stdout is not a terminal and "... | d" is not used..
fq -n '[1,2,3] | tobytes' | cat > binary
fq -n '[1,2,3] | tobytes | d' | cat > hexdump
2023-05-11 14:15:00 +02:00
Mattias Wadman
a200d3eee0
Merge pull request #415 from pnsafonov/postgres
Postgres
2023-05-06 09:07:13 +02:00
Mattias Wadman
684a083813 interp,decode: Support decode group argument
Will be used for probing via -d <string> and filename
2023-05-03 10:01:13 +02:00
Pavel Safonov
bb2659d442 Merge remote-tracking branch 'master' into postgres_merge_master_0 2023-05-03 09:44:02 +03:00
Mattias Wadman
c51271399c interp: Add skip_gaps option for tovalue/-V
Skips gap fields in struct and arrays.

Gaps fields are bit ranges that a decoder did not add any fields for.
Note that skipping gaps in arrays will affect indexes.
2023-04-30 13:04:39 +02:00