1
1
mirror of https://github.com/wader/fq.git synced 2024-11-26 21:55:57 +03:00
Commit Graph

373 Commits

Author SHA1 Message Date
Mattias Wadman
427ce78df6 interp: Add --value-output/-V option to do tovalue before output
Idea is to skip display so that JSON is outputted instead of showing tree and hexdump etc.
2023-04-13 11:29:16 +02:00
Mattias Wadman
9a1ef84cdd interp: Allow and convert JQValues:s (ex decode value) in function arg objects
Before it failed when passing a decode value, like an integer, string etc, as a object value
to an internal function (written in go), ex some_function({name: <decode value>}).

A future improvement could be to allow an internal function to say that it wants a "raw"
decode value etc and not one converted to primitive type.
2023-04-12 12:17:46 +02:00
Mattias Wadman
3dd2c61d3c interp: Fix input completion regression in sub-REPLs
readline Config was used to pass completer function per readline call,
was changed in #612 and caused regression. Now use our own member in
stdOS to pass it instead.

Add a test but test script completer is implemented differently.
2023-03-10 13:57:21 +01:00
Mattias Wadman
cc52a4419d id3v2: Decode subframes for CTOC and add struct for headers 2023-03-05 19:29:22 +01:00
Mattias Wadman
c75a83c829 help: Show default option value as JSON 2023-03-04 09:53:33 +01:00
Mattias Wadman
dec433fc53 help,markdown: Fix double line breaks when converting to text
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00
Mattias Wadman
73db6587a0 interp: Exit with error if -o name=@path fails to be read, also document 2023-02-26 21:18:36 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
a1bb630a2a doc,fq: Improve cli help and some cleanup 2023-02-15 11:04:39 +01:00
Mattias Wadman
dc79a73b72 interp,json: Move error handling to colorjson
Cancel error from ValueFn etc will be return by Marshal instead
2023-02-07 16:57:54 +01:00
Mattias Wadman
5c8e115106 colorjson: Refactor to option struct 2023-02-07 16:18:32 +01:00
Mattias Wadman
8b49b42fa3 interp: Wrap Binary in decodeValue to fix prompt issue with bits/bytes format 2023-01-04 15:30:05 +01:00
Mattias Wadman
ca27e4266e doc: Add _parent for decode values and clenaup doc a bit 2023-01-03 11:21:09 +01:00
Mattias Wadman
6c229d7382 interp: Don't auto convert to binary for string functions, is just confusing
Instead use tobytes etc if binary matching is what is wanted
2022-12-22 12:13:23 +01:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
9e5a072e50 mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders 2022-12-15 12:10:06 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
48522e3cb8 mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one 2022-12-14 11:34:53 +01:00
Mattias Wadman
578b84d4cf interp,display: Add workaround for go 1.18 when escaping 0x7f 2022-12-10 11:34:21 +01:00
Mattias Wadman
6c86945125 gojq: Update rebased fq fork
Fix issue with JQValue:s in assig/update paths.

Rebase on upstream master, commits since last rebase:
c1a3ebd (tag: v0.12.10, origin/main) bump up version to 0.12.10
5294c9e update CHANGELOG.md for v0.12.10
319a797 update dependencies
20ca575 add notes on precedence of unary operators against variable binding
62ffdf7 fix break in try catch syntax
35c123b improve the order of test cases
fe0e20a improve performance of ascii_downcase, ascii_upcase functions
76648c7 improve match/2 to reduce syntax tree nodes just like splits/2
6a683bb improve the install targets in Makefile
6631dd3 escape the null character in csv, tsv, sh formats (close #200)
5253817 fix path value validation on getpath function (fix #198)
2022-12-07 16:45:08 +01:00
Mattias Wadman
423bab9e33 dev,test: Use jqtest code from jqjq for jq tests 2022-12-05 20:52:27 +01:00
Mattias Wadman
6fc84a885c doc,dev: Add more usage and dev tips 2022-12-03 19:00:47 +01:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
8d69f1fb23 interp: Change default bits_format=string
I think this is more intuitive but might in some case cause very large JSON output
but maybe that less common or expected. In does cases i think you either want to
use some other bits_format (md5, truncate, etc) or you delete/transform the jq value
before turn it into JSON.

Strings in gojq are binary safe so you can use to hold raw bytes. But note that
convert the binary into JSON is lossy, same as the JSON standard.

Add bits_format option documentation.
2022-12-01 17:49:34 +01:00
Mattias Wadman
ba88a68466 interp: mimic jq: if expr arg is given read stdin even if tty 2022-11-29 18:39:57 +01:00
Mattias Wadman
646f32d563 matroska: Fix path tests and make _tree_path more robust 2022-11-20 20:16:41 +01:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
c3a0686c9e mp3_frame_tags: Refactor and rename xing format to mp3_frame_tags
Make more sense to use a more general name when there is both info and xing
and possibly vbri in the future. Also both xing and info seems to have extension
fields

Remove some old tests that are not really needed anymore.
2022-11-16 17:32:03 +01:00
Mattias Wadman
41b2d1ad1b cli: Better decode error help 2022-11-02 12:28:53 +01:00
Mattias Wadman
2606916753 yaml,fuzz: gojq.Normalize value to fix type panic 2022-10-11 19:22:49 +02:00
Mattias Wadman
c4219d69db interp: Fix interrupt panic for cli eval
Rewrite cli query to use display as output query, same as for repl.
Without this "tovalue" values that don't print in the current eval
will end up in the paren eval and be written using an output without
a cancel context.
2022-09-29 18:17:57 +02:00
Mattias Wadman
00ee10a1a1 interp: Make to{actual,sym} behave similar to tovalue
Before if you did toactual it behaved differently and even gave error for
a scalar that only had actual value set. Now both to{actal,sym} works
similar to tovalue but will force if actual or sym value should be used.
2022-09-25 17:53:12 +02:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
fe64530e11 csv: Add tsv and header example
Also fix string esacpe in fuzz options parser
2022-09-15 01:16:45 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
00a50662ea markdown: Add decoder 2022-09-10 18:26:45 +02:00
Mattias Wadman
768df3012c interp,decode: For struct use map to lookup field
Will make it faster for struct with logs of fields and seems to
not cuase any significant difference for small structs.

All this really needs a rewrite somehow, maybe refactor into interfaces somehow? getting messy.
2022-09-01 11:35:19 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
913f5780f4 columnwriter,dump: Add Column interface and refactor into BarColumn and MultiLineColumn
This removes bar column drawing responsility from already complicated dump code.

Start of dump code refactor that will enable configurable columns and proper column truncate/wrap.
2022-08-24 21:48:56 +02:00
Mattias Wadman
d8792fd104 interp,dump: Correctly flush columns if data will be shown
Before addrbar ended up in data and also redudant colunms was added
2022-08-24 21:14:26 +02:00
Mattias Wadman
8fc43533a9 prores_frame: Add basic container and frame header decoder 2022-08-24 19:33:35 +02:00
Takashi Oguma
934ed9a809
wasm: initial version 2022-08-18 21:40:30 +09:00
Mattias Wadman
ff29fe7154
Merge pull request #377 from wader/force-unicode
interp: Add --unicode-output/-U to force use of unicode
2022-08-17 21:49:40 +02:00
Mattias Wadman
c7559b59a0 interp: Add --unicode-output/-U to force use of unicode
Currently only used to make dump output a bit nicer.
2022-08-17 21:40:42 +02:00
Mattias Wadman
cf15661e9c mp3_frame: Add LSF support and fix incorrect main data handling
Some mpeg versions use LSF (low sampling frequency) when encoding
low sample rates which changes a bit how the side info is layed out.

Also skip trying to separate data and other_data as after reading up
a bit on how main data begin and bit reservoir works i think it was done
incorrectly. To know the real audio data length i think you need to decode
the huffman tables. Instead just have a "audio_data" field which are the
bits with audo data for current and possibly following frames.
2022-08-17 21:36:39 +02:00
Mattias Wadman
701c67c136 jsonl: Add decoder, also tojsonl encoder 2022-08-15 19:43:59 +02:00
Mattias Wadman
d898732c3d gojq: Update fq fork, new scope function, rawstring, stricter integers
Update repl completion for new scope/0 function, now reports name/arity and is more correct
Support for experimental `rawstring` literals, similar to go raw string literals
Sticter number literals, now 0b,0o and 0x fail it they have invalud symbols, ex: 0b123
2022-08-14 22:40:44 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00