1
1
mirror of https://github.com/wader/fq.git synced 2024-11-30 18:08:16 +03:00
Commit Graph

343 Commits

Author SHA1 Message Date
Mattias Wadman
c4219d69db interp: Fix interrupt panic for cli eval
Rewrite cli query to use display as output query, same as for repl.
Without this "tovalue" values that don't print in the current eval
will end up in the paren eval and be written using an output without
a cancel context.
2022-09-29 18:17:57 +02:00
Mattias Wadman
00ee10a1a1 interp: Make to{actual,sym} behave similar to tovalue
Before if you did toactual it behaved differently and even gave error for
a scalar that only had actual value set. Now both to{actal,sym} works
similar to tovalue but will force if actual or sym value should be used.
2022-09-25 17:53:12 +02:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
fe64530e11 csv: Add tsv and header example
Also fix string esacpe in fuzz options parser
2022-09-15 01:16:45 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
00a50662ea markdown: Add decoder 2022-09-10 18:26:45 +02:00
Mattias Wadman
768df3012c interp,decode: For struct use map to lookup field
Will make it faster for struct with logs of fields and seems to
not cuase any significant difference for small structs.

All this really needs a rewrite somehow, maybe refactor into interfaces somehow? getting messy.
2022-09-01 11:35:19 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
913f5780f4 columnwriter,dump: Add Column interface and refactor into BarColumn and MultiLineColumn
This removes bar column drawing responsility from already complicated dump code.

Start of dump code refactor that will enable configurable columns and proper column truncate/wrap.
2022-08-24 21:48:56 +02:00
Mattias Wadman
d8792fd104 interp,dump: Correctly flush columns if data will be shown
Before addrbar ended up in data and also redudant colunms was added
2022-08-24 21:14:26 +02:00
Mattias Wadman
8fc43533a9 prores_frame: Add basic container and frame header decoder 2022-08-24 19:33:35 +02:00
Takashi Oguma
934ed9a809
wasm: initial version 2022-08-18 21:40:30 +09:00
Mattias Wadman
ff29fe7154
Merge pull request #377 from wader/force-unicode
interp: Add --unicode-output/-U to force use of unicode
2022-08-17 21:49:40 +02:00
Mattias Wadman
c7559b59a0 interp: Add --unicode-output/-U to force use of unicode
Currently only used to make dump output a bit nicer.
2022-08-17 21:40:42 +02:00
Mattias Wadman
cf15661e9c mp3_frame: Add LSF support and fix incorrect main data handling
Some mpeg versions use LSF (low sampling frequency) when encoding
low sample rates which changes a bit how the side info is layed out.

Also skip trying to separate data and other_data as after reading up
a bit on how main data begin and bit reservoir works i think it was done
incorrectly. To know the real audio data length i think you need to decode
the huffman tables. Instead just have a "audio_data" field which are the
bits with audo data for current and possibly following frames.
2022-08-17 21:36:39 +02:00
Mattias Wadman
701c67c136 jsonl: Add decoder, also tojsonl encoder 2022-08-15 19:43:59 +02:00
Mattias Wadman
d898732c3d gojq: Update fq fork, new scope function, rawstring, stricter integers
Update repl completion for new scope/0 function, now reports name/arity and is more correct
Support for experimental `rawstring` literals, similar to go raw string literals
Sticter number literals, now 0b,0o and 0x fail it they have invalud symbols, ex: 0b123
2022-08-14 22:40:44 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
2022-08-09 18:09:41 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
725c8e83ab macho: Split into macho/macho_fat, fix offset issue and add string decoding
Split fat macho into own decoder macho_fat. This also fixes issue with section
offset etc not being correct as they are from the start of each embedded file.

Make all address and offset field be in hex.

Decode __cstring, __ustring and __cfstring sections.

Fix LC_ENCRYPTION_INFO_64 missing pading issue.

Skip ranging for __bss and __common as they dont have any data in the file.

Simplifed magic handling a bit and add symbols.

Simplified state struct field, had redudant struct.
2022-08-02 14:36:53 +02:00
Mattias Wadman
bc6cffde61 lint,decode,fuzz:: Fix nilerr warnings, one real one should be ignored for now 2022-08-01 18:47:24 +02:00
Mattias Wadman
ed4247836c doc,interp: Update and add more examples 2022-07-30 20:21:35 +02:00
Mattias Wadman
4eae7ffd99 interp,doc: Add -R raw string slurp hint to -s help 2022-07-29 16:37:42 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
fa20c74c22 xml,csv,interp: Handle JQValue when string normalizing
Otherwise nested array/object JQValue:s could end up being JSON marshalled
2022-07-26 17:43:59 +02:00
Mattias Wadman
af35b2848d gojq: Preserve keys order for to_entries when used with JQValue
Feels like what one would expect when doing to_entries on a decode value struct etc,
that is get the pairs in same order as decoder wants.

Quite sure this used to the case but was change to sorted key order when
to_entries was move from bein implemented in jq to be internal for performance
reasons. Now internal gojq to_entries has a special case for JQValue objects.

Also use new gojq.JQType* constants for type strings and switch binary type
to look more like strings. Can still use _exttype to get "binary".
2022-07-26 15:55:58 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
78aa96b0ac dev: Cleanup some code to fix a bunch of new linter warnings 2022-07-19 18:56:09 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
f5be5180f0 interp: Cleanup and clarify some format naming 2022-07-16 22:04:21 +02:00
Mattias Wadman
9b2e474ea8 gojqextra: Simplify function type helpers
No need for env to be a function
2022-07-16 21:55:47 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
64f3e5c717 fairplay: Add basic SPC decoder and PSSH system id 2022-07-07 17:11:55 +02:00
Mattias Wadman
ab8c728aaa interp: Rename --decode-file to --argdecode to be more consistent
There is still an alias for --decode-file
2022-07-04 12:28:37 +02:00
Mattias Wadman
417255b72b bitcoin: Add blkdat, block, transcation and script decoder 2022-07-04 09:59:51 +02:00
Mattias Wadman
a3c33fc1f4 interp: Add group/0
Also add tests for count, count_by, delta and delta_by
2022-06-21 15:18:12 +02:00
Mattias Wadman
c7701851b0 interp: Extract to/from map/struct to own package 2022-06-20 21:24:59 +02:00
Mattias Wadman
32361deefe gojqextra: Cleanup gojq type cast code 2022-06-20 18:02:35 +02:00
Mattias Wadman
81a014ce49 interp: Make empty _finally fin error on error
Ex: _finally(error(a); empty)
This is not used by fq atm
2022-06-18 00:00:54 +02:00
Mattias Wadman
0863374f8c doc: Correct bencode spec URL
Also rename _tojq value_sep to object_sep and fix formats_diagram.jq to use tomd5
2022-06-13 18:49:34 +02:00
Mattias Wadman
a64c28d927 dump: Skip JQValueEx if there are not options
I used to tojson and would crash in binary values are there were
not optins deciding how to format the strinh. Now just treat as utf8 string.
2022-06-07 16:39:45 +02:00
Mattias Wadman
c1a0cda5a8 gojq: Update rebased fq fork
Fix tostring for non-string JQValue:s, is now JSON
Update EOF error
2022-06-01 17:27:45 +02:00
Mattias Wadman
6b0880002d interp: Cast jq value to go value properly for encoding functions
Some encoding fuctions accept binary used string as input type, should be any.
Add cast helper functions, hopefully can be useful in future for even nicer
function bindings api.
2022-05-30 18:27:12 +02:00
Mattias Wadman
735c443b37 interp: Improve type normalization and use it for toyaml and totoml
Use smallest int type for int64, uint6 and *big.Int

Fixes integer serialization for yaml and toml for small integers, othweise
they will end up as strings.
2022-05-28 20:33:37 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
7b27e506f1 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.

Add test with playread and widevine pssh boxes

Related to #282
2022-05-28 13:42:38 +02:00