1
1
mirror of https://github.com/wader/fq.git synced 2024-12-02 04:05:35 +03:00
Commit Graph

57 Commits

Author SHA1 Message Date
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
a3c33fc1f4 interp: Add group/0
Also add tests for count, count_by, delta and delta_by
2022-06-21 15:18:12 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
95e619659b interp: Add internal _is_<type> helpers 2022-05-20 16:49:00 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
3f50bb90a6 interp: Rework formats and options help
Replace --formats with --help formats
Add --help options
2022-03-03 18:25:20 +01:00
Mattias Wadman
33efb02af0 interp,repl: Add path and value colors to prompt 2022-03-03 18:25:19 +01:00
Mattias Wadman
0a043f9096 repl,interp: Refactor repl and slurp
Now repl, slurp and help implemented using same query rewrite.
Include filename context in error if possible.
Add spew function that does opposite of slurp.
Start of help infra, not done or documented yet.
Show error pointer on parse error.
Rename internal eval to _eval and make eval be wrapper that
does rewrite and has various eror handling etc.
Nicer repl, slupr and help errors.
2022-03-01 18:31:00 +01:00
Mattias Wadman
9e4f264193 dev: Add .jq-lsp.jq to add additional builtins for jq-lsp
Fixes missing function errors for fq builtins
Add includes to fix some other missing function errors

Add COMPLETION_TIMEOUT env to increase comepletion timeout during test using -race
2022-02-16 19:52:42 +01:00
Mattias Wadman
48a19cb82c interp: Add paste function to allow pasting text into REPL etc
Also refactor readline and eval args into option struct and partinally start
addressing some side effects during completion.
2022-02-11 17:59:06 +01:00
Mattias Wadman
26d9650b0e interp: Refactor radix* into toradix($base)/fromradix($base)
There is also toradix($base; $table)/fromradix($base; $table) for custom symbols and bases.
Add radix tests

Part of cleaning up all convert/tranformation functions into to/from*.

Remove number_to_bytes as with better binary support it can be written as:
1234 | tobytes | [.[range(.size)]] and soon as 1234 | tobytes | explode
2022-02-11 01:28:39 +01:00
Mattias Wadman
0312c92c31 asn1_ber: Add more doc and multiple outputs for frompem 2022-02-07 17:41:05 +01:00
Mattias Wadman
0829c167cc asn1_ber: Add decoder
Basic support, can also do CER and DER but without any extra validation.
No schema support.

Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.

Related to #20
2022-02-06 22:13:03 +01:00
Mattias Wadman
e365f22084 interp: Cleanup stdio usage and functions 2022-01-05 21:41:11 +01:00
Mattias Wadman
fa350c6165 interp: Move display to jq 2021-12-06 21:08:40 +01:00
Mattias Wadman
01a407a462 interp: Rename s/bufferrange/buffer and cleanup 2021-11-30 21:10:26 +01:00
Mattias Wadman
8d442b82ee interp: Cleanup stdin reading and add more option tests 2021-11-30 12:25:56 +01:00
Mattias Wadman
07b421011f interp: add topath/0 and make todescription return null if there is none
Move out more code from funcs.jq
2021-11-19 11:44:23 +01:00
Mattias Wadman
f9f866000e interp,decode: Add force option to ignore asserts 2021-11-16 13:03:56 +01:00
Mattias Wadman
1b32b42f93 decode: Major decode API refactor
Generate more code
More generic and comfortable API
Improve and Update format decoder to new API
Add some more format tests
2021-11-15 21:12:07 +01:00
Mattias Wadman
178032e0a8 interp: Fix $opts shadowing in decode 2021-11-12 00:22:57 +01:00
Mattias Wadman
6a15625587 interp,decode: Refactor out Scalar from Value and merge Array/Struct into Compound
Also add tosym, toactual
2021-11-05 17:29:22 +01:00
Mattias Wadman
93322bc434 interp: Implement format/0 in jq 2021-11-03 10:58:33 +01:00
Mattias Wadman
96cc1283cd interp: Eval options in jq instead of calling jq from go
Simpler and causes less weird performance issues
2021-11-01 17:05:36 +01:00
Mattias Wadman
67898cb44a interp: Cleanup, use BufferRange for _open, progress for all decode 2021-10-30 01:17:14 +02:00
Mattias Wadman
bc1b3bf819 todo: Add note about symbols and iprint improvments 2021-10-22 11:01:48 +02:00
Mattias Wadman
eedfd16b40 interp: Replace find with overloaded match that support buffer 2021-10-20 21:16:13 +02:00
Mattias Wadman
7f36f703dd Add to/bytes/bits[range] 2021-10-17 14:43:37 +02:00
Mattias Wadman
e86b45bd1a Add *grep/1/2 and find/1/2 2021-10-13 01:27:04 +02:00
Mattias Wadman
1c3c65b191 interp: Rework buffer, still confusing 2021-10-05 22:26:05 +02:00
Mattias Wadman
618c1ea79f interp: Refactor/Rename BufferView
TODO: fix bit level dump again
2021-09-28 01:46:32 +02:00
Mattias Wadman
c997536f14 interp: Add root, buffer_root, format_root, parent and parents 2021-09-28 01:46:29 +02:00
Mattias Wadman
0cce5ec61f interp: Reorganize, move out repl and options, more functions to funcs.jq 2021-09-22 21:08:36 +02:00
Mattias Wadman
976e992c31 interp: Move progress logic to jq 2021-09-20 17:47:40 +02:00
Mattias Wadman
c770a754dc funcs: make in_bytes/bits_range more generic 2021-09-14 12:55:53 +02:00
Mattias Wadman
d350971fa2 cli: Add --decode-file VAR PATH support 2021-09-12 13:08:55 +02:00
Mattias Wadman
962d84dc68 funcs: Make intdiv truncate to int 2021-09-12 13:08:55 +02:00
Mattias Wadman
48517c7c60 cli: Unbreak part of completion 2021-09-12 13:08:55 +02:00
Mattias Wadman
0a97f86f32 doc: Cleanup and note about repl limit 2021-09-12 13:08:55 +02:00
Mattias Wadman
834f4a5893 fq: use jq functions for all display alises 2021-09-12 13:08:55 +02:00
Mattias Wadman
905c0ab74e fq: Add chunk/1 2021-09-12 13:08:55 +02:00
Mattias Wadman
25f5ad7566 fq,cli: Rename chunk to streaks, cleanup 2021-09-12 13:08:55 +02:00
Mattias Wadman
dfcefc1b55 fq: Make format/0 native for performance 2021-09-12 13:08:53 +02:00
Mattias Wadman
45afbe6201 wip 2021-09-12 13:08:53 +02:00
Mattias Wadman
e2ff2a277a cli: Move help/0 to inter.jq, better help for -n 2021-09-12 13:08:53 +02:00
Mattias Wadman
01d8a902eb funcs: Add delta/0 delta_by/0 2021-09-12 13:08:53 +02:00
Mattias Wadman
96f7a750b9 funcs: Fix typo add count/0 2021-09-12 13:08:53 +02:00
Mattias Wadman
6a8ba315fa number_to_bytes: Force int to make it work with bigint 2021-09-12 13:08:52 +02:00