Mattias Wadman
41551de331
interp,repl: Improved eval and output interrupt
...
repl now rewrites query to do implicit display inside the sub eval.
This makes it possible to interrupt eval and output in a better and faster way.
Make JSON encoder fail early on errors.
Add more interrupt tests.
2022-03-11 16:22:30 +01:00
Mattias Wadman
6f10745aa5
interp: Fix interrupt regression after query rewrite refactor
...
Also add test for it
2022-03-11 12:41:49 +01:00
Mattias Wadman
dff7e7dadb
interp: Cleanup binary regexp overloading and add explode
...
Add explode for binary that explode into an array of the binary unit tobits
Remove scan_toend as it feels a bit weird and one can use tobytes | .[match(...):] instead
Some general cleanup also
2022-03-09 20:17:12 +01:00
Mattias Wadman
56ae4a0c70
test: Make expect cli test more robust
...
Wait for prompt before sending ^D. Seems to fix issue with expect on ubuntu focal.
Thanks @KristianKarl for report
2022-03-07 16:22:41 +01:00
Mattias Wadman
eeb59152a6
interp: Make help output less wide
2022-03-03 22:02:33 +01:00
Mattias Wadman
9cb4205b1d
repl: Correct error arrow position in color mode
2022-03-03 21:24:03 +01:00
Mattias Wadman
3f50bb90a6
interp: Rework formats and options help
...
Replace --formats with --help formats
Add --help options
2022-03-03 18:25:20 +01:00
Mattias Wadman
33efb02af0
interp,repl: Add path and value colors to prompt
2022-03-03 18:25:19 +01:00
Mattias Wadman
fe8183b500
interp: Color parse in jq
2022-03-03 17:55:24 +01:00
Mattias Wadman
be6f00935f
gojq: Rebase fq fork and add support for non-10 base for tonumber
...
tonumber now works with non-10 number literals
Fix try precedences changes
Fix .[]? error handling changes
2022-03-02 23:46:20 +01:00
Mattias Wadman
0a043f9096
repl,interp: Refactor repl and slurp
...
Now repl, slurp and help implemented using same query rewrite.
Include filename context in error if possible.
Add spew function that does opposite of slurp.
Start of help infra, not done or documented yet.
Show error pointer on parse error.
Rename internal eval to _eval and make eval be wrapper that
does rewrite and has various eror handling etc.
Nicer repl, slupr and help errors.
2022-03-01 18:31:00 +01:00
Mattias Wadman
e0f493fa0b
Merge pull request #51 from Akaame/43_macho_support
...
[WIP] 43: Initial impl for macho support.
2022-02-28 10:22:50 +01:00
Mattias Wadman
f66f115e11
interp: Make _finally handle null, call fin once and last
...
null value from f did not call fin.
fin was not always called last
Now fin is onyl called once
Might be cause of some weird repl bugs.
2022-02-24 18:39:03 +01:00
Mattias Wadman
03f450f82e
interp: Skip redundant includes
...
Speeds up interp quite a bit.
Also good as i've start to use more includes to make code easier to follow
where thigns come from and also makes jq-lsp happier.
Also rename evalContext to evalInstace to make it less confused with context.
2022-02-24 12:44:26 +01:00
siddik.acil
2021b05408
macho: make actual
2022-02-21 20:32:17 +01:00
siddik.acil
b78ed02fb3
macho: barebones decoder impl
2022-02-21 20:31:12 +01:00
Mattias Wadman
0bc1171914
interp: Move opts eval to options.jq
...
Also made _main a bit nicer
2022-02-20 21:48:01 +01:00
Mattias Wadman
ca8cdadbe3
repl: Add comments and query from/to helper
2022-02-18 20:59:21 +01:00
Mattias Wadman
e238f292af
repl: Speedup multi input to sub-repl
...
Rewrite non-sub-repl queries as "q" -> ".[] | q" and pass in inputs as array.
Before each input did an eval.
Also fixup error message a bit, skip line:column when they dont make sense.
2022-02-18 01:13:45 +01:00
Mattias Wadman
cd2cbef6fe
decode: Some cleanup
...
Fix PeekFind arg order
2022-02-17 00:25:24 +01:00
Mattias Wadman
9e4f264193
dev: Add .jq-lsp.jq to add additional builtins for jq-lsp
...
Fixes missing function errors for fq builtins
Add includes to fix some other missing function errors
Add COMPLETION_TIMEOUT env to increase comepletion timeout during test using -race
2022-02-16 19:52:42 +01:00
Mattias Wadman
9dc59e5d39
interp: Move _is_decode_value to jq
...
Make _extype work like type
2022-02-15 22:20:32 +01:00
Xentripetal
9636613ec6
Merge branch 'master' of https://github.com/wader/fq into avro
2022-02-12 22:16:50 -06:00
Mattias Wadman
b86da7aecc
doc: Add inital decoder API documentation
2022-02-12 16:13:29 +01:00
Mattias Wadman
b55ca2cd5d
gojq: Rebase fq branch
...
Add bin,hex,oct bigint number literal tests
2022-02-12 13:03:38 +01:00
Mattias Wadman
af053811cf
repl,interp: Make stdio work during completion
...
Unbreaks completion using options
2022-02-11 20:04:07 +01:00
Mattias Wadman
37d07926c1
Merge pull request #144 from wader/repl-complete-extkeys-without-underscore
...
repl: Fix completion of non-underscore extkeys
2022-02-11 18:42:36 +01:00
Mattias Wadman
bd9be2c533
repl: Fix completion of non-underscore extkeys
...
For example .size for binary
2022-02-11 18:30:17 +01:00
Mattias Wadman
48a19cb82c
interp: Add paste function to allow pasting text into REPL etc
...
Also refactor readline and eval args into option struct and partinally start
addressing some side effects during completion.
2022-02-11 17:59:06 +01:00
Mattias Wadman
77ab667bfa
interp: Use absolute path in errors
...
Make it easier to find parse error in init.jq etc
2022-02-11 16:34:20 +01:00
Mattias Wadman
26d9650b0e
interp: Refactor radix* into toradix($base)/fromradix($base)
...
There is also toradix($base; $table)/fromradix($base; $table) for custom symbols and bases.
Add radix tests
Part of cleaning up all convert/tranformation functions into to/from*.
Remove number_to_bytes as with better binary support it can be written as:
1234 | tobytes | [.[range(.size)]] and soon as 1234 | tobytes | explode
2022-02-11 01:28:39 +01:00
Mattias Wadman
0d74e879fb
bitio,doc: Even more cleanup
2022-02-09 01:35:47 +01:00
Mattias Wadman
82aeb35591
bitio: More doc cleanup
2022-02-09 01:03:50 +01:00
Mattias Wadman
06245d1295
binary,decode,doc: Rename buffer to binary and add some documentation
...
Rename buffer to binary. Still some work left what to call buffer/binary in decode code.
Document decode value and binary type
Fix proper unit padding for tobytes and add still undocumenated extra padding argument.
Add some additional binary tests
2022-02-08 22:20:28 +01:00
Xentripetal
5b11385ea4
Pull latest from wader
2022-02-07 17:35:33 -06:00
Mattias Wadman
2dc2a24f92
Merge pull request #132 from wader/asn1-frompem-doc
...
asn1_ber: Add more doc and multiple outputs for frompem
2022-02-07 18:04:12 +01:00
Mattias Wadman
0312c92c31
asn1_ber: Add more doc and multiple outputs for frompem
2022-02-07 17:41:05 +01:00
Mattias Wadman
0a1a5610a9
interp: Add missing default opts for tovalue
2022-02-07 17:11:51 +01:00
Mattias Wadman
0829c167cc
asn1_ber: Add decoder
...
Basic support, can also do CER and DER but without any extra validation.
No schema support.
Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.
Related to #20
2022-02-06 22:13:03 +01:00
Mattias Wadman
be71eb01ac
decode: Rename LenFn() to LimitedFn, add FramedFn and document
...
I think framed makes more sense, but not sure
2022-02-06 13:49:00 +01:00
Mattias Wadman
032cb7d3d7
Merge pull request #126 from wader/interp-allow-force-as-cli-option
...
interp: Support force decode as -o force=true
2022-02-05 17:30:37 +01:00
Mattias Wadman
366f6b185c
interp: Support force decode as -o force=true
2022-02-05 17:24:42 +01:00
Mattias Wadman
d854ed5737
bitio: Cleanup documentation a bit
2022-02-05 16:54:54 +01:00
Mattias Wadman
7c5215347d
bitio,decode: Refactor bitio usage and make buffer slicing more correct
...
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.
bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.
Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
2022-02-04 21:41:53 +01:00
Mattias Wadman
4ab6381dc4
decode: Add scalars args to FieldRootBitBuf
...
Also move *Value.TryScalarFn to value.go
2022-02-01 22:24:24 +01:00
Mattias Wadman
b66e3287f2
Merge pull request #112 from wader/interp-cleanup-d-alises
...
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
2022-01-29 12:28:46 +01:00
Mattias Wadman
2efc3cba9b
Merge pull request #115 from wader/doc-example-usages
...
doc,interp: Add some example usages to cli help
2022-01-29 12:26:08 +01:00
Mattias Wadman
7bc2521938
doc,interp: Add some example usages to cli help
2022-01-29 12:21:51 +01:00
Mattias Wadman
ca68e6a1e4
interp: Add Platform() method to OS interface
...
Felt bad that main should have to know about it
2022-01-29 11:31:36 +01:00
Mattias Wadman
fc0aacb654
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
...
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
2022-01-28 18:25:38 +01:00
Mattias Wadman
6cd1c38fbf
decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers
2022-01-28 17:18:50 +01:00
Mattias Wadman
b2e5a83860
Merge pull request #110 from wader/doc-improve-use-other-func-notation
...
doc: Use f($a; $b) instead of jq f/2 notation
2022-01-27 10:10:12 +01:00
Mattias Wadman
c52a1a2364
doc: Use f($a; $b) instead of jq f/2 notation
...
Also some general polish
2022-01-27 09:55:45 +01:00
Mattias Wadman
233d86a384
fq: Add arch and os to --version
2022-01-26 18:09:59 +01:00
Mattias Wadman
898dfec1d1
lint: Fix typeassert and case exhaustive warnings
2022-01-25 17:23:41 +01:00
Mattias Wadman
e792598c69
interp: Add grep_by/1 to recursively match using a filter
...
Refactor other grep functions to use it
2022-01-24 18:26:25 +01:00
Mattias Wadman
bc107f82e4
Merge pull request #101 from wader/dump-sym-correct-color
...
interp: Use correct sym color
2022-01-24 17:53:46 +01:00
Mattias Wadman
c31ec2a388
interp: Use correct sym color
2022-01-24 16:57:35 +01:00
Mattias Wadman
b8efd8e549
fuzz: Fuzz all formats
...
Fix string readers to not panic on make slice with bogus sizes
2022-01-24 16:56:24 +01:00
Mattias Wadman
21ce6574fa
Merge pull request #96 from wader/pcapng-fatal-on-negzero-blocklen
...
pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
2022-01-23 13:39:04 +01:00
Mattias Wadman
d1943dad49
pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
...
Also add dev/fuzzbytes.go tool to convert fuzz input to raw bytes
2022-01-23 13:34:56 +01:00
Mattias Wadman
bf7a483f87
interp: Fix handling of group decode error from stdin
...
null input to open means stdin but it as also erroneously also used as object key
to store the error.
2022-01-23 12:41:31 +01:00
Mattias Wadman
34a09c40a7
Merge pull request #90 from wader/decode-stable-sort
...
decode: Use stable sort for values to not change order or values with…
2022-01-20 16:34:47 +01:00
Mattias Wadman
9198e18c10
decode: Add FieldArrayRootBitBufFn for completeness
...
Should rethink these, not very nice api
2022-01-20 16:27:52 +01:00
Mattias Wadman
cf8a50c150
decode: Use stable sort for values to not change order or values with same range start
...
Some decoders might relay on values added with same start to be kept in same order.
2022-01-20 16:21:50 +01:00
Xentripetal
6e1f338ac3
Pull latest from wader
2022-01-19 19:39:12 -06:00
Mattias Wadman
ff5c0b8195
decode: Error on negative number of bits when reading numbers
...
Maybe should even give error on zero bits? maybe only allow for zero bit for unsigned ints?
2022-01-18 12:01:04 +01:00
Mattias Wadman
548a065388
decode,interp: Finish up nil value support
...
Update msgpack, bson and cbor to use it.
2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878
num,mathextra: Rename num package to mathextra
...
Think it makes more sense
2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae
decode,interp: Add arbitrary large integer support (BigInt)
...
Was already handled in fq in various places as gojq uses them
Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
2022-01-15 19:00:42 +01:00
Xentripetal
788fd0258f
Pull latest from wader
2022-01-13 20:59:56 -06:00
Mattias Wadman
0b0f28e966
cbor: Add decoder
...
Does not decode sematic tag types
Also fixes broken float16 support
Fixes #71
2022-01-12 17:56:03 +01:00
Mattias Wadman
149cb3f45a
interp: Add torepr/0 that converts decode value into what it reptresents
...
Ex: fq -d msgpack torepr file.msgpack
Willoutput the JSON representation of the msgpack
Make per format *_torepr functions internal
2022-01-12 17:33:58 +01:00
Mattias Wadman
05df5a9092
Merge pull request #69 from wader/msgpack
...
msgpack: Add decoder
2022-01-10 21:38:59 +01:00
Mattias Wadman
9770b0046b
repl: Make prompt for array and iter more jqish
...
Really need to refactor script.go some day into a proper parser
2022-01-10 20:02:43 +01:00
Mattias Wadman
a6bf62ce19
msgpack: Add decoder
...
Fixes #25
2022-01-10 19:33:09 +01:00
Mattias Wadman
af8e7efc61
bencode: Add decoder
...
Closes #63
2022-01-10 10:52:14 +01:00
Mattias Wadman
b3a09805a4
interp: Document bit opts funcs and add some error tests
2022-01-09 10:43:07 +01:00
Mattias Wadman
be0ef80c8d
interp,fq: Make bit operators normal functions
...
Uses same rule as jq math functions, 1 arg uses input, more than one all passed as args.
So "bnot 1" -> "1 | bnot", "1 bsl 1" -> "bsl(1; 1)"
Don't think it's worth changing the jq syntax for this and also it could make fq
scripts no compatiblr with other jq tools. Non-10 base number literals are still allowed
but should probably not be used in scripts, only in repl and with cli.
2022-01-09 01:06:36 +01:00
Mattias Wadman
a3cfcd0ad2
interp: Add stderr again for jq compat
...
Also add tests for print* and stderr
2022-01-07 11:30:09 +01:00
Mattias Wadman
628f0f403d
bson: Add decoder
...
Only supports basic types for now
2022-01-06 00:51:37 +01:00
Mattias Wadman
681dbc26b7
elf,ar: Add ar decoder, improved elf decoder
...
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
2022-01-05 22:05:09 +01:00
Mattias Wadman
f6abcff1fc
Merge pull request #54 from wader/interp-stdio-cleanup
...
interp: Cleanup stdio usage and functions
2022-01-05 21:44:48 +01:00
Mattias Wadman
e365f22084
interp: Cleanup stdio usage and functions
2022-01-05 21:41:11 +01:00
Mattias Wadman
b6515c8b15
interp: Remove buffer smartness for regexp match functions
...
Just confusing and breaks symbolic grep for decode values (as they are buffers).
To get buffer use tobytes functions intstead.
2022-01-05 21:13:39 +01:00
Mattias Wadman
a41f0d4b5f
windows: Correct @builtin include path join
...
Failed because go embed paths always uses /
Switch to always use forward slash paths internally
Related to #21
2022-01-04 22:10:53 +01:00
Mattias Wadman
b9aef39ed4
pcap,pcapng,bsd_loopback_frame: Add decoder, refactor link frame into a group
...
Also fix incorrect struct name s/UDPDatagramIn/UDPPayloadIn/
2022-01-03 21:06:27 +01:00
Mattias Wadman
6db7218093
doc: Some more dev infomration
2021-12-31 17:13:16 +01:00
Xentripetal
7a8e3ca26e
Hook into registry, add codecs
2021-12-28 13:05:10 -06:00
Mattias Wadman
55b1d5c3e7
interp: Move _registry to decode
...
More related to decode than interp. Maybe should have a format.go?
2021-12-25 16:29:45 +01:00
Mattias Wadman
46b59d0ebf
crc: Unbreak build on 32-bit arch
...
Thanks Alexey Dokuchaev @danfe for report and fix.
Fixes #30
2021-12-25 15:47:54 +01:00
Mattias Wadman
f7c7801bbe
interp: Rework buffer regex support
...
Now all regexp functions should work similar to string version for buffers.
Added scan_toend that works as scan but outputs buffer from match to end of buffer,
useful when trying to decode a format and you dont know or care to limit.
2021-12-17 16:23:44 +01:00
Mattias Wadman
f4f63835dc
interp: Add ._index for values in arrays
2021-12-10 18:19:11 +01:00
Mattias Wadman
e823475d3c
interp: Fix help a bit
2021-12-09 21:11:24 +01:00
Mattias Wadman
f3480026bf
sll2: fuzz: Limit address length to max 8 bytes
2021-12-09 10:38:34 +01:00
Mattias Wadman
6ed2e2e72e
interp: dump: Indicate arrays using jq-syntax
...
Related to #16
2021-12-08 16:24:28 +01:00
Mattias Wadman
c2131bb4a0
flac: Cleanup scalar usage and fix incorrect sample rates
...
Also split DisplayFormat into actual and sym
2021-12-08 00:39:52 +01:00
Mattias Wadman
9b683cd839
deocde: Cleanup some io panic(err)
2021-12-07 17:33:57 +01:00
Mattias Wadman
713ffe4770
scalar: Add Require* and Require/Validate/AssertRange*
2021-12-07 15:50:55 +01:00
Mattias Wadman
fa350c6165
interp: Move display to jq
2021-12-06 21:08:40 +01:00