wader/fq
1
1
Fork 0
mirror of https://github.com/wader/fq.git synced 2024-12-26 23:15:04 +03:00
Code Issues Projects Releases Wiki Activity
2,235 Commits 36 Branches 23 Tags 62 MiB
a4cfdcf518
Commit Graph

518 Commits

Author SHA1 Message Date
Mattias Wadman
32361deefe gojqextra: Cleanup gojq type cast code 2022-06-20 18:02:35 +02:00
Mattias Wadman
81a014ce49 interp: Make empty _finally fin error on error 
Ex: _finally(error(a); empty)
This is not used by fq atm
 2022-06-18 00:00:54 +02:00
Mattias Wadman
0863374f8c doc: Correct bencode spec URL 
Also rename _tojq value_sep to object_sep and fix formats_diagram.jq to use tomd5
 2022-06-13 18:49:34 +02:00
Mattias Wadman
a64c28d927 dump: Skip JQValueEx if there are not options 
I used to tojson and would crash in binary values are there were
not optins deciding how to format the strinh. Now just treat as utf8 string.
 2022-06-07 16:39:45 +02:00
Mattias Wadman
c1a0cda5a8 gojq: Update rebased fq fork 
Fix tostring for non-string JQValue:s, is now JSON
Update EOF error
 2022-06-01 17:27:45 +02:00
Mattias Wadman
6b0880002d interp: Cast jq value to go value properly for encoding functions 
Some encoding fuctions accept binary used string as input type, should be any.
Add cast helper functions, hopefully can be useful in future for even nicer
function bindings api.
 2022-05-30 18:27:12 +02:00
Mattias Wadman
735c443b37 interp: Improve type normalization and use it for toyaml and totoml 
Use smallest int type for int64, uint6 and *big.Int

Fixes integer serialization for yaml and toml for small integers, othweise
they will end up as strings.
 2022-05-28 20:33:37 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes 
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
 2022-05-28 16:31:20 +02:00
Mattias Wadman
7b27e506f1 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio 
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.

Add test with playread and widevine pssh boxes

Related to #282
 2022-05-28 13:42:38 +02:00
Mattias Wadman
8021492197 interp: help: Fix incorrect options example 2022-05-26 12:52:51 +02:00
Mattias Wadman
9a7ce14886 gojq: Update rebased fq fork 
Use new gojq.TypeOf and gojq.Preview
 2022-05-23 21:33:12 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage 
Make cwd for a test script the directory where the script is.
Use relative paths
 2022-05-21 20:03:25 +02:00
Mattias Wadman
95e619659b interp: Add internal _is_<type> helpers 2022-05-20 16:49:00 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18 
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
 2022-05-20 15:23:16 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now 
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
 2022-05-20 12:38:43 +02:00
Mattias Wadman
8dde3ef5b1 interp: Fix crash when including relatve path when no search paths are set 2022-05-17 18:05:10 +02:00
Mattias Wadman
469116f3b3
Merge pull request #260 from wader/mono-unicode-fix 
interp: dump: Fix column truncate issue with unicode bars
 2022-05-11 00:44:50 +02:00
Mattias Wadman
dff3cc1199 interp: dump: Fix column truncate issue with unicode bars 2022-05-11 00:37:52 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots 
Hopefully makes it a bit easier to read and spot
 2022-05-11 00:00:39 +02:00
Mattias Wadman
76161a1b99 scalar,mp4,gzip,tar: Add timestamp to description 
Scalar api for this needs some work
 2022-05-07 13:15:13 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming 
Should replace most of this with generics at some point
 2022-05-07 12:46:34 +02:00
Mattias Wadman
c4dd518e04 decode: Make compound range sort optional 
Some formats might want to control child order
mp4: Keep tracks in track id order
dns: Keep label component order
elf: Keep seciton order
macho: Keep command and section order
 2022-05-03 16:16:09 +02:00
Mattias Wadman
3ce660a2cf decode: Keep decode tree on RangeFn error 2022-05-03 10:57:43 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options 
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
 2022-05-01 17:08:30 +02:00
Mattias Wadman
21ad628add interp: dump: Show field name for compound values in arrays 
mp3: Remove one level of struct for granales
dump: Makes it a bit easier to understand array of arrays/structs.
avro_ocf: Rename ocf.go -> avro_ocr.go to have same name of source file as format
pcap,pcapng: Rename tcp flow struct to tcp_connection, makes more sense i think
mp4: Add ISOBMFF to format description
 2022-04-21 17:47:15 +02:00
Mattias Wadman
afd724bf9c gojq: Rebase fq fork. Fixes JQValue path tracking when iterating 
Add some decode value path and assign tests
 2022-04-19 19:00:37 +02:00
Mattias Wadman
6b4dbbf415
Merge pull request #234 from wader/field-truncate2 
interp: Don't truncate last display column
 2022-04-19 18:30:17 +02:00
Mattias Wadman
607202bb67 interp: Don't truncate last display column 
Wrapping lines might be annoying but not seeing things even worse i think.
Hopefully once column config is in jq it could be an option.
 2022-04-19 18:19:42 +02:00
Mattias Wadman
087d12419e interp: Simpler and more efficient hexdump 
Should not need to copy bits, just use existing bitreader
 2022-04-19 18:12:58 +02:00
Mattias Wadman
dabad85080 interp: Proper display column truncate 
Also speed up by using less string allocs
 2022-04-16 18:48:21 +02:00
Mattias Wadman
b35b1804be decode,format: Add d.FieldFormatOrRaw(Len) 2022-04-11 23:01:45 +02:00
Mattias Wadman
518f6af4a8 gojq: Rebase fq fork 
Upstream changes:
Many performance improvements
Error message improvments
Lots of refactoring

Most of the JQValue interface changes in gojq fork had to be reworked but
resultet in a much nicer and cleaner changeset.

fq changes:
Assignment to JQValue (like a decode value) now shallowly converts the value into
a jq value before assigning. Was a bit hacky as it was and this makes JQValue
behave more like real jq values. This also fixes some advanced indexing issues.

Actual custom path updates will be something for the future.
 2022-04-11 19:23:15 +02:00
Mattias Wadman
6f03471d15 interp: Paths with a array as root was missing start dot 2022-04-08 16:47:40 +02:00
Mattias Wadman
f8fd6b7f48 mp4: Add more HEIF boxes 
Add scalar.*ToDesceription mapper and refactor to use it
 2022-04-07 19:11:49 +02:00
Mattias Wadman
5ff67e4cfe formats: Sym and field name cleanup to be more jq friendly 2022-04-05 13:59:57 +02:00
Mattias Wadman
e8dc7112b6 ipv6,icmpv6: Add decoder 
Refactor to use groups between network layers to make them less coupled and reusable:
link_frame (ethernet etc)
inet_packet (ipv4,ipv6 etc)
ip_packet (tcp packet (not stream), udp segment etc)

Rename data to payload as i think it makes more sense for network data
 2022-04-03 17:43:51 +02:00
Mattias Wadman
d8aaf30345 rtmp,amf0: Add decoders 
Basic RTMP support, used via tcp_stream group (pcap etc) or manually.
Basic AMF0 support, mostly what is used in RTMP.
 2022-04-01 13:07:20 +02:00
Mattias Wadman
e8678ca88e interp: Remove opts refactor leftover 2022-03-28 15:37:38 +02:00
Mattias Wadman
74978c9d08 hevc: Add hevc_vps, hevc_sps and hevc_pps decoders 
Some extensions and other things still missing
 2022-03-24 18:56:56 +01:00
Mattias Wadman
d376520f11 interp: Remove to*range pad argument and fix stdout padding issue 
Padding could end up double.
Remove to*range($pad) as it probably just confusing to be able
to pad an existing range, what to show in hexdump etc? zero bits
that do not actuall exist at that range?

Add tests and binary stdout support to tests
 2022-03-17 20:15:35 +01:00
Mattias Wadman
9cb4b57a45 interp,cli: Handle ctrl-c properly 
Should supress cancel error message
Add CLI ctlr-c/ctrl-d tests and make them more robust
 2022-03-15 22:38:12 +01:00
Mattias Wadman
41551de331 interp,repl: Improved eval and output interrupt 
repl now rewrites query to do implicit display inside the sub eval.
This makes it possible to interrupt eval and output in a better and faster way.

Make JSON encoder fail early on errors.

Add more interrupt tests.
 2022-03-11 16:22:30 +01:00
Mattias Wadman
6f10745aa5 interp: Fix interrupt regression after query rewrite refactor 
Also add test for it
 2022-03-11 12:41:49 +01:00
Mattias Wadman
dff7e7dadb interp: Cleanup binary regexp overloading and add explode 
Add explode for binary that explode into an array of the binary unit tobits
Remove scan_toend as it feels a bit weird and one can use tobytes | .[match(...):] instead
Some general cleanup also
 2022-03-09 20:17:12 +01:00
Mattias Wadman
56ae4a0c70 test: Make expect cli test more robust 
Wait for prompt before sending ^D. Seems to fix issue with expect on ubuntu focal.

Thanks @KristianKarl for report
 2022-03-07 16:22:41 +01:00
Mattias Wadman
eeb59152a6 interp: Make help output less wide 2022-03-03 22:02:33 +01:00
Mattias Wadman
9cb4205b1d repl: Correct error arrow position in color mode 2022-03-03 21:24:03 +01:00
Mattias Wadman
3f50bb90a6 interp: Rework formats and options help 
Replace --formats with --help formats
Add --help options
 2022-03-03 18:25:20 +01:00
Mattias Wadman
33efb02af0 interp,repl: Add path and value colors to prompt 2022-03-03 18:25:19 +01:00
Mattias Wadman
fe8183b500 interp: Color parse in jq 2022-03-03 17:55:24 +01:00
Mattias Wadman
be6f00935f gojq: Rebase fq fork and add support for non-10 base for tonumber 
tonumber now works with non-10 number literals
Fix try precedences changes
Fix .[]? error handling changes
 2022-03-02 23:46:20 +01:00
Mattias Wadman
0a043f9096 repl,interp: Refactor repl and slurp 
Now repl, slurp and help implemented using same query rewrite.
Include filename context in error if possible.
Add spew function that does opposite of slurp.
Start of help infra, not done or documented yet.
Show error pointer on parse error.
Rename internal eval to _eval and make eval be wrapper that
does rewrite and has various eror handling etc.
Nicer repl, slupr and help errors.
 2022-03-01 18:31:00 +01:00
Mattias Wadman
e0f493fa0b
Merge pull request #51 from Akaame/43_macho_support 
[WIP] 43: Initial impl for macho support.
 2022-02-28 10:22:50 +01:00
Mattias Wadman
f66f115e11 interp: Make _finally handle null, call fin once and last 
null value from f did not call fin.
fin was not always called last
Now fin is onyl called once

Might be cause of some weird repl bugs.
 2022-02-24 18:39:03 +01:00
Mattias Wadman
03f450f82e interp: Skip redundant includes 
Speeds up interp quite a bit.
Also good as i've start to use more includes to make code easier to follow
where thigns come from and also makes jq-lsp happier.

Also rename evalContext to evalInstace to make it less confused with context.
 2022-02-24 12:44:26 +01:00
siddik.acil
2021b05408 macho: make actual 2022-02-21 20:32:17 +01:00
siddik.acil
b78ed02fb3 macho: barebones decoder impl 2022-02-21 20:31:12 +01:00
Mattias Wadman
0bc1171914 interp: Move opts eval to options.jq 
Also made _main a bit nicer
 2022-02-20 21:48:01 +01:00
Mattias Wadman
ca8cdadbe3 repl: Add comments and query from/to helper 2022-02-18 20:59:21 +01:00
Mattias Wadman
e238f292af repl: Speedup multi input to sub-repl 
Rewrite non-sub-repl queries as "q" -> ".[] | q" and pass in inputs as array.
Before each input did an eval.

Also fixup error message a bit, skip line:column when they dont make sense.
 2022-02-18 01:13:45 +01:00
Mattias Wadman
cd2cbef6fe decode: Some cleanup 
Fix PeekFind arg order
 2022-02-17 00:25:24 +01:00
Mattias Wadman
9e4f264193 dev: Add .jq-lsp.jq to add additional builtins for jq-lsp 
Fixes missing function errors for fq builtins
Add includes to fix some other missing function errors

Add COMPLETION_TIMEOUT env to increase comepletion timeout during test using -race
 2022-02-16 19:52:42 +01:00
Mattias Wadman
9dc59e5d39 interp: Move _is_decode_value to jq 
Make _extype work like type
 2022-02-15 22:20:32 +01:00
Xentripetal
9636613ec6 Merge branch 'master' of https://github.com/wader/fq into avro 2022-02-12 22:16:50 -06:00
Mattias Wadman
b86da7aecc doc: Add inital decoder API documentation 2022-02-12 16:13:29 +01:00
Mattias Wadman
b55ca2cd5d gojq: Rebase fq branch 
Add bin,hex,oct bigint number literal tests
 2022-02-12 13:03:38 +01:00
Mattias Wadman
af053811cf repl,interp: Make stdio work during completion 
Unbreaks completion using options
 2022-02-11 20:04:07 +01:00
Mattias Wadman
37d07926c1
Merge pull request #144 from wader/repl-complete-extkeys-without-underscore 
repl: Fix completion of non-underscore extkeys
 2022-02-11 18:42:36 +01:00
Mattias Wadman
bd9be2c533 repl: Fix completion of non-underscore extkeys 
For example .size for binary
 2022-02-11 18:30:17 +01:00
Mattias Wadman
48a19cb82c interp: Add paste function to allow pasting text into REPL etc 
Also refactor readline and eval args into option struct and partinally start
addressing some side effects during completion.
 2022-02-11 17:59:06 +01:00
Mattias Wadman
77ab667bfa interp: Use absolute path in errors 
Make it easier to find parse error in init.jq etc
 2022-02-11 16:34:20 +01:00
Mattias Wadman
26d9650b0e interp: Refactor radix* into toradix($base)/fromradix($base) 
There is also toradix($base; $table)/fromradix($base; $table) for custom symbols and bases.
Add radix tests

Part of cleaning up all convert/tranformation functions into to/from*.

Remove number_to_bytes as with better binary support it can be written as:
1234 | tobytes | [.[range(.size)]] and soon as 1234 | tobytes | explode
 2022-02-11 01:28:39 +01:00
Mattias Wadman
0d74e879fb bitio,doc: Even more cleanup 2022-02-09 01:35:47 +01:00
Mattias Wadman
82aeb35591 bitio: More doc cleanup 2022-02-09 01:03:50 +01:00
Mattias Wadman
06245d1295 binary,decode,doc: Rename buffer to binary and add some documentation 
Rename buffer to binary. Still some work left what to call buffer/binary in decode code.
Document decode value and binary type
Fix proper unit padding for tobytes and add still undocumenated extra padding argument.
Add some additional binary tests
 2022-02-08 22:20:28 +01:00
Xentripetal
5b11385ea4 Pull latest from wader 2022-02-07 17:35:33 -06:00
Mattias Wadman
2dc2a24f92
Merge pull request #132 from wader/asn1-frompem-doc 
asn1_ber: Add more doc and multiple outputs for frompem
 2022-02-07 18:04:12 +01:00
Mattias Wadman
0312c92c31 asn1_ber: Add more doc and multiple outputs for frompem 2022-02-07 17:41:05 +01:00
Mattias Wadman
0a1a5610a9 interp: Add missing default opts for tovalue 2022-02-07 17:11:51 +01:00
Mattias Wadman
0829c167cc asn1_ber: Add decoder 
Basic support, can also do CER and DER but without any extra validation.
No schema support.

Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.

Related to #20
 2022-02-06 22:13:03 +01:00
Mattias Wadman
be71eb01ac decode: Rename LenFn() to LimitedFn, add FramedFn and document 
I think framed makes more sense, but not sure
 2022-02-06 13:49:00 +01:00
Mattias Wadman
032cb7d3d7
Merge pull request #126 from wader/interp-allow-force-as-cli-option 
interp: Support force decode as -o force=true
 2022-02-05 17:30:37 +01:00
Mattias Wadman
366f6b185c interp: Support force decode as -o force=true 2022-02-05 17:24:42 +01:00
Mattias Wadman
d854ed5737 bitio: Cleanup documentation a bit 2022-02-05 16:54:54 +01:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct 
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
 2022-02-04 21:41:53 +01:00
Mattias Wadman
4ab6381dc4 decode: Add scalars args to FieldRootBitBuf 
Also move *Value.TryScalarFn to value.go
 2022-02-01 22:24:24 +01:00
Mattias Wadman
b66e3287f2
Merge pull request #112 from wader/interp-cleanup-d-alises 
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
 2022-01-29 12:28:46 +01:00
Mattias Wadman
2efc3cba9b
Merge pull request #115 from wader/doc-example-usages 
doc,interp: Add some example usages to cli help
 2022-01-29 12:26:08 +01:00
Mattias Wadman
7bc2521938 doc,interp: Add some example usages to cli help 2022-01-29 12:21:51 +01:00
Mattias Wadman
ca68e6a1e4 interp: Add Platform() method to OS interface 
Felt bad that main should have to know about it
 2022-01-29 11:31:36 +01:00
Mattias Wadman
fc0aacb654 interp: Cleanup display aliases, now: d, da, dd, dv, ddv 
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
 2022-01-28 18:25:38 +01:00
Mattias Wadman
6cd1c38fbf decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers 2022-01-28 17:18:50 +01:00
Mattias Wadman
b2e5a83860
Merge pull request #110 from wader/doc-improve-use-other-func-notation 
doc: Use f($a; $b) instead of jq f/2 notation
 2022-01-27 10:10:12 +01:00
Mattias Wadman
c52a1a2364 doc: Use f($a; $b) instead of jq f/2 notation 
Also some general polish
 2022-01-27 09:55:45 +01:00
Mattias Wadman
233d86a384 fq: Add arch and os to --version 2022-01-26 18:09:59 +01:00
Mattias Wadman
898dfec1d1 lint: Fix typeassert and case exhaustive warnings 2022-01-25 17:23:41 +01:00
Mattias Wadman
e792598c69 interp: Add grep_by/1 to recursively match using a filter 
Refactor other grep functions to use it
 2022-01-24 18:26:25 +01:00
Mattias Wadman
bc107f82e4
Merge pull request #101 from wader/dump-sym-correct-color 
interp: Use correct sym color
 2022-01-24 17:53:46 +01:00
Mattias Wadman
c31ec2a388 interp: Use correct sym color 2022-01-24 16:57:35 +01:00
Mattias Wadman
b8efd8e549 fuzz: Fuzz all formats 
Fix string readers to not panic on make slice with bogus sizes
 2022-01-24 16:56:24 +01:00
Mattias Wadman
21ce6574fa
Merge pull request #96 from wader/pcapng-fatal-on-negzero-blocklen 
pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
 2022-01-23 13:39:04 +01:00
Mattias Wadman
d1943dad49 pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0 
Also add dev/fuzzbytes.go tool to convert fuzz input to raw bytes
 2022-01-23 13:34:56 +01:00
Mattias Wadman
bf7a483f87 interp: Fix handling of group decode error from stdin 
null input to open means stdin but it as also erroneously also used as object key
to store the error.
 2022-01-23 12:41:31 +01:00
Mattias Wadman
34a09c40a7
Merge pull request #90 from wader/decode-stable-sort 
decode: Use stable sort for values to not change order or values with…
 2022-01-20 16:34:47 +01:00
Mattias Wadman
9198e18c10 decode: Add FieldArrayRootBitBufFn for completeness 
Should rethink these, not very nice api
 2022-01-20 16:27:52 +01:00
Mattias Wadman
cf8a50c150 decode: Use stable sort for values to not change order or values with same range start 
Some decoders might relay on values added with same start to be kept in same order.
 2022-01-20 16:21:50 +01:00
Xentripetal
6e1f338ac3 Pull latest from wader 2022-01-19 19:39:12 -06:00
Mattias Wadman
ff5c0b8195 decode: Error on negative number of bits when reading numbers 
Maybe should even give error on zero bits? maybe only allow for zero bit for unsigned ints?
 2022-01-18 12:01:04 +01:00
Mattias Wadman
548a065388 decode,interp: Finish up nil value support 
Update msgpack, bson and cbor to use it.
 2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra 
Think it makes more sense
 2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae decode,interp: Add arbitrary large integer support (BigInt) 
Was already handled in fq in various places as gojq uses them

Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
 2022-01-15 19:00:42 +01:00
Xentripetal
788fd0258f Pull latest from wader 2022-01-13 20:59:56 -06:00
Mattias Wadman
0b0f28e966 cbor: Add decoder 
Does not decode sematic tag types
Also fixes broken float16 support

Fixes #71
 2022-01-12 17:56:03 +01:00
Mattias Wadman
149cb3f45a interp: Add torepr/0 that converts decode value into what it reptresents 
Ex: fq -d msgpack torepr file.msgpack
Willoutput the JSON representation of the msgpack

Make per format *_torepr functions internal
 2022-01-12 17:33:58 +01:00
Mattias Wadman
05df5a9092
Merge pull request #69 from wader/msgpack 
msgpack: Add decoder
 2022-01-10 21:38:59 +01:00
Mattias Wadman
9770b0046b repl: Make prompt for array and iter more jqish 
Really need to refactor script.go some day into a proper parser
 2022-01-10 20:02:43 +01:00
Mattias Wadman
a6bf62ce19 msgpack: Add decoder 
Fixes #25
 2022-01-10 19:33:09 +01:00
Mattias Wadman
af8e7efc61 bencode: Add decoder 
Closes #63
 2022-01-10 10:52:14 +01:00
Mattias Wadman
b3a09805a4 interp: Document bit opts funcs and add some error tests 2022-01-09 10:43:07 +01:00
Mattias Wadman
be0ef80c8d interp,fq: Make bit operators normal functions 
Uses same rule as jq math functions, 1 arg uses input, more than one all passed as args.
So "bnot 1" -> "1 | bnot", "1 bsl 1" -> "bsl(1; 1)"

Don't think it's worth changing the jq syntax for this and also it could make fq
scripts no compatiblr with other jq tools. Non-10 base number literals are still allowed
but should probably not be used in scripts, only in repl and with cli.
 2022-01-09 01:06:36 +01:00
Mattias Wadman
a3cfcd0ad2 interp: Add stderr again for jq compat 
Also add tests for print* and stderr
 2022-01-07 11:30:09 +01:00
Mattias Wadman
628f0f403d bson: Add decoder 
Only supports basic types for now
 2022-01-06 00:51:37 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder 
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
 2022-01-05 22:05:09 +01:00
Mattias Wadman
f6abcff1fc
Merge pull request #54 from wader/interp-stdio-cleanup 
interp: Cleanup stdio usage and functions
 2022-01-05 21:44:48 +01:00
Mattias Wadman
e365f22084 interp: Cleanup stdio usage and functions 2022-01-05 21:41:11 +01:00
Mattias Wadman
b6515c8b15 interp: Remove buffer smartness for regexp match functions 
Just confusing and breaks symbolic grep for decode values (as they are buffers).
To get buffer use tobytes functions intstead.
 2022-01-05 21:13:39 +01:00
Mattias Wadman
a41f0d4b5f windows: Correct @builtin include path join 
Failed because go embed paths always uses /
Switch to always use forward slash paths internally

Related to #21
 2022-01-04 22:10:53 +01:00
Mattias Wadman
b9aef39ed4 pcap,pcapng,bsd_loopback_frame: Add decoder, refactor link frame into a group 
Also fix incorrect struct name s/UDPDatagramIn/UDPPayloadIn/
 2022-01-03 21:06:27 +01:00
Mattias Wadman
6db7218093 doc: Some more dev infomration 2021-12-31 17:13:16 +01:00
Xentripetal
7a8e3ca26e Hook into registry, add codecs 2021-12-28 13:05:10 -06:00
Mattias Wadman
55b1d5c3e7 interp: Move _registry to decode 
More related to decode than interp. Maybe should have a format.go?
 2021-12-25 16:29:45 +01:00
Mattias Wadman
46b59d0ebf crc: Unbreak build on 32-bit arch 
Thanks Alexey Dokuchaev @danfe for report and fix.

Fixes #30
 2021-12-25 15:47:54 +01:00
Mattias Wadman
f7c7801bbe interp: Rework buffer regex support 
Now all regexp functions should work similar to string version for buffers.
Added scan_toend that works as scan but outputs buffer from match to end of buffer,
useful when trying to decode a format and you dont know or care to limit.
 2021-12-17 16:23:44 +01:00
Mattias Wadman
f4f63835dc interp: Add ._index for values in arrays 2021-12-10 18:19:11 +01:00
Mattias Wadman
e823475d3c interp: Fix help a bit 2021-12-09 21:11:24 +01:00
Mattias Wadman
f3480026bf sll2: fuzz: Limit address length to max 8 bytes 2021-12-09 10:38:34 +01:00
Mattias Wadman
6ed2e2e72e interp: dump: Indicate arrays using jq-syntax 
Related to #16
 2021-12-08 16:24:28 +01:00
Mattias Wadman
c2131bb4a0 flac: Cleanup scalar usage and fix incorrect sample rates 
Also split DisplayFormat into actual and sym
 2021-12-08 00:39:52 +01:00
Mattias Wadman
9b683cd839 deocde: Cleanup some io panic(err) 2021-12-07 17:33:57 +01:00
Mattias Wadman
713ffe4770 scalar: Add Require* and Require/Validate/AssertRange* 2021-12-07 15:50:55 +01:00
Mattias Wadman
fa350c6165 interp: Move display to jq 2021-12-06 21:08:40 +01:00
Mattias Wadman
4558192e7c interp: Cleanup buffer code and implement ExtType() 2021-12-06 21:08:40 +01:00
Mattias Wadman
75aa47556a decode: Generate Try?Scalar* methods for readers too 2021-12-04 19:05:16 +01:00
Mattias Wadman
9aec91a675 interp: match: Fix issue with regexp meta characters when matching using a buffer 2021-12-04 17:24:43 +01:00
Mattias Wadman
d48ebc12ee decode: Simplify Compound.Children 
Not a array reference anymore but instead pass around Compound reference
 2021-12-03 00:06:11 +01:00
Mattias Wadman
36e5562e5b interp: Rename value.* to decode.* as it makes more sense 2021-12-02 21:43:16 +01:00
Mattias Wadman
1b7b2f9641 interp: Use gojqextra.NonUpdatableTypeError 2021-12-02 21:41:38 +01:00
Mattias Wadman
2fc0a71a47 decode: Refactor scalar usage 
Move scalar into own package.
Split scalar code into decode related scalar code (that reads etc) and
scalar code that just transform the scalar value.
Use a scalar.Mapper interface instead of just a function.
Make mappers, assert and validat impement the interface.
 2021-12-02 17:39:26 +01:00
Mattias Wadman
01a407a462 interp: Rename s/bufferrange/buffer and cleanup 2021-11-30 21:10:26 +01:00
Mattias Wadman
4eccb1ee71 interp: Fix broken aes_ctr, should return buffer instead of []byte 2021-11-30 15:30:34 +01:00
Mattias Wadman
f801cc0af7 decode: Rename s/FieldTryFormat/TryFieldFormat for consistency 2021-11-30 15:29:41 +01:00
Mattias Wadman
d1e1cd98c2 decode: Fix walk root depth issue causing dump to indent incorrectly 2021-11-30 12:51:52 +01:00
Mattias Wadman
8d442b82ee interp: Cleanup stdin reading and add more option tests 2021-11-30 12:25:56 +01:00
Mattias Wadman
f55b1af6ac inet: Add tcp and ipv4 reassembly 
Also add tcp_stream and udp_payload to decode content
 2021-11-29 18:42:18 +01:00
Mattias Wadman
9f2dddc6e1 interp,gojqextra: Make buffers values even more lazy and error early on non-scalar calls 2021-11-29 14:51:00 +01:00
Mattias Wadman
afb1050355 interp: Move _decode_value to value.jq 2021-11-29 14:26:59 +01:00
Mattias Wadman
8e5442f023 interp: Limit how often decode progress fn is called 2021-11-29 14:26:59 +01:00
Mattias Wadman
3bf1a57cfa avc: Cleanup and add color names etc 2021-11-28 17:29:56 +01:00
Mattias Wadman
6034ad7d67 interp: More sure stdOS stops the signal forward gorutine 2021-11-25 13:35:20 +01:00
Mattias Wadman
7b7faaf02b pcap: Add pcap, pcapng, ether8023, ipv4, udp, udp 2021-11-24 18:13:00 +01:00
Mattias Wadman
9d116df799 decode: Rework use of TryFieldReaderRangeFormat 
Handle failure better
 2021-11-24 15:25:27 +01:00
Mattias Wadman
4f8d03731d format,decode: Some crc and endian refactor 2021-11-24 12:01:01 +01:00
Mattias Wadman
e6cb708d38 mpeg: Nicer sym and description 2021-11-22 17:02:07 +01:00
Mattias Wadman
69e4eea920 interp: Better error if format/group is not found 2021-11-22 13:28:18 +01:00
Mattias Wadman
abcecb8156 interp: Remove unused []byte type 2021-11-22 01:46:06 +01:00
Mattias Wadman
fdb811ec54 gojqextra,decode: Add generic lazy JQValue 2021-11-22 01:23:59 +01:00
Mattias Wadman
1047d909ac doc: Document io packages a bit 2021-11-21 21:55:53 +01:00
Mattias Wadman
7c4b0b3844 bitio: Simplify by embedding reader 2021-11-21 21:55:09 +01:00
Mattias Wadman
aab32cf2db gojqextra,interp: Add lazy string to speed usage of decode value buffer where string is not used 2021-11-21 21:27:44 +01:00
Mattias Wadman
f40320b04c decode: Remove D.Scalar* and add d.(Try)FieldScala*Fn instead 
Idea is scalar fn should not read
 2021-11-21 13:08:18 +01:00
Mattias Wadman
0480a2f722 decode: Some format decode and sub buffer work 2021-11-20 18:56:59 +01:00
Mattias Wadman
278e909a2f mp4: Add tapt, prof, enof and clap boxes 2021-11-19 15:29:49 +01:00
Mattias Wadman
9cba69e64d interp: Use todescription in tests 2021-11-19 11:56:58 +01:00
Mattias Wadman
07b421011f interp: add topath/0 and make todescription return null if there is none 
Move out more code from funcs.jq
 2021-11-19 11:44:23 +01:00
Mattias Wadman
07c7daaf92 doc: Some rewording 2021-11-19 00:23:04 +01:00
Mattias Wadman
d838d2f613 zip: Add format decoder 2021-11-19 00:22:26 +01:00
Mattias Wadman
ffb5adff1d registry: Move to pkg/registry, feels better 2021-11-18 01:22:21 +01:00
Mattias Wadman
7af191d974 decode: Move io helper into *D 2021-11-18 01:17:41 +01:00
Mattias Wadman
6fba1a8125 decode: Fix bitbuf root handling a bit 2021-11-18 01:17:15 +01:00
Mattias Wadman
c083a9e475 decode: Fix MapRawToScalar regression 2021-11-17 17:14:19 +01:00
Mattias Wadman
986d5ecc50 decode: Move registry package to decode/registry and add a format group type 2021-11-17 16:59:56 +01:00
Mattias Wadman
5d98a6949a decode: Refactor Error/Fatal into printf functions 2021-11-17 16:26:13 +01:00
Mattias Wadman
ede2e77975 decode: Nicer scalar template and add doc 2021-11-17 16:13:10 +01:00
Mattias Wadman
826c5094f2 decode: Add Generated header for scalar 2021-11-17 10:19:10 +01:00
Mattias Wadman
b66ed32614 decode,interp: Make fuzzing work again and cleanup fatal/error code 2021-11-16 17:11:26 +01:00
Mattias Wadman
f9f866000e interp,decode: Add force option to ignore asserts 2021-11-16 13:03:56 +01:00
Mattias Wadman
5cd5633374 interp: Move jq function impls closer to where they belong 2021-11-16 11:43:11 +01:00
Mattias Wadman
996be0fae3 interp: Move more options code into options.jq 2021-11-16 10:49:18 +01:00
Mattias Wadman
3fafee8559 interp: Clearer help for -d 2021-11-16 08:43:29 +01:00
Mattias Wadman
1b32b42f93 decode: Major decode API refactor 
Generate more code
More generic and comfortable API
Improve and Update format decoder to new API
Add some more format tests
 2021-11-15 21:12:07 +01:00
Mattias Wadman
fcbfc29154 make: Move testjq.sh into pkg/interp 2021-11-12 00:23:16 +01:00
Mattias Wadman
178032e0a8 interp: Fix $opts shadowing in decode 2021-11-12 00:22:57 +01:00
Mattias Wadman
cc5f405554 interp,format: Update tests after decode refactor/tosym/toactual 2021-11-05 17:29:23 +01:00
Mattias Wadman
6a15625587 interp,decode: Refactor out Scalar from Value and merge Array/Struct into Compound 
Also add tosym, toactual
 2021-11-05 17:29:22 +01:00
Mattias Wadman
6cacc9b398 interp: Move *CtxWriter to ioextra, some comments cleanup 2021-11-05 17:28:29 +01:00
Mattias Wadman
ff2077bc0f interp: Simplify Function, aliases done in jq now 2021-11-05 17:28:29 +01:00
Mattias Wadman
93322bc434 interp: Implement format/0 in jq 2021-11-03 10:58:33 +01:00
Mattias Wadman
571bf29ab2 Change project title 2021-11-01 17:47:36 +01:00
Mattias Wadman
58ba84ff93 decode,interp: Add RecoverableErrorer interface instead of enumerate 2021-11-01 17:41:39 +01:00
Mattias Wadman
96cc1283cd interp: Eval options in jq instead of calling jq from go 
Simpler and causes less weird performance issues
 2021-11-01 17:05:36 +01:00