1
1
mirror of https://github.com/wader/fq.git synced 2024-12-25 06:12:30 +03:00
Commit Graph

453 Commits

Author SHA1 Message Date
Mattias Wadman
ff29fe7154
Merge pull request #377 from wader/force-unicode
interp: Add --unicode-output/-U to force use of unicode
2022-08-17 21:49:40 +02:00
Mattias Wadman
c7559b59a0 interp: Add --unicode-output/-U to force use of unicode
Currently only used to make dump output a bit nicer.
2022-08-17 21:40:42 +02:00
Mattias Wadman
cf15661e9c mp3_frame: Add LSF support and fix incorrect main data handling
Some mpeg versions use LSF (low sampling frequency) when encoding
low sample rates which changes a bit how the side info is layed out.

Also skip trying to separate data and other_data as after reading up
a bit on how main data begin and bit reservoir works i think it was done
incorrectly. To know the real audio data length i think you need to decode
the huffman tables. Instead just have a "audio_data" field which are the
bits with audo data for current and possibly following frames.
2022-08-17 21:36:39 +02:00
Mattias Wadman
701c67c136 jsonl: Add decoder, also tojsonl encoder 2022-08-15 19:43:59 +02:00
Mattias Wadman
0d44b9376c tar: Some number fields can be empty 2022-08-15 14:57:16 +02:00
Mattias Wadman
d898732c3d gojq: Update fq fork, new scope function, rawstring, stricter integers
Update repl completion for new scope/0 function, now reports name/arity and is more correct
Support for experimental `rawstring` literals, similar to go raw string literals
Sticter number literals, now 0b,0o and 0x fail it they have invalud symbols, ex: 0b123
2022-08-14 22:40:44 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
2022-08-09 18:09:41 +02:00
Mattias Wadman
502f451cc3 decode: Refactor to use scalar type assert helper 2022-08-08 18:09:49 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
5d9ffead52 decode,scalar: Map empty string also else sym might ends up nil
Later mappers should be able to rely on sym being a value of a certain type.
In this crash case tar.go  had this:
d.FieldUTF8NullFixedLen("mtime", 12, scalar.SymUParseUint(8), scalar.DescriptionSymUUnixTime)
If mtime was empty string SymUParseUint was not even called so did not bail out running
all mappers.
2022-08-04 12:23:31 +02:00
Mattias Wadman
725c8e83ab macho: Split into macho/macho_fat, fix offset issue and add string decoding
Split fat macho into own decoder macho_fat. This also fixes issue with section
offset etc not being correct as they are from the start of each embedded file.

Make all address and offset field be in hex.

Decode __cstring, __ustring and __cfstring sections.

Fix LC_ENCRYPTION_INFO_64 missing pading issue.

Skip ranging for __bss and __common as they dont have any data in the file.

Simplifed magic handling a bit and add symbols.

Simplified state struct field, had redudant struct.
2022-08-02 14:36:53 +02:00
Mattias Wadman
888a471245
Merge pull request #350 from wader/bump-github-golangci-lint-1.47.3
Update github-golangci-lint from 1.47.2 to 1.47.3
2022-08-01 18:54:35 +02:00
Mattias Wadman
bc6cffde61 lint,decode,fuzz:: Fix nilerr warnings, one real one should be ignored for now 2022-08-01 18:47:24 +02:00
Mattias Wadman
840292ba6e decode: Simplify compound range sort behaviour
Doing it thru a propery in the decode fn feels a bit hidden and will
also not get set on failed decoding.

Now array is not range sorted, logic is you care about index number and ordering.
Struct is range sorted as you will prefer to fields by name.
2022-08-01 17:17:54 +02:00
Mattias Wadman
ed4247836c doc,interp: Update and add more examples 2022-07-30 20:21:35 +02:00
Mattias Wadman
4eae7ffd99 interp,doc: Add -R raw string slurp hint to -s help 2022-07-29 16:37:42 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
fa20c74c22 xml,csv,interp: Handle JQValue when string normalizing
Otherwise nested array/object JQValue:s could end up being JSON marshalled
2022-07-26 17:43:59 +02:00
Mattias Wadman
af35b2848d gojq: Preserve keys order for to_entries when used with JQValue
Feels like what one would expect when doing to_entries on a decode value struct etc,
that is get the pairs in same order as decoder wants.

Quite sure this used to the case but was change to sorted key order when
to_entries was move from bein implemented in jq to be internal for performance
reasons. Now internal gojq to_entries has a special case for JQValue objects.

Also use new gojq.JQType* constants for type strings and switch binary type
to look more like strings. Can still use _exttype to get "binary".
2022-07-26 15:55:58 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
8bb4a6d27c decode: Range decode with new decoder to preserve bit reader
Fix issue with modified decoder on panic causing other panics as the ranges gets messed up
2022-07-22 19:11:56 +02:00
Mattias Wadman
78aa96b0ac dev: Cleanup some code to fix a bunch of new linter warnings 2022-07-19 18:56:09 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
f5be5180f0 interp: Cleanup and clarify some format naming 2022-07-16 22:04:21 +02:00
Mattias Wadman
9b2e474ea8 gojqextra: Simplify function type helpers
No need for env to be a function
2022-07-16 21:55:47 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
64f3e5c717 fairplay: Add basic SPC decoder and PSSH system id 2022-07-07 17:11:55 +02:00
Mattias Wadman
ab8c728aaa interp: Rename --decode-file to --argdecode to be more consistent
There is still an alias for --decode-file
2022-07-04 12:28:37 +02:00
Mattias Wadman
417255b72b bitcoin: Add blkdat, block, transcation and script decoder 2022-07-04 09:59:51 +02:00
Mattias Wadman
d4fe00d65c
Merge pull request #299 from wader/interp-add-group
interp: Add group/0
2022-06-30 12:31:37 +02:00
Mattias Wadman
a6a97136b7 decode: Cleanup Try<f>/<f> pairs
Decode API design is that Try* returns error non-Try panics.
Also rename Must* as they should panic and introduce some new Try* functions
that were missing.
2022-06-30 12:13:46 +02:00
Mattias Wadman
a3c33fc1f4 interp: Add group/0
Also add tests for count, count_by, delta and delta_by
2022-06-21 15:18:12 +02:00
Mattias Wadman
c7701851b0 interp: Extract to/from map/struct to own package 2022-06-20 21:24:59 +02:00
Mattias Wadman
32361deefe gojqextra: Cleanup gojq type cast code 2022-06-20 18:02:35 +02:00
Mattias Wadman
81a014ce49 interp: Make empty _finally fin error on error
Ex: _finally(error(a); empty)
This is not used by fq atm
2022-06-18 00:00:54 +02:00
Mattias Wadman
0863374f8c doc: Correct bencode spec URL
Also rename _tojq value_sep to object_sep and fix formats_diagram.jq to use tomd5
2022-06-13 18:49:34 +02:00
Mattias Wadman
a64c28d927 dump: Skip JQValueEx if there are not options
I used to tojson and would crash in binary values are there were
not optins deciding how to format the strinh. Now just treat as utf8 string.
2022-06-07 16:39:45 +02:00
Mattias Wadman
c1a0cda5a8 gojq: Update rebased fq fork
Fix tostring for non-string JQValue:s, is now JSON
Update EOF error
2022-06-01 17:27:45 +02:00
Mattias Wadman
6b0880002d interp: Cast jq value to go value properly for encoding functions
Some encoding fuctions accept binary used string as input type, should be any.
Add cast helper functions, hopefully can be useful in future for even nicer
function bindings api.
2022-05-30 18:27:12 +02:00
Mattias Wadman
735c443b37 interp: Improve type normalization and use it for toyaml and totoml
Use smallest int type for int64, uint6 and *big.Int

Fixes integer serialization for yaml and toml for small integers, othweise
they will end up as strings.
2022-05-28 20:33:37 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
7b27e506f1 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.

Add test with playread and widevine pssh boxes

Related to #282
2022-05-28 13:42:38 +02:00
Mattias Wadman
8021492197 interp: help: Fix incorrect options example 2022-05-26 12:52:51 +02:00
Mattias Wadman
9a7ce14886 gojq: Update rebased fq fork
Use new gojq.TypeOf and gojq.Preview
2022-05-23 21:33:12 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage
Make cwd for a test script the directory where the script is.
Use relative paths
2022-05-21 20:03:25 +02:00
Mattias Wadman
95e619659b interp: Add internal _is_<type> helpers 2022-05-20 16:49:00 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
2022-05-20 12:38:43 +02:00
Mattias Wadman
8dde3ef5b1 interp: Fix crash when including relatve path when no search paths are set 2022-05-17 18:05:10 +02:00
Mattias Wadman
469116f3b3
Merge pull request #260 from wader/mono-unicode-fix
interp: dump: Fix column truncate issue with unicode bars
2022-05-11 00:44:50 +02:00
Mattias Wadman
dff3cc1199 interp: dump: Fix column truncate issue with unicode bars 2022-05-11 00:37:52 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots
Hopefully makes it a bit easier to read and spot
2022-05-11 00:00:39 +02:00
Mattias Wadman
76161a1b99 scalar,mp4,gzip,tar: Add timestamp to description
Scalar api for this needs some work
2022-05-07 13:15:13 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming
Should replace most of this with generics at some point
2022-05-07 12:46:34 +02:00
Mattias Wadman
c4dd518e04 decode: Make compound range sort optional
Some formats might want to control child order
mp4: Keep tracks in track id order
dns: Keep label component order
elf: Keep seciton order
macho: Keep command and section order
2022-05-03 16:16:09 +02:00
Mattias Wadman
3ce660a2cf decode: Keep decode tree on RangeFn error 2022-05-03 10:57:43 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
21ad628add interp: dump: Show field name for compound values in arrays
mp3: Remove one level of struct for granales
dump: Makes it a bit easier to understand array of arrays/structs.
avro_ocf: Rename ocf.go -> avro_ocr.go to have same name of source file as format
pcap,pcapng: Rename tcp flow struct to tcp_connection, makes more sense i think
mp4: Add ISOBMFF to format description
2022-04-21 17:47:15 +02:00
Mattias Wadman
afd724bf9c gojq: Rebase fq fork. Fixes JQValue path tracking when iterating
Add some decode value path and assign tests
2022-04-19 19:00:37 +02:00
Mattias Wadman
6b4dbbf415
Merge pull request #234 from wader/field-truncate2
interp: Don't truncate last display column
2022-04-19 18:30:17 +02:00
Mattias Wadman
607202bb67 interp: Don't truncate last display column
Wrapping lines might be annoying but not seeing things even worse i think.
Hopefully once column config is in jq it could be an option.
2022-04-19 18:19:42 +02:00
Mattias Wadman
087d12419e interp: Simpler and more efficient hexdump
Should not need to copy bits, just use existing bitreader
2022-04-19 18:12:58 +02:00
Mattias Wadman
dabad85080 interp: Proper display column truncate
Also speed up by using less string allocs
2022-04-16 18:48:21 +02:00
Mattias Wadman
b35b1804be decode,format: Add d.FieldFormatOrRaw(Len) 2022-04-11 23:01:45 +02:00
Mattias Wadman
518f6af4a8 gojq: Rebase fq fork
Upstream changes:
Many performance improvements
Error message improvments
Lots of refactoring

Most of the JQValue interface changes in gojq fork had to be reworked but
resultet in a much nicer and cleaner changeset.

fq changes:
Assignment to JQValue (like a decode value) now shallowly converts the value into
a jq value before assigning. Was a bit hacky as it was and this makes JQValue
behave more like real jq values. This also fixes some advanced indexing issues.

Actual custom path updates will be something for the future.
2022-04-11 19:23:15 +02:00
Mattias Wadman
6f03471d15 interp: Paths with a array as root was missing start dot 2022-04-08 16:47:40 +02:00
Mattias Wadman
f8fd6b7f48 mp4: Add more HEIF boxes
Add scalar.*ToDesceription mapper and refactor to use it
2022-04-07 19:11:49 +02:00
Mattias Wadman
5ff67e4cfe formats: Sym and field name cleanup to be more jq friendly 2022-04-05 13:59:57 +02:00
Mattias Wadman
e8dc7112b6 ipv6,icmpv6: Add decoder
Refactor to use groups between network layers to make them less coupled and reusable:
link_frame (ethernet etc)
inet_packet (ipv4,ipv6 etc)
ip_packet (tcp packet (not stream), udp segment etc)

Rename data to payload as i think it makes more sense for network data
2022-04-03 17:43:51 +02:00
Mattias Wadman
d8aaf30345 rtmp,amf0: Add decoders
Basic RTMP support, used via tcp_stream group (pcap etc) or manually.
Basic AMF0 support, mostly what is used in RTMP.
2022-04-01 13:07:20 +02:00
Mattias Wadman
e8678ca88e interp: Remove opts refactor leftover 2022-03-28 15:37:38 +02:00
Mattias Wadman
74978c9d08 hevc: Add hevc_vps, hevc_sps and hevc_pps decoders
Some extensions and other things still missing
2022-03-24 18:56:56 +01:00
Mattias Wadman
d376520f11 interp: Remove to*range pad argument and fix stdout padding issue
Padding could end up double.
Remove to*range($pad) as it probably just confusing to be able
to pad an existing range, what to show in hexdump etc? zero bits
that do not actuall exist at that range?

Add tests and binary stdout support to tests
2022-03-17 20:15:35 +01:00
Mattias Wadman
9cb4b57a45 interp,cli: Handle ctrl-c properly
Should supress cancel error message
Add CLI ctlr-c/ctrl-d tests and make them more robust
2022-03-15 22:38:12 +01:00
Mattias Wadman
41551de331 interp,repl: Improved eval and output interrupt
repl now rewrites query to do implicit display inside the sub eval.
This makes it possible to interrupt eval and output in a better and faster way.

Make JSON encoder fail early on errors.

Add more interrupt tests.
2022-03-11 16:22:30 +01:00
Mattias Wadman
6f10745aa5 interp: Fix interrupt regression after query rewrite refactor
Also add test for it
2022-03-11 12:41:49 +01:00
Mattias Wadman
dff7e7dadb interp: Cleanup binary regexp overloading and add explode
Add explode for binary that explode into an array of the binary unit tobits
Remove scan_toend as it feels a bit weird and one can use tobytes | .[match(...):] instead
Some general cleanup also
2022-03-09 20:17:12 +01:00
Mattias Wadman
56ae4a0c70 test: Make expect cli test more robust
Wait for prompt before sending ^D. Seems to fix issue with expect on ubuntu focal.

Thanks @KristianKarl for report
2022-03-07 16:22:41 +01:00
Mattias Wadman
eeb59152a6 interp: Make help output less wide 2022-03-03 22:02:33 +01:00
Mattias Wadman
9cb4205b1d repl: Correct error arrow position in color mode 2022-03-03 21:24:03 +01:00
Mattias Wadman
3f50bb90a6 interp: Rework formats and options help
Replace --formats with --help formats
Add --help options
2022-03-03 18:25:20 +01:00
Mattias Wadman
33efb02af0 interp,repl: Add path and value colors to prompt 2022-03-03 18:25:19 +01:00
Mattias Wadman
fe8183b500 interp: Color parse in jq 2022-03-03 17:55:24 +01:00
Mattias Wadman
be6f00935f gojq: Rebase fq fork and add support for non-10 base for tonumber
tonumber now works with non-10 number literals
Fix try precedences changes
Fix .[]? error handling changes
2022-03-02 23:46:20 +01:00
Mattias Wadman
0a043f9096 repl,interp: Refactor repl and slurp
Now repl, slurp and help implemented using same query rewrite.
Include filename context in error if possible.
Add spew function that does opposite of slurp.
Start of help infra, not done or documented yet.
Show error pointer on parse error.
Rename internal eval to _eval and make eval be wrapper that
does rewrite and has various eror handling etc.
Nicer repl, slupr and help errors.
2022-03-01 18:31:00 +01:00
Mattias Wadman
e0f493fa0b
Merge pull request #51 from Akaame/43_macho_support
[WIP] 43: Initial impl for macho support.
2022-02-28 10:22:50 +01:00
Mattias Wadman
f66f115e11 interp: Make _finally handle null, call fin once and last
null value from f did not call fin.
fin was not always called last
Now fin is onyl called once

Might be cause of some weird repl bugs.
2022-02-24 18:39:03 +01:00
Mattias Wadman
03f450f82e interp: Skip redundant includes
Speeds up interp quite a bit.
Also good as i've start to use more includes to make code easier to follow
where thigns come from and also makes jq-lsp happier.

Also rename evalContext to evalInstace to make it less confused with context.
2022-02-24 12:44:26 +01:00
siddik.acil
2021b05408 macho: make actual 2022-02-21 20:32:17 +01:00
siddik.acil
b78ed02fb3 macho: barebones decoder impl 2022-02-21 20:31:12 +01:00
Mattias Wadman
0bc1171914 interp: Move opts eval to options.jq
Also made _main a bit nicer
2022-02-20 21:48:01 +01:00
Mattias Wadman
ca8cdadbe3 repl: Add comments and query from/to helper 2022-02-18 20:59:21 +01:00
Mattias Wadman
e238f292af repl: Speedup multi input to sub-repl
Rewrite non-sub-repl queries as "q" -> ".[] | q" and pass in inputs as array.
Before each input did an eval.

Also fixup error message a bit, skip line:column when they dont make sense.
2022-02-18 01:13:45 +01:00
Mattias Wadman
cd2cbef6fe decode: Some cleanup
Fix PeekFind arg order
2022-02-17 00:25:24 +01:00
Mattias Wadman
9e4f264193 dev: Add .jq-lsp.jq to add additional builtins for jq-lsp
Fixes missing function errors for fq builtins
Add includes to fix some other missing function errors

Add COMPLETION_TIMEOUT env to increase comepletion timeout during test using -race
2022-02-16 19:52:42 +01:00
Mattias Wadman
9dc59e5d39 interp: Move _is_decode_value to jq
Make _extype work like type
2022-02-15 22:20:32 +01:00
Xentripetal
9636613ec6 Merge branch 'master' of https://github.com/wader/fq into avro 2022-02-12 22:16:50 -06:00
Mattias Wadman
b86da7aecc doc: Add inital decoder API documentation 2022-02-12 16:13:29 +01:00