1
1
mirror of https://github.com/wader/fq.git synced 2024-11-27 14:14:58 +03:00
Commit Graph

82 Commits

Author SHA1 Message Date
Mattias Wadman
a4a332bf4e formats: Clenaup naming a bit 2023-05-01 13:19:04 +02:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
b04a650bae flac_picture,mpeg: Fix trailing ")" typo in map sym and description 2023-01-25 14:01:34 +01:00
Mattias Wadman
a85da29544 decode: Make FieldFormat usage more consistent
Will make it easier to refactor later on
2022-12-19 19:12:57 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
fb2a2b949c flac,fuzz: Fatal error on negative partition sample count 2022-10-18 13:40:48 +02:00
Mattias Wadman
7859be1e5d flac_frame: Properly decode zero escape sample size 2022-10-11 17:05:19 +02:00
Mattias Wadman
bafd1f564e decode,fuzz: Signed integer (S) read require at least one bit
Otherwise there is no sign bit and it will panic on shifting.

Maybe should require 2? now -0 and +1 hmm
2022-10-11 12:12:28 +02:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Martijn van Beurden
fb583e2cc0 Add 32 bps FLAC to test 2022-08-18 14:27:43 +02:00
Martijn van Beurden
c1d5b2b1c3 Add sample size entry to list for 32bps flac streams 2022-08-18 14:16:55 +02:00
Mattias Wadman
da6b9413fd
Merge pull request #381 from wader/flac-fix-gen-script
flac: Make gen script generate correct fqtest files
2022-08-18 11:48:00 +02:00
Mattias Wadman
8a19978b2f flac: Make gen script generate correct fqtest files 2022-08-18 11:37:07 +02:00
Mattias Wadman
2bfbe9a94a flac_frame: Cleanup some dev lefterovers and todos 2022-08-18 11:13:45 +02:00
Mattias Wadman
b6c4c27688
Merge pull request #379 from ktmf01/fix-flac-raw-partition
Fix decoding of FLAC raw entropy partition
2022-08-18 11:04:00 +02:00
Martijn van Beurden
3f209c46ca Fix decoding of FLAC raw entropy partition 2022-08-18 10:11:57 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
2022-08-09 18:09:41 +02:00
Mattias Wadman
78aa96b0ac dev: Cleanup some code to fix a bunch of new linter warnings 2022-07-19 18:56:09 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
a6a97136b7 decode: Cleanup Try<f>/<f> pairs
Decode API design is that Try* returns error non-Try panics.
Also rename Must* as they should panic and introduce some new Try* functions
that were missing.
2022-06-30 12:13:46 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage
Make cwd for a test script the directory where the script is.
Use relative paths
2022-05-21 20:03:25 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
2022-05-20 12:38:43 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots
Hopefully makes it a bit easier to read and spot
2022-05-11 00:00:39 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming
Should replace most of this with generics at some point
2022-05-07 12:46:34 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
b35b1804be decode,format: Add d.FieldFormatOrRaw(Len) 2022-04-11 23:01:45 +02:00
Mattias Wadman
5ff67e4cfe formats: Sym and field name cleanup to be more jq friendly 2022-04-05 13:59:57 +02:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
2022-02-04 21:41:53 +01:00
Mattias Wadman
fc0aacb654 interp: Cleanup display aliases, now: d, da, dd, dv, ddv
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
2022-01-28 18:25:38 +01:00
Mattias Wadman
263f1aeb66 flac: Don't allow zero subframe sample size
Spec says sample size can be 4-32 but that is probably before correcting for waste bits
2022-01-18 11:58:57 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra
Think it makes more sense
2022-01-15 19:00:42 +01:00
Mattias Wadman
bfc977b810 png: Decode PLTE and tRNS chunks and cleanup syms a bit 2022-01-07 14:31:18 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
2022-01-05 22:05:09 +01:00
Mattias Wadman
6ed2e2e72e interp: dump: Indicate arrays using jq-syntax
Related to #16
2021-12-08 16:24:28 +01:00
Mattias Wadman
c2131bb4a0 flac: Cleanup scalar usage and fix incorrect sample rates
Also split DisplayFormat into actual and sym
2021-12-08 00:39:52 +01:00
Mattias Wadman
2fc0a71a47 decode: Refactor scalar usage
Move scalar into own package.
Split scalar code into decode related scalar code (that reads etc) and
scalar code that just transform the scalar value.
Use a scalar.Mapper interface instead of just a function.
Make mappers, assert and validat impement the interface.
2021-12-02 17:39:26 +01:00
Mattias Wadman
f55b1af6ac inet: Add tcp and ipv4 reassembly
Also add tcp_stream and udp_payload to decode content
2021-11-29 18:42:18 +01:00
Mattias Wadman
4f0bf9228f flac: Make md5_calculated be a buffer 2021-11-24 12:31:42 +01:00
Mattias Wadman
4f8d03731d format,decode: Some crc and endian refactor 2021-11-24 12:01:01 +01:00
Mattias Wadman
fc76907c07 png: Add proper color type 2021-11-23 17:04:21 +01:00
Mattias Wadman
2af08da9f7 flac: Fix block_size regression 2021-11-23 16:59:50 +01:00
Mattias Wadman
d1b514edeb format: Some claeanup 2021-11-19 16:44:06 +01:00
Mattias Wadman
7af191d974 decode: Move io helper into *D 2021-11-18 01:17:41 +01:00
Mattias Wadman
6fba1a8125 decode: Fix bitbuf root handling a bit 2021-11-18 01:17:15 +01:00
Mattias Wadman
986d5ecc50 decode: Move registry package to decode/registry and add a format group type 2021-11-17 16:59:56 +01:00