1
1
mirror of https://github.com/wader/fq.git synced 2024-12-24 05:41:38 +03:00
Commit Graph

46 Commits

Author SHA1 Message Date
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
Mattias Wadman
23ae4d978b decode,interp: Make synthetic values more visible and not act as decode values 2023-10-07 17:40:16 +02:00
Mattias Wadman
2c505feec8 help,markdown: Rewrote and made text rendering nicer
Plan is to use it for man page etc and also some ansi renderer
2023-05-08 00:07:56 +02:00
Mattias Wadman
a4a332bf4e formats: Clenaup naming a bit 2023-05-01 13:19:04 +02:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
dec433fc53 help,markdown: Fix double line breaks when converting to text
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
2022-08-09 18:09:41 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
5d67df47e2 avro_ocf: Fix panic on missing meta schema 2022-07-23 22:20:03 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
a6a97136b7 decode: Cleanup Try<f>/<f> pairs
Decode API design is that Try* returns error non-Try panics.
Also rename Must* as they should panic and introduce some new Try* functions
that were missing.
2022-06-30 12:13:46 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
2022-05-20 12:38:43 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots
Hopefully makes it a bit easier to read and spot
2022-05-11 00:00:39 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming
Should replace most of this with generics at some point
2022-05-07 12:46:34 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
21ad628add interp: dump: Show field name for compound values in arrays
mp3: Remove one level of struct for granales
dump: Makes it a bit easier to understand array of arrays/structs.
avro_ocf: Rename ocf.go -> avro_ocr.go to have same name of source file as format
pcap,pcapng: Rename tcp flow struct to tcp_connection, makes more sense i think
mp4: Add ISOBMFF to format description
2022-04-21 17:47:15 +02:00
Xentripetal
69c745d3c4 simplify scalar usage 2022-02-12 22:16:46 -06:00
Xentripetal
0909fb6dad Comment on snappy decompression 2022-02-10 08:55:56 -06:00
Xentripetal
21cfc70c38 Dates need to specify UTC too 2022-02-10 08:51:51 -06:00
Xentripetal
0300c955c5 Take heading off doc to match make doc format 2022-02-09 20:59:29 -06:00
Xentripetal
2605bce4e3 Lint and add basic doc 2022-02-09 20:46:12 -06:00
Xentripetal
4508241b1f Cleanup snappy 2022-02-09 20:39:31 -06:00
Xentripetal
6f57cdbfd0 Timestamps should be UTC 2022-02-09 18:16:26 -06:00
Xentripetal
75b8496178 Fix lint 2022-02-07 23:10:25 -06:00
Xentripetal
31c4c0d3c4 Support snappy and deflate codecs 2022-02-07 23:03:50 -06:00
Xentripetal
66ca1f109e Change tests to use new verbose syntax 2022-02-07 17:39:37 -06:00
Xentripetal
5b11385ea4 Pull latest from wader 2022-02-07 17:35:33 -06:00
Xentripetal
6fc1efd902 Add test case with all data types 2022-01-19 23:53:33 -06:00
Xentripetal
ee184075b7 Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing. 2022-01-19 21:00:44 -06:00
Xentripetal
16849c8f35 Update linting 2022-01-19 19:44:42 -06:00
Xentripetal
de64a99e72 cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes 2022-01-19 19:35:50 -06:00
Xentripetal
251053ef3c Initial pass on logical types 2022-01-13 23:32:19 -06:00
Xentripetal
68e85a2ddf Use existing scalar description helper 2021-12-29 17:01:40 -06:00
Xentripetal
07ddf36f1c Cleanup for linting 2021-12-29 16:46:56 -06:00
Xentripetal
4b809a73f6 Change avro codec to funcs 2021-12-29 16:12:36 -06:00
Xentripetal
7345b8c7c2 Cleanup 2021-12-28 13:09:54 -06:00
Xentripetal
7a8e3ca26e Hook into registry, add codecs 2021-12-28 13:05:10 -06:00
Xentripetal
d6ca48182e initial work for avro OCF files 2021-12-28 13:01:53 -06:00