1
1
mirror of https://github.com/wader/fq.git synced 2024-11-29 23:27:12 +03:00
Commit Graph

98 Commits

Author SHA1 Message Date
bump
2c38cf1324 Update github-golangci-lint to 1.60.1 from 1.59.1
Release notes https://github.com/golangci/golangci-lint/releases/tag/v1.60.1

Require go 1.21 and fix various lint warnings and use new features
2024-08-14 20:05:40 +02:00
Mattias Wadman
80bccc91ed opus,vorbis: More sym snake_case 2024-03-27 18:30:27 +01:00
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
Mattias Wadman
1d14ea51cd matroska: Decode ebml date type
Also some refactor of shared date decode functions to allow unit
2023-10-17 11:10:48 +02:00
Mattias Wadman
23ae4d978b decode,interp: Make synthetic values more visible and not act as decode values 2023-10-07 17:40:16 +02:00
Mattias Wadman
c07673a07a matroska: Update spec and regenerate 2023-08-09 13:59:21 +02:00
Mattias Wadman
2c505feec8 help,markdown: Rewrote and made text rendering nicer
Plan is to use it for man page etc and also some ansi renderer
2023-05-08 00:07:56 +02:00
Mattias Wadman
a4a332bf4e formats: Clenaup naming a bit 2023-05-01 13:19:04 +02:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
8941b13932 matroska: file_data: Fallback to raw if probe fails 2023-04-20 20:08:40 +02:00
Mattias Wadman
2a2b64dda2 matroska: Update ebml specification
Also add license note about ebml_matroska.xml
2023-04-13 12:19:41 +02:00
Mattias Wadman
0b6ef2a9d8 golangci-lint: Disable revive unused-parameter and update for new default config
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
2023-03-18 22:56:58 +01:00
Mattias Wadman
980ecdba82 decode: Add float 80 reader
Rename *d.Bits to UintBits as it return a uint
Add *d.Bits that return []byte
2023-03-10 01:15:55 +01:00
Mattias Wadman
dec433fc53 help,markdown: Fix double line breaks when converting to text
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
9aaf2ddf27 matroska: Add unknown size test and add description to ebml header 2023-02-10 10:29:25 +01:00
Mattias Wadman
c890a2899f matroska: Update spec and make refs in descriptions look nicer 2023-02-09 23:18:17 +01:00
Mattias Wadman
a8d0bf4d3e matroska: Assume master with unknown size has ended if a valid parent is found
Major refactor of ebml code generator, now decoupled from fq code
Cleanup element descriptions a bit, a bit shorter and less clutter
Cleanup old comments
2023-02-09 20:10:47 +01:00
Mattias Wadman
0d14d7b40f matroska: Handle unknown size for non-master types a bit better
For non-master types assume size to be rest of file
Still does not follow the ebml spec
2023-02-08 12:14:40 +01:00
Mattias Wadman
b60aceca9e matroska: Add decode_samples option
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
2023-02-08 11:31:41 +01:00
Mattias Wadman
b04a650bae flac_picture,mpeg: Fix trailing ")" typo in map sym and description 2023-01-25 14:01:34 +01:00
Mattias Wadman
a85da29544 decode: Make FieldFormat usage more consistent
Will make it easier to refactor later on
2022-12-19 19:12:57 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
646f32d563 matroska: Fix path tests and make _tree_path more robust 2022-11-20 20:16:41 +01:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
e748079e87 matroska: Update spec and regenerate
The new type "TargetType" used in labels is bit confusing, should be handled differently?
2022-10-08 15:45:46 +02:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
7749e1b5b9 matroska: Add proper lacing support 2022-08-28 21:03:26 +02:00
Mattias Wadman
cf15661e9c mp3_frame: Add LSF support and fix incorrect main data handling
Some mpeg versions use LSF (low sampling frequency) when encoding
low sample rates which changes a bit how the side info is layed out.

Also skip trying to separate data and other_data as after reading up
a bit on how main data begin and bit reservoir works i think it was done
incorrectly. To know the real audio data length i think you need to decode
the huffman tables. Instead just have a "audio_data" field which are the
bits with audo data for current and possibly following frames.
2022-08-17 21:36:39 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
2e407386ae matroska: Strip newlines in description
Will mess up dump formatting
2022-08-12 14:46:35 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
2022-08-09 18:09:41 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
c92f4f13c1 matroska: Update ebml_matroska.xml and allow unknown ids
Update spec from https://raw.githubusercontent.com/ietf-wg-cellar/matroska-specification/master/ebml_matroska.xml
Allow unknown ids, decodeas "unknown" type with just binary data
Change "type" field to just a string, there is no type id number.
Make all type field symbols consistently lowercase
2022-06-06 21:43:42 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage
Make cwd for a test script the directory where the script is.
Use relative paths
2022-05-21 20:03:25 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
2022-05-20 12:38:43 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots
Hopefully makes it a bit easier to read and spot
2022-05-11 00:00:39 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
21ad628add interp: dump: Show field name for compound values in arrays
mp3: Remove one level of struct for granales
dump: Makes it a bit easier to understand array of arrays/structs.
avro_ocf: Rename ocf.go -> avro_ocr.go to have same name of source file as format
pcap,pcapng: Rename tcp flow struct to tcp_connection, makes more sense i think
mp4: Add ISOBMFF to format description
2022-04-21 17:47:15 +02:00
Mattias Wadman
f8d79a5777 matroska: More sym cleanup 2022-04-06 19:48:26 +02:00
Mattias Wadman
5ff67e4cfe formats: Sym and field name cleanup to be more jq friendly 2022-04-05 13:59:57 +02:00
Mattias Wadman
c0202483b9 hevc_vpc,hevc_sps: Use same nameing for profile as in spec 2022-03-25 09:48:28 +01:00
Mattias Wadman
74978c9d08 hevc: Add hevc_vps, hevc_sps and hevc_pps decoders
Some extensions and other things still missing
2022-03-24 18:56:56 +01:00