1
1
mirror of https://github.com/wader/fq.git synced 2024-11-30 09:58:13 +03:00
Commit Graph

146 Commits

Author SHA1 Message Date
Mattias Wadman
dec433fc53 help,markdown: Fix double line breaks when converting to text
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
a1bb630a2a doc,fq: Improve cli help and some cleanup 2023-02-15 11:04:39 +01:00
Mattias Wadman
b60aceca9e matroska: Add decode_samples option
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
2023-02-08 11:31:41 +01:00
Mattias Wadman
a2cdb3d6c9
Merge pull request #558 from wader/mp4-udta-with-no-length
mp4: udta: Handle box with value rest of box
2023-01-25 16:34:58 +01:00
Mattias Wadman
6340365841 mp4: udta: Handle box with value rest of box
Try distinguish by probing length field. Should probably be improved, what does ffmpeg do?

Regression from c3e3b3e90d #553
2023-01-25 16:24:28 +01:00
Mattias Wadman
b04a650bae flac_picture,mpeg: Fix trailing ")" typo in map sym and description 2023-01-25 14:01:34 +01:00
Mattias Wadman
d645e71008
Merge pull request #554 from wader/mp4-ftyp-qt-minor-desc
mp4: Decode qt minor verison as YYYY.MM description
2023-01-16 12:23:42 +01:00
Mattias Wadman
f386a5158e mp4: Decode qt minor verison as YYYY.MM description 2023-01-16 12:13:19 +01:00
Mattias Wadman
c3e3b3e90d mp4: Decode udta metadata boxes without meta box 2023-01-16 12:03:53 +01:00
Mattias Wadman
c2795a7c2f
Merge pull request #552 from wader/mp4-tkhd-flags
mp4: Decode tkhd flags
2023-01-13 18:02:53 +01:00
Mattias Wadman
3555dc6721 mp4: Decode tkhd flags 2023-01-13 17:49:03 +01:00
Mattias Wadman
c49012dbca mp4: sgpd,sbgp: Change grouping_type to a string
Is defined as int32 but is a string in practive it seems
2023-01-05 19:18:48 +01:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
a85da29544 decode: Make FieldFormat usage more consistent
Will make it easier to refactor later on
2022-12-19 19:12:57 +01:00
Mattias Wadman
1dea40e6af mp4,doc: Add JSON box tree example and reorder a bit 2022-12-15 14:15:31 +01:00
Mattias Wadman
7b60b24ab7 mp4: Add pdin box support 2022-12-15 11:42:05 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
b1b3b63d27 mp4: Add namespace to mdta ilst boxes 2022-12-14 10:49:02 +01:00
Mattias Wadman
9ac453a17d mp4: Fix typo in sample flags sample_is_depended_on description 2022-12-09 17:22:17 +01:00
Mattias Wadman
5fb81a148b mp4: Add sym and description for traf sample flags 2022-12-09 16:22:43 +01:00
Mattias Wadman
1d6ce2c07a mp4: Decode hint and hnti child boxes 2022-12-08 20:47:33 +01:00
Mattias Wadman
83ccedc506 mp4,decode: Properly decode ilst items (both mdta and mdir)
Refactor mp4 decoder to be simpler and have fallback for unknown box type
Cleanup some old ilst hacks
Add generic string reader to decode API that takes an encoding parameters
2022-12-08 15:56:22 +01:00
Mattias Wadman
a23fe618b0 mp4: sidx version 1 segment_duration is s64 2022-12-05 17:04:05 +01:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
ef2d5232ba mp4: Add png mapping
Also support samples decode for tracks without hdlr box
2022-11-25 19:49:18 +01:00
Mattias Wadman
d75748d816 mp4: Decode more sample flags 2022-11-23 14:29:13 +01:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
cb3dc80252 decode,tar: Add scalar description and Try* helpers
Rework time helpers to use new functions
Fix panic in tar decoder where sym value might be missing

Still not very happy about the API but it's getting better.
2022-10-04 17:18:51 +02:00
Mattias Wadman
20a15372ba mp4: Fix data_format typo 2022-10-02 23:31:38 +02:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
2655ba09c5 mp4: More codec names (from mp4ra.org) 2022-09-12 11:25:43 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
7749e1b5b9 matroska: Add proper lacing support 2022-08-28 21:03:26 +02:00
Mattias Wadman
8fc43533a9 prores_frame: Add basic container and frame header decoder 2022-08-24 19:33:35 +02:00
Mattias Wadman
cf15661e9c mp3_frame: Add LSF support and fix incorrect main data handling
Some mpeg versions use LSF (low sampling frequency) when encoding
low sample rates which changes a bit how the side info is layed out.

Also skip trying to separate data and other_data as after reading up
a bit on how main data begin and bit reservoir works i think it was done
incorrectly. To know the real audio data length i think you need to decode
the huffman tables. Instead just have a "audio_data" field which are the
bits with audo data for current and possibly following frames.
2022-08-17 21:36:39 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
2022-08-09 18:09:41 +02:00
Mattias Wadman
545dac8c86 test: Update tests, go 1.19 uses \xff instead of \u00ff 2022-08-05 00:37:28 +02:00
Mattias Wadman
840292ba6e decode: Simplify compound range sort behaviour
Doing it thru a propery in the decode fn feels a bit hidden and will
also not get set on failed decoding.

Now array is not range sorted, logic is you care about index number and ordering.
Struct is range sorted as you will prefer to fields by name.
2022-08-01 17:17:54 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
cec2dede92
Merge pull request #330 from wader/mp4-dont-sort-samples
mp4: Don't range sort samples, keep sample table order
2022-07-22 16:46:38 +02:00
Mattias Wadman
c6d0d89c06 mp4: Don't range sort samples, keep sample table order 2022-07-22 16:35:00 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
ea2cc3c2ec mp4: Don't decode encrypted samples 2022-07-07 19:02:10 +02:00
Mattias Wadman
64f3e5c717 fairplay: Add basic SPC decoder and PSSH system id 2022-07-07 17:11:55 +02:00