1
1
mirror of https://github.com/wader/fq.git synced 2024-12-25 14:23:18 +03:00
Commit Graph

83 Commits

Author SHA1 Message Date
Mattias Wadman
c4dd518e04 decode: Make compound range sort optional
Some formats might want to control child order
mp4: Keep tracks in track id order
dns: Keep label component order
elf: Keep seciton order
macho: Keep command and section order
2022-05-03 16:16:09 +02:00
Mattias Wadman
3ce660a2cf decode: Keep decode tree on RangeFn error 2022-05-03 10:57:43 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
b35b1804be decode,format: Add d.FieldFormatOrRaw(Len) 2022-04-11 23:01:45 +02:00
Mattias Wadman
cd2cbef6fe decode: Some cleanup
Fix PeekFind arg order
2022-02-17 00:25:24 +01:00
Mattias Wadman
06245d1295 binary,decode,doc: Rename buffer to binary and add some documentation
Rename buffer to binary. Still some work left what to call buffer/binary in decode code.
Document decode value and binary type
Fix proper unit padding for tobytes and add still undocumenated extra padding argument.
Add some additional binary tests
2022-02-08 22:20:28 +01:00
Mattias Wadman
be71eb01ac decode: Rename LenFn() to LimitedFn, add FramedFn and document
I think framed makes more sense, but not sure
2022-02-06 13:49:00 +01:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
2022-02-04 21:41:53 +01:00
Mattias Wadman
4ab6381dc4 decode: Add scalars args to FieldRootBitBuf
Also move *Value.TryScalarFn to value.go
2022-02-01 22:24:24 +01:00
Mattias Wadman
898dfec1d1 lint: Fix typeassert and case exhaustive warnings 2022-01-25 17:23:41 +01:00
Mattias Wadman
b8efd8e549 fuzz: Fuzz all formats
Fix string readers to not panic on make slice with bogus sizes
2022-01-24 16:56:24 +01:00
Mattias Wadman
d1943dad49 pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
Also add dev/fuzzbytes.go tool to convert fuzz input to raw bytes
2022-01-23 13:34:56 +01:00
Mattias Wadman
34a09c40a7
Merge pull request #90 from wader/decode-stable-sort
decode: Use stable sort for values to not change order or values with…
2022-01-20 16:34:47 +01:00
Mattias Wadman
9198e18c10 decode: Add FieldArrayRootBitBufFn for completeness
Should rethink these, not very nice api
2022-01-20 16:27:52 +01:00
Mattias Wadman
cf8a50c150 decode: Use stable sort for values to not change order or values with same range start
Some decoders might relay on values added with same start to be kept in same order.
2022-01-20 16:21:50 +01:00
Mattias Wadman
ff5c0b8195 decode: Error on negative number of bits when reading numbers
Maybe should even give error on zero bits? maybe only allow for zero bit for unsigned ints?
2022-01-18 12:01:04 +01:00
Mattias Wadman
548a065388 decode,interp: Finish up nil value support
Update msgpack, bson and cbor to use it.
2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra
Think it makes more sense
2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae decode,interp: Add arbitrary large integer support (BigInt)
Was already handled in fq in various places as gojq uses them

Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
2022-01-15 19:00:42 +01:00
Mattias Wadman
0b0f28e966 cbor: Add decoder
Does not decode sematic tag types
Also fixes broken float16 support

Fixes #71
2022-01-12 17:56:03 +01:00
Mattias Wadman
149cb3f45a interp: Add torepr/0 that converts decode value into what it reptresents
Ex: fq -d msgpack torepr file.msgpack
Willoutput the JSON representation of the msgpack

Make per format *_torepr functions internal
2022-01-12 17:33:58 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
2022-01-05 22:05:09 +01:00
Mattias Wadman
6db7218093 doc: Some more dev infomration 2021-12-31 17:13:16 +01:00
Mattias Wadman
f4f63835dc interp: Add ._index for values in arrays 2021-12-10 18:19:11 +01:00
Mattias Wadman
f3480026bf sll2: fuzz: Limit address length to max 8 bytes 2021-12-09 10:38:34 +01:00
Mattias Wadman
9b683cd839 deocde: Cleanup some io panic(err) 2021-12-07 17:33:57 +01:00
Mattias Wadman
713ffe4770 scalar: Add Require* and Require/Validate/AssertRange* 2021-12-07 15:50:55 +01:00
Mattias Wadman
75aa47556a decode: Generate Try?Scalar* methods for readers too 2021-12-04 19:05:16 +01:00
Mattias Wadman
d48ebc12ee decode: Simplify Compound.Children
Not a array reference anymore but instead pass around Compound reference
2021-12-03 00:06:11 +01:00
Mattias Wadman
2fc0a71a47 decode: Refactor scalar usage
Move scalar into own package.
Split scalar code into decode related scalar code (that reads etc) and
scalar code that just transform the scalar value.
Use a scalar.Mapper interface instead of just a function.
Make mappers, assert and validat impement the interface.
2021-12-02 17:39:26 +01:00
Mattias Wadman
f801cc0af7 decode: Rename s/FieldTryFormat/TryFieldFormat for consistency 2021-11-30 15:29:41 +01:00
Mattias Wadman
d1e1cd98c2 decode: Fix walk root depth issue causing dump to indent incorrectly 2021-11-30 12:51:52 +01:00
Mattias Wadman
f55b1af6ac inet: Add tcp and ipv4 reassembly
Also add tcp_stream and udp_payload to decode content
2021-11-29 18:42:18 +01:00
Mattias Wadman
3bf1a57cfa avc: Cleanup and add color names etc 2021-11-28 17:29:56 +01:00
Mattias Wadman
7b7faaf02b pcap: Add pcap, pcapng, ether8023, ipv4, udp, udp 2021-11-24 18:13:00 +01:00
Mattias Wadman
9d116df799 decode: Rework use of TryFieldReaderRangeFormat
Handle failure better
2021-11-24 15:25:27 +01:00
Mattias Wadman
4f8d03731d format,decode: Some crc and endian refactor 2021-11-24 12:01:01 +01:00
Mattias Wadman
f40320b04c decode: Remove D.Scalar* and add d.(Try)FieldScala*Fn instead
Idea is scalar fn should not read
2021-11-21 13:08:18 +01:00
Mattias Wadman
0480a2f722 decode: Some format decode and sub buffer work 2021-11-20 18:56:59 +01:00
Mattias Wadman
278e909a2f mp4: Add tapt, prof, enof and clap boxes 2021-11-19 15:29:49 +01:00
Mattias Wadman
d838d2f613 zip: Add format decoder 2021-11-19 00:22:26 +01:00
Mattias Wadman
ffb5adff1d registry: Move to pkg/registry, feels better 2021-11-18 01:22:21 +01:00
Mattias Wadman
7af191d974 decode: Move io helper into *D 2021-11-18 01:17:41 +01:00
Mattias Wadman
6fba1a8125 decode: Fix bitbuf root handling a bit 2021-11-18 01:17:15 +01:00
Mattias Wadman
c083a9e475 decode: Fix MapRawToScalar regression 2021-11-17 17:14:19 +01:00
Mattias Wadman
986d5ecc50 decode: Move registry package to decode/registry and add a format group type 2021-11-17 16:59:56 +01:00
Mattias Wadman
5d98a6949a decode: Refactor Error/Fatal into printf functions 2021-11-17 16:26:13 +01:00
Mattias Wadman
ede2e77975 decode: Nicer scalar template and add doc 2021-11-17 16:13:10 +01:00
Mattias Wadman
826c5094f2 decode: Add Generated header for scalar 2021-11-17 10:19:10 +01:00
Mattias Wadman
b66ed32614 decode,interp: Make fuzzing work again and cleanup fatal/error code 2021-11-16 17:11:26 +01:00