1
1
mirror of https://github.com/wader/fq.git synced 2024-11-26 10:33:53 +03:00
Commit Graph

107 Commits

Author SHA1 Message Date
David McDonald
bcccde2358 Fixes and embeds documentation 2022-09-24 12:57:11 -05:00
David McDonald
5f61994087 adds function for decoding fixed sized arrays 2022-09-23 20:50:41 -05:00
David McDonald
d784db69bc Adds support for Apple Binary Plist, version 00 2022-09-23 20:21:32 -05:00
Mattias Wadman
a6429ffe7a decode: Remove RangeSorted flag as we can decide on array/struct instead 2022-09-01 17:45:28 +02:00
Mattias Wadman
768df3012c interp,decode: For struct use map to lookup field
Will make it faster for struct with logs of fields and seems to
not cuase any significant difference for small structs.

All this really needs a rewrite somehow, maybe refactor into interfaces somehow? getting messy.
2022-09-01 11:35:19 +02:00
Pavel Safonov
7cd43b4919 perfomance: increase performance by map usage 2022-09-01 09:36:50 +03:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
326dada7ca decode: Add LEB128 readers 2022-08-20 12:05:44 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
502f451cc3 decode: Refactor to use scalar type assert helper 2022-08-08 18:09:49 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
725c8e83ab macho: Split into macho/macho_fat, fix offset issue and add string decoding
Split fat macho into own decoder macho_fat. This also fixes issue with section
offset etc not being correct as they are from the start of each embedded file.

Make all address and offset field be in hex.

Decode __cstring, __ustring and __cfstring sections.

Fix LC_ENCRYPTION_INFO_64 missing pading issue.

Skip ranging for __bss and __common as they dont have any data in the file.

Simplifed magic handling a bit and add symbols.

Simplified state struct field, had redudant struct.
2022-08-02 14:36:53 +02:00
Mattias Wadman
840292ba6e decode: Simplify compound range sort behaviour
Doing it thru a propery in the decode fn feels a bit hidden and will
also not get set on failed decoding.

Now array is not range sorted, logic is you care about index number and ordering.
Struct is range sorted as you will prefer to fields by name.
2022-08-01 17:17:54 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
8bb4a6d27c decode: Range decode with new decoder to preserve bit reader
Fix issue with modified decoder on panic causing other panics as the ranges gets messed up
2022-07-22 19:11:56 +02:00
Mattias Wadman
78aa96b0ac dev: Cleanup some code to fix a bunch of new linter warnings 2022-07-19 18:56:09 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
417255b72b bitcoin: Add blkdat, block, transcation and script decoder 2022-07-04 09:59:51 +02:00
Mattias Wadman
a6a97136b7 decode: Cleanup Try<f>/<f> pairs
Decode API design is that Try* returns error non-Try panics.
Also rename Must* as they should panic and introduce some new Try* functions
that were missing.
2022-06-30 12:13:46 +02:00
Mattias Wadman
7b27e506f1 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.

Add test with playread and widevine pssh boxes

Related to #282
2022-05-28 13:42:38 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
c4dd518e04 decode: Make compound range sort optional
Some formats might want to control child order
mp4: Keep tracks in track id order
dns: Keep label component order
elf: Keep seciton order
macho: Keep command and section order
2022-05-03 16:16:09 +02:00
Mattias Wadman
3ce660a2cf decode: Keep decode tree on RangeFn error 2022-05-03 10:57:43 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
b35b1804be decode,format: Add d.FieldFormatOrRaw(Len) 2022-04-11 23:01:45 +02:00
Mattias Wadman
cd2cbef6fe decode: Some cleanup
Fix PeekFind arg order
2022-02-17 00:25:24 +01:00
Mattias Wadman
06245d1295 binary,decode,doc: Rename buffer to binary and add some documentation
Rename buffer to binary. Still some work left what to call buffer/binary in decode code.
Document decode value and binary type
Fix proper unit padding for tobytes and add still undocumenated extra padding argument.
Add some additional binary tests
2022-02-08 22:20:28 +01:00
Mattias Wadman
be71eb01ac decode: Rename LenFn() to LimitedFn, add FramedFn and document
I think framed makes more sense, but not sure
2022-02-06 13:49:00 +01:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
2022-02-04 21:41:53 +01:00
Mattias Wadman
4ab6381dc4 decode: Add scalars args to FieldRootBitBuf
Also move *Value.TryScalarFn to value.go
2022-02-01 22:24:24 +01:00
Mattias Wadman
898dfec1d1 lint: Fix typeassert and case exhaustive warnings 2022-01-25 17:23:41 +01:00
Mattias Wadman
b8efd8e549 fuzz: Fuzz all formats
Fix string readers to not panic on make slice with bogus sizes
2022-01-24 16:56:24 +01:00
Mattias Wadman
d1943dad49 pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
Also add dev/fuzzbytes.go tool to convert fuzz input to raw bytes
2022-01-23 13:34:56 +01:00
Mattias Wadman
34a09c40a7
Merge pull request #90 from wader/decode-stable-sort
decode: Use stable sort for values to not change order or values with…
2022-01-20 16:34:47 +01:00
Mattias Wadman
9198e18c10 decode: Add FieldArrayRootBitBufFn for completeness
Should rethink these, not very nice api
2022-01-20 16:27:52 +01:00
Mattias Wadman
cf8a50c150 decode: Use stable sort for values to not change order or values with same range start
Some decoders might relay on values added with same start to be kept in same order.
2022-01-20 16:21:50 +01:00
Mattias Wadman
ff5c0b8195 decode: Error on negative number of bits when reading numbers
Maybe should even give error on zero bits? maybe only allow for zero bit for unsigned ints?
2022-01-18 12:01:04 +01:00
Mattias Wadman
548a065388 decode,interp: Finish up nil value support
Update msgpack, bson and cbor to use it.
2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra
Think it makes more sense
2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae decode,interp: Add arbitrary large integer support (BigInt)
Was already handled in fq in various places as gojq uses them

Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
2022-01-15 19:00:42 +01:00
Mattias Wadman
0b0f28e966 cbor: Add decoder
Does not decode sematic tag types
Also fixes broken float16 support

Fixes #71
2022-01-12 17:56:03 +01:00
Mattias Wadman
149cb3f45a interp: Add torepr/0 that converts decode value into what it reptresents
Ex: fq -d msgpack torepr file.msgpack
Willoutput the JSON representation of the msgpack

Make per format *_torepr functions internal
2022-01-12 17:33:58 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
2022-01-05 22:05:09 +01:00
Mattias Wadman
6db7218093 doc: Some more dev infomration 2021-12-31 17:13:16 +01:00
Mattias Wadman
f4f63835dc interp: Add ._index for values in arrays 2021-12-10 18:19:11 +01:00
Mattias Wadman
f3480026bf sll2: fuzz: Limit address length to max 8 bytes 2021-12-09 10:38:34 +01:00
Mattias Wadman
9b683cd839 deocde: Cleanup some io panic(err) 2021-12-07 17:33:57 +01:00