siddik.acil
12eb7cc536
macho: use FieldUTF8NullFixedLen
2022-02-21 20:31:12 +01:00
siddik.acil
b5fe9ce655
macho: change registry description
2022-02-21 20:31:12 +01:00
siddik.acil
1feb81c9cd
macho: little-endian to little_endian
2022-02-21 20:31:12 +01:00
siddik.acil
bf214d5e4e
macho: add nolint suppression to const defs
2022-02-21 20:31:12 +01:00
siddik.acil
9206d9d8ab
macho: magicToHex to scalar.Hex
2022-02-21 20:31:12 +01:00
siddik.acil
b78ed02fb3
macho: barebones decoder impl
2022-02-21 20:31:12 +01:00
Mattias Wadman
e238f292af
repl: Speedup multi input to sub-repl
...
Rewrite non-sub-repl queries as "q" -> ".[] | q" and pass in inputs as array.
Before each input did an eval.
Also fixup error message a bit, skip line:column when they dont make sense.
2022-02-18 01:13:45 +01:00
Mattias Wadman
cd2cbef6fe
decode: Some cleanup
...
Fix PeekFind arg order
2022-02-17 00:25:24 +01:00
Mattias Wadman
c6a90cfc1d
doc,asn1_ber: Add more documentation
2022-02-16 20:02:23 +01:00
Mattias Wadman
9dc59e5d39
interp: Move _is_decode_value to jq
...
Make _extype work like type
2022-02-15 22:20:32 +01:00
Xentripetal
9636613ec6
Merge branch 'master' of https://github.com/wader/fq into avro
2022-02-12 22:16:50 -06:00
Xentripetal
69c745d3c4
simplify scalar usage
2022-02-12 22:16:46 -06:00
Mattias Wadman
c58ba28da1
mpeg_spu: Fatal error on unknown cmd
...
Also fixes fuzzer issues
2022-02-12 16:11:41 +01:00
Mattias Wadman
6090b65eca
fuzz: Make it compile again and run one format per fuzz
...
Seems to run into timeouts (1s) if not as some formats can ends up allocating a lot
2022-02-12 00:22:39 +01:00
Xentripetal
0909fb6dad
Comment on snappy decompression
2022-02-10 08:55:56 -06:00
Xentripetal
21cfc70c38
Dates need to specify UTC too
2022-02-10 08:51:51 -06:00
Xentripetal
0300c955c5
Take heading off doc to match make doc format
2022-02-09 20:59:29 -06:00
Xentripetal
2605bce4e3
Lint and add basic doc
2022-02-09 20:46:12 -06:00
Xentripetal
4508241b1f
Cleanup snappy
2022-02-09 20:39:31 -06:00
Xentripetal
6f57cdbfd0
Timestamps should be UTC
2022-02-09 18:16:26 -06:00
Xentripetal
75b8496178
Fix lint
2022-02-07 23:10:25 -06:00
Xentripetal
31c4c0d3c4
Support snappy and deflate codecs
2022-02-07 23:03:50 -06:00
Xentripetal
66ca1f109e
Change tests to use new verbose syntax
2022-02-07 17:39:37 -06:00
Xentripetal
5b11385ea4
Pull latest from wader
2022-02-07 17:35:33 -06:00
Mattias Wadman
0312c92c31
asn1_ber: Add more doc and multiple outputs for frompem
2022-02-07 17:41:05 +01:00
Mattias Wadman
3bab3d659a
aac_frame: Only try decode object types we know about
...
Fixes issue failing on unknown or future aac standards
Also add missing object type passing for matroska
2022-02-07 16:02:56 +01:00
Mattias Wadman
45b00aabd4
mp4: Add stz2 support
2022-02-07 12:35:39 +01:00
Mattias Wadman
0829c167cc
asn1_ber: Add decoder
...
Basic support, can also do CER and DER but without any extra validation.
No schema support.
Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.
Related to #20
2022-02-06 22:13:03 +01:00
Mattias Wadman
5fab619db7
Merge pull request #128 from wader/fuzz-log-seed-number
...
fuzz: List seed numbers and make it build again
2022-02-05 18:42:35 +01:00
Mattias Wadman
e1bdfdf8ef
fuzz: List seed numbers and make it build again
2022-02-05 18:38:38 +01:00
Mattias Wadman
092609bee1
mp4: Add video preview (pnot) and JPEG 2000 (jP) signatures
2022-02-05 17:03:18 +01:00
Mattias Wadman
7c5215347d
bitio,decode: Refactor bitio usage and make buffer slicing more correct
...
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.
bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.
Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
2022-02-04 21:41:53 +01:00
Mattias Wadman
2ab395a03b
protobuf: Add note about sub message decoding
...
Fix and cleanup some other doc also
2022-02-01 22:22:19 +01:00
Mattias Wadman
b66e3287f2
Merge pull request #112 from wader/interp-cleanup-d-alises
...
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
2022-01-29 12:28:46 +01:00
Mattias Wadman
8e47fb1a85
doc,matroska: Fix filesname in example
2022-01-29 12:20:47 +01:00
Mattias Wadman
a8664ed5ba
doc: Add per format documentation
2022-01-29 12:01:36 +01:00
Mattias Wadman
fc0aacb654
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
...
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
2022-01-28 18:25:38 +01:00
Mattias Wadman
6cd1c38fbf
decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers
2022-01-28 17:18:50 +01:00
Mattias Wadman
8d2d88f4d0
mp3: Decrease max sync seek length between frames to 4k
...
This might break mp3s with large id3v2 chunks embedded into the stream but
i think that is rare and probably means it is a dump of a mp3 radio stream etc.
An alternative approach is to also look for id3v2 etc between frames but then the mp3
root value probably should be an array of mix of things instead of a struct, or rename
header/footer to metadata?
Related to #103
2022-01-25 16:08:43 +01:00
Mattias Wadman
4f490e5618
Merge pull request #99 from wader/mpeg-spu-fatal-infinit-loop
...
mpeg_spu: Fatal error on infinite loop
2022-01-24 17:53:56 +01:00
Mattias Wadman
b8efd8e549
fuzz: Fuzz all formats
...
Fix string readers to not panic on make slice with bogus sizes
2022-01-24 16:56:24 +01:00
Mattias Wadman
febce5a5a4
mpeg_spu: Fatal error on infinite loop
2022-01-24 16:53:55 +01:00
Mattias Wadman
d555c3248c
mp4,fuzz: Fatal error on infinite sgpd box entries
...
Also rename array to entries
2022-01-23 14:24:57 +01:00
Mattias Wadman
d1943dad49
pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0
...
Also add dev/fuzzbytes.go tool to convert fuzz input to raw bytes
2022-01-23 13:34:56 +01:00
Mattias Wadman
778a1a4178
zip: Assert signature not validate
2022-01-22 16:26:03 +01:00
Mattias Wadman
85371173b5
id3v2: Should assert not validate magic
...
Fixes issue with mp3 frame that happens to have a 2,3 or 4 at byte 3 being seen as id3v2 header
2022-01-22 16:16:29 +01:00
Mattias Wadman
cf8a50c150
decode: Use stable sort for values to not change order or values with same range start
...
Some decoders might relay on values added with same start to be kept in same order.
2022-01-20 16:21:50 +01:00
Xentripetal
6fc1efd902
Add test case with all data types
2022-01-19 23:53:33 -06:00
Xentripetal
ee184075b7
Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing.
2022-01-19 21:00:44 -06:00
Xentripetal
16849c8f35
Update linting
2022-01-19 19:44:42 -06:00
Xentripetal
6e1f338ac3
Pull latest from wader
2022-01-19 19:39:12 -06:00
Xentripetal
de64a99e72
cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes
2022-01-19 19:35:50 -06:00
Mattias Wadman
ba1edef793
tar: Allow more than 2 zero end blocks at end
2022-01-18 12:54:50 +01:00
Mattias Wadman
39d4d846d7
Merge pull request #86 from wader/tar-dont-assume-endmarker
...
tar: Don't assume there is a end marker
2022-01-18 12:20:59 +01:00
Mattias Wadman
5921d76bf0
tar: Don't assume there is a end marker
...
GNU tar docs says:
At the end of the archive file there are two 512-byte blocks filled with binary zeros as an
end-of-file marker. A reasonable system should write such end-of-file marker at the end of
an archive, but must not assume that such a block exists when reading an archive.
In particular GNU tar always issues a warning if it does not encounter it.
Also clean up API usage a bit
2022-01-18 12:12:34 +01:00
Mattias Wadman
263f1aeb66
flac: Don't allow zero subframe sample size
...
Spec says sample size can be 4-32 but that is probably before correcting for waste bits
2022-01-18 11:58:57 +01:00
Mattias Wadman
61bf2ce644
mp4: Refactor sample decode into something more sane
...
Also fatal error instead of silent ignore if index goes outside stsc or stco tables.
2022-01-17 13:38:11 +01:00
Mattias Wadman
548a065388
decode,interp: Finish up nil value support
...
Update msgpack, bson and cbor to use it.
2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878
num,mathextra: Rename num package to mathextra
...
Think it makes more sense
2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae
decode,interp: Add arbitrary large integer support (BigInt)
...
Was already handled in fq in various places as gojq uses them
Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
2022-01-15 19:00:42 +01:00
Mattias Wadman
4a1e8590fa
mp4: Improved stsz handling
...
Track size/count instead of just sizes which should decrease memory usage
making count sanity check unnecessary.
Fixes issue with huge mp4 files (80gb+) with lots of samples.
2022-01-15 17:51:10 +01:00
Xentripetal
251053ef3c
Initial pass on logical types
2022-01-13 23:32:19 -06:00
Xentripetal
788fd0258f
Pull latest from wader
2022-01-13 20:59:56 -06:00
Mattias Wadman
729a6cae1e
formats: Sort and make lists less likely to cause collision
2022-01-12 18:35:28 +01:00
Mattias Wadman
0b0f28e966
cbor: Add decoder
...
Does not decode sematic tag types
Also fixes broken float16 support
Fixes #71
2022-01-12 17:56:03 +01:00
Mattias Wadman
149cb3f45a
interp: Add torepr/0 that converts decode value into what it reptresents
...
Ex: fq -d msgpack torepr file.msgpack
Willoutput the JSON representation of the msgpack
Make per format *_torepr functions internal
2022-01-12 17:33:58 +01:00
Mattias Wadman
a6bf62ce19
msgpack: Add decoder
...
Fixes #25
2022-01-10 19:33:09 +01:00
Mattias Wadman
af8e7efc61
bencode: Add decoder
...
Closes #63
2022-01-10 10:52:14 +01:00
siddik.acil
0cf486dcf1
elf: fix all-platforms naming typo
2022-01-08 13:42:08 +01:00
Mattias Wadman
c05fe732b6
Merge pull request #59 from wader/png-plte-trns-cleanup
...
png: Decode PLTE and tRNS chunks and cleanup syms a bit
2022-01-07 20:13:37 +01:00
Mattias Wadman
bfc977b810
png: Decode PLTE and tRNS chunks and cleanup syms a bit
2022-01-07 14:31:18 +01:00
Mattias Wadman
edd0ae198b
tcp,flow: By default allow missing syn/ack for now
...
Is probably what you usually want
2022-01-07 12:02:38 +01:00
Mattias Wadman
628f0f403d
bson: Add decoder
...
Only supports basic types for now
2022-01-06 00:51:37 +01:00
Mattias Wadman
681dbc26b7
elf,ar: Add ar decoder, improved elf decoder
...
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
2022-01-05 22:05:09 +01:00
Mattias Wadman
af23eb8269
pcap: Don't fail if incl_len > spanlen
...
Documentation wrong? seems to happen in real pcap files
2022-01-03 21:27:14 +01:00
Mattias Wadman
b9aef39ed4
pcap,pcapng,bsd_loopback_frame: Add decoder, refactor link frame into a group
...
Also fix incorrect struct name s/UDPDatagramIn/UDPPayloadIn/
2022-01-03 21:06:27 +01:00
Mattias Wadman
b8671137dd
matroska: Assert sane tag size only for strings
...
Fixes issue with big webm/matroska files
Can be ignored with force decode
2022-01-01 19:37:12 +01:00
Xentripetal
68e85a2ddf
Use existing scalar description helper
2021-12-29 17:01:40 -06:00
Xentripetal
07ddf36f1c
Cleanup for linting
2021-12-29 16:46:56 -06:00
Xentripetal
4b809a73f6
Change avro codec to funcs
2021-12-29 16:12:36 -06:00
Xentripetal
7345b8c7c2
Cleanup
2021-12-28 13:09:54 -06:00
Xentripetal
7a8e3ca26e
Hook into registry, add codecs
2021-12-28 13:05:10 -06:00
Xentripetal
d6ca48182e
initial work for avro OCF files
2021-12-28 13:01:53 -06:00
Mattias Wadman
eb9698fce7
mp4,ctts: Seem more usable to treat sample count/offset as signed
...
Seems to different between specs
Modify avc.fqtest to generate a ctts box
2021-12-20 11:19:38 +01:00
Mattias Wadman
9f08af31df
mpeg,aac: Factor out escape value decoding
2021-12-14 17:34:38 +01:00
Mattias Wadman
28a3b71bd4
mp4: Cleanup sample decode code
2021-12-12 15:25:57 +01:00
Mattias Wadman
91217e8210
tiff: Fix endian typo and cleanup todos
2021-12-12 11:26:00 +01:00
Mattias Wadman
07a2ebe4b7
tiff,fuzz: Fatal error on infinite ifd loops
2021-12-12 11:20:01 +01:00
Mattias Wadman
c149732182
mp4,trun,fuzz: Limit number of constant sample entries
2021-12-12 09:55:07 +01:00
Mattias Wadman
2b2320d411
pcap,flows: fuzz: Handle broken packets more nicely
...
Still need to understand how gopacket is suppose to handle broken things
2021-12-09 14:19:17 +01:00
Mattias Wadman
62785291fe
mp4: fuzz: Make sure stsz has sane number of entries on constant sample size
...
Should be redone to not use an array
2021-12-09 13:00:10 +01:00
Mattias Wadman
f3480026bf
sll2: fuzz: Limit address length to max 8 bytes
2021-12-09 10:38:34 +01:00
Mattias Wadman
6ed2e2e72e
interp: dump: Indicate arrays using jq-syntax
...
Related to #16
2021-12-08 16:24:28 +01:00
Mattias Wadman
406263b485
mp4: Add comment about hdlr.component_name prefix byte
2021-12-08 10:45:10 +01:00
Mattias Wadman
c2131bb4a0
flac: Cleanup scalar usage and fix incorrect sample rates
...
Also split DisplayFormat into actual and sym
2021-12-08 00:39:52 +01:00
Mattias Wadman
1d7ace3899
pcap,pcapng,tcp: Use capture length not original length
2021-12-07 18:47:31 +01:00
Mattias Wadman
e91b22b324
matroska,ebml: Use scalar and require sane tag size
2021-12-07 15:51:46 +01:00
Mattias Wadman
45026ebf21
tar: Cleanup constant usage a bit more
2021-12-06 21:08:40 +01:00
Mattias Wadman
b525d0b3c8
pcap: fuzz: Skip ssl2 packet if too short
...
Should report error somehow?
2021-12-06 19:05:04 +01:00
Mattias Wadman
38509683a7
udp: Use proper udp payload format var name
2021-12-06 15:42:49 +01:00
