wader/fq
1
1
Fork 0
mirror of https://github.com/wader/fq.git synced 2024-11-24 11:16:09 +03:00
Code Issues Projects Releases Wiki Activity
948 Commits 33 Branches 23 Tags 59 MiB
8228ecaeee
Commit Graph

309 Commits

Author SHA1 Message Date
Mattias Wadman
0312c92c31 asn1_ber: Add more doc and multiple outputs for frompem 2022-02-07 17:41:05 +01:00
Mattias Wadman
3bab3d659a aac_frame: Only try decode object types we know about 
Fixes issue failing on unknown or future aac standards
Also add missing object type passing for matroska
 2022-02-07 16:02:56 +01:00
Mattias Wadman
45b00aabd4 mp4: Add stz2 support 2022-02-07 12:35:39 +01:00
Mattias Wadman
0829c167cc asn1_ber: Add decoder 
Basic support, can also do CER and DER but without any extra validation.
No schema support.

Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.

Related to #20
 2022-02-06 22:13:03 +01:00
Mattias Wadman
5fab619db7
Merge pull request #128 from wader/fuzz-log-seed-number 
fuzz: List seed numbers and make it build again
 2022-02-05 18:42:35 +01:00
Mattias Wadman
e1bdfdf8ef fuzz: List seed numbers and make it build again 2022-02-05 18:38:38 +01:00
Mattias Wadman
092609bee1 mp4: Add video preview (pnot) and JPEG 2000 (jP) signatures 2022-02-05 17:03:18 +01:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct 
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
 2022-02-04 21:41:53 +01:00
Mattias Wadman
2ab395a03b protobuf: Add note about sub message decoding 
Fix and cleanup some other doc also
 2022-02-01 22:22:19 +01:00
Mattias Wadman
b66e3287f2
Merge pull request #112 from wader/interp-cleanup-d-alises 
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
 2022-01-29 12:28:46 +01:00
Mattias Wadman
8e47fb1a85 doc,matroska: Fix filesname in example 2022-01-29 12:20:47 +01:00
Mattias Wadman
a8664ed5ba doc: Add per format documentation 2022-01-29 12:01:36 +01:00
Mattias Wadman
fc0aacb654 interp: Cleanup display aliases, now: d, da, dd, dv, ddv 
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
 2022-01-28 18:25:38 +01:00
Mattias Wadman
6cd1c38fbf decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers 2022-01-28 17:18:50 +01:00
Mattias Wadman
8d2d88f4d0 mp3: Decrease max sync seek length between frames to 4k 
This might break mp3s with large id3v2 chunks embedded into the stream but
i think that is rare and probably means it is a dump of a mp3 radio stream etc.

An alternative approach is to also look for id3v2 etc between frames but then the mp3
root value probably should be an array of mix of things instead of a struct, or rename
header/footer to metadata?

Related to #103
 2022-01-25 16:08:43 +01:00
Mattias Wadman
4f490e5618
Merge pull request #99 from wader/mpeg-spu-fatal-infinit-loop 
mpeg_spu: Fatal error on infinite loop
 2022-01-24 17:53:56 +01:00
Mattias Wadman
b8efd8e549 fuzz: Fuzz all formats 
Fix string readers to not panic on make slice with bogus sizes
 2022-01-24 16:56:24 +01:00
Mattias Wadman
febce5a5a4 mpeg_spu: Fatal error on infinite loop 2022-01-24 16:53:55 +01:00
Mattias Wadman
d555c3248c mp4,fuzz: Fatal error on infinite sgpd box entries 
Also rename array to entries
 2022-01-23 14:24:57 +01:00
Mattias Wadman
d1943dad49 pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0 
Also add dev/fuzzbytes.go tool to convert fuzz input to raw bytes
 2022-01-23 13:34:56 +01:00
Mattias Wadman
778a1a4178 zip: Assert signature not validate 2022-01-22 16:26:03 +01:00
Mattias Wadman
85371173b5 id3v2: Should assert not validate magic 
Fixes issue with mp3 frame that happens to have a 2,3 or 4 at byte 3 being seen as id3v2 header
 2022-01-22 16:16:29 +01:00
Mattias Wadman
cf8a50c150 decode: Use stable sort for values to not change order or values with same range start 
Some decoders might relay on values added with same start to be kept in same order.
 2022-01-20 16:21:50 +01:00
Xentripetal
6fc1efd902 Add test case with all data types 2022-01-19 23:53:33 -06:00
Xentripetal
ee184075b7 Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing. 2022-01-19 21:00:44 -06:00
Xentripetal
16849c8f35 Update linting 2022-01-19 19:44:42 -06:00
Xentripetal
6e1f338ac3 Pull latest from wader 2022-01-19 19:39:12 -06:00
Xentripetal
de64a99e72 cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes 2022-01-19 19:35:50 -06:00
Mattias Wadman
ba1edef793 tar: Allow more than 2 zero end blocks at end 2022-01-18 12:54:50 +01:00
Mattias Wadman
39d4d846d7
Merge pull request #86 from wader/tar-dont-assume-endmarker 
tar: Don't assume there is a end marker
 2022-01-18 12:20:59 +01:00
Mattias Wadman
5921d76bf0 tar: Don't assume there is a end marker 
GNU tar docs says:
At the end of the archive file there are two 512-byte blocks filled with binary zeros as an
end-of-file marker. A reasonable system should write such end-of-file marker at the end of
an archive, but must not assume that such a block exists when reading an archive.
In particular GNU tar always issues a warning if it does not encounter it.

Also clean up API usage a bit
 2022-01-18 12:12:34 +01:00
Mattias Wadman
263f1aeb66 flac: Don't allow zero subframe sample size 
Spec says sample size can be 4-32 but that is probably before correcting for waste bits
 2022-01-18 11:58:57 +01:00
Mattias Wadman
61bf2ce644 mp4: Refactor sample decode into something more sane 
Also fatal error instead of silent ignore if index goes outside stsc or stco tables.
 2022-01-17 13:38:11 +01:00
Mattias Wadman
548a065388 decode,interp: Finish up nil value support 
Update msgpack, bson and cbor to use it.
 2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra 
Think it makes more sense
 2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae decode,interp: Add arbitrary large integer support (BigInt) 
Was already handled in fq in various places as gojq uses them

Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
 2022-01-15 19:00:42 +01:00
Mattias Wadman
4a1e8590fa mp4: Improved stsz handling 
Track size/count instead of just sizes which should decrease memory usage
making count sanity check unnecessary.

Fixes issue with huge mp4 files (80gb+) with lots of samples.
 2022-01-15 17:51:10 +01:00
Xentripetal
251053ef3c Initial pass on logical types 2022-01-13 23:32:19 -06:00
Xentripetal
788fd0258f Pull latest from wader 2022-01-13 20:59:56 -06:00
Mattias Wadman
729a6cae1e formats: Sort and make lists less likely to cause collision 2022-01-12 18:35:28 +01:00
Mattias Wadman
0b0f28e966 cbor: Add decoder 
Does not decode sematic tag types
Also fixes broken float16 support

Fixes #71
 2022-01-12 17:56:03 +01:00
Mattias Wadman
149cb3f45a interp: Add torepr/0 that converts decode value into what it reptresents 
Ex: fq -d msgpack torepr file.msgpack
Willoutput the JSON representation of the msgpack

Make per format *_torepr functions internal
 2022-01-12 17:33:58 +01:00
Mattias Wadman
a6bf62ce19 msgpack: Add decoder 
Fixes #25
 2022-01-10 19:33:09 +01:00
Mattias Wadman
af8e7efc61 bencode: Add decoder 
Closes #63
 2022-01-10 10:52:14 +01:00
siddik.acil
0cf486dcf1 elf: fix all-platforms naming typo 2022-01-08 13:42:08 +01:00
Mattias Wadman
c05fe732b6
Merge pull request #59 from wader/png-plte-trns-cleanup 
png: Decode PLTE and tRNS chunks and cleanup syms a bit
 2022-01-07 20:13:37 +01:00
Mattias Wadman
bfc977b810 png: Decode PLTE and tRNS chunks and cleanup syms a bit 2022-01-07 14:31:18 +01:00
Mattias Wadman
edd0ae198b tcp,flow: By default allow missing syn/ack for now 
Is probably what you usually want
 2022-01-07 12:02:38 +01:00
Mattias Wadman
628f0f403d bson: Add decoder 
Only supports basic types for now
 2022-01-06 00:51:37 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder 
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
 2022-01-05 22:05:09 +01:00
Mattias Wadman
af23eb8269 pcap: Don't fail if incl_len > spanlen 
Documentation wrong? seems to happen in real pcap files
 2022-01-03 21:27:14 +01:00
Mattias Wadman
b9aef39ed4 pcap,pcapng,bsd_loopback_frame: Add decoder, refactor link frame into a group 
Also fix incorrect struct name s/UDPDatagramIn/UDPPayloadIn/
 2022-01-03 21:06:27 +01:00
Mattias Wadman
b8671137dd matroska: Assert sane tag size only for strings 
Fixes issue with big webm/matroska files
Can be ignored with force decode
 2022-01-01 19:37:12 +01:00
Xentripetal
68e85a2ddf Use existing scalar description helper 2021-12-29 17:01:40 -06:00
Xentripetal
07ddf36f1c Cleanup for linting 2021-12-29 16:46:56 -06:00
Xentripetal
4b809a73f6 Change avro codec to funcs 2021-12-29 16:12:36 -06:00
Xentripetal
7345b8c7c2 Cleanup 2021-12-28 13:09:54 -06:00
Xentripetal
7a8e3ca26e Hook into registry, add codecs 2021-12-28 13:05:10 -06:00
Xentripetal
d6ca48182e initial work for avro OCF files 2021-12-28 13:01:53 -06:00
Mattias Wadman
eb9698fce7 mp4,ctts: Seem more usable to treat sample count/offset as signed 
Seems to different between specs
Modify avc.fqtest to generate a ctts box
 2021-12-20 11:19:38 +01:00
Mattias Wadman
9f08af31df mpeg,aac: Factor out escape value decoding 2021-12-14 17:34:38 +01:00
Mattias Wadman
28a3b71bd4 mp4: Cleanup sample decode code 2021-12-12 15:25:57 +01:00
Mattias Wadman
91217e8210 tiff: Fix endian typo and cleanup todos 2021-12-12 11:26:00 +01:00
Mattias Wadman
07a2ebe4b7 tiff,fuzz: Fatal error on infinite ifd loops 2021-12-12 11:20:01 +01:00
Mattias Wadman
c149732182 mp4,trun,fuzz: Limit number of constant sample entries 2021-12-12 09:55:07 +01:00
Mattias Wadman
2b2320d411 pcap,flows: fuzz: Handle broken packets more nicely 
Still need to understand how gopacket is suppose to handle broken things
 2021-12-09 14:19:17 +01:00
Mattias Wadman
62785291fe mp4: fuzz: Make sure stsz has sane number of entries on constant sample size 
Should be redone to not use an array
 2021-12-09 13:00:10 +01:00
Mattias Wadman
f3480026bf sll2: fuzz: Limit address length to max 8 bytes 2021-12-09 10:38:34 +01:00
Mattias Wadman
6ed2e2e72e interp: dump: Indicate arrays using jq-syntax 
Related to #16
 2021-12-08 16:24:28 +01:00
Mattias Wadman
406263b485 mp4: Add comment about hdlr.component_name prefix byte 2021-12-08 10:45:10 +01:00
Mattias Wadman
c2131bb4a0 flac: Cleanup scalar usage and fix incorrect sample rates 
Also split DisplayFormat into actual and sym
 2021-12-08 00:39:52 +01:00
Mattias Wadman
1d7ace3899 pcap,pcapng,tcp: Use capture length not original length 2021-12-07 18:47:31 +01:00
Mattias Wadman
e91b22b324 matroska,ebml: Use scalar and require sane tag size 2021-12-07 15:51:46 +01:00
Mattias Wadman
45026ebf21 tar: Cleanup constant usage a bit more 2021-12-06 21:08:40 +01:00
Mattias Wadman
b525d0b3c8 pcap: fuzz: Skip ssl2 packet if too short 
Should report error somehow?
 2021-12-06 19:05:04 +01:00
Mattias Wadman
38509683a7 udp: Use proper udp payload format var name 2021-12-06 15:42:49 +01:00
Mattias Wadman
3601fe3bcb gzip: fuzz: Don't uncompress on unknown compress method (nil create reader fn) 2021-12-06 15:41:39 +01:00
Mattias Wadman
91cc6d80f7 tar: Fix size decode regression after cleanup 2021-12-06 14:15:54 +01:00
Mattias Wadman
e260830454 webp: Cleanup endian usage 2021-12-04 19:15:54 +01:00
Mattias Wadman
dd883b35bd wav: Cleanup endian usage 2021-12-04 19:14:07 +01:00
Mattias Wadman
57e9f418c8 vorbis_packet: Cleanup endian usage 2021-12-04 19:12:24 +01:00
Mattias Wadman
dc1aea3030 opus: Cleanup endian usage and fix incorrect preskip decode 2021-12-04 19:12:24 +01:00
Mattias Wadman
6a8d77b05c vorbis_comment: Cleanup endian usage and naming a bit 2021-12-04 19:12:24 +01:00
Mattias Wadman
eb4718fbdd tar: Cleanup api usage 2021-12-04 19:05:26 +01:00
Mattias Wadman
d48ebc12ee decode: Simplify Compound.Children 
Not a array reference anymore but instead pass around Compound reference
 2021-12-03 00:06:11 +01:00
Mattias Wadman
2fc0a71a47 decode: Refactor scalar usage 
Move scalar into own package.
Split scalar code into decode related scalar code (that reads etc) and
scalar code that just transform the scalar value.
Use a scalar.Mapper interface instead of just a function.
Make mappers, assert and validat impement the interface.
 2021-12-02 17:39:26 +01:00
Mattias Wadman
f801cc0af7 decode: Rename s/FieldTryFormat/TryFieldFormat for consistency 2021-11-30 15:29:41 +01:00
Mattias Wadman
c0eebcc23f format: Remove unused ProtoBufType 2021-11-30 13:13:14 +01:00
Mattias Wadman
7f7698601f ogg: Cleanup bitio in format out, maybe later 2021-11-30 13:12:14 +01:00
Mattias Wadman
f55b1af6ac inet: Add tcp and ipv4 reassembly 
Also add tcp_stream and udp_payload to decode content
 2021-11-29 18:42:18 +01:00
Mattias Wadman
be0fdbe793 vp9: Add profile and fix reserved_zero field collision 2021-11-29 16:14:59 +01:00
Mattias Wadman
db586eb3f9 mp4,matroska: Add *_path/0 variant that uses format_root 2021-11-29 13:19:23 +01:00
Mattias Wadman
c8fad57df7 tiff: Fix reading of mluc tags with multiple records 2021-11-29 11:55:25 +01:00
Mattias Wadman
646f902ca7 vpx_ccr: Add color names 2021-11-29 00:37:22 +01:00
Mattias Wadman
3bf1a57cfa avc: Cleanup and add color names etc 2021-11-28 17:29:56 +01:00
Mattias Wadman
7b7faaf02b pcap: Add pcap, pcapng, ether8023, ipv4, udp, udp 2021-11-24 18:13:00 +01:00
Mattias Wadman
9d116df799 decode: Rework use of TryFieldReaderRangeFormat 
Handle failure better
 2021-11-24 15:25:27 +01:00
Mattias Wadman
4f0bf9228f flac: Make md5_calculated be a buffer 2021-11-24 12:31:42 +01:00
Mattias Wadman
4f8d03731d format,decode: Some crc and endian refactor 2021-11-24 12:01:01 +01:00
Mattias Wadman
fc76907c07 png: Add proper color type 2021-11-23 17:04:21 +01:00
Mattias Wadman
2af08da9f7 flac: Fix block_size regression 2021-11-23 16:59:50 +01:00
Mattias Wadman
e47888e24c mp4: Fix 64bit size regression 2021-11-23 11:24:09 +01:00
Mattias Wadman
e6cb708d38 mpeg: Nicer sym and description 2021-11-22 17:02:07 +01:00
Mattias Wadman
2e71fa1380 mp4: Add smhd box 2021-11-22 13:26:32 +01:00
Mattias Wadman
27e477061e apev2: Fatal if > 1000 tags 2021-11-22 01:45:30 +01:00
Mattias Wadman
fdb811ec54 gojqextra,decode: Add generic lazy JQValue 2021-11-22 01:23:59 +01:00
Mattias Wadman
5ad048daf4 tar: Fix 0 trim regression 2021-11-22 00:45:13 +01:00
Mattias Wadman
aab32cf2db gojqextra,interp: Add lazy string to speed usage of decode value buffer where string is not used 2021-11-21 21:27:44 +01:00
Mattias Wadman
ee611a489a gzip,bzip2: Calculate CRC 2021-11-21 20:13:42 +01:00
Mattias Wadman
606c0b67ae format: Add vorbis-comment-picture test, add .gitignore and cleanup 2021-11-21 19:32:52 +01:00
Mattias Wadman
f40320b04c decode: Remove D.Scalar* and add d.(Try)FieldScala*Fn instead 
Idea is scalar fn should not read
 2021-11-21 13:08:18 +01:00
Mattias Wadman
046f2fd965 mp3: Don't allow more than 64k between frames 
Should rethink this
 2021-11-21 12:02:40 +01:00
Mattias Wadman
9029143acc zip: Fix nested decode for none compress 2021-11-21 12:01:09 +01:00
Mattias Wadman
0480a2f722 decode: Some format decode and sub buffer work 2021-11-20 18:56:59 +01:00
Mattias Wadman
d1b514edeb format: Some claeanup 2021-11-19 16:44:06 +01:00
Mattias Wadman
527f917698 mp3: Error if > 5 unique header configs 
Hopefully make it less likely to missprobe
 2021-11-19 16:22:18 +01:00
Mattias Wadman
278e909a2f mp4: Add tapt, prof, enof and clap boxes 2021-11-19 15:29:49 +01:00
Mattias Wadman
d838d2f613 zip: Add format decoder 2021-11-19 00:22:26 +01:00
Mattias Wadman
5344c7e3de icc: Add mluc support 2021-11-18 11:22:07 +01:00
Mattias Wadman
ffb5adff1d registry: Move to pkg/registry, feels better 2021-11-18 01:22:21 +01:00
Mattias Wadman
7af191d974 decode: Move io helper into *D 2021-11-18 01:17:41 +01:00
Mattias Wadman
6fba1a8125 decode: Fix bitbuf root handling a bit 2021-11-18 01:17:15 +01:00
Mattias Wadman
c083a9e475 decode: Fix MapRawToScalar regression 2021-11-17 17:14:19 +01:00
Mattias Wadman
986d5ecc50 decode: Move registry package to decode/registry and add a format group type 2021-11-17 16:59:56 +01:00
Mattias Wadman
3cea849101 dns: Cleanup a bit 2021-11-17 16:30:01 +01:00
Mattias Wadman
5d98a6949a decode: Refactor Error/Fatal into printf functions 2021-11-17 16:26:13 +01:00
Mattias Wadman
ede2e77975 decode: Nicer scalar template and add doc 2021-11-17 16:13:10 +01:00
Mattias Wadman
b66ed32614 decode,interp: Make fuzzing work again and cleanup fatal/error code 2021-11-16 17:11:26 +01:00
Mattias Wadman
f9f866000e interp,decode: Add force option to ignore asserts 2021-11-16 13:03:56 +01:00
Mattias Wadman
1b32b42f93 decode: Major decode API refactor 
Generate more code
More generic and comfortable API
Improve and Update format decoder to new API
Add some more format tests
 2021-11-15 21:12:07 +01:00
Mattias Wadman
cc5f405554 interp,format: Update tests after decode refactor/tosym/toactual 2021-11-05 17:29:23 +01:00
Mattias Wadman
6a15625587 interp,decode: Refactor out Scalar from Value and merge Array/Struct into Compound 
Also add tosym, toactual
 2021-11-05 17:29:22 +01:00
Mattias Wadman
13fae09172 interp: Don't print context cancel 2021-11-01 13:01:05 +01:00
Mattias Wadman
67898cb44a interp: Cleanup, use BufferRange for _open, progress for all decode 2021-10-30 01:17:14 +02:00
Mattias Wadman
88eade9ab9 ogg: Add flac support 
Refactor flac_streaminfo and flac_metadata from flac_metadatablocks
Some deocde bitbuf api cleanup
 2021-10-29 18:30:16 +02:00
Mattias Wadman
c1d9b4d2d5 flac_metadatablocks: typ > 127 can't happen, add app id 2021-10-23 23:06:03 +02:00
Mattias Wadman
bc9951cc59 flac_frame: Fail if trying to decode outside block size 2021-10-20 13:55:34 +02:00
Mattias Wadman
4af5739712 interp: Rework string/buffer for decode values 2021-10-20 01:31:59 +02:00
Mattias Wadman
49d2e617f9 tar: Unbreak num parsing and add test 2021-10-19 22:36:03 +02:00
Mattias Wadman
344f628403 flac: Calculate correct md5 when total samples count is zero 2021-10-19 18:05:22 +02:00
Mattias Wadman
1f26d4f233 flac_frame: Correctly read escaped samples and also a bit less allocations 2021-10-19 17:27:05 +02:00
Mattias Wadman
ce044baf7e flac: Cleanup 2021-10-19 15:44:15 +02:00
Mattias Wadman
5052bae18e decode,interp: Refactor to allow decode/fillgap a range 2021-10-19 14:38:11 +02:00
Mattias Wadman
2b35d28f3b flac_picture,mp4: avif images and data fallback is image format fails 2021-10-19 01:58:07 +02:00
Mattias Wadman
6ff5acae03 gif: Support GIF87a 2021-10-19 01:53:37 +02:00
Mattias Wadman
b643e22877 flac_frame: Support non-8 bit align sample size 2021-10-19 01:52:59 +02:00
Mattias Wadman
ec97eca054 format: Split default.go into format.go and shared.go 2021-10-18 11:51:50 +02:00
Mattias Wadman
984ba1aa43 flac_metadatablocks: type >= 127 is invalid 2021-10-13 21:40:20 +02:00
Mattias Wadman
509b8f8c50 flac_picture: Add picture_type names 2021-10-13 21:34:14 +02:00
Mattias Wadman
691688022f fqtest: Refactor our script part to own package 
Hopefully can be reused for docs generation
 2021-10-06 18:49:03 +02:00