wader/fq
1
1
Fork 0
mirror of https://github.com/wader/fq.git synced 2024-11-24 03:05:22 +03:00
Code Issues Projects Releases Wiki Activity
1,208 Commits 33 Branches 23 Tags 59 MiB
3298d181fe
Commit Graph

390 Commits

Author SHA1 Message Date
Mattias Wadman
6f03471d15 interp: Paths with a array as root was missing start dot 2022-04-08 16:47:40 +02:00
Mattias Wadman
f8fd6b7f48 mp4: Add more HEIF boxes 
Add scalar.*ToDesceription mapper and refactor to use it
 2022-04-07 19:11:49 +02:00
Mattias Wadman
f8d79a5777 matroska: More sym cleanup 2022-04-06 19:48:26 +02:00
Mattias Wadman
09385c612e id3v2: Add 2.0 PIC support 2022-04-06 13:21:28 +02:00
Mattias Wadman
5ff67e4cfe formats: Sym and field name cleanup to be more jq friendly 2022-04-05 13:59:57 +02:00
Mattias Wadman
34cbe48753 amf0: Decode strings in more detail 2022-04-05 11:01:59 +02:00
Mattias Wadman
788b0ac197 rtmp,amf0: Improve decoders, aac asc, chunk stream interrupt, fix amf0 ecma arrays 
rtmp: Handle chunk streams that have been interrupted (capture terminated in middle of stream etc)
rtmp: Decode AAC ASC
rtmp: Add ffmpeg client/server stream test
rtmp: Decode user control messages
rtmo: Decode all data messages
amf0: Fix ecma array decoding
 2022-04-04 14:42:16 +02:00
Mattias Wadman
e8dc7112b6 ipv6,icmpv6: Add decoder 
Refactor to use groups between network layers to make them less coupled and reusable:
link_frame (ethernet etc)
inet_packet (ipv4,ipv6 etc)
ip_packet (tcp packet (not stream), udp segment etc)

Rename data to payload as i think it makes more sense for network data
 2022-04-03 17:43:51 +02:00
Mattias Wadman
d8aaf30345 rtmp,amf0: Add decoders 
Basic RTMP support, used via tcp_stream group (pcap etc) or manually.
Basic AMF0 support, mostly what is used in RTMP.
 2022-04-01 13:07:20 +02:00
Mattias Wadman
c0202483b9 hevc_vpc,hevc_sps: Use same nameing for profile as in spec 2022-03-25 09:48:28 +01:00
Mattias Wadman
74978c9d08 hevc: Add hevc_vps, hevc_sps and hevc_pps decoders 
Some extensions and other things still missing
 2022-03-24 18:56:56 +01:00
Mattias Wadman
f34ebd839b mp4: Add more HEIF boxes 2022-03-23 17:04:17 +01:00
bump
e5f61e22e9 Update github-go-version from 1.17.7, 1.17.7, 1.17.7 to 1.18.0 
Replace strings.Title and and fix deprecation warning
 2022-03-18 21:09:42 +01:00
Mattias Wadman
b2a865eac7 avc_sps: Add chroma format name mapping 2022-03-18 12:22:08 +01:00
Mattias Wadman
27e761570a format: Simplify torepr, no need for _f function 2022-03-08 17:09:48 +01:00
Mattias Wadman
920629f5fd doc: Regenerate and fix macho section size 
Make macho format documentaion a bit more consistent.
 2022-03-03 09:55:53 +01:00
Mattias Wadman
b157751a0a mp4: Reformat and use dv in test 2022-03-01 11:59:28 +01:00
Mattias Wadman
f8690e6c59 mp4: Add colr box support 2022-03-01 11:43:50 +01:00
Mattias Wadman
e0f493fa0b
Merge pull request #51 from Akaame/43_macho_support 
[WIP] 43: Initial impl for macho support.
 2022-02-28 10:22:50 +01:00
siddik.acil
b0911af204 macho: docs review changes 2022-02-27 19:31:18 +01:00
siddik.acil
04eae939eb macho: add basic docs 2022-02-26 19:58:40 +01:00
Mattias Wadman
206dcd02a1 fuzz: Include more testdata seed files 2022-02-22 01:04:58 +01:00
siddik.acil
228757b92a macho: review fixes 2022-02-21 20:32:18 +01:00
siddik.acil
78699f3ace macho: parse flags individually 2022-02-21 20:32:18 +01:00
siddik.acil
3991c51ab5 macho: handle unknown lc_commands better 2022-02-21 20:32:18 +01:00
siddik.acil
ef2919b300 macho: introduce arm and fat tests 2022-02-21 20:32:18 +01:00
siddik.acil
d1f093ce80 macho: fix fat header decode bug 2022-02-21 20:32:18 +01:00
siddik.acil
d47e04c494 fixup! macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff 2022-02-21 20:32:18 +01:00
siddik.acil
c5918d234b macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff 2022-02-21 20:32:18 +01:00
siddik.acil
5ee9a23c14 macho: review fixes 2022-02-21 20:32:18 +01:00
siddik.acil
5c97420989 macho: TS string to UTC 2022-02-21 20:32:18 +01:00
siddik.acil
2ccb8087b0 macho: update test cases v to dv 2022-02-21 20:32:18 +01:00
siddik.acil
e3daee7d94 macho: simplify thread state decoder 2022-02-21 20:32:18 +01:00
siddik.acil
70c9d5196c macho: thread state visualization 2022-02-21 20:32:18 +01:00
siddik.acil
a4789dc18f macho: timestamp mapper 2022-02-21 20:32:18 +01:00
siddik.acil
66feebc565 macho: change parseFlags impl for ordered results 2022-02-21 20:32:18 +01:00
siddik.acil
74abe99095 macho: update tests 
parseFlags reorders flags every time the program is executed which is non-deterministic as golang maps are unordered.
 2022-02-21 20:32:18 +01:00
siddik.acil
a86e704310 macho: add section type parsing 2022-02-21 20:32:18 +01:00
siddik.acil
20e5be3f57 macho: delete ar decoder code 2022-02-21 20:32:18 +01:00
siddik.acil
5a48cb30a6 macho: refactor prebound_dylib 2022-02-21 20:32:18 +01:00
siddik.acil
4016ad0ba3 macho: parse segment section flags 2022-02-21 20:32:18 +01:00
siddik.acil
a29bfca5ad macho: expand filetypes and header flags 2022-02-21 20:32:17 +01:00
siddik.acil
98c9840d52 macho: linting changes for ar parse 2022-02-21 20:32:17 +01:00
siddik.acil
3334750328 macho: reuse ar decoder 2022-02-21 20:32:17 +01:00
siddik.acil
e199d2190f macho: basic impl for ar and fat file parsing 
untested
 2022-02-21 20:32:17 +01:00
siddik.acil
5c5bd87914 macho: add fqtest actualization 
leave libbbb.a out
 2022-02-21 20:32:17 +01:00
siddik.acil
0d648928ce macho: fix null in segname sectname 
add OPCODE_DECODER comment for future
 2022-02-21 20:32:17 +01:00
siddik.acil
141a8e84be macho: mach_header_X to header 
arch_bits as value fields
 2022-02-21 20:32:17 +01:00
siddik.acil
2638f419eb macho: add darwin_amd64 tests 2022-02-21 20:32:17 +01:00
siddik.acil
70b84cded9 macho: ntools fix LC_MAIN fix 2022-02-21 20:32:17 +01:00
siddik.acil
efdd0bf5e8 macho: discard lc_ and lowercase command names 2022-02-21 20:32:17 +01:00
siddik.acil
5e95d1c3fe macho: add cpuSubTypes 2022-02-21 20:32:17 +01:00
siddik.acil
fb0654ec7a macho: fix FieldUTF8NullFixedLen for segname 2022-02-21 20:32:17 +01:00
siddik.acil
9eb71dc662 macho: generate doc via make doc 2022-02-21 20:32:13 +01:00
siddik.acil
5f4ad41005 macho: use FieldUTF8NullFixedLen for segname 2022-02-21 20:31:12 +01:00
siddik.acil
333a3243f8 macho: add scalar.Hex mapper to addr fields 2022-02-21 20:31:12 +01:00
siddik.acil
90b94631bc macho: adopt plural-singular scheme for FieldStructArrayLoop 2022-02-21 20:31:12 +01:00
siddik.acil
12eb7cc536 macho: use FieldUTF8NullFixedLen 2022-02-21 20:31:12 +01:00
siddik.acil
b5fe9ce655 macho: change registry description 2022-02-21 20:31:12 +01:00
siddik.acil
1feb81c9cd macho: little-endian to little_endian 2022-02-21 20:31:12 +01:00
siddik.acil
bf214d5e4e macho: add nolint suppression to const defs 2022-02-21 20:31:12 +01:00
siddik.acil
9206d9d8ab macho: magicToHex to scalar.Hex 2022-02-21 20:31:12 +01:00
siddik.acil
b78ed02fb3 macho: barebones decoder impl 2022-02-21 20:31:12 +01:00
Mattias Wadman
e238f292af repl: Speedup multi input to sub-repl 
Rewrite non-sub-repl queries as "q" -> ".[] | q" and pass in inputs as array.
Before each input did an eval.

Also fixup error message a bit, skip line:column when they dont make sense.
 2022-02-18 01:13:45 +01:00
Mattias Wadman
cd2cbef6fe decode: Some cleanup 
Fix PeekFind arg order
 2022-02-17 00:25:24 +01:00
Mattias Wadman
c6a90cfc1d doc,asn1_ber: Add more documentation 2022-02-16 20:02:23 +01:00
Mattias Wadman
9dc59e5d39 interp: Move _is_decode_value to jq 
Make _extype work like type
 2022-02-15 22:20:32 +01:00
Xentripetal
9636613ec6 Merge branch 'master' of https://github.com/wader/fq into avro 2022-02-12 22:16:50 -06:00
Xentripetal
69c745d3c4 simplify scalar usage 2022-02-12 22:16:46 -06:00
Mattias Wadman
c58ba28da1 mpeg_spu: Fatal error on unknown cmd 
Also fixes fuzzer issues
 2022-02-12 16:11:41 +01:00
Mattias Wadman
6090b65eca fuzz: Make it compile again and run one format per fuzz 
Seems to run into timeouts (1s) if not as some formats can ends up allocating a lot
 2022-02-12 00:22:39 +01:00
Xentripetal
0909fb6dad Comment on snappy decompression 2022-02-10 08:55:56 -06:00
Xentripetal
21cfc70c38 Dates need to specify UTC too 2022-02-10 08:51:51 -06:00
Xentripetal
0300c955c5 Take heading off doc to match make doc format 2022-02-09 20:59:29 -06:00
Xentripetal
2605bce4e3 Lint and add basic doc 2022-02-09 20:46:12 -06:00
Xentripetal
4508241b1f Cleanup snappy 2022-02-09 20:39:31 -06:00
Xentripetal
6f57cdbfd0 Timestamps should be UTC 2022-02-09 18:16:26 -06:00
Xentripetal
75b8496178 Fix lint 2022-02-07 23:10:25 -06:00
Xentripetal
31c4c0d3c4 Support snappy and deflate codecs 2022-02-07 23:03:50 -06:00
Xentripetal
66ca1f109e Change tests to use new verbose syntax 2022-02-07 17:39:37 -06:00
Xentripetal
5b11385ea4 Pull latest from wader 2022-02-07 17:35:33 -06:00
Mattias Wadman
0312c92c31 asn1_ber: Add more doc and multiple outputs for frompem 2022-02-07 17:41:05 +01:00
Mattias Wadman
3bab3d659a aac_frame: Only try decode object types we know about 
Fixes issue failing on unknown or future aac standards
Also add missing object type passing for matroska
 2022-02-07 16:02:56 +01:00
Mattias Wadman
45b00aabd4 mp4: Add stz2 support 2022-02-07 12:35:39 +01:00
Mattias Wadman
0829c167cc asn1_ber: Add decoder 
Basic support, can also do CER and DER but without any extra validation.
No schema support.

Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.

Related to #20
 2022-02-06 22:13:03 +01:00
Mattias Wadman
5fab619db7
Merge pull request #128 from wader/fuzz-log-seed-number 
fuzz: List seed numbers and make it build again
 2022-02-05 18:42:35 +01:00
Mattias Wadman
e1bdfdf8ef fuzz: List seed numbers and make it build again 2022-02-05 18:38:38 +01:00
Mattias Wadman
092609bee1 mp4: Add video preview (pnot) and JPEG 2000 (jP) signatures 2022-02-05 17:03:18 +01:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct 
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
 2022-02-04 21:41:53 +01:00
Mattias Wadman
2ab395a03b protobuf: Add note about sub message decoding 
Fix and cleanup some other doc also
 2022-02-01 22:22:19 +01:00
Mattias Wadman
b66e3287f2
Merge pull request #112 from wader/interp-cleanup-d-alises 
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
 2022-01-29 12:28:46 +01:00
Mattias Wadman
8e47fb1a85 doc,matroska: Fix filesname in example 2022-01-29 12:20:47 +01:00
Mattias Wadman
a8664ed5ba doc: Add per format documentation 2022-01-29 12:01:36 +01:00
Mattias Wadman
fc0aacb654 interp: Cleanup display aliases, now: d, da, dd, dv, ddv 
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
 2022-01-28 18:25:38 +01:00
Mattias Wadman
6cd1c38fbf decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers 2022-01-28 17:18:50 +01:00
Mattias Wadman
8d2d88f4d0 mp3: Decrease max sync seek length between frames to 4k 
This might break mp3s with large id3v2 chunks embedded into the stream but
i think that is rare and probably means it is a dump of a mp3 radio stream etc.

An alternative approach is to also look for id3v2 etc between frames but then the mp3
root value probably should be an array of mix of things instead of a struct, or rename
header/footer to metadata?

Related to #103
 2022-01-25 16:08:43 +01:00
Mattias Wadman
4f490e5618
Merge pull request #99 from wader/mpeg-spu-fatal-infinit-loop 
mpeg_spu: Fatal error on infinite loop
 2022-01-24 17:53:56 +01:00
Mattias Wadman
b8efd8e549 fuzz: Fuzz all formats 
Fix string readers to not panic on make slice with bogus sizes
 2022-01-24 16:56:24 +01:00
Mattias Wadman
febce5a5a4 mpeg_spu: Fatal error on infinite loop 2022-01-24 16:53:55 +01:00
Mattias Wadman
d555c3248c mp4,fuzz: Fatal error on infinite sgpd box entries 
Also rename array to entries
 2022-01-23 14:24:57 +01:00
Mattias Wadman
d1943dad49 pcapng,fuzz: Fix infinite loop by fatal error on block length <= 0 
Also add dev/fuzzbytes.go tool to convert fuzz input to raw bytes
 2022-01-23 13:34:56 +01:00
Mattias Wadman
778a1a4178 zip: Assert signature not validate 2022-01-22 16:26:03 +01:00
Mattias Wadman
85371173b5 id3v2: Should assert not validate magic 
Fixes issue with mp3 frame that happens to have a 2,3 or 4 at byte 3 being seen as id3v2 header
 2022-01-22 16:16:29 +01:00
Mattias Wadman
cf8a50c150 decode: Use stable sort for values to not change order or values with same range start 
Some decoders might relay on values added with same start to be kept in same order.
 2022-01-20 16:21:50 +01:00
Xentripetal
6fc1efd902 Add test case with all data types 2022-01-19 23:53:33 -06:00
Xentripetal
ee184075b7 Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing. 2022-01-19 21:00:44 -06:00
Xentripetal
16849c8f35 Update linting 2022-01-19 19:44:42 -06:00
Xentripetal
6e1f338ac3 Pull latest from wader 2022-01-19 19:39:12 -06:00
Xentripetal
de64a99e72 cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes 2022-01-19 19:35:50 -06:00
Mattias Wadman
ba1edef793 tar: Allow more than 2 zero end blocks at end 2022-01-18 12:54:50 +01:00
Mattias Wadman
39d4d846d7
Merge pull request #86 from wader/tar-dont-assume-endmarker 
tar: Don't assume there is a end marker
 2022-01-18 12:20:59 +01:00
Mattias Wadman
5921d76bf0 tar: Don't assume there is a end marker 
GNU tar docs says:
At the end of the archive file there are two 512-byte blocks filled with binary zeros as an
end-of-file marker. A reasonable system should write such end-of-file marker at the end of
an archive, but must not assume that such a block exists when reading an archive.
In particular GNU tar always issues a warning if it does not encounter it.

Also clean up API usage a bit
 2022-01-18 12:12:34 +01:00
Mattias Wadman
263f1aeb66 flac: Don't allow zero subframe sample size 
Spec says sample size can be 4-32 but that is probably before correcting for waste bits
 2022-01-18 11:58:57 +01:00
Mattias Wadman
61bf2ce644 mp4: Refactor sample decode into something more sane 
Also fatal error instead of silent ignore if index goes outside stsc or stco tables.
 2022-01-17 13:38:11 +01:00
Mattias Wadman
548a065388 decode,interp: Finish up nil value support 
Update msgpack, bson and cbor to use it.
 2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra 
Think it makes more sense
 2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae decode,interp: Add arbitrary large integer support (BigInt) 
Was already handled in fq in various places as gojq uses them

Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
 2022-01-15 19:00:42 +01:00
Mattias Wadman
4a1e8590fa mp4: Improved stsz handling 
Track size/count instead of just sizes which should decrease memory usage
making count sanity check unnecessary.

Fixes issue with huge mp4 files (80gb+) with lots of samples.
 2022-01-15 17:51:10 +01:00
Xentripetal
251053ef3c Initial pass on logical types 2022-01-13 23:32:19 -06:00
Xentripetal
788fd0258f Pull latest from wader 2022-01-13 20:59:56 -06:00
Mattias Wadman
729a6cae1e formats: Sort and make lists less likely to cause collision 2022-01-12 18:35:28 +01:00
Mattias Wadman
0b0f28e966 cbor: Add decoder 
Does not decode sematic tag types
Also fixes broken float16 support

Fixes #71
 2022-01-12 17:56:03 +01:00
Mattias Wadman
149cb3f45a interp: Add torepr/0 that converts decode value into what it reptresents 
Ex: fq -d msgpack torepr file.msgpack
Willoutput the JSON representation of the msgpack

Make per format *_torepr functions internal
 2022-01-12 17:33:58 +01:00
Mattias Wadman
a6bf62ce19 msgpack: Add decoder 
Fixes #25
 2022-01-10 19:33:09 +01:00
Mattias Wadman
af8e7efc61 bencode: Add decoder 
Closes #63
 2022-01-10 10:52:14 +01:00
siddik.acil
0cf486dcf1 elf: fix all-platforms naming typo 2022-01-08 13:42:08 +01:00
Mattias Wadman
c05fe732b6
Merge pull request #59 from wader/png-plte-trns-cleanup 
png: Decode PLTE and tRNS chunks and cleanup syms a bit
 2022-01-07 20:13:37 +01:00
Mattias Wadman
bfc977b810 png: Decode PLTE and tRNS chunks and cleanup syms a bit 2022-01-07 14:31:18 +01:00
Mattias Wadman
edd0ae198b tcp,flow: By default allow missing syn/ack for now 
Is probably what you usually want
 2022-01-07 12:02:38 +01:00
Mattias Wadman
628f0f403d bson: Add decoder 
Only supports basic types for now
 2022-01-06 00:51:37 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder 
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
 2022-01-05 22:05:09 +01:00
Mattias Wadman
af23eb8269 pcap: Don't fail if incl_len > spanlen 
Documentation wrong? seems to happen in real pcap files
 2022-01-03 21:27:14 +01:00
Mattias Wadman
b9aef39ed4 pcap,pcapng,bsd_loopback_frame: Add decoder, refactor link frame into a group 
Also fix incorrect struct name s/UDPDatagramIn/UDPPayloadIn/
 2022-01-03 21:06:27 +01:00
Mattias Wadman
b8671137dd matroska: Assert sane tag size only for strings 
Fixes issue with big webm/matroska files
Can be ignored with force decode
 2022-01-01 19:37:12 +01:00
Xentripetal
68e85a2ddf Use existing scalar description helper 2021-12-29 17:01:40 -06:00
Xentripetal
07ddf36f1c Cleanup for linting 2021-12-29 16:46:56 -06:00
Xentripetal
4b809a73f6 Change avro codec to funcs 2021-12-29 16:12:36 -06:00
Xentripetal
7345b8c7c2 Cleanup 2021-12-28 13:09:54 -06:00
Xentripetal
7a8e3ca26e Hook into registry, add codecs 2021-12-28 13:05:10 -06:00
Xentripetal
d6ca48182e initial work for avro OCF files 2021-12-28 13:01:53 -06:00
Mattias Wadman
eb9698fce7 mp4,ctts: Seem more usable to treat sample count/offset as signed 
Seems to different between specs
Modify avc.fqtest to generate a ctts box
 2021-12-20 11:19:38 +01:00
Mattias Wadman
9f08af31df mpeg,aac: Factor out escape value decoding 2021-12-14 17:34:38 +01:00
Mattias Wadman
28a3b71bd4 mp4: Cleanup sample decode code 2021-12-12 15:25:57 +01:00
Mattias Wadman
91217e8210 tiff: Fix endian typo and cleanup todos 2021-12-12 11:26:00 +01:00
Mattias Wadman
07a2ebe4b7 tiff,fuzz: Fatal error on infinite ifd loops 2021-12-12 11:20:01 +01:00
Mattias Wadman
c149732182 mp4,trun,fuzz: Limit number of constant sample entries 2021-12-12 09:55:07 +01:00
Mattias Wadman
2b2320d411 pcap,flows: fuzz: Handle broken packets more nicely 
Still need to understand how gopacket is suppose to handle broken things
 2021-12-09 14:19:17 +01:00
Mattias Wadman
62785291fe mp4: fuzz: Make sure stsz has sane number of entries on constant sample size 
Should be redone to not use an array
 2021-12-09 13:00:10 +01:00
Mattias Wadman
f3480026bf sll2: fuzz: Limit address length to max 8 bytes 2021-12-09 10:38:34 +01:00
Mattias Wadman
6ed2e2e72e interp: dump: Indicate arrays using jq-syntax 
Related to #16
 2021-12-08 16:24:28 +01:00