wader/fq
1
1
Fork 0
mirror of https://github.com/wader/fq.git synced 2024-12-01 02:30:32 +03:00
Code Issues Projects Releases Wiki Activity
1,172 Commits 33 Branches 23 Tags 61 MiB
c233215aba
Commit Graph

370 Commits

Author SHA1 Message Date
Mattias Wadman
e641274487 zip: Add zip64 support and uncompress option 
{uncompress:boolean} uncompress and probe, default true
 2022-05-26 12:46:50 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage 
Make cwd for a test script the directory where the script is.
Use relative paths
 2022-05-21 20:03:25 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18 
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
 2022-05-20 15:23:16 +02:00
Mattias Wadman
7bfa88cdb5
Merge pull request #271 from wader/mp3-cleanup 
mp3: Use d.FieldValueU and some cleanup
 2022-05-20 13:15:31 +02:00
Mattias Wadman
3780375d76 mp3: Use d.FieldValueU and some cleanup 2022-05-20 12:50:40 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now 
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
 2022-05-20 12:38:43 +02:00
Mattias Wadman
ba844eb06a mp4: Skip fields for pcm samples for now 
Will produce one field per sample which is a bit too much.
Maybe in some future fq can have smart array types that could handle this better.

Also add data_format per track to make it easier to understand things.
 2022-05-19 16:00:42 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots 
Hopefully makes it a bit easier to read and spot
 2022-05-11 00:00:39 +02:00
Mattias Wadman
903c7198b4
Merge pull request #258 from wader/mp4-mvhd-tkhd-mdhd-mehd-version1 
mp4: mvhd,tkhd,mdhd,mehd: Add version 1 support
 2022-05-10 00:48:47 +02:00
Mattias Wadman
2e328180ae mp4: mvhd,tkhd,mdhd,mehd: Add version 1 support 
Also fix incorrect mehd decode
 2022-05-10 00:21:04 +02:00
Mattias Wadman
efa5e23aaa icc_profile: Correctly clamp align padding on EOF 
Also code was mixing bytes and bit units, how could this have worked?
 2022-05-09 13:14:20 +02:00
Mattias Wadman
8228ecaeee mp4: Add track id field and add track for tfhd with unseen track_id 
Fragmened mp4s can have no tkhd box or tkhd box with a track_id that does not match
any tfhd box track_id. Make is possible to see samples for these fragmented tracks
ny adding tracks for unseen track_ids.

Also add id field for tracks.
 2022-05-08 19:02:21 +02:00
Mattias Wadman
76161a1b99 scalar,mp4,gzip,tar: Add timestamp to description 
Scalar api for this needs some work
 2022-05-07 13:15:13 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming 
Should replace most of this with generics at some point
 2022-05-07 12:46:34 +02:00
Mattias Wadman
34cf5442b3 tcp: Split into client/server structs and add skipped_bytes and has_start/end per direction 
Feels clenaer and removes _client/server field prefixes
 2022-05-06 16:52:18 +02:00
Mattias Wadman
51ea1a31b9
Merge pull request #250 from wader/wav-bencode-random-fixes 
wav,bencode: Random fixes
 2022-05-05 20:21:16 +02:00
Mattias Wadman
1aaaefb0ac wav,bencode,mpeg_ps_packet,id3v1: Random fixes 
wav: Rest of file should description not symbolic value
bencode: Remove redundant torepr example
mpeg_ps_packet,id3v1: Remove whitespace in symbolic value
 2022-05-05 19:45:05 +02:00
Mattias Wadman
0e02bb669c mp4: iinf: Only assume sub boxes for version 0 2022-05-04 15:35:00 +02:00
Mattias Wadman
b0096bc170 avc_pps: Correct check if there is more rbsp data 
Also add rbsp_stop_one_bit field
 2022-05-04 12:34:40 +02:00
Mattias Wadman
c4dd518e04 decode: Make compound range sort optional 
Some formats might want to control child order
mp4: Keep tracks in track id order
dns: Keep label component order
elf: Keep seciton order
macho: Keep command and section order
 2022-05-03 16:16:09 +02:00
Mattias Wadman
3ce660a2cf decode: Keep decode tree on RangeFn error 2022-05-03 10:57:43 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options 
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
 2022-05-01 17:08:30 +02:00
Mattias Wadman
d6c31dac2b macho: Add section and encryption_info raw data fields 2022-04-25 20:26:35 +02:00
Mattias Wadman
21ad628add interp: dump: Show field name for compound values in arrays 
mp3: Remove one level of struct for granales
dump: Makes it a bit easier to understand array of arrays/structs.
avro_ocf: Rename ocf.go -> avro_ocr.go to have same name of source file as format
pcap,pcapng: Rename tcp flow struct to tcp_connection, makes more sense i think
mp4: Add ISOBMFF to format description
 2022-04-21 17:47:15 +02:00
Mattias Wadman
b35b1804be decode,format: Add d.FieldFormatOrRaw(Len) 2022-04-11 23:01:45 +02:00
Mattias Wadman
9a053f0a3d
Merge pull request #226 from wader/macho-cleanup 
macho: Cleanup syms and remove redundant fat_arch struct
 2022-04-11 22:17:46 +02:00
Mattias Wadman
5424eed703 macho: Cleanup syms and remove redundant fat_arch struct 2022-04-11 22:09:51 +02:00
Mattias Wadman
39ba5c4da6 mpeg_pes: Support mpeg1 and some cleanup 2022-04-11 21:48:34 +02:00
Mattias Wadman
963a7af0d0
Merge pull request #222 from wader/display-path-root-array-incorrect 
interp: Paths with a array as root was missing start dot
 2022-04-08 19:03:10 +02:00
Mattias Wadman
5d25bbc2e1 tcp,udp: Refactor and make port matching better 
rtmp: check port based on tcp direction
dns,rtmp: refactor to use tcp and udp port match helpers
flow: add has start/end to tcp to know if stream has missing bytes
 2022-04-08 17:44:39 +02:00
Mattias Wadman
6f03471d15 interp: Paths with a array as root was missing start dot 2022-04-08 16:47:40 +02:00
Mattias Wadman
f8fd6b7f48 mp4: Add more HEIF boxes 
Add scalar.*ToDesceription mapper and refactor to use it
 2022-04-07 19:11:49 +02:00
Mattias Wadman
f8d79a5777 matroska: More sym cleanup 2022-04-06 19:48:26 +02:00
Mattias Wadman
09385c612e id3v2: Add 2.0 PIC support 2022-04-06 13:21:28 +02:00
Mattias Wadman
5ff67e4cfe formats: Sym and field name cleanup to be more jq friendly 2022-04-05 13:59:57 +02:00
Mattias Wadman
34cbe48753 amf0: Decode strings in more detail 2022-04-05 11:01:59 +02:00
Mattias Wadman
788b0ac197 rtmp,amf0: Improve decoders, aac asc, chunk stream interrupt, fix amf0 ecma arrays 
rtmp: Handle chunk streams that have been interrupted (capture terminated in middle of stream etc)
rtmp: Decode AAC ASC
rtmp: Add ffmpeg client/server stream test
rtmp: Decode user control messages
rtmo: Decode all data messages
amf0: Fix ecma array decoding
 2022-04-04 14:42:16 +02:00
Mattias Wadman
e8dc7112b6 ipv6,icmpv6: Add decoder 
Refactor to use groups between network layers to make them less coupled and reusable:
link_frame (ethernet etc)
inet_packet (ipv4,ipv6 etc)
ip_packet (tcp packet (not stream), udp segment etc)

Rename data to payload as i think it makes more sense for network data
 2022-04-03 17:43:51 +02:00
Mattias Wadman
d8aaf30345 rtmp,amf0: Add decoders 
Basic RTMP support, used via tcp_stream group (pcap etc) or manually.
Basic AMF0 support, mostly what is used in RTMP.
 2022-04-01 13:07:20 +02:00
Mattias Wadman
c0202483b9 hevc_vpc,hevc_sps: Use same nameing for profile as in spec 2022-03-25 09:48:28 +01:00
Mattias Wadman
74978c9d08 hevc: Add hevc_vps, hevc_sps and hevc_pps decoders 
Some extensions and other things still missing
 2022-03-24 18:56:56 +01:00
Mattias Wadman
f34ebd839b mp4: Add more HEIF boxes 2022-03-23 17:04:17 +01:00
bump
e5f61e22e9 Update github-go-version from 1.17.7, 1.17.7, 1.17.7 to 1.18.0 
Replace strings.Title and and fix deprecation warning
 2022-03-18 21:09:42 +01:00
Mattias Wadman
b2a865eac7 avc_sps: Add chroma format name mapping 2022-03-18 12:22:08 +01:00
Mattias Wadman
27e761570a format: Simplify torepr, no need for _f function 2022-03-08 17:09:48 +01:00
Mattias Wadman
920629f5fd doc: Regenerate and fix macho section size 
Make macho format documentaion a bit more consistent.
 2022-03-03 09:55:53 +01:00
Mattias Wadman
b157751a0a mp4: Reformat and use dv in test 2022-03-01 11:59:28 +01:00
Mattias Wadman
f8690e6c59 mp4: Add colr box support 2022-03-01 11:43:50 +01:00
Mattias Wadman
e0f493fa0b
Merge pull request #51 from Akaame/43_macho_support 
[WIP] 43: Initial impl for macho support.
 2022-02-28 10:22:50 +01:00
siddik.acil
b0911af204 macho: docs review changes 2022-02-27 19:31:18 +01:00
siddik.acil
04eae939eb macho: add basic docs 2022-02-26 19:58:40 +01:00
Mattias Wadman
206dcd02a1 fuzz: Include more testdata seed files 2022-02-22 01:04:58 +01:00
siddik.acil
228757b92a macho: review fixes 2022-02-21 20:32:18 +01:00
siddik.acil
78699f3ace macho: parse flags individually 2022-02-21 20:32:18 +01:00
siddik.acil
3991c51ab5 macho: handle unknown lc_commands better 2022-02-21 20:32:18 +01:00
siddik.acil
ef2919b300 macho: introduce arm and fat tests 2022-02-21 20:32:18 +01:00
siddik.acil
d1f093ce80 macho: fix fat header decode bug 2022-02-21 20:32:18 +01:00
siddik.acil
d47e04c494 fixup! macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff 2022-02-21 20:32:18 +01:00
siddik.acil
c5918d234b macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff 2022-02-21 20:32:18 +01:00
siddik.acil
5ee9a23c14 macho: review fixes 2022-02-21 20:32:18 +01:00
siddik.acil
5c97420989 macho: TS string to UTC 2022-02-21 20:32:18 +01:00
siddik.acil
2ccb8087b0 macho: update test cases v to dv 2022-02-21 20:32:18 +01:00
siddik.acil
e3daee7d94 macho: simplify thread state decoder 2022-02-21 20:32:18 +01:00
siddik.acil
70c9d5196c macho: thread state visualization 2022-02-21 20:32:18 +01:00
siddik.acil
a4789dc18f macho: timestamp mapper 2022-02-21 20:32:18 +01:00
siddik.acil
66feebc565 macho: change parseFlags impl for ordered results 2022-02-21 20:32:18 +01:00
siddik.acil
74abe99095 macho: update tests 
parseFlags reorders flags every time the program is executed which is non-deterministic as golang maps are unordered.
 2022-02-21 20:32:18 +01:00
siddik.acil
a86e704310 macho: add section type parsing 2022-02-21 20:32:18 +01:00
siddik.acil
20e5be3f57 macho: delete ar decoder code 2022-02-21 20:32:18 +01:00
siddik.acil
5a48cb30a6 macho: refactor prebound_dylib 2022-02-21 20:32:18 +01:00
siddik.acil
4016ad0ba3 macho: parse segment section flags 2022-02-21 20:32:18 +01:00
siddik.acil
a29bfca5ad macho: expand filetypes and header flags 2022-02-21 20:32:17 +01:00
siddik.acil
98c9840d52 macho: linting changes for ar parse 2022-02-21 20:32:17 +01:00
siddik.acil
3334750328 macho: reuse ar decoder 2022-02-21 20:32:17 +01:00
siddik.acil
e199d2190f macho: basic impl for ar and fat file parsing 
untested
 2022-02-21 20:32:17 +01:00
siddik.acil
5c5bd87914 macho: add fqtest actualization 
leave libbbb.a out
 2022-02-21 20:32:17 +01:00
siddik.acil
0d648928ce macho: fix null in segname sectname 
add OPCODE_DECODER comment for future
 2022-02-21 20:32:17 +01:00
siddik.acil
141a8e84be macho: mach_header_X to header 
arch_bits as value fields
 2022-02-21 20:32:17 +01:00
siddik.acil
2638f419eb macho: add darwin_amd64 tests 2022-02-21 20:32:17 +01:00
siddik.acil
70b84cded9 macho: ntools fix LC_MAIN fix 2022-02-21 20:32:17 +01:00
siddik.acil
efdd0bf5e8 macho: discard lc_ and lowercase command names 2022-02-21 20:32:17 +01:00
siddik.acil
5e95d1c3fe macho: add cpuSubTypes 2022-02-21 20:32:17 +01:00
siddik.acil
fb0654ec7a macho: fix FieldUTF8NullFixedLen for segname 2022-02-21 20:32:17 +01:00
siddik.acil
9eb71dc662 macho: generate doc via make doc 2022-02-21 20:32:13 +01:00
siddik.acil
5f4ad41005 macho: use FieldUTF8NullFixedLen for segname 2022-02-21 20:31:12 +01:00
siddik.acil
333a3243f8 macho: add scalar.Hex mapper to addr fields 2022-02-21 20:31:12 +01:00
siddik.acil
90b94631bc macho: adopt plural-singular scheme for FieldStructArrayLoop 2022-02-21 20:31:12 +01:00
siddik.acil
12eb7cc536 macho: use FieldUTF8NullFixedLen 2022-02-21 20:31:12 +01:00
siddik.acil
b5fe9ce655 macho: change registry description 2022-02-21 20:31:12 +01:00
siddik.acil
1feb81c9cd macho: little-endian to little_endian 2022-02-21 20:31:12 +01:00
siddik.acil
bf214d5e4e macho: add nolint suppression to const defs 2022-02-21 20:31:12 +01:00
siddik.acil
9206d9d8ab macho: magicToHex to scalar.Hex 2022-02-21 20:31:12 +01:00
siddik.acil
b78ed02fb3 macho: barebones decoder impl 2022-02-21 20:31:12 +01:00
Mattias Wadman
e238f292af repl: Speedup multi input to sub-repl 
Rewrite non-sub-repl queries as "q" -> ".[] | q" and pass in inputs as array.
Before each input did an eval.

Also fixup error message a bit, skip line:column when they dont make sense.
 2022-02-18 01:13:45 +01:00
Mattias Wadman
cd2cbef6fe decode: Some cleanup 
Fix PeekFind arg order
 2022-02-17 00:25:24 +01:00
Mattias Wadman
c6a90cfc1d doc,asn1_ber: Add more documentation 2022-02-16 20:02:23 +01:00
Mattias Wadman
9dc59e5d39 interp: Move _is_decode_value to jq 
Make _extype work like type
 2022-02-15 22:20:32 +01:00
Xentripetal
9636613ec6 Merge branch 'master' of https://github.com/wader/fq into avro 2022-02-12 22:16:50 -06:00
Xentripetal
69c745d3c4 simplify scalar usage 2022-02-12 22:16:46 -06:00
Mattias Wadman
c58ba28da1 mpeg_spu: Fatal error on unknown cmd 
Also fixes fuzzer issues
 2022-02-12 16:11:41 +01:00