wader/fq
1
1
Fork 0
mirror of https://github.com/wader/fq.git synced 2024-12-24 13:52:02 +03:00
Code Issues Projects Releases Wiki Activity
1,548 Commits 36 Branches 23 Tags 61 MiB
6f0c5a9682
Commit Graph

555 Commits

Author SHA1 Message Date
Mattias Wadman
421b2b3071 bplist: Fix unknown field for singletons and add torepr tests 2022-10-04 14:30:54 +02:00
Mattias Wadman
09ea08f3aa
Merge pull request #427 from dgmcdona/bplist 
Adds support for Apple Binary Plist, version 00
 2022-10-04 14:25:53 +02:00
Mattias Wadman
20a15372ba mp4: Fix data_format typo 2022-10-02 23:31:38 +02:00
Mattias Wadman
0fdc03a4f1 doc: Fix some incorrect example prompts 2022-09-27 13:42:31 +02:00
Mattias Wadman
d6316d5cbf wav: Decode smpl chunk 2022-09-26 11:28:02 +02:00
David McDonald
1e47f4f2f7 updates tests post integer-bug fix 2022-09-25 00:48:43 -05:00
David McDonald
ebae938d0e Fixes bug in integer parsing 2022-09-25 00:43:21 -05:00
David McDonald
bcccde2358 Fixes and embeds documentation 2022-09-24 12:57:11 -05:00
David McDonald
3d8ea1deac updates torepr for data type 2022-09-23 22:12:30 -05:00
David McDonald
12836abe26 updates fqtest 2022-09-23 21:16:28 -05:00
David McDonald
5f61994087 adds function for decoding fixed sized arrays 2022-09-23 20:50:41 -05:00
David McDonald
3deceeebd6 fixes from PR comments 2022-09-23 20:23:29 -05:00
David McDonald
3198602d25 removed unused return type 2022-09-23 20:23:29 -05:00
David McDonald
369f4016c1 removed unneccessary type conversions 2022-09-23 20:23:29 -05:00
David McDonald
6b04f2de1c Documentation cleanup 2022-09-23 20:23:29 -05:00
David McDonald
368d183b73 Size check on nBits to save memory 2022-09-23 20:23:29 -05:00
David McDonald
a77cec921f Added documentation and tests, fixed bad date parsing 2022-09-23 20:23:23 -05:00
David McDonald
5711f29084 Code fixes from PR, still need to add tests and testdata 2022-09-23 20:21:32 -05:00
David McDonald
d784db69bc Adds support for Apple Binary Plist, version 00 2022-09-23 20:21:32 -05:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format 
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
 2022-09-22 23:07:58 +02:00
Mattias Wadman
725ab1b17b doc,html,xml: Add more documentation and examples 2022-09-20 17:35:27 +02:00
Mattias Wadman
01be59e459
Merge pull request #428 from wader/ar-allow-empty-integerr 
ar: Integer strings might be empty
 2022-09-17 16:17:47 +02:00
Mattias Wadman
f08f44f12a ar: Integer strings might be empty 2022-09-17 15:36:34 +02:00
Mattias Wadman
fe64530e11 csv: Add tsv and header example 
Also fix string esacpe in fuzz options parser
 2022-09-15 01:16:45 +02:00
Mattias Wadman
88009ee9a3
Merge pull request #425 from wader/wasm-doc 
wasm: Add some documentation
 2022-09-13 09:59:17 +02:00
Mattias Wadman
1385b5d0ba wasm: Add some documentation 2022-09-12 11:53:48 +02:00
Mattias Wadman
2655ba09c5 mp4: More codec names (from mp4ra.org) 2022-09-12 11:25:43 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again 
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
 2022-09-11 00:53:29 +02:00
Mattias Wadman
00a50662ea markdown: Add decoder 2022-09-10 18:26:45 +02:00
Mattias Wadman
5382d46a9a elf: Basic program header notes decoding 
Used in coredump to record various statees but also used in executables.

Next step would be to decode the note descriptions, like register info, file mappings etc.
Maybe a bit tricky as it is os/abi specific.
 2022-09-09 14:35:35 +02:00
Mattias Wadman
12105d8c1d elf: Treat missing string tables as empty to be more robust 
Makes it possible to decode shellcode snippets etc:

$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | elf | d'
    |00 01 02 03 04 05 06 07 08 09 0a 0b 0c 0d 0e 0f|0123456789abcdef|.{}: (elf)
    |                                               |                |  error: elf: BitBufRange: failed at position 0 (read size 4294967296 seek pos 0): outside buffer
    |                                               |                |  header{}:
    |                                               |                |    ident{}:
0x00|7f 45 4c 46                                    |.ELF            |      magic: raw bits (valid)
0x00|            02                                 |    .           |      class: 64 (2)
0x00|               01                              |     .          |      data: "little_endian" (1)
0x00|                  01                           |      .         |      version: 1
0x00|                     00                        |       .        |      os_abi: "sysv" (0)
0x00|                        00                     |        .       |      abi_version: 0
0x00|                           00 00 00 00 00 00 00|         .......|      pad: raw bits (all zero)
0x10|02 00                                          |..              |    type: "exec" (0x2)
0x10|      3e 00                                    |  >.            |    machine: "x86_64" (0x3e) (AMD x86-64)
0x10|            01 00 00 00                        |    ....        |    version: 1
0x10|                        78 00 40 00 00 00 00 00|        x.@.....|    entry: 4194424
0x20|40 00 00 00 00 00 00 00                        |@.......        |    phoff: 64
0x20|                        00 00 00 00 00 00 00 00|        ........|    shoff: 0
0x30|00 00 00 00                                    |....            |    flags: 0
0x30|            40 00                              |    @.          |    ehsize: 64
0x30|                  38 00                        |      8.        |    phentsize: 56
0x30|                        01 00                  |        ..      |    phnum: 1
0x30|                              00 00            |          ..    |    shentsize: 0
0x30|                                    00 00      |            ..  |    shnum: 0
0x30|                                          00 00|              ..|    shstrndx: 0
    |                                               |                |  program_headers[0:1]:
    |                                               |                |    [0]{}: program_header
0x40|01 00 00 00                                    |....            |      type: "load" (1) (Loadable segment)
    |                                               |                |      flags{}:
0x40|            05                                 |    .           |        unused0: 0
0x40|            05                                 |    .           |        r: true
0x40|            05                                 |    .           |        w: false
0x40|            05                                 |    .           |        x: true
0x40|               00 00 00                        |     ...        |        unused1: 0
0x40|                        00 00 00 00 00 00 00 00|        ........|      offset: 0x0
0x50|00 00 40 00 00 00 00 00                        |..@.....        |      vaddr: 0x400000
0x50|                        00 00 40 00 00 00 00 00|        ..@.....|      paddr: 0x400000
0x60|00 00 00 00 01 00 00 00                        |........        |      filesz: 4294967296
0x60|                        00 00 00 00 01 00 00 00|        ........|      memsz: 4294967296
0x70|00 00 20 00 00 00 00 00                        |.. .....        |      align: 2097152
0x70|                        b0 3c 66 bf 06 00 0f 05|        .<f.....|  unknown0: raw bits
$ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' | ndisasm -b 64 -
00000000  B03C              mov al,0x3c
00000002  66BF0600          mov di,0x6
00000006  0F05              syscall
 2022-09-06 11:31:48 +02:00
Mattias Wadman
8016352b3b html: Handle html with leading doctype etc 2022-09-01 22:58:48 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
e0bd489668
Merge pull request #404 from wader/matroska-lacing 
matroska: Add proper lacing support
 2022-08-28 21:13:50 +02:00
Mattias Wadman
7749e1b5b9 matroska: Add proper lacing support 2022-08-28 21:03:26 +02:00
Mattias Wadman
004406de65 bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat 
Makes bitcoin_blkdat fails fast as it is part of probe group.
Speeds up reading a big JSON file etc.
 2022-08-26 12:07:53 +02:00
Mattias Wadman
095e1161b7 xml: Switch from "-" to "@" as attribute prefix and make it an option 
Seems to be more common.
 2022-08-25 17:15:23 +02:00
Mattias Wadman
716323cea1 xml: Even more namespace fixes 
fromXML*: Collect namespaces before resolving to not relay on attribute order.
toXML*: Sort attributes, prefer non-namescaped first.
 2022-08-25 15:45:09 +02:00
Mattias Wadman
d8792fd104 interp,dump: Correctly flush columns if data will be shown 
Before addrbar ended up in data and also redudant colunms was added
 2022-08-24 21:14:26 +02:00
Mattias Wadman
8fc43533a9 prores_frame: Add basic container and frame header decoder 2022-08-24 19:33:35 +02:00
@0xb17bea125
f73965d2a3
wasm: add wasm to probe list 2022-08-23 21:19:07 +09:00
@0xb17bea125
ae4529c491
wasm: run golangci-lint run --fix 2022-08-23 21:12:29 +09:00
@0xb17bea125
bcc0dfd9f0
wasm: fix comment format 2022-08-23 21:09:56 +09:00
@0xb17bea125
2eb1750574
wasm: update tests 2022-08-23 21:07:44 +09:00
@0xb17bea125
cd037c514d
wasm: add comment to clarify lazy initialization 2022-08-23 17:46:00 +09:00
@0xb17bea125
db8021c931
wasm: define and use constants for some insturctions 2022-08-23 08:42:34 +09:00
@0xb17bea125
e5cf1731e3
wasm: make the godoc formatter happy 2022-08-22 21:30:49 +09:00
@0xb17bea125
d5d9e738b6
wasm: use FieldULEB128() / FieldSLEB128() defined in the upstream 2022-08-22 21:09:26 +09:00
bitbears-dev
89cb0d8e91
Merge branch 'wader:master' into wasm-support 2022-08-22 21:01:19 +09:00
@0xb17bea125
289ddf59fb
wasm: fix lint issues 2022-08-22 21:00:54 +09:00
@0xb17bea125
00869b3701
wasm: avoid race condition 2022-08-22 21:00:02 +09:00
Mattias Wadman
f24d685a5b xml: Keep track of default namespace and skip it element names 
Refactor element name handling a bit, return it instead, feels nicer.
 2022-08-21 10:11:59 +02:00
Mattias Wadman
326dada7ca decode: Add LEB128 readers 2022-08-20 12:05:44 +02:00
Mattias Wadman
3290c8a46b
Merge pull request #386 from wader/gopacket-update-sll2 
gopacket: Switch/update to new fork, remove SLL2 hack
 2022-08-20 09:37:23 +02:00
@0xb17bea125
7401d141e8
wasm: use WRITE_ACTUAL=1 to generate .fqtest files 2022-08-20 16:11:33 +09:00
@0xb17bea125
e1691dec4f
wasm: remove unused function 2022-08-20 16:04:18 +09:00
@0xb17bea125
fead68de50
wasm: tidy up 2022-08-20 15:50:14 +09:00
@0xb17bea125
cbb5a8ed00
wasm: further simplification 2022-08-20 15:25:11 +09:00
@0xb17bea125
ae08bf70d4
wasm: use s.ActualU() instead of s.Actual.(uint64) 2022-08-20 15:02:52 +09:00
@0xb17bea125
3298d181fe
wasm: to be able to probe 2022-08-20 14:55:16 +09:00
Mattias Wadman
394717cae8 gopacket: Switch/update to new fork, remove SLL2 hack 
SLL2 support merged upstream https://github.com/gopacket/gopacket/pull/1

This also cuts down a bit on some indirect deps
 2022-08-20 01:25:03 +02:00
@0xb17bea125
b0f3fec855
wasm: remove nolint:unparam which is no longer needed 2022-08-19 18:49:19 +09:00
@0xb17bea125
2037b86abd
wasm: use map, not switch 2022-08-19 18:47:50 +09:00
@0xb17bea125
0ad5a8ec2f
wasm: use underscores for symbol values 2022-08-19 14:50:56 +09:00
@0xb17bea125
63f4a7267c
wasm: use scalar.UToSymStr for simplicity 2022-08-19 14:35:16 +09:00
@0xb17bea125
3fca7cc09a
wasm: fix lint issues 2022-08-19 09:05:46 +09:00
Mattias Wadman
71900c2a9d xml: Correctly sort if one #seq is found and allow negative seq numbers 2022-08-18 23:08:31 +02:00
Takashi Oguma
934ed9a809
wasm: initial version 2022-08-18 21:40:30 +09:00
Martijn van Beurden
fb583e2cc0 Add 32 bps FLAC to test 2022-08-18 14:27:43 +02:00
Martijn van Beurden
c1d5b2b1c3 Add sample size entry to list for 32bps flac streams 2022-08-18 14:16:55 +02:00
Mattias Wadman
74c7dc4eaf pcap: Add ns support and add header field 2022-08-18 14:09:42 +02:00
Mattias Wadman
da6b9413fd
Merge pull request #381 from wader/flac-fix-gen-script 
flac: Make gen script generate correct fqtest files
 2022-08-18 11:48:00 +02:00
Mattias Wadman
8a19978b2f flac: Make gen script generate correct fqtest files 2022-08-18 11:37:07 +02:00
Mattias Wadman
2bfbe9a94a flac_frame: Cleanup some dev lefterovers and todos 2022-08-18 11:13:45 +02:00
Mattias Wadman
b6c4c27688
Merge pull request #379 from ktmf01/fix-flac-raw-partition 
Fix decoding of FLAC raw entropy partition
 2022-08-18 11:04:00 +02:00
Martijn van Beurden
3f209c46ca Fix decoding of FLAC raw entropy partition 2022-08-18 10:11:57 +02:00
Mattias Wadman
cf15661e9c mp3_frame: Add LSF support and fix incorrect main data handling 
Some mpeg versions use LSF (low sampling frequency) when encoding
low sample rates which changes a bit how the side info is layed out.

Also skip trying to separate data and other_data as after reading up
a bit on how main data begin and bit reservoir works i think it was done
incorrectly. To know the real audio data length i think you need to decode
the huffman tables. Instead just have a "audio_data" field which are the
bits with audo data for current and possibly following frames.
 2022-08-17 21:36:39 +02:00
Mattias Wadman
701c67c136 jsonl: Add decoder, also tojsonl encoder 2022-08-15 19:43:59 +02:00
Mattias Wadman
3623eac365 yaml: Error on trailing yaml/json 
Turns our yaml is a superset of json
 2022-08-15 19:14:25 +02:00
Mattias Wadman
0d44b9376c tar: Some number fields can be empty 2022-08-15 14:57:16 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
2e407386ae matroska: Strip newlines in description 
Will mess up dump formatting
 2022-08-12 14:46:35 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?> 
Also more context in error messages and refactor trim function
 2022-08-11 23:48:29 +02:00
Mattias Wadman
c233215aba dns: Rename isTCP to hasLengthHeader 2022-08-10 13:35:01 +02:00
Mattias Wadman
46dca8cdc5 dns: Don't use dns (udp) format for tcp also 
Fixes issue with TCP streams being probed as dns over TCP
 2022-08-09 18:15:34 +02:00
Mattias Wadman
29005c70bf interp,dump: Show address bar for root, nested roots and on format change 
Also increase indent for nested roots a bit.

Makes it a bit easier to read i think.
 2022-08-09 18:09:41 +02:00
Mattias Wadman
4dec7ff64b
Merge pull request #364 from wader/decode-refactor-scalar-type-assert 
decode: Refactor to use scalar type assert helper
 2022-08-08 18:17:51 +02:00
Mattias Wadman
fbe7bc9af6
Merge pull request #363 from wader/zip-correct-max-peek 
zip: Correctly limit max EOCD find
 2022-08-08 18:15:55 +02:00
Mattias Wadman
502f451cc3 decode: Refactor to use scalar type assert helper 2022-08-08 18:09:49 +02:00
Mattias Wadman
d607bee180 zip: Correctly limit max EOCD find 2022-08-08 18:04:48 +02:00
Mattias Wadman
b2d4e6d94d macho: Decode cmd symtab symbols 2022-08-08 18:02:59 +02:00
Mattias Wadman
545dac8c86 test: Update tests, go 1.19 uses \xff instead of \u00ff 2022-08-05 00:37:28 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
b135701ccd
Merge pull request #359 from wader/zip-skip-header-check 
zip: Skip header assert as there are zip files with other things appe…
 2022-08-04 12:53:09 +02:00
Mattias Wadman
19b70899ca zip: Skip header assert as there are zip files with other things appended 2022-08-04 12:33:33 +02:00
Mattias Wadman
f4e0137244 xml,html: Always include attrs and children in array mode 
Was hard to use when you dont know what indexes things will have. They are sill optioal for toxml
 2022-08-04 11:44:59 +02:00
Mattias Wadman
725c8e83ab macho: Split into macho/macho_fat, fix offset issue and add string decoding 
Split fat macho into own decoder macho_fat. This also fixes issue with section
offset etc not being correct as they are from the start of each embedded file.

Make all address and offset field be in hex.

Decode __cstring, __ustring and __cfstring sections.

Fix LC_ENCRYPTION_INFO_64 missing pading issue.

Skip ranging for __bss and __common as they dont have any data in the file.

Simplifed magic handling a bit and add symbols.

Simplified state struct field, had redudant struct.
 2022-08-02 14:36:53 +02:00
Mattias Wadman
888a471245
Merge pull request #350 from wader/bump-github-golangci-lint-1.47.3 
Update github-golangci-lint from 1.47.2 to 1.47.3
 2022-08-01 18:54:35 +02:00
Mattias Wadman
bc6cffde61 lint,decode,fuzz:: Fix nilerr warnings, one real one should be ignored for now 2022-08-01 18:47:24 +02:00
Mattias Wadman
840292ba6e decode: Simplify compound range sort behaviour 
Doing it thru a propery in the decode fn feels a bit hidden and will
also not get set on failed decoding.

Now array is not range sorted, logic is you care about index number and ordering.
Struct is range sorted as you will prefer to fields by name.
 2022-08-01 17:17:54 +02:00
Mattias Wadman
3613b6d4dd elf: Remove redundant program_header struct 2022-07-29 13:41:21 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
0326eeb697 asn1_ber,goreleaser: Convert CRLF to LF in asn1 xml tests, hopefully fixes goreleaer workspace dirty error 2022-07-24 12:38:24 +02:00
Mattias Wadman
e843e31b20
Merge pull request #336 from wader/avro-fix-missing-schema-crash 
avro_ocf: Fix panic on missing meta schema
 2022-07-24 11:24:51 +02:00
Mattias Wadman
5d67df47e2 avro_ocf: Fix panic on missing meta schema 2022-07-23 22:20:03 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions 
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
 2022-07-23 21:48:45 +02:00
Mattias Wadman
8b5cc89641
Merge pull request #334 from wader/hevc_vps-max-vps-layers 
hevc_vps,fuzz: Error on too many vps layers
 2022-07-22 22:56:31 +02:00
Mattias Wadman
fd302093e1 hevc_vps,fuzz: Error on too many vps layers 2022-07-22 22:48:25 +02:00
Mattias Wadman
e0a391de56
Merge pull request #332 from wader/elf-error-on-large-strtab 
elf,fuzz: Error on too large string table
 2022-07-22 22:47:33 +02:00
Mattias Wadman
40481f6692 elf,fuzz: Error on too large string table 
Fixes alloc/hang of giant string
 2022-07-22 22:35:45 +02:00
Mattias Wadman
75169a65b5 asn1: Add regression test for range decode fix ##330 2022-07-22 22:32:37 +02:00
Mattias Wadman
cec2dede92
Merge pull request #330 from wader/mp4-dont-sort-samples 
mp4: Don't range sort samples, keep sample table order
 2022-07-22 16:46:38 +02:00
Mattias Wadman
c6d0d89c06 mp4: Don't range sort samples, keep sample table order 2022-07-22 16:35:00 +02:00
Mattias Wadman
7537905593
Merge pull request #325 from wader/unbreak-fuzz-test 
fuzz: Replace built tag with FUZZTEST env and use new interp api
 2022-07-20 18:31:31 +02:00
Mattias Wadman
2464ebc238 fuzz: Replace built tag with FUZZTEST env and use new interp api 
Build tag hides build errors
 2022-07-20 18:24:07 +02:00
Mattias Wadman
78aa96b0ac dev: Cleanup some code to fix a bunch of new linter warnings 2022-07-19 18:56:09 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
f5be5180f0 interp: Cleanup and clarify some format naming 2022-07-16 22:04:21 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers 
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
 2022-07-16 19:24:13 +02:00
Mattias Wadman
ea2cc3c2ec mp4: Don't decode encrypted samples 2022-07-07 19:02:10 +02:00
Mattias Wadman
64f3e5c717 fairplay: Add basic SPC decoder and PSSH system id 2022-07-07 17:11:55 +02:00
Mattias Wadman
417255b72b bitcoin: Add blkdat, block, transcation and script decoder 2022-07-04 09:59:51 +02:00
Mattias Wadman
f66a359c39 elf: Use correct offset to dynamic linking string table 
String table used for "needed" tags (the string seen with ldd etc) was using address as
offset which seem to work as it's usually the same as correct offset. But this is not true
for some ELFs. Then we have to look up offset by finding the section by address and then use
its offset for the string table. Hope this is the correct way.

Not sure how to produce a small test for this.
 2022-06-30 23:30:56 +02:00
Mattias Wadman
a6a97136b7 decode: Cleanup Try<f>/<f> pairs 
Decode API design is that Try* returns error non-Try panics.
Also rename Must* as they should panic and introduce some new Try* functions
that were missing.
 2022-06-30 12:13:46 +02:00
Mattias Wadman
0863374f8c doc: Correct bencode spec URL 
Also rename _tojq value_sep to object_sep and fix formats_diagram.jq to use tomd5
 2022-06-13 18:49:34 +02:00
Mattias Wadman
7d25fbfdbb mp4: Properly use trun data offset 
Each trun has it's own data offset, before the last offset was wrongly used
for all truns. Could also cause sample ranges to be beyond EOF.

tenc: Decode default constant iv

Fixes #292
 2022-06-07 17:43:45 +02:00
Mattias Wadman
c92f4f13c1 matroska: Update ebml_matroska.xml and allow unknown ids 
Update spec from https://raw.githubusercontent.com/ietf-wg-cellar/matroska-specification/master/ebml_matroska.xml
Allow unknown ids, decodeas "unknown" type with just binary data
Change "type" field to just a string, there is no type id number.
Make all type field symbols consistently lowercase
 2022-06-06 21:43:42 +02:00
Mattias Wadman
6b00297edd mp4,senc: Refactor current track/moof tracking and add senc box support 
Keep track of curren using box data stack instead of refrences in the context.
 2022-06-03 21:13:49 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes 
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
 2022-05-28 16:31:20 +02:00
Mattias Wadman
7b27e506f1 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio 
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.

Add test with playread and widevine pssh boxes

Related to #282
 2022-05-28 13:42:38 +02:00
Mattias Wadman
aa694e3f20 zip: s/Decompress/Uncompress/ 2022-05-26 19:36:06 +02:00
Mattias Wadman
47350e46a6 zip: Add uncompress=false test and some docs 2022-05-26 16:39:01 +02:00
Mattias Wadman
8021492197 interp: help: Fix incorrect options example 2022-05-26 12:52:51 +02:00
Mattias Wadman
e641274487 zip: Add zip64 support and uncompress option 
{uncompress:boolean} uncompress and probe, default true
 2022-05-26 12:46:50 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage 
Make cwd for a test script the directory where the script is.
Use relative paths
 2022-05-21 20:03:25 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18 
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
 2022-05-20 15:23:16 +02:00
Mattias Wadman
7bfa88cdb5
Merge pull request #271 from wader/mp3-cleanup 
mp3: Use d.FieldValueU and some cleanup
 2022-05-20 13:15:31 +02:00
Mattias Wadman
3780375d76 mp3: Use d.FieldValueU and some cleanup 2022-05-20 12:50:40 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now 
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
 2022-05-20 12:38:43 +02:00
Mattias Wadman
ba844eb06a mp4: Skip fields for pcm samples for now 
Will produce one field per sample which is a bit too much.
Maybe in some future fq can have smart array types that could handle this better.

Also add data_format per track to make it easier to understand things.
 2022-05-19 16:00:42 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots 
Hopefully makes it a bit easier to read and spot
 2022-05-11 00:00:39 +02:00
Mattias Wadman
903c7198b4
Merge pull request #258 from wader/mp4-mvhd-tkhd-mdhd-mehd-version1 
mp4: mvhd,tkhd,mdhd,mehd: Add version 1 support
 2022-05-10 00:48:47 +02:00
Mattias Wadman
2e328180ae mp4: mvhd,tkhd,mdhd,mehd: Add version 1 support 
Also fix incorrect mehd decode
 2022-05-10 00:21:04 +02:00
Mattias Wadman
efa5e23aaa icc_profile: Correctly clamp align padding on EOF 
Also code was mixing bytes and bit units, how could this have worked?
 2022-05-09 13:14:20 +02:00
Mattias Wadman
8228ecaeee mp4: Add track id field and add track for tfhd with unseen track_id 
Fragmened mp4s can have no tkhd box or tkhd box with a track_id that does not match
any tfhd box track_id. Make is possible to see samples for these fragmented tracks
ny adding tracks for unseen track_ids.

Also add id field for tracks.
 2022-05-08 19:02:21 +02:00
Mattias Wadman
76161a1b99 scalar,mp4,gzip,tar: Add timestamp to description 
Scalar api for this needs some work
 2022-05-07 13:15:13 +02:00
Mattias Wadman
9133f0e527 scalar: Add *Fn type to map value and clearer naming 
Should replace most of this with generics at some point
 2022-05-07 12:46:34 +02:00
Mattias Wadman
34cf5442b3 tcp: Split into client/server structs and add skipped_bytes and has_start/end per direction 
Feels clenaer and removes _client/server field prefixes
 2022-05-06 16:52:18 +02:00