1
1
mirror of https://github.com/wader/fq.git synced 2024-11-24 11:16:09 +03:00
Commit Graph

362 Commits

Author SHA1 Message Date
Mattias Wadman
545dac8c86 test: Update tests, go 1.19 uses \xff instead of \u00ff 2022-08-05 00:37:28 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
b135701ccd
Merge pull request #359 from wader/zip-skip-header-check
zip: Skip header assert as there are zip files with other things appe…
2022-08-04 12:53:09 +02:00
Mattias Wadman
19b70899ca zip: Skip header assert as there are zip files with other things appended 2022-08-04 12:33:33 +02:00
Mattias Wadman
f4e0137244 xml,html: Always include attrs and children in array mode
Was hard to use when you dont know what indexes things will have. They are sill optioal for toxml
2022-08-04 11:44:59 +02:00
Mattias Wadman
725c8e83ab macho: Split into macho/macho_fat, fix offset issue and add string decoding
Split fat macho into own decoder macho_fat. This also fixes issue with section
offset etc not being correct as they are from the start of each embedded file.

Make all address and offset field be in hex.

Decode __cstring, __ustring and __cfstring sections.

Fix LC_ENCRYPTION_INFO_64 missing pading issue.

Skip ranging for __bss and __common as they dont have any data in the file.

Simplifed magic handling a bit and add symbols.

Simplified state struct field, had redudant struct.
2022-08-02 14:36:53 +02:00
Mattias Wadman
888a471245
Merge pull request #350 from wader/bump-github-golangci-lint-1.47.3
Update github-golangci-lint from 1.47.2 to 1.47.3
2022-08-01 18:54:35 +02:00
Mattias Wadman
bc6cffde61 lint,decode,fuzz:: Fix nilerr warnings, one real one should be ignored for now 2022-08-01 18:47:24 +02:00
Mattias Wadman
840292ba6e decode: Simplify compound range sort behaviour
Doing it thru a propery in the decode fn feels a bit hidden and will
also not get set on failed decoding.

Now array is not range sorted, logic is you care about index number and ordering.
Struct is range sorted as you will prefer to fields by name.
2022-08-01 17:17:54 +02:00
Mattias Wadman
3613b6d4dd elf: Remove redundant program_header struct 2022-07-29 13:41:21 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
0326eeb697 asn1_ber,goreleaser: Convert CRLF to LF in asn1 xml tests, hopefully fixes goreleaer workspace dirty error 2022-07-24 12:38:24 +02:00
Mattias Wadman
e843e31b20
Merge pull request #336 from wader/avro-fix-missing-schema-crash
avro_ocf: Fix panic on missing meta schema
2022-07-24 11:24:51 +02:00
Mattias Wadman
5d67df47e2 avro_ocf: Fix panic on missing meta schema 2022-07-23 22:20:03 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
8b5cc89641
Merge pull request #334 from wader/hevc_vps-max-vps-layers
hevc_vps,fuzz: Error on too many vps layers
2022-07-22 22:56:31 +02:00
Mattias Wadman
fd302093e1 hevc_vps,fuzz: Error on too many vps layers 2022-07-22 22:48:25 +02:00
Mattias Wadman
e0a391de56
Merge pull request #332 from wader/elf-error-on-large-strtab
elf,fuzz: Error on too large string table
2022-07-22 22:47:33 +02:00
Mattias Wadman
40481f6692 elf,fuzz: Error on too large string table
Fixes alloc/hang of giant string
2022-07-22 22:35:45 +02:00
Mattias Wadman
75169a65b5 asn1: Add regression test for range decode fix ##330 2022-07-22 22:32:37 +02:00
Mattias Wadman
cec2dede92
Merge pull request #330 from wader/mp4-dont-sort-samples
mp4: Don't range sort samples, keep sample table order
2022-07-22 16:46:38 +02:00
Mattias Wadman
c6d0d89c06 mp4: Don't range sort samples, keep sample table order 2022-07-22 16:35:00 +02:00
Mattias Wadman
7537905593
Merge pull request #325 from wader/unbreak-fuzz-test
fuzz: Replace built tag with FUZZTEST env and use new interp api
2022-07-20 18:31:31 +02:00
Mattias Wadman
2464ebc238 fuzz: Replace built tag with FUZZTEST env and use new interp api
Build tag hides build errors
2022-07-20 18:24:07 +02:00
Mattias Wadman
78aa96b0ac dev: Cleanup some code to fix a bunch of new linter warnings 2022-07-19 18:56:09 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
f5be5180f0 interp: Cleanup and clarify some format naming 2022-07-16 22:04:21 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
ea2cc3c2ec mp4: Don't decode encrypted samples 2022-07-07 19:02:10 +02:00
Mattias Wadman
64f3e5c717 fairplay: Add basic SPC decoder and PSSH system id 2022-07-07 17:11:55 +02:00
Mattias Wadman
417255b72b bitcoin: Add blkdat, block, transcation and script decoder 2022-07-04 09:59:51 +02:00
Mattias Wadman
f66a359c39 elf: Use correct offset to dynamic linking string table
String table used for "needed" tags (the string seen with ldd etc) was using address as
offset which seem to work as it's usually the same as correct offset. But this is not true
for some ELFs. Then we have to look up offset by finding the section by address and then use
its offset for the string table. Hope this is the correct way.

Not sure how to produce a small test for this.
2022-06-30 23:30:56 +02:00
Mattias Wadman
a6a97136b7 decode: Cleanup Try<f>/<f> pairs
Decode API design is that Try* returns error non-Try panics.
Also rename Must* as they should panic and introduce some new Try* functions
that were missing.
2022-06-30 12:13:46 +02:00
Mattias Wadman
0863374f8c doc: Correct bencode spec URL
Also rename _tojq value_sep to object_sep and fix formats_diagram.jq to use tomd5
2022-06-13 18:49:34 +02:00
Mattias Wadman
7d25fbfdbb mp4: Properly use trun data offset
Each trun has it's own data offset, before the last offset was wrongly used
for all truns. Could also cause sample ranges to be beyond EOF.

tenc: Decode default constant iv

Fixes #292
2022-06-07 17:43:45 +02:00
Mattias Wadman
c92f4f13c1 matroska: Update ebml_matroska.xml and allow unknown ids
Update spec from https://raw.githubusercontent.com/ietf-wg-cellar/matroska-specification/master/ebml_matroska.xml
Allow unknown ids, decodeas "unknown" type with just binary data
Change "type" field to just a string, there is no type id number.
Make all type field symbols consistently lowercase
2022-06-06 21:43:42 +02:00
Mattias Wadman
6b00297edd mp4,senc: Refactor current track/moof tracking and add senc box support
Keep track of curren using box data stack instead of refrences in the context.
2022-06-03 21:13:49 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
7b27e506f1 mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio
PSSH decoding probably got broken in some refactoring. system id is 16 bytes,
also scalar mapper should use cloned reader to not affect reader which in the case
broke reading system id bytes.

Add test with playread and widevine pssh boxes

Related to #282
2022-05-28 13:42:38 +02:00
Mattias Wadman
aa694e3f20 zip: s/Decompress/Uncompress/ 2022-05-26 19:36:06 +02:00
Mattias Wadman
47350e46a6 zip: Add uncompress=false test and some docs 2022-05-26 16:39:01 +02:00
Mattias Wadman
8021492197 interp: help: Fix incorrect options example 2022-05-26 12:52:51 +02:00
Mattias Wadman
e641274487 zip: Add zip64 support and uncompress option
{uncompress:boolean} uncompress and probe, default true
2022-05-26 12:46:50 +02:00
Mattias Wadman
377af133e0 fqtest: Cleanup path usage
Make cwd for a test script the directory where the script is.
Use relative paths
2022-05-21 20:03:25 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
7bfa88cdb5
Merge pull request #271 from wader/mp3-cleanup
mp3: Use d.FieldValueU and some cleanup
2022-05-20 13:15:31 +02:00
Mattias Wadman
3780375d76 mp3: Use d.FieldValueU and some cleanup 2022-05-20 12:50:40 +02:00
Mattias Wadman
2dc509ab2f interp: Refactor dump and revert #259 for now
Causes address bar to not be shown in some cases.
Will have to rethink and redo the whole dump thing somehow.
2022-05-20 12:38:43 +02:00
Mattias Wadman
ba844eb06a mp4: Skip fields for pcm samples for now
Will produce one field per sample which is a bit too much.
Maybe in some future fq can have smart array types that could handle this better.

Also add data_format per track to make it easier to understand things.
2022-05-19 16:00:42 +02:00
Mattias Wadman
5109df4a50 interp: dump: Show address bar for nested roots
Hopefully makes it a bit easier to read and spot
2022-05-11 00:00:39 +02:00