1
1
mirror of https://github.com/wader/fq.git synced 2024-12-23 13:22:58 +03:00
Commit Graph

58 Commits

Author SHA1 Message Date
David McDonald
34c560c2f5 Merge branch 'master' of https://github.com/wader/fq into macos_bookmark 2022-12-05 22:01:44 -06:00
@0xb17bea125
2ee6360b97 support tzif (time zone information format) 2022-12-04 21:02:39 +09:00
David McDonald
63a3ca20c4 removes underscore from apple_bookmark package name 2022-12-01 23:31:57 -06:00
David McDonald
4a28e44f44 changes decoder package name from bookmark to apple_bookmark 2022-12-01 23:31:57 -06:00
David McDonald
5146f28dbb fixes broken test for all.fqtest 2022-12-01 23:31:57 -06:00
David McDonald
0287ffa4dc decoding well but torepr needs work 2022-12-01 23:31:56 -06:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
0cf46e1196 wav,avi,avc_au: Add avi decoder and refactor wav decoder
avi and wav now share  common riff decode code
avc_au decoder can now handle annexb format
2022-11-19 13:11:27 +01:00
Mattias Wadman
c3a0686c9e mp3_frame_tags: Refactor and rename xing format to mp3_frame_tags
Make more sense to use a more general name when there is both info and xing
and possibly vbri in the future. Also both xing and info seems to have extension
fields

Remove some old tests that are not really needed anymore.
2022-11-16 17:32:03 +01:00
Mattias Wadman
421b2b3071 bplist: Fix unknown field for singletons and add torepr tests 2022-10-04 14:30:54 +02:00
David McDonald
a77cec921f Added documentation and tests, fixed bad date parsing 2022-09-23 20:23:23 -05:00
David McDonald
d784db69bc Adds support for Apple Binary Plist, version 00 2022-09-23 20:21:32 -05:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
725ab1b17b doc,html,xml: Add more documentation and examples 2022-09-20 17:35:27 +02:00
Mattias Wadman
fe64530e11 csv: Add tsv and header example
Also fix string esacpe in fuzz options parser
2022-09-15 01:16:45 +02:00
Mattias Wadman
1385b5d0ba wasm: Add some documentation 2022-09-12 11:53:48 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
00a50662ea markdown: Add decoder 2022-09-10 18:26:45 +02:00
Mattias Wadman
004406de65 bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat
Makes bitcoin_blkdat fails fast as it is part of probe group.
Speeds up reading a big JSON file etc.
2022-08-26 12:07:53 +02:00
Mattias Wadman
095e1161b7 xml: Switch from "-" to "@" as attribute prefix and make it an option
Seems to be more common.
2022-08-25 17:15:23 +02:00
Mattias Wadman
8fc43533a9 prores_frame: Add basic container and frame header decoder 2022-08-24 19:33:35 +02:00
@0xb17bea125
f73965d2a3
wasm: add wasm to probe list 2022-08-23 21:19:07 +09:00
Takashi Oguma
934ed9a809
wasm: initial version 2022-08-18 21:40:30 +09:00
Mattias Wadman
701c67c136 jsonl: Add decoder, also tojsonl encoder 2022-08-15 19:43:59 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
725c8e83ab macho: Split into macho/macho_fat, fix offset issue and add string decoding
Split fat macho into own decoder macho_fat. This also fixes issue with section
offset etc not being correct as they are from the start of each embedded file.

Make all address and offset field be in hex.

Decode __cstring, __ustring and __cfstring sections.

Fix LC_ENCRYPTION_INFO_64 missing pading issue.

Skip ranging for __bss and __common as they dont have any data in the file.

Simplifed magic handling a bit and add symbols.

Simplified state struct field, had redudant struct.
2022-08-02 14:36:53 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
64f3e5c717 fairplay: Add basic SPC decoder and PSSH system id 2022-07-07 17:11:55 +02:00
Mattias Wadman
417255b72b bitcoin: Add blkdat, block, transcation and script decoder 2022-07-04 09:59:51 +02:00
Mattias Wadman
0863374f8c doc: Correct bencode spec URL
Also rename _tojq value_sep to object_sep and fix formats_diagram.jq to use tomd5
2022-06-13 18:49:34 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
aa694e3f20 zip: s/Decompress/Uncompress/ 2022-05-26 19:36:06 +02:00
Mattias Wadman
47350e46a6 zip: Add uncompress=false test and some docs 2022-05-26 16:39:01 +02:00
Mattias Wadman
8021492197 interp: help: Fix incorrect options example 2022-05-26 12:52:51 +02:00
Mattias Wadman
e641274487 zip: Add zip64 support and uncompress option
{uncompress:boolean} uncompress and probe, default true
2022-05-26 12:46:50 +02:00
Mattias Wadman
1aaaefb0ac wav,bencode,mpeg_ps_packet,id3v1: Random fixes
wav: Rest of file should description not symbolic value
bencode: Remove redundant torepr example
mpeg_ps_packet,id3v1: Remove whitespace in symbolic value
2022-05-05 19:45:05 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
d8aaf30345 rtmp,amf0: Add decoders
Basic RTMP support, used via tcp_stream group (pcap etc) or manually.
Basic AMF0 support, mostly what is used in RTMP.
2022-04-01 13:07:20 +02:00
siddik.acil
b78ed02fb3 macho: barebones decoder impl 2022-02-21 20:31:12 +01:00
Xentripetal
5b11385ea4 Pull latest from wader 2022-02-07 17:35:33 -06:00
Mattias Wadman
0829c167cc asn1_ber: Add decoder
Basic support, can also do CER and DER but without any extra validation.
No schema support.

Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.

Related to #20
2022-02-06 22:13:03 +01:00
Xentripetal
16849c8f35 Update linting 2022-01-19 19:44:42 -06:00
Xentripetal
788fd0258f Pull latest from wader 2022-01-13 20:59:56 -06:00
Mattias Wadman
0b0f28e966 cbor: Add decoder
Does not decode sematic tag types
Also fixes broken float16 support

Fixes #71
2022-01-12 17:56:03 +01:00
Mattias Wadman
a6bf62ce19 msgpack: Add decoder
Fixes #25
2022-01-10 19:33:09 +01:00
Mattias Wadman
af8e7efc61 bencode: Add decoder
Closes #63
2022-01-10 10:52:14 +01:00
Mattias Wadman
628f0f403d bson: Add decoder
Only supports basic types for now
2022-01-06 00:51:37 +01:00
Mattias Wadman
681dbc26b7 elf,ar: Add ar decoder, improved elf decoder
Add x86, 386, arm64, arm v6, arm v7 linux elf test files
2022-01-05 22:05:09 +01:00
Xentripetal
7a8e3ca26e Hook into registry, add codecs 2021-12-28 13:05:10 -06:00