wader/fq
1
1
Fork 0
mirror of https://github.com/wader/fq.git synced 2024-11-29 12:14:17 +03:00
Code Issues Projects Releases Wiki Activity
2,770 Commits 33 Branches 23 Tags 59 MiB
a4372cb127
Commit Graph

1059 Commits

Author SHA1 Message Date
Ronsor
14f233d21e moc3: update tests 2023-09-18 18:22:50 -07:00
Ronsor
f7eb0279b3 moc3: Fix field order in blend_shape_keyform_bindings structure, version detection in count_info 2023-09-18 18:21:07 -07:00
Mattias Wadman
0699c80bac mp3_frame_xing: Detect lame ext more similar to ffmpeg and mediainfo 2023-09-08 12:09:35 +02:00
Mattias Wadman
0efe5a2c64 id3v2: Handle W000-WZZZ,W00-WZZ URL frames 2023-09-04 15:41:57 +02:00
Mattias Wadman
bedd719b3a mp4: Decode emsg box 2023-08-31 17:33:18 +02:00
Mattias Wadman
312d8078a7 mp4: Decode cslg box 2023-08-31 15:00:29 +02:00
Mattias Wadman
e50028ac00 mp4,mpeg_es: Decode iods box and MP4_IOD_Tag OD 2023-08-30 14:57:41 +02:00
Mattias Wadman
0cefc46b8b golangci: Fix gosec aliasing warnings 2023-08-21 19:26:21 +02:00
Mattias Wadman
c503bc1313 html: Add forgotten re test 2023-08-21 19:26:00 +02:00
Mattias Wadman
f15f9bc1a4 doc,moc3,caff: Add author and regenerate docs 2023-08-21 10:04:37 +02:00
Mattias Wadman
2eae4c2fd5
Merge pull request #747 from Ronsor/live2d-formats 
moc3 and caff: add decoder
 2023-08-20 19:06:50 +02:00
Ronsor
833b06363c moc3: update test data 2023-08-20 09:55:59 -07:00
Ronsor
fac1e68395 moc3: count_info: extra space is reserved, not normal alignment/padding 2023-08-19 21:09:50 -07:00
Ronsor
e424e2932e moc3: eliminate gaps and properly handle padding, fix version 5 format decoding 2023-08-19 21:03:54 -07:00
Ronsor
b3759de7f0 caff: run go fmt 2023-08-19 07:18:33 -07:00
Ronsor
013cc2f68f caff: eliminate gaps and specify unused fields 2023-08-19 07:07:10 -07:00
Ronsor
4dd0f6d88b caff: run go fmt 2023-08-18 20:02:26 -07:00
Ronsor
d3073c6459 moc3: add test data for new version 5 2023-08-18 20:02:12 -07:00
Ronsor
fa1157221b caff: obfuscation key is a signed integer, add test data 
The obfuscation key is actually a signed integer, and thanks to two's complement
arithmetic and sign extension, this is actually significant.
 2023-08-18 19:45:09 -07:00
Ronsor
03ba71b67f moc3: add support for version 5 2023-08-18 17:42:17 -07:00
Ronsor
ce40fd19d7 moc3: consistency - scales array contains value elements, not scale elements 2023-08-18 12:02:55 -07:00
Ronsor
3caf34e32a moc3: nicer tree structure, use more meaningful names for array elements 2023-08-18 11:42:16 -07:00
Ronsor
6a3fecd294 caff: include uncompressed bits for proper decompressed entries that can't be decoded as a format 2023-08-18 09:25:10 -07:00
Ronsor
441fcd09c8 moc3, caff: update tests and README 2023-08-18 07:41:17 -07:00
Ronsor
23e660f4ca caff: minor formatting changes 2023-08-18 07:40:54 -07:00
Ronsor
6d10a25bb4 moc3: update certain array element names, explicitly mark unused or reserved space 2023-08-18 07:40:31 -07:00
Mattias Wadman
456a6a4fb5 protobuf_widevine: Make protection_scheme constants less magic 2023-08-18 11:24:57 +02:00
Ronsor
20f02e79b7 moc3: remove dead code 2023-08-17 20:13:35 -07:00
Ronsor
092662ec85 moc3: initial implementation 2023-08-17 20:07:55 -07:00
Ronsor
29084e358e caff: remove dead code 2023-08-17 13:19:28 -07:00
Ronsor
da41a8d311 caff: initial implementation 2023-08-17 13:17:01 -07:00
Mattias Wadman
4ed8c6df18
Merge pull request #738 from wader/gomod-update 
gomod: Update x/exp and gomarkdown
 2023-08-09 14:57:28 +02:00
Mattias Wadman
0d0147643a gomod: Update x/exp and gomarkdown 
Add cmp package from go 1.21 to have cmp.Compare to make sort easier
 2023-08-09 14:36:51 +02:00
Mattias Wadman
c07673a07a matroska: Update spec and regenerate 2023-08-09 13:59:21 +02:00
Mattias Wadman
56fec2aa7e elf: Fix broken static and segfault tests 2023-08-02 22:13:33 +02:00
Mattias Wadman
44f0060205 dev,jq: Reformat jq code to look more the same 2023-07-27 13:21:04 +02:00
Mattias Wadman
97194ad8bc mp4: Nicer major brand and handle some qt brand short strings better 2023-07-26 12:15:51 +02:00
Babz
3af0462c68 luajit: file null terminator: raw bits, validate 2023-07-25 18:34:44 +02:00
bump
8e75dc9bc8 Update gomod-BurntSushi/toml to 1.3.2 from 1.2.1 
Source diff 1.2.1..1.3.2 https://github.com/BurntSushi/toml/compare/v1.2.1..v1.3.2
 2023-07-05 17:36:12 +02:00
Mattias Wadman
8c75509e13 wav: Decode bext chunk 2023-07-03 11:17:50 +02:00
Mattias Wadman
5605ebe5c5
Merge pull request #709 from dlatchx/luajit 
luajit: add decoder
 2023-06-22 01:45:37 +02:00
Babz
c42fb9e769 luajit: typo 2023-06-22 01:30:28 +02:00
Babz
64c92da68b luajit: remove unused variable 2023-06-22 01:27:20 +02:00
Babz
5d9a08c6e9 luajit: fix regression: (u64 vs i64) 2023-06-21 23:57:59 +02:00
Babz
dd594f47eb luajit: tests: rename lua source file 2023-06-21 23:53:55 +02:00
Babz
751ee5a328 luajit: explain LuaJITDecodeKNum, fix negative in bug 2023-06-21 23:53:44 +02:00
Babz
64c11bedfa luajit: improve debuginfo decoding 2023-06-21 22:33:23 +02:00
Babz
eb819dd479 luajit: tests: improve coverage 2023-06-21 20:53:47 +02:00
Babz
441d246de6 luajit: standardize field names (key/value/type ect.) 2023-06-21 20:42:15 +02:00
Babz
3561c08a8c luajit: fallbackUintMapSymStr 2023-06-21 20:34:56 +02:00
Babz
c3a123ad84 luajit: remove unecessary dependency 2023-06-21 18:47:13 +02:00
Sergei Kuzmin
e869d8af03 sgpd box entries parsing 
Previously entries were parsed as raw bytes, now KIDs/IVs are
explicitely parsed.
 2023-06-20 21:57:27 -07:00
Babz
93c96965e0 luajit: add to probe group 2023-06-21 01:41:03 +02:00
Babz
a83576a83a luajit: clarify description 2023-06-21 01:39:23 +02:00
Babz
b24063be02 luajit: *.fqtest: add comments for generating .luac from source 2023-06-21 01:25:35 +02:00
Babz
52ce81817f luajit: split in smaller decode functions 2023-06-21 01:02:38 +02:00
Babz
bdf158be9e luajit: add luajit.md 2023-06-21 00:17:27 +02:00
Babz
715f850d50 luajit: opcodes: implement scalar.UintMapper 2023-06-20 23:55:37 +02:00
Babz
08ae661c15 luajit: use UTF8 strings 2023-06-20 23:33:46 +02:00
Babz
32300a3ffa luajit: check binary.Read() error 2023-06-20 23:31:35 +02:00
Babz
e44f5c00b8 luajit: magic number: raw bits, check with assert 2023-06-20 23:28:34 +02:00
Babz
29ab66b3eb luajit: lowercase flags 2023-06-20 23:11:17 +02:00
Babz
db2e621468 go fmt 2023-06-20 23:02:31 +02:00
Babz
1afdf8b1cf luajit: initial support 2023-06-20 18:24:49 +02:00
Mattias Wadman
1da8069190 mp4: udta: Improve length/lang box probe and support empty value 2023-05-17 11:55:13 +02:00
Mattias Wadman
493848a7e5 mp4: Use correct epoch for quicktime timestamps 
Somehow was 1904-01-04 should be 1904-01-01
 2023-05-16 13:10:56 +02:00
Mattias Wadman
aec2635ea6
Merge pull request #672 from Rogach/pr/fix-tohex 
fix to_hex function handling of non-ascii characters
 2023-05-16 11:02:03 +02:00
Platon Pronko
b0e4da28de fix non-ascii characters handling in to_hex and to_base64 functions 
to_hex and to_base64 functions were expecting `string` as an input type,
which caused the value to be CastFn'ed to string,
which in turn resulted in raw bytes being cast to []rune (in makeDecodeValueOut).
This operation replaces invalid UTF-8 bytes with 0xFFFD, which then were
passed on to the hex/base64 encoders, resulting in incorrect output.

This patch fixes it by expecting `any` as an input type,
which allows the function to correctly read raw bytes of the input data.
 2023-05-16 09:44:35 +08:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex 
interp: Add hex bits format
 2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df interp: Add hex bits format 
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d

Also made unknown bit formats an error.
 2023-05-15 17:31:38 +02:00
Platon Pronko
a3361e70ba bitcoin: fix witness item structs 
This commit fixes witness item decoding such that they are represented
as structs with `item_size` and `item` fields, instead of a flat list
of raw bytes.
 2023-05-15 18:36:26 +08:00
Mattias Wadman
e2eb667091 html: Add to probe group 
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.

Add lazyre package to help delay compile of RE and make it concurrency safe.
 2023-05-11 19:07:18 +02:00
Mattias Wadman
222cd88b77 bits,bytes: Behave as binary instead of raw decode value 
I think this is closer to what one would expect.
With this fq -d bytes 'scan("...")' will match as a binary and also return binaries,
before as raw decode value one would have to use tobytes to get the binary.
 2023-05-11 18:45:16 +02:00
Mattias Wadman
2c505feec8 help,markdown: Rewrote and made text rendering nicer 
Plan is to use it for man page etc and also some ansi renderer
 2023-05-08 00:07:56 +02:00
Mattias Wadman
a200d3eee0
Merge pull request #415 from pnsafonov/postgres 
Postgres
 2023-05-06 09:07:13 +02:00
Pavel Safonov
e6a9cdbecf postgres: doc 2023-05-06 08:54:56 +03:00
Pavel Safonov
c9350de352 postgres: use bit stream instead of masks to get flags 2023-05-06 08:42:16 +03:00
Pavel Safonov
e839191679 postgres: btree refactored by Mattias Wadman 2023-05-05 11:28:38 +03:00
Pavel Safonov
c23bc4216f postgres: fix line endings in error messages, simplify code, add comments 2023-05-04 09:16:55 +03:00
Pavel Safonov
08c535239b postgres: refactoring 2023-05-04 08:45:42 +03:00
Pavel Safonov
f6f8d5c037 postgres: refactoring 2023-05-03 15:40:32 +03:00
Pavel Safonov
9508a209ea postgres: fix lint 2023-05-03 15:28:29 +03:00
Mattias Wadman
684a083813 interp,decode: Support decode group argument 
Will be used for probing via -d <string> and filename
 2023-05-03 10:01:13 +02:00
Pavel Safonov
e5f15c5fed postgres: fix compilation, fix tests 2023-05-03 10:10:03 +03:00
Pavel Safonov
bb2659d442 Merge remote-tracking branch 'master' into postgres_merge_master_0 2023-05-03 09:44:02 +03:00
Mattias Wadman
f237db2794 id3v2: Decode CTOC flags 2023-05-02 13:43:58 +02:00
Mattias Wadman
d010dcec06 id3v2: Add WXXX (desc/url) frame support 2023-05-02 11:54:11 +02:00
Mattias Wadman
ddd7b0e439
Merge pull request #650 from matthewdale/fix-bson 
bson: support all non-deprecated types and fix int/uint bugs
 2023-05-02 09:33:25 +02:00
Matt Dale
40630d39b2
bson: fix doc formatting and add author info 2023-05-01 18:41:52 -07:00
Mattias Wadman
d6f785c6a6 pcap: Add forgotten help test 2023-05-01 16:06:22 +02:00
Mattias Wadman
a4a332bf4e formats: Clenaup naming a bit 2023-05-01 13:19:04 +02:00
Mattias Wadman
7adc1e7030 mp4: Better description for QuickTime terminator atom 
Addition improvement could be to only allow it in quicktime ftyp brand
and also not as a root book. Currently trailing zero bytes could be
mistaken as a terminator atom.
 2023-05-01 12:32:25 +02:00
Matt Dale
9a982d0a76
bson: add BSON test file generator module and correct BSON format docs 2023-04-30 21:25:57 -07:00
Matt Dale
2017ff8766
bson: support all non-deprecated types and fix int/uint bugs 2023-04-30 10:28:42 -07:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct 
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
 2023-04-29 20:02:34 +02:00
Mattias Wadman
b2cb5c3c1b
Merge pull request #645 from wader/matroska-filedata-fallback 
matroska: file_data: Fallback to raw if probe fails
 2023-04-20 20:37:26 +02:00
Mattias Wadman
8941b13932 matroska: file_data: Fallback to raw if probe fails 2023-04-20 20:08:40 +02:00
Martin Castillo
b3b6cd0ee3 gzip.go: fix typo in variablename: delfate 2023-04-19 15:51:55 +02:00
Pavel Safonov
c9b263e925 postgres: version 15 support 2023-04-17 17:12:46 +03:00
Mattias Wadman
3c6d31b0fe mp4: ctts,infe,iinf,trun: More ISOMFF version handling 2023-04-17 11:43:54 +02:00
Mattias Wadman
2a2b64dda2 matroska: Update ebml specification 
Also add license note about ebml_matroska.xml
 2023-04-13 12:19:41 +02:00
Mattias Wadman
e1b02312d0 wav: Cleanup avi leftovers 2023-04-11 19:34:34 +02:00
Mattias Wadman
82da99c917 msgpack: Add str, array and object type tests 
Related to #634
 2023-04-08 13:09:15 +02:00
Mattias Wadman
97360d6ff9 msgpack: fixstr length field is 5 bits 
Fixes #634
 2023-04-08 08:45:26 +02:00
Mattias Wadman
0b6ef2a9d8 golangci-lint: Disable revive unused-parameter and update for new default config 
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
 2023-03-18 22:56:58 +01:00
Mattias Wadman
980ecdba82 decode: Add float 80 reader 
Rename *d.Bits to UintBits as it return a uint
Add *d.Bits that return []byte
 2023-03-10 01:15:55 +01:00
Mattias Wadman
dc4a82eeed aiff: Add basic decoder 2023-03-09 15:16:52 +01:00
Mattias Wadman
cc52a4419d id3v2: Decode subframes for CTOC and add struct for headers 2023-03-05 19:29:22 +01:00
Mattias Wadman
9852f56b74 tls: Add TLS 1.0, 1.1, 1.2 decode and decryption 
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
  and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.

What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet

Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.

Will be base for decoding http2 and other TLS based on protocols.

Fixes #587
 2023-03-05 13:52:12 +01:00
Mattias Wadman
c75a83c829 help: Show default option value as JSON 2023-03-04 09:53:33 +01:00
Mattias Wadman
dec433fc53 help,markdown: Fix double line breaks when converting to text 
Also fix ugly last line break hack
 2023-03-02 23:38:11 +01:00
Mattias Wadman
6c032455eb pcap,pcapng,ipv4,ipv6: Support raw link type (ipv4 or ipv6) 2023-03-02 18:37:00 +01:00
Mattias Wadman
bdd6718ddb zip: Correctly peek for zip64 EOCD 2023-02-22 22:22:36 +01:00
Mattias Wadman
5228fdd6cd zip: Correctly look for and decode both zip32/64 EOCD record 
There will always be zip(32) EOCD but optinally a zip64 EOCD

Related to #586
 2023-02-22 19:34:28 +01:00
Mattias Wadman
56edb59e83 toml,xml: Fail fast on invalid content 
encoding/xml and github.com/BurntSushi/toml both reads a lot before detecting
that it can't decode. Now we instead read one UTF-8 and make sure it's valid
xml or toml.

Should speed up probing

Related to #586 bigzero-zip.zip
 2023-02-22 16:23:21 +01:00
Mattias Wadman
d4ea6632fa pcap: Add ipv4 fragments tcp test 2023-02-20 11:45:12 +01:00
Mattias Wadman
c8666eeb04 ipv4_packet,ipv6_packet,sll_packet,sll2_packet: Support ipv4/ipv6 link frames and pass correct in arg 
ipv4/ipv6 in sll* accidentally worked as it passed wrong arg with ether type
 2023-02-20 01:34:38 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor 
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
 2023-02-18 21:38:51 +01:00
Mattias Wadman
a1bb630a2a doc,fq: Improve cli help and some cleanup 2023-02-15 11:04:39 +01:00
Mattias Wadman
9aaf2ddf27 matroska: Add unknown size test and add description to ebml header 2023-02-10 10:29:25 +01:00
Mattias Wadman
c890a2899f matroska: Update spec and make refs in descriptions look nicer 2023-02-09 23:18:17 +01:00
Mattias Wadman
a8d0bf4d3e matroska: Assume master with unknown size has ended if a valid parent is found 
Major refactor of ebml code generator, now decoupled from fq code
Cleanup element descriptions a bit, a bit shorter and less clutter
Cleanup old comments
 2023-02-09 20:10:47 +01:00
Mattias Wadman
0d14d7b40f matroska: Handle unknown size for non-master types a bit better 
For non-master types assume size to be rest of file
Still does not follow the ebml spec
 2023-02-08 12:14:40 +01:00
Mattias Wadman
b60aceca9e matroska: Add decode_samples option 
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
 2023-02-08 11:31:41 +01:00
Mattias Wadman
dc79a73b72 interp,json: Move error handling to colorjson 
Cancel error from ValueFn etc will be return by Marshal instead
 2023-02-07 16:57:54 +01:00
Mattias Wadman
5c8e115106 colorjson: Refactor to option struct 2023-02-07 16:18:32 +01:00
Mattias Wadman
62e2cef5c2 tcp_segment: Decode standard options and rename maxseg to mss 2023-01-31 12:42:16 +01:00
Mattias Wadman
1eb5e502af tcp: Ignore TCP option check for now as it seems unreliable in dumps 
For example MSS can be to small in local dumps
 2023-01-28 20:49:20 +01:00
Mattias Wadman
a2cdb3d6c9
Merge pull request #558 from wader/mp4-udta-with-no-length 
mp4: udta: Handle box with value rest of box
 2023-01-25 16:34:58 +01:00
Mattias Wadman
6340365841 mp4: udta: Handle box with value rest of box 
Try distinguish by probing length field. Should probably be improved, what does ffmpeg do?

Regression from c3e3b3e90d #553
 2023-01-25 16:24:28 +01:00
Mattias Wadman
0d1fdb5edf
Merge pull request #557 from wader/mp3-max-unknown-opt 
mp3: Add max_unknown option to fail decode if too much unknown bits
 2023-01-25 16:00:49 +01:00
Mattias Wadman
2d82c05f64 mp3: Add max_unknown option to fail decode if too much unknown bits 
Hopefully help fix even more miss-detections
 2023-01-25 15:27:31 +01:00
Mattias Wadman
b04a650bae flac_picture,mpeg: Fix trailing ")" typo in map sym and description 2023-01-25 14:01:34 +01:00
Mattias Wadman
d645e71008
Merge pull request #554 from wader/mp4-ftyp-qt-minor-desc 
mp4: Decode qt minor verison as YYYY.MM description
 2023-01-16 12:23:42 +01:00
Mattias Wadman
f386a5158e mp4: Decode qt minor verison as YYYY.MM description 2023-01-16 12:13:19 +01:00
Mattias Wadman
c3e3b3e90d mp4: Decode udta metadata boxes without meta box 2023-01-16 12:03:53 +01:00
Mattias Wadman
c2795a7c2f
Merge pull request #552 from wader/mp4-tkhd-flags 
mp4: Decode tkhd flags
 2023-01-13 18:02:53 +01:00
Mattias Wadman
3555dc6721 mp4: Decode tkhd flags 2023-01-13 17:49:03 +01:00
Mattias Wadman
c49012dbca mp4: sgpd,sbgp: Change grouping_type to a string 
Is defined as int32 but is a string in practive it seems
 2023-01-05 19:18:48 +01:00
Mattias Wadman
8b49b42fa3 interp: Wrap Binary in decodeValue to fix prompt issue with bits/bytes format 2023-01-04 15:30:05 +01:00
Pavel Safonov
bffa00835d postgres: fix error in tests 2022-12-26 14:47:23 +03:00
Pavel Safonov
97bbc22a1d postgres: add page arg in pg_btree, change args names in pg_heap 2022-12-22 11:22:02 +03:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format> 
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
 2022-12-21 17:48:39 +01:00
Mattias Wadman
8f39ef6335 bplist: Harmonize ns_keyed_archive jq style a bit 
Let root arg be a lambda so torepr can be done once
 2022-12-20 00:56:19 +01:00
Mattias Wadman
6f354bb350
Merge pull request #527 from dgmcdona/apple_package 
decode: add ns_keyed_archiver, restructure apple decoders into package
 2022-12-19 22:47:29 +01:00
David McDonald
129b4b7026 bplist: doc: update docs to reflect changes to ns_keyed_archiver 2022-12-19 15:24:57 -06:00
David McDonald
a9047c0278 bplist: updates from_ns_keyed_archiver to do automatic torepr based on format detection 2022-12-19 15:18:05 -06:00
David McDonald
9dab3c606c bplist: minor fix to from_ns_keyed_archiver 2022-12-19 14:22:17 -06:00
David McDonald
448c3efb28 bplist: update docs with from_ns_keyed_archiver reference, add error case to function 2022-12-19 13:30:34 -06:00
Mattias Wadman
a85da29544 decode: Make FieldFormat usage more consistent 
Will make it easier to refactor later on
 2022-12-19 19:12:57 +01:00
David McDonald
cba72dbdf0 bplist: added overload for from_ns_keyed_archiver jq func 2022-12-19 02:04:23 -06:00
David McDonald
98eab8cb5b decode: rename parameter for consistency 2022-12-17 13:36:16 -06:00
David McDonald
93f2aa5d73 decode: change PosLoopDetector to use generics 2022-12-17 13:26:40 -06:00
David McDonald
3232f9cc15 decode: moves PosLoopDetector into its own package 2022-12-17 02:18:48 -06:00
David McDonald
fa368bb790 decode: updates all.go with correct macho path 2022-12-17 01:57:23 -06:00
David McDonald
7c9504c727 decode: moves macho decoder to apple package 2022-12-17 01:54:59 -06:00
David McDonald
708346784f decode: remove dead code from ns_keyed_archiver 2022-12-17 01:52:46 -06:00
David McDonald
3ec0ba3f25 decode: add ns_keyed_archiver, restructure apple decoder into apple package 2022-12-17 01:47:27 -06:00
Mattias Wadman
1dea40e6af mp4,doc: Add JSON box tree example and reorder a bit 2022-12-15 14:15:31 +01:00
Mattias Wadman
502e1e3059
Merge pull request #525 from wader/mp3-tags-split-decoder 
mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,v…
 2022-12-15 12:20:16 +01:00
Mattias Wadman
9e5a072e50 mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders 2022-12-15 12:10:06 +01:00
Mattias Wadman
7b60b24ab7 mp4: Add pdin box support 2022-12-15 11:42:05 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types 
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
 2022-12-14 16:23:58 +01:00
Mattias Wadman
48522e3cb8 mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one 2022-12-14 11:34:53 +01:00
Mattias Wadman
0c11ab5354
Merge pull request #521 from wader/mp4-ilst-namespace 
mp4: Add namespace to mdta ilst boxes
 2022-12-14 11:26:28 +01:00
Mattias Wadman
b1b3b63d27 mp4: Add namespace to mdta ilst boxes 2022-12-14 10:49:02 +01:00
David McDonald
7fb674b532 decode: unexport methods 2022-12-13 00:47:58 -06:00
David McDonald
f747873dbc decode: implements lost and found for unreferenced objects 2022-12-12 23:48:33 -06:00
David McDonald
a873819ea2 decode: fixes endian of unicode strings 2022-12-10 21:04:56 -06:00
Mattias Wadman
0b9426976f
Merge pull request #493 from dgmcdona/macos_bookmark 
Apple bookmarkData
 2022-12-10 08:58:33 +01:00
David McDonald
bdb8166261 decode: removed unnecessary struct 2022-12-09 20:13:25 -06:00
Mattias Wadman
9ac453a17d mp4: Fix typo in sample flags sample_is_depended_on description 2022-12-09 17:22:17 +01:00
Mattias Wadman
5fb81a148b mp4: Add sym and description for traf sample flags 2022-12-09 16:22:43 +01:00
David McDonald
7e98b53894 decode: fix type on defer function call, test: add loop.fqtest 2022-12-08 15:01:16 -06:00
Mattias Wadman
1d6ce2c07a mp4: Decode hint and hnti child boxes 2022-12-08 20:47:33 +01:00
Mattias Wadman
83ccedc506 mp4,decode: Properly decode ilst items (both mdta and mdir) 
Refactor mp4 decoder to be simpler and have fallback for unknown box type
Cleanup some old ilst hacks
Add generic string reader to decode API that takes an encoding parameters
 2022-12-08 15:56:22 +01:00
David McDonald
8d5dcff8e3 test: applebookmark: adds problematic test case 2022-12-08 01:48:59 -06:00
David McDonald
330d5f7f28 decode: apple_bookmark: simplifies flag decoding 2022-12-08 01:47:00 -06:00
David McDonald
a162e07b76 decode: minor change to method receiver name 2022-12-08 01:33:38 -06:00
David McDonald
04379df8d4 decode: revert decode.D back, place posLoopDetector in apple_bookmark 2022-12-07 16:25:34 -06:00
David McDonald
63a4e80c8b test: fixed doc test 2022-12-06 19:46:25 -06:00
David McDonald
3942db79c2 pkg/decode/D: Adds PushAndPop, Push, Pop methods. doc: adds help_applebookmark.fqtestdecode: converts applebookmark to use new d.PushAndPop method 2022-12-06 19:40:28 -06:00
David McDonald
34c560c2f5 Merge branch 'master' of https://github.com/wader/fq into macos_bookmark 2022-12-05 22:01:44 -06:00
David McDonald
7ab446626e decode: remove unused field from decoder, unused parens from torepr 2022-12-05 21:37:36 -06:00
Mattias Wadman
47a568e076 text,test: Unbeak base64 tests 2022-12-05 20:51:30 +01:00
Mattias Wadman
a23fe618b0 mp4: sidx version 1 segment_duration is s64 2022-12-05 17:04:05 +01:00
Mattias Wadman
2c4862c9fb
Merge pull request #498 from bitbears-dev/master 
support tzif (time zone information format)
 2022-12-04 13:21:19 +01:00
@0xb17bea125
fd22426b3d tzif: add help_tzif.fqtest 2022-12-04 21:04:15 +09:00
@0xb17bea125
c4e7fc7919 tzif: moved document to tzif.md 2022-12-04 21:04:13 +09:00
@0xb17bea125
4481a77a1c tzif: use scalar.Fn() to define a mapper ad hoc 2022-12-04 21:02:39 +09:00
@0xb17bea125
abde823a79 tzif: use PeekFindByte() to find end of the string 2022-12-04 21:02:39 +09:00
@0xb17bea125
2ee6360b97 support tzif (time zone information format) 2022-12-04 21:02:39 +09:00
Mattias Wadman
42debe5871 dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg 
Update format dev docs to only generate README.md and doc/foramts.md to not
cause formats.svg conflicts.
 2022-12-04 12:37:10 +01:00
David McDonald
b45f9fa67b decode: improve stack push/pop 2022-12-03 15:27:59 -06:00
David McDonald
62f377c2fe doc: fixes snippet for recursive bookmark searching 2022-12-03 15:24:49 -06:00
Mattias Wadman
6fc84a885c doc,dev: Add more usage and dev tips 2022-12-03 19:00:47 +01:00
David McDonald
d199793af6 created stack type 2022-12-02 00:29:29 -06:00
David McDonald
b2aeac6ad3 updates bplist fq tests 2022-12-01 23:45:48 -06:00
David McDonald
55f4f1aa04 moved a flag bit fields into correct positions 2022-12-01 23:31:57 -06:00
David McDonald
c7d00b87b6 fixes unknown bit ranges 2022-12-01 23:31:57 -06:00
David McDonald
cb15b37175 added checks to prevent infinite looping and recursion 2022-12-01 23:31:57 -06:00
David McDonald
9bd65f93b7 migrates tests to per-sample files 2022-12-01 23:31:57 -06:00
David McDonald
7d13cf7312 adds flag parsing to applebookmark 2022-12-01 23:31:57 -06:00
David McDonald
1352598af9 removed commented out line 2022-12-01 23:31:57 -06:00
David McDonald
a23ac8f553 updates fqtest for torepr in apple_bookmarkdata 2022-12-01 23:31:57 -06:00
David McDonald
1e1fc551eb fixed one more snake_case letter 2022-12-01 23:31:57 -06:00
David McDonald
d0b76cae23 fixes broken test and removes long link from markdown body 2022-12-01 23:31:57 -06:00
David McDonald
c244533540 added some sfl2 test files to bplist package 2022-12-01 23:31:57 -06:00
David McDonald
64351e8b44 fixes tests and adds torepr test 2022-12-01 23:31:57 -06:00
David McDonald
f7d7a49fb6 missed a letter on last commit - converting to snake_case 2022-12-01 23:31:57 -06:00
David McDonald
0c216dff48 refactors some decoder logic in apple_bookmark for better querying 2022-12-01 23:31:57 -06:00
David McDonald
44c91d8296 tweaks apple_bookmark markdown documentation 2022-12-01 23:31:57 -06:00
David McDonald
f535ad3de9 fixes spacing in jq files 2022-12-01 23:31:57 -06:00
David McDonald
a351c346f6 removes unused function 2022-12-01 23:31:57 -06:00
David McDonald
253033cc4c fixes broken uid parsing in plist decoder 2022-12-01 23:31:57 -06:00
David McDonald
63a3ca20c4 removes underscore from apple_bookmark package name 2022-12-01 23:31:57 -06:00
David McDonald
4a28e44f44 changes decoder package name from bookmark to apple_bookmark 2022-12-01 23:31:57 -06:00
David McDonald
81269430fe removed unnecessary conversions 2022-12-01 23:31:57 -06:00
David McDonald
d0b044c2f0 converts to snake_case and refactors decode helper 2022-12-01 23:31:57 -06:00
David McDonald
5146f28dbb fixes broken test for all.fqtest 2022-12-01 23:31:57 -06:00
David McDonald
5b1455e777 removed unused function 2022-12-01 23:31:57 -06:00
David McDonald
6f4d1cb1de updated documentation 2022-12-01 23:31:57 -06:00
David McDonald
dbc6fccdda updated doc with apple reference 2022-12-01 23:31:57 -06:00
David McDonald
34db9d7fea regenerated docs, added tests, fixed torepr 2022-12-01 23:31:57 -06:00
David McDonald
8f930aacee forgot to add bookmark.jq in last commit 2022-12-01 23:31:57 -06:00
David McDonald
0287ffa4dc decoding well but torepr needs work 2022-12-01 23:31:56 -06:00
David McDonald
71b17d0382 apple bookmarkdata decoder initial commit 2022-12-01 23:31:56 -06:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#" 
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
 2022-12-01 20:43:30 +01:00
Mattias Wadman
8d69f1fb23 interp: Change default bits_format=string 
I think this is more intuitive but might in some case cause very large JSON output
but maybe that less common or expected. In does cases i think you either want to
use some other bits_format (md5, truncate, etc) or you delete/transform the jq value
before turn it into JSON.

Strings in gojq are binary safe so you can use to hold raw bytes. But note that
convert the binary into JSON is lossy, same as the JSON standard.

Add bits_format option documentation.
 2022-12-01 17:49:34 +01:00
Mattias Wadman
42d9f2c27d interp,help: Properly count line length when breaking on whole words 
Whitespace between words were not counted
 2022-11-29 18:59:32 +01:00
Mattias Wadman
ef2d5232ba mp4: Add png mapping 
Also support samples decode for tracks without hdlr box
 2022-11-25 19:49:18 +01:00
Mattias Wadman
d75748d816 mp4: Decode more sample flags 2022-11-23 14:29:13 +01:00
Pavel Safonov
0ea20e6827 postgres: fail on error in pg_heap 2022-11-23 14:14:51 +03:00
Pavel Safonov
6fe61cd634 Merge branch 'wader_master' into postgres_wal_removed 2022-11-23 13:57:50 +03:00
Pavel Safonov
dafbf4b7ed postgres: lint fixes 2022-11-23 13:56:49 +03:00
Pavel Safonov
60709e5a00 postgres: remove pg_wal. Failed to implement. 2022-11-23 13:55:07 +03:00
Mattias Wadman
646f32d563 matroska: Fix path tests and make _tree_path more robust 2022-11-20 20:16:41 +01:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary 
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
 2022-11-20 19:51:12 +01:00
Mattias Wadman
0cf46e1196 wav,avi,avc_au: Add avi decoder and refactor wav decoder 
avi and wav now share  common riff decode code
avc_au decoder can now handle annexb format
 2022-11-19 13:11:27 +01:00
Mattias Wadman
c3a0686c9e mp3_frame_tags: Refactor and rename xing format to mp3_frame_tags 
Make more sense to use a more general name when there is both info and xing
and possibly vbri in the future. Also both xing and info seems to have extension
fields

Remove some old tests that are not really needed anymore.
 2022-11-16 17:32:03 +01:00
Mattias Wadman
2de875393a mp3_frame: Fix issue calc frame size for some configs 
Seems have to use the 144000*bitrate/sameplrate to get correct size or
one would miss one byte (odd size rounded to even?)
 2022-11-16 15:55:10 +01:00
Pavel Safonov
015b770585 postgres: wal support multiple xlog_body for wal record 2022-11-10 11:32:25 +03:00
Pavel Safonov
e8bb1692f6 postgres: fix pg_wal when XLogRecord size is more than page size 2022-11-09 16:57:56 +03:00
Pavel Safonov
5664c0a498 postgres: refactor ItemIdData 2022-11-07 13:35:23 +03:00
Pavel Safonov
5bb8654423 postgres: remove lsn parameter in pg_wal 2022-11-07 12:05:33 +03:00
Pavel Safonov
9c7f20b6a0 Merge branch 'postgres' into postgres_wal_sync_with_postgres 2022-11-01 08:26:17 +03:00
Pavel Safonov
cdf283101f Merge branch 'wader_master' into postgres 2022-11-01 08:25:07 +03:00
Pavel Safonov
a4d904e1c2 postgres: refactoring 2022-10-31 14:02:08 +03:00
Pavel Safonov
721c1ab3bf psotgres: refactoring 2022-10-31 13:56:56 +03:00
Pavel Safonov
6e2e44d682 postgres: remove arg in pg_btree 2022-10-31 12:55:09 +03:00