1
1
mirror of https://github.com/wader/fq.git synced 2024-12-01 19:12:34 +03:00
Commit Graph

1059 Commits

Author SHA1 Message Date
Babz
c3a123ad84 luajit: remove unecessary dependency 2023-06-21 18:47:13 +02:00
Sergei Kuzmin
e869d8af03 sgpd box entries parsing
Previously entries were parsed as raw bytes, now KIDs/IVs are
explicitely parsed.
2023-06-20 21:57:27 -07:00
Babz
93c96965e0 luajit: add to probe group 2023-06-21 01:41:03 +02:00
Babz
a83576a83a luajit: clarify description 2023-06-21 01:39:23 +02:00
Babz
b24063be02 luajit: *.fqtest: add comments for generating .luac from source 2023-06-21 01:25:35 +02:00
Babz
52ce81817f luajit: split in smaller decode functions 2023-06-21 01:02:38 +02:00
Babz
bdf158be9e luajit: add luajit.md 2023-06-21 00:17:27 +02:00
Babz
715f850d50 luajit: opcodes: implement scalar.UintMapper 2023-06-20 23:55:37 +02:00
Babz
08ae661c15 luajit: use UTF8 strings 2023-06-20 23:33:46 +02:00
Babz
32300a3ffa luajit: check binary.Read() error 2023-06-20 23:31:35 +02:00
Babz
e44f5c00b8 luajit: magic number: raw bits, check with assert 2023-06-20 23:28:34 +02:00
Babz
29ab66b3eb luajit: lowercase flags 2023-06-20 23:11:17 +02:00
Babz
db2e621468 go fmt 2023-06-20 23:02:31 +02:00
Babz
1afdf8b1cf luajit: initial support 2023-06-20 18:24:49 +02:00
Mattias Wadman
1da8069190 mp4: udta: Improve length/lang box probe and support empty value 2023-05-17 11:55:13 +02:00
Mattias Wadman
493848a7e5 mp4: Use correct epoch for quicktime timestamps
Somehow was 1904-01-04 should be 1904-01-01
2023-05-16 13:10:56 +02:00
Mattias Wadman
aec2635ea6
Merge pull request #672 from Rogach/pr/fix-tohex
fix to_hex function handling of non-ascii characters
2023-05-16 11:02:03 +02:00
Platon Pronko
b0e4da28de fix non-ascii characters handling in to_hex and to_base64 functions
to_hex and to_base64 functions were expecting `string` as an input type,
which caused the value to be CastFn'ed to string,
which in turn resulted in raw bytes being cast to []rune (in makeDecodeValueOut).
This operation replaces invalid UTF-8 bytes with 0xFFFD, which then were
passed on to the hex/base64 encoders, resulting in incorrect output.

This patch fixes it by expecting `any` as an input type,
which allows the function to correctly read raw bytes of the input data.
2023-05-16 09:44:35 +08:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex
interp: Add hex bits format
2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df interp: Add hex bits format
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d

Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Platon Pronko
a3361e70ba bitcoin: fix witness item structs
This commit fixes witness item decoding such that they are represented
as structs with `item_size` and `item` fields, instead of a flat list
of raw bytes.
2023-05-15 18:36:26 +08:00
Mattias Wadman
e2eb667091 html: Add to probe group
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.

Add lazyre package to help delay compile of RE and make it concurrency safe.
2023-05-11 19:07:18 +02:00
Mattias Wadman
222cd88b77 bits,bytes: Behave as binary instead of raw decode value
I think this is closer to what one would expect.
With this fq -d bytes 'scan("...")' will match as a binary and also return binaries,
before as raw decode value one would have to use tobytes to get the binary.
2023-05-11 18:45:16 +02:00
Mattias Wadman
2c505feec8 help,markdown: Rewrote and made text rendering nicer
Plan is to use it for man page etc and also some ansi renderer
2023-05-08 00:07:56 +02:00
Mattias Wadman
a200d3eee0
Merge pull request #415 from pnsafonov/postgres
Postgres
2023-05-06 09:07:13 +02:00
Pavel Safonov
e6a9cdbecf postgres: doc 2023-05-06 08:54:56 +03:00
Pavel Safonov
c9350de352 postgres: use bit stream instead of masks to get flags 2023-05-06 08:42:16 +03:00
Pavel Safonov
e839191679 postgres: btree refactored by Mattias Wadman 2023-05-05 11:28:38 +03:00
Pavel Safonov
c23bc4216f postgres: fix line endings in error messages, simplify code, add comments 2023-05-04 09:16:55 +03:00
Pavel Safonov
08c535239b postgres: refactoring 2023-05-04 08:45:42 +03:00
Pavel Safonov
f6f8d5c037 postgres: refactoring 2023-05-03 15:40:32 +03:00
Pavel Safonov
9508a209ea postgres: fix lint 2023-05-03 15:28:29 +03:00
Mattias Wadman
684a083813 interp,decode: Support decode group argument
Will be used for probing via -d <string> and filename
2023-05-03 10:01:13 +02:00
Pavel Safonov
e5f15c5fed postgres: fix compilation, fix tests 2023-05-03 10:10:03 +03:00
Pavel Safonov
bb2659d442 Merge remote-tracking branch 'master' into postgres_merge_master_0 2023-05-03 09:44:02 +03:00
Mattias Wadman
f237db2794 id3v2: Decode CTOC flags 2023-05-02 13:43:58 +02:00
Mattias Wadman
d010dcec06 id3v2: Add WXXX (desc/url) frame support 2023-05-02 11:54:11 +02:00
Mattias Wadman
ddd7b0e439
Merge pull request #650 from matthewdale/fix-bson
bson: support all non-deprecated types and fix int/uint bugs
2023-05-02 09:33:25 +02:00
Matt Dale
40630d39b2
bson: fix doc formatting and add author info 2023-05-01 18:41:52 -07:00
Mattias Wadman
d6f785c6a6 pcap: Add forgotten help test 2023-05-01 16:06:22 +02:00
Mattias Wadman
a4a332bf4e formats: Clenaup naming a bit 2023-05-01 13:19:04 +02:00
Mattias Wadman
7adc1e7030 mp4: Better description for QuickTime terminator atom
Addition improvement could be to only allow it in quicktime ftyp brand
and also not as a root book. Currently trailing zero bytes could be
mistaken as a terminator atom.
2023-05-01 12:32:25 +02:00
Matt Dale
9a982d0a76
bson: add BSON test file generator module and correct BSON format docs 2023-04-30 21:25:57 -07:00
Matt Dale
2017ff8766
bson: support all non-deprecated types and fix int/uint bugs 2023-04-30 10:28:42 -07:00
Mattias Wadman
b08ef00dd1 decode,interp: Refactor format groups into a proper struct
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.

This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group  (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
b2cb5c3c1b
Merge pull request #645 from wader/matroska-filedata-fallback
matroska: file_data: Fallback to raw if probe fails
2023-04-20 20:37:26 +02:00
Mattias Wadman
8941b13932 matroska: file_data: Fallback to raw if probe fails 2023-04-20 20:08:40 +02:00
Martin Castillo
b3b6cd0ee3 gzip.go: fix typo in variablename: delfate 2023-04-19 15:51:55 +02:00
Pavel Safonov
c9b263e925 postgres: version 15 support 2023-04-17 17:12:46 +03:00
Mattias Wadman
3c6d31b0fe mp4: ctts,infe,iinf,trun: More ISOMFF version handling 2023-04-17 11:43:54 +02:00
Mattias Wadman
2a2b64dda2 matroska: Update ebml specification
Also add license note about ebml_matroska.xml
2023-04-13 12:19:41 +02:00
Mattias Wadman
e1b02312d0 wav: Cleanup avi leftovers 2023-04-11 19:34:34 +02:00
Mattias Wadman
82da99c917 msgpack: Add str, array and object type tests
Related to #634
2023-04-08 13:09:15 +02:00
Mattias Wadman
97360d6ff9 msgpack: fixstr length field is 5 bits
Fixes #634
2023-04-08 08:45:26 +02:00
Mattias Wadman
0b6ef2a9d8 golangci-lint: Disable revive unused-parameter and update for new default config
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
2023-03-18 22:56:58 +01:00
Mattias Wadman
980ecdba82 decode: Add float 80 reader
Rename *d.Bits to UintBits as it return a uint
Add *d.Bits that return []byte
2023-03-10 01:15:55 +01:00
Mattias Wadman
dc4a82eeed aiff: Add basic decoder 2023-03-09 15:16:52 +01:00
Mattias Wadman
cc52a4419d id3v2: Decode subframes for CTOC and add struct for headers 2023-03-05 19:29:22 +01:00
Mattias Wadman
9852f56b74 tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
  and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.

What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet

Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.

Will be base for decoding http2 and other TLS based on protocols.

Fixes #587
2023-03-05 13:52:12 +01:00
Mattias Wadman
c75a83c829 help: Show default option value as JSON 2023-03-04 09:53:33 +01:00
Mattias Wadman
dec433fc53 help,markdown: Fix double line breaks when converting to text
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00
Mattias Wadman
6c032455eb pcap,pcapng,ipv4,ipv6: Support raw link type (ipv4 or ipv6) 2023-03-02 18:37:00 +01:00
Mattias Wadman
bdd6718ddb zip: Correctly peek for zip64 EOCD 2023-02-22 22:22:36 +01:00
Mattias Wadman
5228fdd6cd zip: Correctly look for and decode both zip32/64 EOCD record
There will always be zip(32) EOCD but optinally a zip64 EOCD

Related to #586
2023-02-22 19:34:28 +01:00
Mattias Wadman
56edb59e83 toml,xml: Fail fast on invalid content
encoding/xml and github.com/BurntSushi/toml both reads a lot before detecting
that it can't decode. Now we instead read one UTF-8 and make sure it's valid
xml or toml.

Should speed up probing

Related to #586 bigzero-zip.zip
2023-02-22 16:23:21 +01:00
Mattias Wadman
d4ea6632fa pcap: Add ipv4 fragments tcp test 2023-02-20 11:45:12 +01:00
Mattias Wadman
c8666eeb04 ipv4_packet,ipv6_packet,sll_packet,sll2_packet: Support ipv4/ipv6 link frames and pass correct in arg
ipv4/ipv6 in sll* accidentally worked as it passed wrong arg with ether type
2023-02-20 01:34:38 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
a1bb630a2a doc,fq: Improve cli help and some cleanup 2023-02-15 11:04:39 +01:00
Mattias Wadman
9aaf2ddf27 matroska: Add unknown size test and add description to ebml header 2023-02-10 10:29:25 +01:00
Mattias Wadman
c890a2899f matroska: Update spec and make refs in descriptions look nicer 2023-02-09 23:18:17 +01:00
Mattias Wadman
a8d0bf4d3e matroska: Assume master with unknown size has ended if a valid parent is found
Major refactor of ebml code generator, now decoupled from fq code
Cleanup element descriptions a bit, a bit shorter and less clutter
Cleanup old comments
2023-02-09 20:10:47 +01:00
Mattias Wadman
0d14d7b40f matroska: Handle unknown size for non-master types a bit better
For non-master types assume size to be rest of file
Still does not follow the ebml spec
2023-02-08 12:14:40 +01:00
Mattias Wadman
b60aceca9e matroska: Add decode_samples option
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
2023-02-08 11:31:41 +01:00
Mattias Wadman
dc79a73b72 interp,json: Move error handling to colorjson
Cancel error from ValueFn etc will be return by Marshal instead
2023-02-07 16:57:54 +01:00
Mattias Wadman
5c8e115106 colorjson: Refactor to option struct 2023-02-07 16:18:32 +01:00
Mattias Wadman
62e2cef5c2 tcp_segment: Decode standard options and rename maxseg to mss 2023-01-31 12:42:16 +01:00
Mattias Wadman
1eb5e502af tcp: Ignore TCP option check for now as it seems unreliable in dumps
For example MSS can be to small in local dumps
2023-01-28 20:49:20 +01:00
Mattias Wadman
a2cdb3d6c9
Merge pull request #558 from wader/mp4-udta-with-no-length
mp4: udta: Handle box with value rest of box
2023-01-25 16:34:58 +01:00
Mattias Wadman
6340365841 mp4: udta: Handle box with value rest of box
Try distinguish by probing length field. Should probably be improved, what does ffmpeg do?

Regression from c3e3b3e90d #553
2023-01-25 16:24:28 +01:00
Mattias Wadman
0d1fdb5edf
Merge pull request #557 from wader/mp3-max-unknown-opt
mp3: Add max_unknown option to fail decode if too much unknown bits
2023-01-25 16:00:49 +01:00
Mattias Wadman
2d82c05f64 mp3: Add max_unknown option to fail decode if too much unknown bits
Hopefully help fix even more miss-detections
2023-01-25 15:27:31 +01:00
Mattias Wadman
b04a650bae flac_picture,mpeg: Fix trailing ")" typo in map sym and description 2023-01-25 14:01:34 +01:00
Mattias Wadman
d645e71008
Merge pull request #554 from wader/mp4-ftyp-qt-minor-desc
mp4: Decode qt minor verison as YYYY.MM description
2023-01-16 12:23:42 +01:00
Mattias Wadman
f386a5158e mp4: Decode qt minor verison as YYYY.MM description 2023-01-16 12:13:19 +01:00
Mattias Wadman
c3e3b3e90d mp4: Decode udta metadata boxes without meta box 2023-01-16 12:03:53 +01:00
Mattias Wadman
c2795a7c2f
Merge pull request #552 from wader/mp4-tkhd-flags
mp4: Decode tkhd flags
2023-01-13 18:02:53 +01:00
Mattias Wadman
3555dc6721 mp4: Decode tkhd flags 2023-01-13 17:49:03 +01:00
Mattias Wadman
c49012dbca mp4: sgpd,sbgp: Change grouping_type to a string
Is defined as int32 but is a string in practive it seems
2023-01-05 19:18:48 +01:00
Mattias Wadman
8b49b42fa3 interp: Wrap Binary in decodeValue to fix prompt issue with bits/bytes format 2023-01-04 15:30:05 +01:00
Pavel Safonov
bffa00835d postgres: fix error in tests 2022-12-26 14:47:23 +03:00
Pavel Safonov
97bbc22a1d postgres: add page arg in pg_btree, change args names in pg_heap 2022-12-22 11:22:02 +03:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
8f39ef6335 bplist: Harmonize ns_keyed_archive jq style a bit
Let root arg be a lambda so torepr can be done once
2022-12-20 00:56:19 +01:00
Mattias Wadman
6f354bb350
Merge pull request #527 from dgmcdona/apple_package
decode: add ns_keyed_archiver, restructure apple decoders into package
2022-12-19 22:47:29 +01:00
David McDonald
129b4b7026 bplist: doc: update docs to reflect changes to ns_keyed_archiver 2022-12-19 15:24:57 -06:00
David McDonald
a9047c0278 bplist: updates from_ns_keyed_archiver to do automatic torepr based on format detection 2022-12-19 15:18:05 -06:00
David McDonald
9dab3c606c bplist: minor fix to from_ns_keyed_archiver 2022-12-19 14:22:17 -06:00
David McDonald
448c3efb28 bplist: update docs with from_ns_keyed_archiver reference, add error case to function 2022-12-19 13:30:34 -06:00
Mattias Wadman
a85da29544 decode: Make FieldFormat usage more consistent
Will make it easier to refactor later on
2022-12-19 19:12:57 +01:00
David McDonald
cba72dbdf0 bplist: added overload for from_ns_keyed_archiver jq func 2022-12-19 02:04:23 -06:00
David McDonald
98eab8cb5b decode: rename parameter for consistency 2022-12-17 13:36:16 -06:00
David McDonald
93f2aa5d73 decode: change PosLoopDetector to use generics 2022-12-17 13:26:40 -06:00
David McDonald
3232f9cc15 decode: moves PosLoopDetector into its own package 2022-12-17 02:18:48 -06:00
David McDonald
fa368bb790 decode: updates all.go with correct macho path 2022-12-17 01:57:23 -06:00
David McDonald
7c9504c727 decode: moves macho decoder to apple package 2022-12-17 01:54:59 -06:00
David McDonald
708346784f decode: remove dead code from ns_keyed_archiver 2022-12-17 01:52:46 -06:00
David McDonald
3ec0ba3f25 decode: add ns_keyed_archiver, restructure apple decoder into apple package 2022-12-17 01:47:27 -06:00
Mattias Wadman
1dea40e6af mp4,doc: Add JSON box tree example and reorder a bit 2022-12-15 14:15:31 +01:00
Mattias Wadman
502e1e3059
Merge pull request #525 from wader/mp3-tags-split-decoder
mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,v…
2022-12-15 12:20:16 +01:00
Mattias Wadman
9e5a072e50 mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders 2022-12-15 12:10:06 +01:00
Mattias Wadman
7b60b24ab7 mp4: Add pdin box support 2022-12-15 11:42:05 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
48522e3cb8 mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one 2022-12-14 11:34:53 +01:00
Mattias Wadman
0c11ab5354
Merge pull request #521 from wader/mp4-ilst-namespace
mp4: Add namespace to mdta ilst boxes
2022-12-14 11:26:28 +01:00
Mattias Wadman
b1b3b63d27 mp4: Add namespace to mdta ilst boxes 2022-12-14 10:49:02 +01:00
David McDonald
7fb674b532 decode: unexport methods 2022-12-13 00:47:58 -06:00
David McDonald
f747873dbc decode: implements lost and found for unreferenced objects 2022-12-12 23:48:33 -06:00
David McDonald
a873819ea2 decode: fixes endian of unicode strings 2022-12-10 21:04:56 -06:00
Mattias Wadman
0b9426976f
Merge pull request #493 from dgmcdona/macos_bookmark
Apple bookmarkData
2022-12-10 08:58:33 +01:00
David McDonald
bdb8166261 decode: removed unnecessary struct 2022-12-09 20:13:25 -06:00
Mattias Wadman
9ac453a17d mp4: Fix typo in sample flags sample_is_depended_on description 2022-12-09 17:22:17 +01:00
Mattias Wadman
5fb81a148b mp4: Add sym and description for traf sample flags 2022-12-09 16:22:43 +01:00
David McDonald
7e98b53894 decode: fix type on defer function call, test: add loop.fqtest 2022-12-08 15:01:16 -06:00
Mattias Wadman
1d6ce2c07a mp4: Decode hint and hnti child boxes 2022-12-08 20:47:33 +01:00
Mattias Wadman
83ccedc506 mp4,decode: Properly decode ilst items (both mdta and mdir)
Refactor mp4 decoder to be simpler and have fallback for unknown box type
Cleanup some old ilst hacks
Add generic string reader to decode API that takes an encoding parameters
2022-12-08 15:56:22 +01:00
David McDonald
8d5dcff8e3 test: applebookmark: adds problematic test case 2022-12-08 01:48:59 -06:00
David McDonald
330d5f7f28 decode: apple_bookmark: simplifies flag decoding 2022-12-08 01:47:00 -06:00
David McDonald
a162e07b76 decode: minor change to method receiver name 2022-12-08 01:33:38 -06:00
David McDonald
04379df8d4 decode: revert decode.D back, place posLoopDetector in apple_bookmark 2022-12-07 16:25:34 -06:00
David McDonald
63a4e80c8b test: fixed doc test 2022-12-06 19:46:25 -06:00
David McDonald
3942db79c2 pkg/decode/D: Adds PushAndPop, Push, Pop methods. doc: adds help_applebookmark.fqtestdecode: converts applebookmark to use new d.PushAndPop method 2022-12-06 19:40:28 -06:00
David McDonald
34c560c2f5 Merge branch 'master' of https://github.com/wader/fq into macos_bookmark 2022-12-05 22:01:44 -06:00
David McDonald
7ab446626e decode: remove unused field from decoder, unused parens from torepr 2022-12-05 21:37:36 -06:00
Mattias Wadman
47a568e076 text,test: Unbeak base64 tests 2022-12-05 20:51:30 +01:00
Mattias Wadman
a23fe618b0 mp4: sidx version 1 segment_duration is s64 2022-12-05 17:04:05 +01:00
Mattias Wadman
2c4862c9fb
Merge pull request #498 from bitbears-dev/master
support tzif (time zone information format)
2022-12-04 13:21:19 +01:00
@0xb17bea125
fd22426b3d tzif: add help_tzif.fqtest 2022-12-04 21:04:15 +09:00
@0xb17bea125
c4e7fc7919 tzif: moved document to tzif.md 2022-12-04 21:04:13 +09:00
@0xb17bea125
4481a77a1c tzif: use scalar.Fn() to define a mapper ad hoc 2022-12-04 21:02:39 +09:00
@0xb17bea125
abde823a79 tzif: use PeekFindByte() to find end of the string 2022-12-04 21:02:39 +09:00
@0xb17bea125
2ee6360b97 support tzif (time zone information format) 2022-12-04 21:02:39 +09:00
Mattias Wadman
42debe5871 dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg
Update format dev docs to only generate README.md and doc/foramts.md to not
cause formats.svg conflicts.
2022-12-04 12:37:10 +01:00
David McDonald
b45f9fa67b decode: improve stack push/pop 2022-12-03 15:27:59 -06:00
David McDonald
62f377c2fe doc: fixes snippet for recursive bookmark searching 2022-12-03 15:24:49 -06:00
Mattias Wadman
6fc84a885c doc,dev: Add more usage and dev tips 2022-12-03 19:00:47 +01:00
David McDonald
d199793af6 created stack type 2022-12-02 00:29:29 -06:00
David McDonald
b2aeac6ad3 updates bplist fq tests 2022-12-01 23:45:48 -06:00
David McDonald
55f4f1aa04 moved a flag bit fields into correct positions 2022-12-01 23:31:57 -06:00
David McDonald
c7d00b87b6 fixes unknown bit ranges 2022-12-01 23:31:57 -06:00
David McDonald
cb15b37175 added checks to prevent infinite looping and recursion 2022-12-01 23:31:57 -06:00
David McDonald
9bd65f93b7 migrates tests to per-sample files 2022-12-01 23:31:57 -06:00
David McDonald
7d13cf7312 adds flag parsing to applebookmark 2022-12-01 23:31:57 -06:00
David McDonald
1352598af9 removed commented out line 2022-12-01 23:31:57 -06:00
David McDonald
a23ac8f553 updates fqtest for torepr in apple_bookmarkdata 2022-12-01 23:31:57 -06:00
David McDonald
1e1fc551eb fixed one more snake_case letter 2022-12-01 23:31:57 -06:00
David McDonald
d0b76cae23 fixes broken test and removes long link from markdown body 2022-12-01 23:31:57 -06:00
David McDonald
c244533540 added some sfl2 test files to bplist package 2022-12-01 23:31:57 -06:00
David McDonald
64351e8b44 fixes tests and adds torepr test 2022-12-01 23:31:57 -06:00
David McDonald
f7d7a49fb6 missed a letter on last commit - converting to snake_case 2022-12-01 23:31:57 -06:00
David McDonald
0c216dff48 refactors some decoder logic in apple_bookmark for better querying 2022-12-01 23:31:57 -06:00
David McDonald
44c91d8296 tweaks apple_bookmark markdown documentation 2022-12-01 23:31:57 -06:00
David McDonald
f535ad3de9 fixes spacing in jq files 2022-12-01 23:31:57 -06:00
David McDonald
a351c346f6 removes unused function 2022-12-01 23:31:57 -06:00
David McDonald
253033cc4c fixes broken uid parsing in plist decoder 2022-12-01 23:31:57 -06:00
David McDonald
63a3ca20c4 removes underscore from apple_bookmark package name 2022-12-01 23:31:57 -06:00
David McDonald
4a28e44f44 changes decoder package name from bookmark to apple_bookmark 2022-12-01 23:31:57 -06:00
David McDonald
81269430fe removed unnecessary conversions 2022-12-01 23:31:57 -06:00
David McDonald
d0b044c2f0 converts to snake_case and refactors decode helper 2022-12-01 23:31:57 -06:00
David McDonald
5146f28dbb fixes broken test for all.fqtest 2022-12-01 23:31:57 -06:00
David McDonald
5b1455e777 removed unused function 2022-12-01 23:31:57 -06:00
David McDonald
6f4d1cb1de updated documentation 2022-12-01 23:31:57 -06:00
David McDonald
dbc6fccdda updated doc with apple reference 2022-12-01 23:31:57 -06:00
David McDonald
34db9d7fea regenerated docs, added tests, fixed torepr 2022-12-01 23:31:57 -06:00
David McDonald
8f930aacee forgot to add bookmark.jq in last commit 2022-12-01 23:31:57 -06:00
David McDonald
0287ffa4dc decoding well but torepr needs work 2022-12-01 23:31:56 -06:00
David McDonald
71b17d0382 apple bookmarkdata decoder initial commit 2022-12-01 23:31:56 -06:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
8d69f1fb23 interp: Change default bits_format=string
I think this is more intuitive but might in some case cause very large JSON output
but maybe that less common or expected. In does cases i think you either want to
use some other bits_format (md5, truncate, etc) or you delete/transform the jq value
before turn it into JSON.

Strings in gojq are binary safe so you can use to hold raw bytes. But note that
convert the binary into JSON is lossy, same as the JSON standard.

Add bits_format option documentation.
2022-12-01 17:49:34 +01:00
Mattias Wadman
42d9f2c27d interp,help: Properly count line length when breaking on whole words
Whitespace between words were not counted
2022-11-29 18:59:32 +01:00
Mattias Wadman
ef2d5232ba mp4: Add png mapping
Also support samples decode for tracks without hdlr box
2022-11-25 19:49:18 +01:00
Mattias Wadman
d75748d816 mp4: Decode more sample flags 2022-11-23 14:29:13 +01:00
Pavel Safonov
0ea20e6827 postgres: fail on error in pg_heap 2022-11-23 14:14:51 +03:00
Pavel Safonov
6fe61cd634 Merge branch 'wader_master' into postgres_wal_removed 2022-11-23 13:57:50 +03:00
Pavel Safonov
dafbf4b7ed postgres: lint fixes 2022-11-23 13:56:49 +03:00
Pavel Safonov
60709e5a00 postgres: remove pg_wal. Failed to implement. 2022-11-23 13:55:07 +03:00
Mattias Wadman
646f32d563 matroska: Fix path tests and make _tree_path more robust 2022-11-20 20:16:41 +01:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
0cf46e1196 wav,avi,avc_au: Add avi decoder and refactor wav decoder
avi and wav now share  common riff decode code
avc_au decoder can now handle annexb format
2022-11-19 13:11:27 +01:00
Mattias Wadman
c3a0686c9e mp3_frame_tags: Refactor and rename xing format to mp3_frame_tags
Make more sense to use a more general name when there is both info and xing
and possibly vbri in the future. Also both xing and info seems to have extension
fields

Remove some old tests that are not really needed anymore.
2022-11-16 17:32:03 +01:00
Mattias Wadman
2de875393a mp3_frame: Fix issue calc frame size for some configs
Seems have to use the 144000*bitrate/sameplrate to get correct size or
one would miss one byte (odd size rounded to even?)
2022-11-16 15:55:10 +01:00
Pavel Safonov
015b770585 postgres: wal support multiple xlog_body for wal record 2022-11-10 11:32:25 +03:00
Pavel Safonov
e8bb1692f6 postgres: fix pg_wal when XLogRecord size is more than page size 2022-11-09 16:57:56 +03:00
Pavel Safonov
5664c0a498 postgres: refactor ItemIdData 2022-11-07 13:35:23 +03:00
Pavel Safonov
5bb8654423 postgres: remove lsn parameter in pg_wal 2022-11-07 12:05:33 +03:00
Pavel Safonov
9c7f20b6a0 Merge branch 'postgres' into postgres_wal_sync_with_postgres 2022-11-01 08:26:17 +03:00
Pavel Safonov
cdf283101f Merge branch 'wader_master' into postgres 2022-11-01 08:25:07 +03:00
Pavel Safonov
a4d904e1c2 postgres: refactoring 2022-10-31 14:02:08 +03:00
Pavel Safonov
721c1ab3bf psotgres: refactoring 2022-10-31 13:56:56 +03:00
Pavel Safonov
6e2e44d682 postgres: remove arg in pg_btree 2022-10-31 12:55:09 +03:00
Pavel Safonov
7fd41090e3 postgres: add btree index tests 2022-10-31 11:07:10 +03:00
Pavel Safonov
3c7ed5d73c postgres: add tests data 2022-10-31 09:32:31 +03:00
Pavel Safonov
de68785a8d postgres: add test files 2022-10-31 08:36:57 +03:00
Pavel Safonov
7f7f729c90 postgres: refactoring, tests 2022-10-28 13:55:38 +03:00
Pavel Safonov
069babbc0d postgres: wal refactoring 2022-10-28 13:42:42 +03:00
Pavel Safonov
d7a0f930df postgres: pgpro wal refactoring 2022-10-28 13:13:33 +03:00
Pavel Safonov
e8d8caca41 postgres: pgpro wal implementation 2022-10-28 13:09:09 +03:00
Pavel Safonov
fb7778a5b0 postgres: add wal tests 2022-10-27 12:08:40 +03:00
Pavel Safonov
7dedcbabae postgres: refactoring 2022-10-26 16:37:50 +03:00
Pavel Safonov
bd2bdd649a postgres: wal refactoring 2022-10-26 10:13:50 +03:00
Pavel Safonov
7c92715f2a postgres: refactoring 2022-10-26 10:11:23 +03:00
Pavel Safonov
7c1dfbd090 postgres: wal implementation 2022-10-26 10:10:47 +03:00
Pavel Safonov
26bff1442f postgres: wal refactoing 2022-10-26 10:02:38 +03:00
Pavel Safonov
b09ec2fc94 postgres: wal refactoing 2022-10-26 10:01:21 +03:00
Pavel Safonov
c105fcdd90 postgres: wal impl 2022-10-26 09:56:00 +03:00
Pavel Safonov
a4c1c5b811 postgres: wal const 2022-10-24 13:23:28 +03:00
Pavel Safonov
6aed2387b8 postgres: add pg_wal for pgproee11 as copy of postgres14 2022-10-24 13:04:54 +03:00
Pavel Safonov
f4408fb072 Merge branch 'postgres' into postgres_wal_sync_with_postgres 2022-10-24 13:01:03 +03:00
Pavel Safonov
9f61e63702 postgres: allow all flovours to decode btree index 2022-10-24 09:47:27 +03:00
Pavel Safonov
5ff6273505 postgres: add additional checks in pg_heap 2022-10-24 09:44:42 +03:00
Mattias Wadman
f66e224452 id3v2: In the wild COMM and TXXX frame might not have a null terminator 2022-10-20 22:07:37 +02:00
Mattias Wadman
fb2a2b949c flac,fuzz: Fatal error on negative partition sample count 2022-10-18 13:40:48 +02:00
Pavel Safonov
8f55e177f6 postgres: remove SeekAbs(0) where it's not used. 2022-10-12 10:50:00 +03:00
Pavel Safonov
dd9c7f0420 Merge branch 'postgres' into postgres_wal_sync_with_postgres 2022-10-12 10:43:59 +03:00
Mattias Wadman
2606916753 yaml,fuzz: gojq.Normalize value to fix type panic 2022-10-11 19:22:49 +02:00
Mattias Wadman
7859be1e5d flac_frame: Properly decode zero escape sample size 2022-10-11 17:05:19 +02:00
Mattias Wadman
3a578152f2
Merge pull request #459 from wader/decode-s-must-least-1bit
decode,fuzz: Signed integer (S) read require at least one bit
2022-10-11 12:41:28 +02:00
Mattias Wadman
bafd1f564e decode,fuzz: Signed integer (S) read require at least one bit
Otherwise there is no sign bit and it will panic on shifting.

Maybe should require 2? now -0 and +1 hmm
2022-10-11 12:12:28 +02:00
Mattias Wadman
cef4245b49 fuzz: make fuzz GROUP=mp4 to fuzz one group 2022-10-10 21:07:10 +02:00
Pavel Safonov
83d63c587c Merge branch 'wader_master' into postgres 2022-10-10 13:32:13 +03:00
Pavel Safonov
939c7c17ad postgres: how_to.md 2022-10-10 13:25:43 +03:00
Pavel Safonov
1aa08e336c postgres: add btree, pg_control to how_to.md 2022-10-10 13:24:04 +03:00
Pavel Safonov
2423f86b3a postgres: add how_to.md - how to generate test files for postgres 2022-10-10 13:08:03 +03:00
Mattias Wadman
e9aa4837e9
Merge pull request #454 from dgmcdona/reference_size_bug_fix
fixes reference calculation to use reference size from trailer
2022-10-09 21:48:12 +02:00
David McDonald
b09d61162b makes dictionary key type checking more sensible 2022-10-09 13:34:00 -05:00
Mattias Wadman
2a86d323b4 doc,rtmp,pcap,markdown: Add more examples 2022-10-09 19:18:14 +02:00
Mattias Wadman
e748079e87 matroska: Update spec and regenerate
The new type "TargetType" used in labels is bit confusing, should be handled differently?
2022-10-08 15:45:46 +02:00
David McDonald
ac86f9318b fixes signed integer parsing 2022-10-07 21:33:26 -05:00
David McDonald
2b3adbe8d6 renames test data file 2022-10-07 18:08:20 -05:00
David McDonald
4ee7dd8a38 changes Errorf to Fatalf on infinite loops 2022-10-07 18:04:07 -05:00
David McDonald
4d8dd5c5e3 adds check for recursion in decodeReference, adds test to verify fix 2022-10-07 15:07:05 -05:00
David McDonald
413d425079 gofmt 2022-10-07 13:16:17 -05:00
David McDonald
fb1a91ac3c drop indented else block per lint 2022-10-07 13:15:22 -05:00
David McDonald
236fbc17b5 fixes reference calculation to use reference size from trailer 2022-10-07 13:07:51 -05:00
Pavel Safonov
6fe5f05faf postgres: lint, doc 2022-10-07 14:17:04 +03:00
Pavel Safonov
bedc480a51 postgres: refactoring 2022-10-07 14:12:36 +03:00
Pavel Safonov
e06fa6e139 postgres: refactoring 2022-10-07 13:33:05 +03:00
Pavel Safonov
4676590624 postgres: add argument to calc page's check sum correctly 2022-10-07 12:58:36 +03:00
Pavel Safonov
46e1e337ff postgres: make page size const 2022-10-07 12:04:49 +03:00
Pavel Safonov
03d8fe1c61 postgres: page sum impl 2022-10-07 11:58:03 +03:00
David McDonald
4dd594c14e fixes bad path in test output 2022-10-07 00:50:48 -05:00
David McDonald
46b7ab3268 adds test to verify fix 2022-10-07 00:23:25 -05:00
David McDonald
b08e25ce78 removes unnecessary cast 2022-10-06 23:49:15 -05:00
David McDonald
b7c4576c5e adds necessary cast 2022-10-06 23:47:10 -05:00
David McDonald
f9a1d3f444 fixes calculation of floating point lengths 2022-10-06 23:23:04 -05:00
Pavel Safonov
6588e1dd88 Merge branch 'wader_master' into postgres 2022-10-05 19:39:35 +03:00
Pavel Safonov
e57c3b980b postgres: refactoring 2022-10-05 19:37:42 +03:00
Pavel Safonov
78583731dc postgres: postgres 10 support 2022-10-05 13:16:39 +03:00
Mattias Wadman
16b01211b1 bplist: Make torepr convert to values 2022-10-04 18:12:49 +02:00
Mattias Wadman
cb3dc80252 decode,tar: Add scalar description and Try* helpers
Rework time helpers to use new functions
Fix panic in tar decoder where sym value might be missing

Still not very happy about the API but it's getting better.
2022-10-04 17:18:51 +02:00
Mattias Wadman
421b2b3071 bplist: Fix unknown field for singletons and add torepr tests 2022-10-04 14:30:54 +02:00
Mattias Wadman
09ea08f3aa
Merge pull request #427 from dgmcdona/bplist
Adds support for Apple Binary Plist, version 00
2022-10-04 14:25:53 +02:00
Mattias Wadman
20a15372ba mp4: Fix data_format typo 2022-10-02 23:31:38 +02:00
Mattias Wadman
0fdc03a4f1 doc: Fix some incorrect example prompts 2022-09-27 13:42:31 +02:00
Mattias Wadman
d6316d5cbf wav: Decode smpl chunk 2022-09-26 11:28:02 +02:00
Pavel Safonov
9deab2ea1b postgres: pg_heap fix page_begin, page_end 2022-09-26 09:10:03 +03:00
David McDonald
1e47f4f2f7 updates tests post integer-bug fix 2022-09-25 00:48:43 -05:00
David McDonald
ebae938d0e Fixes bug in integer parsing 2022-09-25 00:43:21 -05:00
David McDonald
bcccde2358 Fixes and embeds documentation 2022-09-24 12:57:11 -05:00
David McDonald
3d8ea1deac updates torepr for data type 2022-09-23 22:12:30 -05:00
David McDonald
12836abe26 updates fqtest 2022-09-23 21:16:28 -05:00
David McDonald
5f61994087 adds function for decoding fixed sized arrays 2022-09-23 20:50:41 -05:00
David McDonald
3deceeebd6 fixes from PR comments 2022-09-23 20:23:29 -05:00
David McDonald
3198602d25 removed unused return type 2022-09-23 20:23:29 -05:00
David McDonald
369f4016c1 removed unneccessary type conversions 2022-09-23 20:23:29 -05:00
David McDonald
6b04f2de1c Documentation cleanup 2022-09-23 20:23:29 -05:00
David McDonald
368d183b73 Size check on nBits to save memory 2022-09-23 20:23:29 -05:00
David McDonald
a77cec921f Added documentation and tests, fixed bad date parsing 2022-09-23 20:23:23 -05:00
David McDonald
5711f29084 Code fixes from PR, still need to add tests and testdata 2022-09-23 20:21:32 -05:00
David McDonald
d784db69bc Adds support for Apple Binary Plist, version 00 2022-09-23 20:21:32 -05:00
Pavel Safonov
b06c9bc28c postgres: pg_control change default flavour to empty string - it uses versions prober. Fix root name in pg_heap. 2022-09-23 14:05:19 +03:00
Pavel Safonov
de3ecf1636 postgres: generate docs by embedded md 2022-09-23 13:50:54 +03:00
Pavel Safonov
90386a651a postgres: add postgres.md to format 2022-09-23 12:50:04 +03:00
Pavel Safonov
94e5b79bbf Merge branch 'wader_master' into postgres 2022-09-23 12:18:06 +03:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Pavel Safonov
296ce68e80 postgres: refactoring 2022-09-22 13:32:03 +03:00
Pavel Safonov
ae838b921b postgres: move postgres.md to formats.md, add btree tests 2022-09-22 13:25:57 +03:00
Pavel Safonov
7f219d8c5b Merge branch 'wader_master' into postgres 2022-09-21 15:41:52 +03:00
Pavel Safonov
7b52149c6b postgres: refactoring 2022-09-21 15:40:47 +03:00
Pavel Safonov
35124bf2ff postgres: pg_heap refactoring 2022-09-21 15:36:09 +03:00
Pavel Safonov
c3ef341106 postgresql: general logic for pg_heap, pg_btree 2022-09-21 15:21:29 +03:00
Pavel Safonov
7dd7dbee51 postgres: pg_heap reafactoring 2022-09-21 13:40:35 +03:00
Pavel Safonov
9f1adb2d81 postgres: change AssertPosBytes to AssertPos (bits) 2022-09-21 13:36:15 +03:00
Pavel Safonov
edb565028d postgres: pg_control refactoring 2022-09-21 13:14:57 +03:00
Pavel Safonov
00de0a96e3 postgres: pg_heap refactoring 2022-09-21 12:49:40 +03:00
Mattias Wadman
725ab1b17b doc,html,xml: Add more documentation and examples 2022-09-20 17:35:27 +02:00
Pavel Safonov
efda7b3299 postgres: btree handle full file 2022-09-20 18:32:18 +03:00
Pavel Safonov
f3f259af2d postgres: btree add free space 2022-09-20 17:19:01 +03:00
Pavel Safonov
87b7acf39a postgres: btree impl 2022-09-20 16:57:24 +03:00
Pavel Safonov
d370f5d925 postgres: btree impl 2022-09-20 10:11:10 +03:00