Babz
c3a123ad84
luajit: remove unecessary dependency
2023-06-21 18:47:13 +02:00
Sergei Kuzmin
e869d8af03
sgpd box entries parsing
...
Previously entries were parsed as raw bytes, now KIDs/IVs are
explicitely parsed.
2023-06-20 21:57:27 -07:00
Babz
93c96965e0
luajit: add to probe group
2023-06-21 01:41:03 +02:00
Babz
a83576a83a
luajit: clarify description
2023-06-21 01:39:23 +02:00
Babz
b24063be02
luajit: *.fqtest: add comments for generating .luac from source
2023-06-21 01:25:35 +02:00
Babz
52ce81817f
luajit: split in smaller decode functions
2023-06-21 01:02:38 +02:00
Babz
bdf158be9e
luajit: add luajit.md
2023-06-21 00:17:27 +02:00
Babz
715f850d50
luajit: opcodes: implement scalar.UintMapper
2023-06-20 23:55:37 +02:00
Babz
08ae661c15
luajit: use UTF8 strings
2023-06-20 23:33:46 +02:00
Babz
32300a3ffa
luajit: check binary.Read() error
2023-06-20 23:31:35 +02:00
Babz
e44f5c00b8
luajit: magic number: raw bits, check with assert
2023-06-20 23:28:34 +02:00
Babz
29ab66b3eb
luajit: lowercase flags
2023-06-20 23:11:17 +02:00
Babz
db2e621468
go fmt
2023-06-20 23:02:31 +02:00
Babz
1afdf8b1cf
luajit: initial support
2023-06-20 18:24:49 +02:00
Mattias Wadman
1da8069190
mp4: udta: Improve length/lang box probe and support empty value
2023-05-17 11:55:13 +02:00
Mattias Wadman
493848a7e5
mp4: Use correct epoch for quicktime timestamps
...
Somehow was 1904-01-04 should be 1904-01-01
2023-05-16 13:10:56 +02:00
Mattias Wadman
aec2635ea6
Merge pull request #672 from Rogach/pr/fix-tohex
...
fix to_hex function handling of non-ascii characters
2023-05-16 11:02:03 +02:00
Platon Pronko
b0e4da28de
fix non-ascii characters handling in to_hex and to_base64 functions
...
to_hex and to_base64 functions were expecting `string` as an input type,
which caused the value to be CastFn'ed to string,
which in turn resulted in raw bytes being cast to []rune (in makeDecodeValueOut).
This operation replaces invalid UTF-8 bytes with 0xFFFD, which then were
passed on to the hex/base64 encoders, resulting in incorrect output.
This patch fixes it by expecting `any` as an input type,
which allows the function to correctly read raw bytes of the input data.
2023-05-16 09:44:35 +08:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex
...
interp: Add hex bits format
2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df
interp: Add hex bits format
...
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d
Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Platon Pronko
a3361e70ba
bitcoin: fix witness item structs
...
This commit fixes witness item decoding such that they are represented
as structs with `item_size` and `item` fields, instead of a flat list
of raw bytes.
2023-05-15 18:36:26 +08:00
Mattias Wadman
e2eb667091
html: Add to probe group
...
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.
Add lazyre package to help delay compile of RE and make it concurrency safe.
2023-05-11 19:07:18 +02:00
Mattias Wadman
222cd88b77
bits,bytes: Behave as binary instead of raw decode value
...
I think this is closer to what one would expect.
With this fq -d bytes 'scan("...")' will match as a binary and also return binaries,
before as raw decode value one would have to use tobytes to get the binary.
2023-05-11 18:45:16 +02:00
Mattias Wadman
2c505feec8
help,markdown: Rewrote and made text rendering nicer
...
Plan is to use it for man page etc and also some ansi renderer
2023-05-08 00:07:56 +02:00
Mattias Wadman
a200d3eee0
Merge pull request #415 from pnsafonov/postgres
...
Postgres
2023-05-06 09:07:13 +02:00
Pavel Safonov
e6a9cdbecf
postgres: doc
2023-05-06 08:54:56 +03:00
Pavel Safonov
c9350de352
postgres: use bit stream instead of masks to get flags
2023-05-06 08:42:16 +03:00
Pavel Safonov
e839191679
postgres: btree refactored by Mattias Wadman
2023-05-05 11:28:38 +03:00
Pavel Safonov
c23bc4216f
postgres: fix line endings in error messages, simplify code, add comments
2023-05-04 09:16:55 +03:00
Pavel Safonov
08c535239b
postgres: refactoring
2023-05-04 08:45:42 +03:00
Pavel Safonov
f6f8d5c037
postgres: refactoring
2023-05-03 15:40:32 +03:00
Pavel Safonov
9508a209ea
postgres: fix lint
2023-05-03 15:28:29 +03:00
Mattias Wadman
684a083813
interp,decode: Support decode group argument
...
Will be used for probing via -d <string> and filename
2023-05-03 10:01:13 +02:00
Pavel Safonov
e5f15c5fed
postgres: fix compilation, fix tests
2023-05-03 10:10:03 +03:00
Pavel Safonov
bb2659d442
Merge remote-tracking branch 'master' into postgres_merge_master_0
2023-05-03 09:44:02 +03:00
Mattias Wadman
f237db2794
id3v2: Decode CTOC flags
2023-05-02 13:43:58 +02:00
Mattias Wadman
d010dcec06
id3v2: Add WXXX (desc/url) frame support
2023-05-02 11:54:11 +02:00
Mattias Wadman
ddd7b0e439
Merge pull request #650 from matthewdale/fix-bson
...
bson: support all non-deprecated types and fix int/uint bugs
2023-05-02 09:33:25 +02:00
Matt Dale
40630d39b2
bson: fix doc formatting and add author info
2023-05-01 18:41:52 -07:00
Mattias Wadman
d6f785c6a6
pcap: Add forgotten help test
2023-05-01 16:06:22 +02:00
Mattias Wadman
a4a332bf4e
formats: Clenaup naming a bit
2023-05-01 13:19:04 +02:00
Mattias Wadman
7adc1e7030
mp4: Better description for QuickTime terminator atom
...
Addition improvement could be to only allow it in quicktime ftyp brand
and also not as a root book. Currently trailing zero bytes could be
mistaken as a terminator atom.
2023-05-01 12:32:25 +02:00
Matt Dale
9a982d0a76
bson: add BSON test file generator module and correct BSON format docs
2023-04-30 21:25:57 -07:00
Matt Dale
2017ff8766
bson: support all non-deprecated types and fix int/uint bugs
2023-04-30 10:28:42 -07:00
Mattias Wadman
b08ef00dd1
decode,interp: Refactor format groups into a proper struct
...
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.
This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
b2cb5c3c1b
Merge pull request #645 from wader/matroska-filedata-fallback
...
matroska: file_data: Fallback to raw if probe fails
2023-04-20 20:37:26 +02:00
Mattias Wadman
8941b13932
matroska: file_data: Fallback to raw if probe fails
2023-04-20 20:08:40 +02:00
Martin Castillo
b3b6cd0ee3
gzip.go: fix typo in variablename: delfate
2023-04-19 15:51:55 +02:00
Pavel Safonov
c9b263e925
postgres: version 15 support
2023-04-17 17:12:46 +03:00
Mattias Wadman
3c6d31b0fe
mp4: ctts,infe,iinf,trun: More ISOMFF version handling
2023-04-17 11:43:54 +02:00
Mattias Wadman
2a2b64dda2
matroska: Update ebml specification
...
Also add license note about ebml_matroska.xml
2023-04-13 12:19:41 +02:00
Mattias Wadman
e1b02312d0
wav: Cleanup avi leftovers
2023-04-11 19:34:34 +02:00
Mattias Wadman
82da99c917
msgpack: Add str, array and object type tests
...
Related to #634
2023-04-08 13:09:15 +02:00
Mattias Wadman
97360d6ff9
msgpack: fixstr length field is 5 bits
...
Fixes #634
2023-04-08 08:45:26 +02:00
Mattias Wadman
0b6ef2a9d8
golangci-lint: Disable revive unused-parameter and update for new default config
...
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
2023-03-18 22:56:58 +01:00
Mattias Wadman
980ecdba82
decode: Add float 80 reader
...
Rename *d.Bits to UintBits as it return a uint
Add *d.Bits that return []byte
2023-03-10 01:15:55 +01:00
Mattias Wadman
dc4a82eeed
aiff: Add basic decoder
2023-03-09 15:16:52 +01:00
Mattias Wadman
cc52a4419d
id3v2: Decode subframes for CTOC and add struct for headers
2023-03-05 19:29:22 +01:00
Mattias Wadman
9852f56b74
tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
...
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.
What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet
Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.
Will be base for decoding http2 and other TLS based on protocols.
Fixes #587
2023-03-05 13:52:12 +01:00
Mattias Wadman
c75a83c829
help: Show default option value as JSON
2023-03-04 09:53:33 +01:00
Mattias Wadman
dec433fc53
help,markdown: Fix double line breaks when converting to text
...
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00
Mattias Wadman
6c032455eb
pcap,pcapng,ipv4,ipv6: Support raw link type (ipv4 or ipv6)
2023-03-02 18:37:00 +01:00
Mattias Wadman
bdd6718ddb
zip: Correctly peek for zip64 EOCD
2023-02-22 22:22:36 +01:00
Mattias Wadman
5228fdd6cd
zip: Correctly look for and decode both zip32/64 EOCD record
...
There will always be zip(32) EOCD but optinally a zip64 EOCD
Related to #586
2023-02-22 19:34:28 +01:00
Mattias Wadman
56edb59e83
toml,xml: Fail fast on invalid content
...
encoding/xml and github.com/BurntSushi/toml both reads a lot before detecting
that it can't decode. Now we instead read one UTF-8 and make sure it's valid
xml or toml.
Should speed up probing
Related to #586 bigzero-zip.zip
2023-02-22 16:23:21 +01:00
Mattias Wadman
d4ea6632fa
pcap: Add ipv4 fragments tcp test
2023-02-20 11:45:12 +01:00
Mattias Wadman
c8666eeb04
ipv4_packet,ipv6_packet,sll_packet,sll2_packet: Support ipv4/ipv6 link frames and pass correct in arg
...
ipv4/ipv6 in sll* accidentally worked as it passed wrong arg with ether type
2023-02-20 01:34:38 +01:00
Mattias Wadman
8e0dde03d0
decode: Support multiple format args and some rename and refactor
...
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.
Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.
This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
a1bb630a2a
doc,fq: Improve cli help and some cleanup
2023-02-15 11:04:39 +01:00
Mattias Wadman
9aaf2ddf27
matroska: Add unknown size test and add description to ebml header
2023-02-10 10:29:25 +01:00
Mattias Wadman
c890a2899f
matroska: Update spec and make refs in descriptions look nicer
2023-02-09 23:18:17 +01:00
Mattias Wadman
a8d0bf4d3e
matroska: Assume master with unknown size has ended if a valid parent is found
...
Major refactor of ebml code generator, now decoupled from fq code
Cleanup element descriptions a bit, a bit shorter and less clutter
Cleanup old comments
2023-02-09 20:10:47 +01:00
Mattias Wadman
0d14d7b40f
matroska: Handle unknown size for non-master types a bit better
...
For non-master types assume size to be rest of file
Still does not follow the ebml spec
2023-02-08 12:14:40 +01:00
Mattias Wadman
b60aceca9e
matroska: Add decode_samples option
...
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
2023-02-08 11:31:41 +01:00
Mattias Wadman
dc79a73b72
interp,json: Move error handling to colorjson
...
Cancel error from ValueFn etc will be return by Marshal instead
2023-02-07 16:57:54 +01:00
Mattias Wadman
5c8e115106
colorjson: Refactor to option struct
2023-02-07 16:18:32 +01:00
Mattias Wadman
62e2cef5c2
tcp_segment: Decode standard options and rename maxseg to mss
2023-01-31 12:42:16 +01:00
Mattias Wadman
1eb5e502af
tcp: Ignore TCP option check for now as it seems unreliable in dumps
...
For example MSS can be to small in local dumps
2023-01-28 20:49:20 +01:00
Mattias Wadman
a2cdb3d6c9
Merge pull request #558 from wader/mp4-udta-with-no-length
...
mp4: udta: Handle box with value rest of box
2023-01-25 16:34:58 +01:00
Mattias Wadman
6340365841
mp4: udta: Handle box with value rest of box
...
Try distinguish by probing length field. Should probably be improved, what does ffmpeg do?
Regression from c3e3b3e90d
#553
2023-01-25 16:24:28 +01:00
Mattias Wadman
0d1fdb5edf
Merge pull request #557 from wader/mp3-max-unknown-opt
...
mp3: Add max_unknown option to fail decode if too much unknown bits
2023-01-25 16:00:49 +01:00
Mattias Wadman
2d82c05f64
mp3: Add max_unknown option to fail decode if too much unknown bits
...
Hopefully help fix even more miss-detections
2023-01-25 15:27:31 +01:00
Mattias Wadman
b04a650bae
flac_picture,mpeg: Fix trailing ")" typo in map sym and description
2023-01-25 14:01:34 +01:00
Mattias Wadman
d645e71008
Merge pull request #554 from wader/mp4-ftyp-qt-minor-desc
...
mp4: Decode qt minor verison as YYYY.MM description
2023-01-16 12:23:42 +01:00
Mattias Wadman
f386a5158e
mp4: Decode qt minor verison as YYYY.MM description
2023-01-16 12:13:19 +01:00
Mattias Wadman
c3e3b3e90d
mp4: Decode udta metadata boxes without meta box
2023-01-16 12:03:53 +01:00
Mattias Wadman
c2795a7c2f
Merge pull request #552 from wader/mp4-tkhd-flags
...
mp4: Decode tkhd flags
2023-01-13 18:02:53 +01:00
Mattias Wadman
3555dc6721
mp4: Decode tkhd flags
2023-01-13 17:49:03 +01:00
Mattias Wadman
c49012dbca
mp4: sgpd,sbgp: Change grouping_type to a string
...
Is defined as int32 but is a string in practive it seems
2023-01-05 19:18:48 +01:00
Mattias Wadman
8b49b42fa3
interp: Wrap Binary in decodeValue to fix prompt issue with bits/bytes format
2023-01-04 15:30:05 +01:00
Pavel Safonov
bffa00835d
postgres: fix error in tests
2022-12-26 14:47:23 +03:00
Pavel Safonov
97bbc22a1d
postgres: add page arg in pg_btree, change args names in pg_heap
2022-12-22 11:22:02 +03:00
Mattias Wadman
e3ae1440c9
interp: Rename to/from<format> functions to to_/from_<format>
...
Feels less cluttered, easier to read and more consistent.
Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.
Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
8f39ef6335
bplist: Harmonize ns_keyed_archive jq style a bit
...
Let root arg be a lambda so torepr can be done once
2022-12-20 00:56:19 +01:00
Mattias Wadman
6f354bb350
Merge pull request #527 from dgmcdona/apple_package
...
decode: add ns_keyed_archiver, restructure apple decoders into package
2022-12-19 22:47:29 +01:00
David McDonald
129b4b7026
bplist: doc: update docs to reflect changes to ns_keyed_archiver
2022-12-19 15:24:57 -06:00
David McDonald
a9047c0278
bplist: updates from_ns_keyed_archiver to do automatic torepr based on format detection
2022-12-19 15:18:05 -06:00
David McDonald
9dab3c606c
bplist: minor fix to from_ns_keyed_archiver
2022-12-19 14:22:17 -06:00
David McDonald
448c3efb28
bplist: update docs with from_ns_keyed_archiver reference, add error case to function
2022-12-19 13:30:34 -06:00
Mattias Wadman
a85da29544
decode: Make FieldFormat usage more consistent
...
Will make it easier to refactor later on
2022-12-19 19:12:57 +01:00
David McDonald
cba72dbdf0
bplist: added overload for from_ns_keyed_archiver jq func
2022-12-19 02:04:23 -06:00
David McDonald
98eab8cb5b
decode: rename parameter for consistency
2022-12-17 13:36:16 -06:00
David McDonald
93f2aa5d73
decode: change PosLoopDetector to use generics
2022-12-17 13:26:40 -06:00
David McDonald
3232f9cc15
decode: moves PosLoopDetector into its own package
2022-12-17 02:18:48 -06:00
David McDonald
fa368bb790
decode: updates all.go with correct macho path
2022-12-17 01:57:23 -06:00
David McDonald
7c9504c727
decode: moves macho decoder to apple package
2022-12-17 01:54:59 -06:00
David McDonald
708346784f
decode: remove dead code from ns_keyed_archiver
2022-12-17 01:52:46 -06:00
David McDonald
3ec0ba3f25
decode: add ns_keyed_archiver, restructure apple decoder into apple package
2022-12-17 01:47:27 -06:00
Mattias Wadman
1dea40e6af
mp4,doc: Add JSON box tree example and reorder a bit
2022-12-15 14:15:31 +01:00
Mattias Wadman
502e1e3059
Merge pull request #525 from wader/mp3-tags-split-decoder
...
mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,v…
2022-12-15 12:20:16 +01:00
Mattias Wadman
9e5a072e50
mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders
2022-12-15 12:10:06 +01:00
Mattias Wadman
7b60b24ab7
mp4: Add pdin box support
2022-12-15 11:42:05 +01:00
Mattias Wadman
9b81d4d3ab
decode: More type safe API and split scalar into multiple types
...
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
48522e3cb8
mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one
2022-12-14 11:34:53 +01:00
Mattias Wadman
0c11ab5354
Merge pull request #521 from wader/mp4-ilst-namespace
...
mp4: Add namespace to mdta ilst boxes
2022-12-14 11:26:28 +01:00
Mattias Wadman
b1b3b63d27
mp4: Add namespace to mdta ilst boxes
2022-12-14 10:49:02 +01:00
David McDonald
7fb674b532
decode: unexport methods
2022-12-13 00:47:58 -06:00
David McDonald
f747873dbc
decode: implements lost and found for unreferenced objects
2022-12-12 23:48:33 -06:00
David McDonald
a873819ea2
decode: fixes endian of unicode strings
2022-12-10 21:04:56 -06:00
Mattias Wadman
0b9426976f
Merge pull request #493 from dgmcdona/macos_bookmark
...
Apple bookmarkData
2022-12-10 08:58:33 +01:00
David McDonald
bdb8166261
decode: removed unnecessary struct
2022-12-09 20:13:25 -06:00
Mattias Wadman
9ac453a17d
mp4: Fix typo in sample flags sample_is_depended_on description
2022-12-09 17:22:17 +01:00
Mattias Wadman
5fb81a148b
mp4: Add sym and description for traf sample flags
2022-12-09 16:22:43 +01:00
David McDonald
7e98b53894
decode: fix type on defer function call, test: add loop.fqtest
2022-12-08 15:01:16 -06:00
Mattias Wadman
1d6ce2c07a
mp4: Decode hint and hnti child boxes
2022-12-08 20:47:33 +01:00
Mattias Wadman
83ccedc506
mp4,decode: Properly decode ilst items (both mdta and mdir)
...
Refactor mp4 decoder to be simpler and have fallback for unknown box type
Cleanup some old ilst hacks
Add generic string reader to decode API that takes an encoding parameters
2022-12-08 15:56:22 +01:00
David McDonald
8d5dcff8e3
test: applebookmark: adds problematic test case
2022-12-08 01:48:59 -06:00
David McDonald
330d5f7f28
decode: apple_bookmark: simplifies flag decoding
2022-12-08 01:47:00 -06:00
David McDonald
a162e07b76
decode: minor change to method receiver name
2022-12-08 01:33:38 -06:00
David McDonald
04379df8d4
decode: revert decode.D back, place posLoopDetector in apple_bookmark
2022-12-07 16:25:34 -06:00
David McDonald
63a4e80c8b
test: fixed doc test
2022-12-06 19:46:25 -06:00
David McDonald
3942db79c2
pkg/decode/D: Adds PushAndPop, Push, Pop methods. doc: adds help_applebookmark.fqtestdecode: converts applebookmark to use new d.PushAndPop method
2022-12-06 19:40:28 -06:00
David McDonald
34c560c2f5
Merge branch 'master' of https://github.com/wader/fq into macos_bookmark
2022-12-05 22:01:44 -06:00
David McDonald
7ab446626e
decode: remove unused field from decoder, unused parens from torepr
2022-12-05 21:37:36 -06:00
Mattias Wadman
47a568e076
text,test: Unbeak base64 tests
2022-12-05 20:51:30 +01:00
Mattias Wadman
a23fe618b0
mp4: sidx version 1 segment_duration is s64
2022-12-05 17:04:05 +01:00
Mattias Wadman
2c4862c9fb
Merge pull request #498 from bitbears-dev/master
...
support tzif (time zone information format)
2022-12-04 13:21:19 +01:00
@0xb17bea125
fd22426b3d
tzif: add help_tzif.fqtest
2022-12-04 21:04:15 +09:00
@0xb17bea125
c4e7fc7919
tzif: moved document to tzif.md
2022-12-04 21:04:13 +09:00
@0xb17bea125
4481a77a1c
tzif: use scalar.Fn() to define a mapper ad hoc
2022-12-04 21:02:39 +09:00
@0xb17bea125
abde823a79
tzif: use PeekFindByte() to find end of the string
2022-12-04 21:02:39 +09:00
@0xb17bea125
2ee6360b97
support tzif (time zone information format)
2022-12-04 21:02:39 +09:00
Mattias Wadman
42debe5871
dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg
...
Update format dev docs to only generate README.md and doc/foramts.md to not
cause formats.svg conflicts.
2022-12-04 12:37:10 +01:00
David McDonald
b45f9fa67b
decode: improve stack push/pop
2022-12-03 15:27:59 -06:00
David McDonald
62f377c2fe
doc: fixes snippet for recursive bookmark searching
2022-12-03 15:24:49 -06:00
Mattias Wadman
6fc84a885c
doc,dev: Add more usage and dev tips
2022-12-03 19:00:47 +01:00
David McDonald
d199793af6
created stack type
2022-12-02 00:29:29 -06:00
David McDonald
b2aeac6ad3
updates bplist fq tests
2022-12-01 23:45:48 -06:00
David McDonald
55f4f1aa04
moved a flag bit fields into correct positions
2022-12-01 23:31:57 -06:00
David McDonald
c7d00b87b6
fixes unknown bit ranges
2022-12-01 23:31:57 -06:00
David McDonald
cb15b37175
added checks to prevent infinite looping and recursion
2022-12-01 23:31:57 -06:00
David McDonald
9bd65f93b7
migrates tests to per-sample files
2022-12-01 23:31:57 -06:00
David McDonald
7d13cf7312
adds flag parsing to applebookmark
2022-12-01 23:31:57 -06:00
David McDonald
1352598af9
removed commented out line
2022-12-01 23:31:57 -06:00
David McDonald
a23ac8f553
updates fqtest for torepr in apple_bookmarkdata
2022-12-01 23:31:57 -06:00
David McDonald
1e1fc551eb
fixed one more snake_case letter
2022-12-01 23:31:57 -06:00
David McDonald
d0b76cae23
fixes broken test and removes long link from markdown body
2022-12-01 23:31:57 -06:00
David McDonald
c244533540
added some sfl2 test files to bplist package
2022-12-01 23:31:57 -06:00
David McDonald
64351e8b44
fixes tests and adds torepr test
2022-12-01 23:31:57 -06:00
David McDonald
f7d7a49fb6
missed a letter on last commit - converting to snake_case
2022-12-01 23:31:57 -06:00
David McDonald
0c216dff48
refactors some decoder logic in apple_bookmark for better querying
2022-12-01 23:31:57 -06:00
David McDonald
44c91d8296
tweaks apple_bookmark markdown documentation
2022-12-01 23:31:57 -06:00
David McDonald
f535ad3de9
fixes spacing in jq files
2022-12-01 23:31:57 -06:00
David McDonald
a351c346f6
removes unused function
2022-12-01 23:31:57 -06:00
David McDonald
253033cc4c
fixes broken uid parsing in plist decoder
2022-12-01 23:31:57 -06:00
David McDonald
63a3ca20c4
removes underscore from apple_bookmark package name
2022-12-01 23:31:57 -06:00
David McDonald
4a28e44f44
changes decoder package name from bookmark to apple_bookmark
2022-12-01 23:31:57 -06:00
David McDonald
81269430fe
removed unnecessary conversions
2022-12-01 23:31:57 -06:00
David McDonald
d0b044c2f0
converts to snake_case and refactors decode helper
2022-12-01 23:31:57 -06:00
David McDonald
5146f28dbb
fixes broken test for all.fqtest
2022-12-01 23:31:57 -06:00
David McDonald
5b1455e777
removed unused function
2022-12-01 23:31:57 -06:00
David McDonald
6f4d1cb1de
updated documentation
2022-12-01 23:31:57 -06:00
David McDonald
dbc6fccdda
updated doc with apple reference
2022-12-01 23:31:57 -06:00
David McDonald
34db9d7fea
regenerated docs, added tests, fixed torepr
2022-12-01 23:31:57 -06:00
David McDonald
8f930aacee
forgot to add bookmark.jq in last commit
2022-12-01 23:31:57 -06:00
David McDonald
0287ffa4dc
decoding well but torepr needs work
2022-12-01 23:31:56 -06:00
David McDonald
71b17d0382
apple bookmarkdata decoder initial commit
2022-12-01 23:31:56 -06:00
Mattias Wadman
e77f776999
decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
...
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
8d69f1fb23
interp: Change default bits_format=string
...
I think this is more intuitive but might in some case cause very large JSON output
but maybe that less common or expected. In does cases i think you either want to
use some other bits_format (md5, truncate, etc) or you delete/transform the jq value
before turn it into JSON.
Strings in gojq are binary safe so you can use to hold raw bytes. But note that
convert the binary into JSON is lossy, same as the JSON standard.
Add bits_format option documentation.
2022-12-01 17:49:34 +01:00
Mattias Wadman
42d9f2c27d
interp,help: Properly count line length when breaking on whole words
...
Whitespace between words were not counted
2022-11-29 18:59:32 +01:00
Mattias Wadman
ef2d5232ba
mp4: Add png mapping
...
Also support samples decode for tracks without hdlr box
2022-11-25 19:49:18 +01:00
Mattias Wadman
d75748d816
mp4: Decode more sample flags
2022-11-23 14:29:13 +01:00
Pavel Safonov
0ea20e6827
postgres: fail on error in pg_heap
2022-11-23 14:14:51 +03:00
Pavel Safonov
6fe61cd634
Merge branch 'wader_master' into postgres_wal_removed
2022-11-23 13:57:50 +03:00
Pavel Safonov
dafbf4b7ed
postgres: lint fixes
2022-11-23 13:56:49 +03:00
Pavel Safonov
60709e5a00
postgres: remove pg_wal. Failed to implement.
2022-11-23 13:55:07 +03:00
Mattias Wadman
646f32d563
matroska: Fix path tests and make _tree_path more robust
2022-11-20 20:16:41 +01:00
Mattias Wadman
c93301fc70
raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
...
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
0cf46e1196
wav,avi,avc_au: Add avi decoder and refactor wav decoder
...
avi and wav now share common riff decode code
avc_au decoder can now handle annexb format
2022-11-19 13:11:27 +01:00
Mattias Wadman
c3a0686c9e
mp3_frame_tags: Refactor and rename xing format to mp3_frame_tags
...
Make more sense to use a more general name when there is both info and xing
and possibly vbri in the future. Also both xing and info seems to have extension
fields
Remove some old tests that are not really needed anymore.
2022-11-16 17:32:03 +01:00
Mattias Wadman
2de875393a
mp3_frame: Fix issue calc frame size for some configs
...
Seems have to use the 144000*bitrate/sameplrate to get correct size or
one would miss one byte (odd size rounded to even?)
2022-11-16 15:55:10 +01:00
Pavel Safonov
015b770585
postgres: wal support multiple xlog_body for wal record
2022-11-10 11:32:25 +03:00
Pavel Safonov
e8bb1692f6
postgres: fix pg_wal when XLogRecord size is more than page size
2022-11-09 16:57:56 +03:00
Pavel Safonov
5664c0a498
postgres: refactor ItemIdData
2022-11-07 13:35:23 +03:00
Pavel Safonov
5bb8654423
postgres: remove lsn parameter in pg_wal
2022-11-07 12:05:33 +03:00
Pavel Safonov
9c7f20b6a0
Merge branch 'postgres' into postgres_wal_sync_with_postgres
2022-11-01 08:26:17 +03:00
Pavel Safonov
cdf283101f
Merge branch 'wader_master' into postgres
2022-11-01 08:25:07 +03:00
Pavel Safonov
a4d904e1c2
postgres: refactoring
2022-10-31 14:02:08 +03:00
Pavel Safonov
721c1ab3bf
psotgres: refactoring
2022-10-31 13:56:56 +03:00
Pavel Safonov
6e2e44d682
postgres: remove arg in pg_btree
2022-10-31 12:55:09 +03:00
Pavel Safonov
7fd41090e3
postgres: add btree index tests
2022-10-31 11:07:10 +03:00
Pavel Safonov
3c7ed5d73c
postgres: add tests data
2022-10-31 09:32:31 +03:00
Pavel Safonov
de68785a8d
postgres: add test files
2022-10-31 08:36:57 +03:00
Pavel Safonov
7f7f729c90
postgres: refactoring, tests
2022-10-28 13:55:38 +03:00
Pavel Safonov
069babbc0d
postgres: wal refactoring
2022-10-28 13:42:42 +03:00
Pavel Safonov
d7a0f930df
postgres: pgpro wal refactoring
2022-10-28 13:13:33 +03:00
Pavel Safonov
e8d8caca41
postgres: pgpro wal implementation
2022-10-28 13:09:09 +03:00
Pavel Safonov
fb7778a5b0
postgres: add wal tests
2022-10-27 12:08:40 +03:00
Pavel Safonov
7dedcbabae
postgres: refactoring
2022-10-26 16:37:50 +03:00
Pavel Safonov
bd2bdd649a
postgres: wal refactoring
2022-10-26 10:13:50 +03:00
Pavel Safonov
7c92715f2a
postgres: refactoring
2022-10-26 10:11:23 +03:00
Pavel Safonov
7c1dfbd090
postgres: wal implementation
2022-10-26 10:10:47 +03:00
Pavel Safonov
26bff1442f
postgres: wal refactoing
2022-10-26 10:02:38 +03:00
Pavel Safonov
b09ec2fc94
postgres: wal refactoing
2022-10-26 10:01:21 +03:00
Pavel Safonov
c105fcdd90
postgres: wal impl
2022-10-26 09:56:00 +03:00
Pavel Safonov
a4c1c5b811
postgres: wal const
2022-10-24 13:23:28 +03:00
Pavel Safonov
6aed2387b8
postgres: add pg_wal for pgproee11 as copy of postgres14
2022-10-24 13:04:54 +03:00
Pavel Safonov
f4408fb072
Merge branch 'postgres' into postgres_wal_sync_with_postgres
2022-10-24 13:01:03 +03:00
Pavel Safonov
9f61e63702
postgres: allow all flovours to decode btree index
2022-10-24 09:47:27 +03:00
Pavel Safonov
5ff6273505
postgres: add additional checks in pg_heap
2022-10-24 09:44:42 +03:00
Mattias Wadman
f66e224452
id3v2: In the wild COMM and TXXX frame might not have a null terminator
2022-10-20 22:07:37 +02:00
Mattias Wadman
fb2a2b949c
flac,fuzz: Fatal error on negative partition sample count
2022-10-18 13:40:48 +02:00
Pavel Safonov
8f55e177f6
postgres: remove SeekAbs(0) where it's not used.
2022-10-12 10:50:00 +03:00
Pavel Safonov
dd9c7f0420
Merge branch 'postgres' into postgres_wal_sync_with_postgres
2022-10-12 10:43:59 +03:00
Mattias Wadman
2606916753
yaml,fuzz: gojq.Normalize value to fix type panic
2022-10-11 19:22:49 +02:00
Mattias Wadman
7859be1e5d
flac_frame: Properly decode zero escape sample size
2022-10-11 17:05:19 +02:00
Mattias Wadman
3a578152f2
Merge pull request #459 from wader/decode-s-must-least-1bit
...
decode,fuzz: Signed integer (S) read require at least one bit
2022-10-11 12:41:28 +02:00
Mattias Wadman
bafd1f564e
decode,fuzz: Signed integer (S) read require at least one bit
...
Otherwise there is no sign bit and it will panic on shifting.
Maybe should require 2? now -0 and +1 hmm
2022-10-11 12:12:28 +02:00
Mattias Wadman
cef4245b49
fuzz: make fuzz GROUP=mp4 to fuzz one group
2022-10-10 21:07:10 +02:00
Pavel Safonov
83d63c587c
Merge branch 'wader_master' into postgres
2022-10-10 13:32:13 +03:00
Pavel Safonov
939c7c17ad
postgres: how_to.md
2022-10-10 13:25:43 +03:00
Pavel Safonov
1aa08e336c
postgres: add btree, pg_control to how_to.md
2022-10-10 13:24:04 +03:00
Pavel Safonov
2423f86b3a
postgres: add how_to.md - how to generate test files for postgres
2022-10-10 13:08:03 +03:00
Mattias Wadman
e9aa4837e9
Merge pull request #454 from dgmcdona/reference_size_bug_fix
...
fixes reference calculation to use reference size from trailer
2022-10-09 21:48:12 +02:00
David McDonald
b09d61162b
makes dictionary key type checking more sensible
2022-10-09 13:34:00 -05:00
Mattias Wadman
2a86d323b4
doc,rtmp,pcap,markdown: Add more examples
2022-10-09 19:18:14 +02:00
Mattias Wadman
e748079e87
matroska: Update spec and regenerate
...
The new type "TargetType" used in labels is bit confusing, should be handled differently?
2022-10-08 15:45:46 +02:00
David McDonald
ac86f9318b
fixes signed integer parsing
2022-10-07 21:33:26 -05:00
David McDonald
2b3adbe8d6
renames test data file
2022-10-07 18:08:20 -05:00
David McDonald
4ee7dd8a38
changes Errorf to Fatalf on infinite loops
2022-10-07 18:04:07 -05:00
David McDonald
4d8dd5c5e3
adds check for recursion in decodeReference, adds test to verify fix
2022-10-07 15:07:05 -05:00
David McDonald
413d425079
gofmt
2022-10-07 13:16:17 -05:00
David McDonald
fb1a91ac3c
drop indented else block per lint
2022-10-07 13:15:22 -05:00
David McDonald
236fbc17b5
fixes reference calculation to use reference size from trailer
2022-10-07 13:07:51 -05:00
Pavel Safonov
6fe5f05faf
postgres: lint, doc
2022-10-07 14:17:04 +03:00
Pavel Safonov
bedc480a51
postgres: refactoring
2022-10-07 14:12:36 +03:00
Pavel Safonov
e06fa6e139
postgres: refactoring
2022-10-07 13:33:05 +03:00
Pavel Safonov
4676590624
postgres: add argument to calc page's check sum correctly
2022-10-07 12:58:36 +03:00
Pavel Safonov
46e1e337ff
postgres: make page size const
2022-10-07 12:04:49 +03:00
Pavel Safonov
03d8fe1c61
postgres: page sum impl
2022-10-07 11:58:03 +03:00
David McDonald
4dd594c14e
fixes bad path in test output
2022-10-07 00:50:48 -05:00
David McDonald
46b7ab3268
adds test to verify fix
2022-10-07 00:23:25 -05:00
David McDonald
b08e25ce78
removes unnecessary cast
2022-10-06 23:49:15 -05:00
David McDonald
b7c4576c5e
adds necessary cast
2022-10-06 23:47:10 -05:00
David McDonald
f9a1d3f444
fixes calculation of floating point lengths
2022-10-06 23:23:04 -05:00
Pavel Safonov
6588e1dd88
Merge branch 'wader_master' into postgres
2022-10-05 19:39:35 +03:00
Pavel Safonov
e57c3b980b
postgres: refactoring
2022-10-05 19:37:42 +03:00
Pavel Safonov
78583731dc
postgres: postgres 10 support
2022-10-05 13:16:39 +03:00
Mattias Wadman
16b01211b1
bplist: Make torepr convert to values
2022-10-04 18:12:49 +02:00
Mattias Wadman
cb3dc80252
decode,tar: Add scalar description and Try* helpers
...
Rework time helpers to use new functions
Fix panic in tar decoder where sym value might be missing
Still not very happy about the API but it's getting better.
2022-10-04 17:18:51 +02:00
Mattias Wadman
421b2b3071
bplist: Fix unknown field for singletons and add torepr tests
2022-10-04 14:30:54 +02:00
Mattias Wadman
09ea08f3aa
Merge pull request #427 from dgmcdona/bplist
...
Adds support for Apple Binary Plist, version 00
2022-10-04 14:25:53 +02:00
Mattias Wadman
20a15372ba
mp4: Fix data_format typo
2022-10-02 23:31:38 +02:00
Mattias Wadman
0fdc03a4f1
doc: Fix some incorrect example prompts
2022-09-27 13:42:31 +02:00
Mattias Wadman
d6316d5cbf
wav: Decode smpl chunk
2022-09-26 11:28:02 +02:00
Pavel Safonov
9deab2ea1b
postgres: pg_heap fix page_begin, page_end
2022-09-26 09:10:03 +03:00
David McDonald
1e47f4f2f7
updates tests post integer-bug fix
2022-09-25 00:48:43 -05:00
David McDonald
ebae938d0e
Fixes bug in integer parsing
2022-09-25 00:43:21 -05:00
David McDonald
bcccde2358
Fixes and embeds documentation
2022-09-24 12:57:11 -05:00
David McDonald
3d8ea1deac
updates torepr for data type
2022-09-23 22:12:30 -05:00
David McDonald
12836abe26
updates fqtest
2022-09-23 21:16:28 -05:00
David McDonald
5f61994087
adds function for decoding fixed sized arrays
2022-09-23 20:50:41 -05:00
David McDonald
3deceeebd6
fixes from PR comments
2022-09-23 20:23:29 -05:00
David McDonald
3198602d25
removed unused return type
2022-09-23 20:23:29 -05:00
David McDonald
369f4016c1
removed unneccessary type conversions
2022-09-23 20:23:29 -05:00
David McDonald
6b04f2de1c
Documentation cleanup
2022-09-23 20:23:29 -05:00
David McDonald
368d183b73
Size check on nBits to save memory
2022-09-23 20:23:29 -05:00
David McDonald
a77cec921f
Added documentation and tests, fixed bad date parsing
2022-09-23 20:23:23 -05:00
David McDonald
5711f29084
Code fixes from PR, still need to add tests and testdata
2022-09-23 20:21:32 -05:00
David McDonald
d784db69bc
Adds support for Apple Binary Plist, version 00
2022-09-23 20:21:32 -05:00
Pavel Safonov
b06c9bc28c
postgres: pg_control change default flavour to empty string - it uses versions prober. Fix root name in pg_heap.
2022-09-23 14:05:19 +03:00
Pavel Safonov
de3ecf1636
postgres: generate docs by embedded md
2022-09-23 13:50:54 +03:00
Pavel Safonov
90386a651a
postgres: add postgres.md to format
2022-09-23 12:50:04 +03:00
Pavel Safonov
94e5b79bbf
Merge branch 'wader_master' into postgres
2022-09-23 12:18:06 +03:00
Mattias Wadman
a7a101ca67
doc,help: Nicer format help and move help tests into each format
...
Also add authors to avro and macho decoders
Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Pavel Safonov
296ce68e80
postgres: refactoring
2022-09-22 13:32:03 +03:00
Pavel Safonov
ae838b921b
postgres: move postgres.md to formats.md, add btree tests
2022-09-22 13:25:57 +03:00
Pavel Safonov
7f219d8c5b
Merge branch 'wader_master' into postgres
2022-09-21 15:41:52 +03:00
Pavel Safonov
7b52149c6b
postgres: refactoring
2022-09-21 15:40:47 +03:00
Pavel Safonov
35124bf2ff
postgres: pg_heap refactoring
2022-09-21 15:36:09 +03:00
Pavel Safonov
c3ef341106
postgresql: general logic for pg_heap, pg_btree
2022-09-21 15:21:29 +03:00
Pavel Safonov
7dd7dbee51
postgres: pg_heap reafactoring
2022-09-21 13:40:35 +03:00
Pavel Safonov
9f1adb2d81
postgres: change AssertPosBytes to AssertPos (bits)
2022-09-21 13:36:15 +03:00
Pavel Safonov
edb565028d
postgres: pg_control refactoring
2022-09-21 13:14:57 +03:00
Pavel Safonov
00de0a96e3
postgres: pg_heap refactoring
2022-09-21 12:49:40 +03:00
Mattias Wadman
725ab1b17b
doc,html,xml: Add more documentation and examples
2022-09-20 17:35:27 +02:00
Pavel Safonov
efda7b3299
postgres: btree handle full file
2022-09-20 18:32:18 +03:00
Pavel Safonov
f3f259af2d
postgres: btree add free space
2022-09-20 17:19:01 +03:00
Pavel Safonov
87b7acf39a
postgres: btree impl
2022-09-20 16:57:24 +03:00
Pavel Safonov
d370f5d925
postgres: btree impl
2022-09-20 10:11:10 +03:00