Babz
751ee5a328
luajit: explain LuaJITDecodeKNum, fix negative in bug
2023-06-21 23:53:44 +02:00
Babz
64c11bedfa
luajit: improve debuginfo decoding
2023-06-21 22:33:23 +02:00
Babz
eb819dd479
luajit: tests: improve coverage
2023-06-21 20:53:47 +02:00
Babz
441d246de6
luajit: standardize field names (key/value/type ect.)
2023-06-21 20:42:15 +02:00
Babz
3561c08a8c
luajit: fallbackUintMapSymStr
2023-06-21 20:34:56 +02:00
Babz
c3a123ad84
luajit: remove unecessary dependency
2023-06-21 18:47:13 +02:00
Sergei Kuzmin
e869d8af03
sgpd box entries parsing
...
Previously entries were parsed as raw bytes, now KIDs/IVs are
explicitely parsed.
2023-06-20 21:57:27 -07:00
Babz
93c96965e0
luajit: add to probe group
2023-06-21 01:41:03 +02:00
Babz
a83576a83a
luajit: clarify description
2023-06-21 01:39:23 +02:00
Babz
b24063be02
luajit: *.fqtest: add comments for generating .luac from source
2023-06-21 01:25:35 +02:00
Babz
52ce81817f
luajit: split in smaller decode functions
2023-06-21 01:02:38 +02:00
Babz
bdf158be9e
luajit: add luajit.md
2023-06-21 00:17:27 +02:00
Babz
715f850d50
luajit: opcodes: implement scalar.UintMapper
2023-06-20 23:55:37 +02:00
Babz
08ae661c15
luajit: use UTF8 strings
2023-06-20 23:33:46 +02:00
Babz
32300a3ffa
luajit: check binary.Read() error
2023-06-20 23:31:35 +02:00
Babz
e44f5c00b8
luajit: magic number: raw bits, check with assert
2023-06-20 23:28:34 +02:00
Babz
29ab66b3eb
luajit: lowercase flags
2023-06-20 23:11:17 +02:00
Babz
db2e621468
go fmt
2023-06-20 23:02:31 +02:00
Babz
1afdf8b1cf
luajit: initial support
2023-06-20 18:24:49 +02:00
Mattias Wadman
1da8069190
mp4: udta: Improve length/lang box probe and support empty value
2023-05-17 11:55:13 +02:00
Mattias Wadman
493848a7e5
mp4: Use correct epoch for quicktime timestamps
...
Somehow was 1904-01-04 should be 1904-01-01
2023-05-16 13:10:56 +02:00
Mattias Wadman
aec2635ea6
Merge pull request #672 from Rogach/pr/fix-tohex
...
fix to_hex function handling of non-ascii characters
2023-05-16 11:02:03 +02:00
Platon Pronko
b0e4da28de
fix non-ascii characters handling in to_hex and to_base64 functions
...
to_hex and to_base64 functions were expecting `string` as an input type,
which caused the value to be CastFn'ed to string,
which in turn resulted in raw bytes being cast to []rune (in makeDecodeValueOut).
This operation replaces invalid UTF-8 bytes with 0xFFFD, which then were
passed on to the hex/base64 encoders, resulting in incorrect output.
This patch fixes it by expecting `any` as an input type,
which allows the function to correctly read raw bytes of the input data.
2023-05-16 09:44:35 +08:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex
...
interp: Add hex bits format
2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df
interp: Add hex bits format
...
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d
Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Platon Pronko
a3361e70ba
bitcoin: fix witness item structs
...
This commit fixes witness item decoding such that they are represented
as structs with `item_size` and `item` fields, instead of a flat list
of raw bytes.
2023-05-15 18:36:26 +08:00
Mattias Wadman
e2eb667091
html: Add to probe group
...
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.
Add lazyre package to help delay compile of RE and make it concurrency safe.
2023-05-11 19:07:18 +02:00
Mattias Wadman
222cd88b77
bits,bytes: Behave as binary instead of raw decode value
...
I think this is closer to what one would expect.
With this fq -d bytes 'scan("...")' will match as a binary and also return binaries,
before as raw decode value one would have to use tobytes to get the binary.
2023-05-11 18:45:16 +02:00
Mattias Wadman
2c505feec8
help,markdown: Rewrote and made text rendering nicer
...
Plan is to use it for man page etc and also some ansi renderer
2023-05-08 00:07:56 +02:00
Mattias Wadman
a200d3eee0
Merge pull request #415 from pnsafonov/postgres
...
Postgres
2023-05-06 09:07:13 +02:00
Pavel Safonov
e6a9cdbecf
postgres: doc
2023-05-06 08:54:56 +03:00
Pavel Safonov
c9350de352
postgres: use bit stream instead of masks to get flags
2023-05-06 08:42:16 +03:00
Pavel Safonov
e839191679
postgres: btree refactored by Mattias Wadman
2023-05-05 11:28:38 +03:00
Pavel Safonov
c23bc4216f
postgres: fix line endings in error messages, simplify code, add comments
2023-05-04 09:16:55 +03:00
Pavel Safonov
08c535239b
postgres: refactoring
2023-05-04 08:45:42 +03:00
Pavel Safonov
f6f8d5c037
postgres: refactoring
2023-05-03 15:40:32 +03:00
Pavel Safonov
9508a209ea
postgres: fix lint
2023-05-03 15:28:29 +03:00
Mattias Wadman
684a083813
interp,decode: Support decode group argument
...
Will be used for probing via -d <string> and filename
2023-05-03 10:01:13 +02:00
Pavel Safonov
e5f15c5fed
postgres: fix compilation, fix tests
2023-05-03 10:10:03 +03:00
Pavel Safonov
bb2659d442
Merge remote-tracking branch 'master' into postgres_merge_master_0
2023-05-03 09:44:02 +03:00
Mattias Wadman
f237db2794
id3v2: Decode CTOC flags
2023-05-02 13:43:58 +02:00
Mattias Wadman
d010dcec06
id3v2: Add WXXX (desc/url) frame support
2023-05-02 11:54:11 +02:00
Mattias Wadman
ddd7b0e439
Merge pull request #650 from matthewdale/fix-bson
...
bson: support all non-deprecated types and fix int/uint bugs
2023-05-02 09:33:25 +02:00
Matt Dale
40630d39b2
bson: fix doc formatting and add author info
2023-05-01 18:41:52 -07:00
Mattias Wadman
d6f785c6a6
pcap: Add forgotten help test
2023-05-01 16:06:22 +02:00
Mattias Wadman
a4a332bf4e
formats: Clenaup naming a bit
2023-05-01 13:19:04 +02:00
Mattias Wadman
7adc1e7030
mp4: Better description for QuickTime terminator atom
...
Addition improvement could be to only allow it in quicktime ftyp brand
and also not as a root book. Currently trailing zero bytes could be
mistaken as a terminator atom.
2023-05-01 12:32:25 +02:00
Matt Dale
9a982d0a76
bson: add BSON test file generator module and correct BSON format docs
2023-04-30 21:25:57 -07:00
Matt Dale
2017ff8766
bson: support all non-deprecated types and fix int/uint bugs
2023-04-30 10:28:42 -07:00
Mattias Wadman
b08ef00dd1
decode,interp: Refactor format groups into a proper struct
...
Replaces []Format with a Group type.
A bit more type safe.
Breaking change for RegisterFormat, now takes a first argument that is a "single" format group.
Lots of naming cleanup.
This is also preparation for decode group argument which will enable doing intresting
probing, ex a format decoder could know it's decode as part of probe group (html could
be probed possibly), or have "arg probe" group for decoder who inspect args to know
if they should probe (-d /path/to/schema etc) to enable nice CLI-ergonomics.
2023-04-29 20:02:34 +02:00
Mattias Wadman
b2cb5c3c1b
Merge pull request #645 from wader/matroska-filedata-fallback
...
matroska: file_data: Fallback to raw if probe fails
2023-04-20 20:37:26 +02:00
Mattias Wadman
8941b13932
matroska: file_data: Fallback to raw if probe fails
2023-04-20 20:08:40 +02:00
Martin Castillo
b3b6cd0ee3
gzip.go: fix typo in variablename: delfate
2023-04-19 15:51:55 +02:00
Pavel Safonov
c9b263e925
postgres: version 15 support
2023-04-17 17:12:46 +03:00
Mattias Wadman
3c6d31b0fe
mp4: ctts,infe,iinf,trun: More ISOMFF version handling
2023-04-17 11:43:54 +02:00
Mattias Wadman
2a2b64dda2
matroska: Update ebml specification
...
Also add license note about ebml_matroska.xml
2023-04-13 12:19:41 +02:00
Mattias Wadman
e1b02312d0
wav: Cleanup avi leftovers
2023-04-11 19:34:34 +02:00
Mattias Wadman
82da99c917
msgpack: Add str, array and object type tests
...
Related to #634
2023-04-08 13:09:15 +02:00
Mattias Wadman
97360d6ff9
msgpack: fixstr length field is 5 bits
...
Fixes #634
2023-04-08 08:45:26 +02:00
Mattias Wadman
0b6ef2a9d8
golangci-lint: Disable revive unused-parameter and update for new default config
...
unused-parameter disabled as i prefer to see the names
new default revive config seems to not warn about capital names
2023-03-18 22:56:58 +01:00
Mattias Wadman
980ecdba82
decode: Add float 80 reader
...
Rename *d.Bits to UintBits as it return a uint
Add *d.Bits that return []byte
2023-03-10 01:15:55 +01:00
Mattias Wadman
dc4a82eeed
aiff: Add basic decoder
2023-03-09 15:16:52 +01:00
Mattias Wadman
cc52a4419d
id3v2: Decode subframes for CTOC and add struct for headers
2023-03-05 19:29:22 +01:00
Mattias Wadman
9852f56b74
tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
...
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.
What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet
Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.
Will be base for decoding http2 and other TLS based on protocols.
Fixes #587
2023-03-05 13:52:12 +01:00
Mattias Wadman
c75a83c829
help: Show default option value as JSON
2023-03-04 09:53:33 +01:00
Mattias Wadman
dec433fc53
help,markdown: Fix double line breaks when converting to text
...
Also fix ugly last line break hack
2023-03-02 23:38:11 +01:00
Mattias Wadman
6c032455eb
pcap,pcapng,ipv4,ipv6: Support raw link type (ipv4 or ipv6)
2023-03-02 18:37:00 +01:00
Mattias Wadman
bdd6718ddb
zip: Correctly peek for zip64 EOCD
2023-02-22 22:22:36 +01:00
Mattias Wadman
5228fdd6cd
zip: Correctly look for and decode both zip32/64 EOCD record
...
There will always be zip(32) EOCD but optinally a zip64 EOCD
Related to #586
2023-02-22 19:34:28 +01:00
Mattias Wadman
56edb59e83
toml,xml: Fail fast on invalid content
...
encoding/xml and github.com/BurntSushi/toml both reads a lot before detecting
that it can't decode. Now we instead read one UTF-8 and make sure it's valid
xml or toml.
Should speed up probing
Related to #586 bigzero-zip.zip
2023-02-22 16:23:21 +01:00
Mattias Wadman
d4ea6632fa
pcap: Add ipv4 fragments tcp test
2023-02-20 11:45:12 +01:00
Mattias Wadman
c8666eeb04
ipv4_packet,ipv6_packet,sll_packet,sll2_packet: Support ipv4/ipv6 link frames and pass correct in arg
...
ipv4/ipv6 in sll* accidentally worked as it passed wrong arg with ether type
2023-02-20 01:34:38 +01:00
Mattias Wadman
8e0dde03d0
decode: Support multiple format args and some rename and refactor
...
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.
Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.
This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
a1bb630a2a
doc,fq: Improve cli help and some cleanup
2023-02-15 11:04:39 +01:00
Mattias Wadman
9aaf2ddf27
matroska: Add unknown size test and add description to ebml header
2023-02-10 10:29:25 +01:00
Mattias Wadman
c890a2899f
matroska: Update spec and make refs in descriptions look nicer
2023-02-09 23:18:17 +01:00
Mattias Wadman
a8d0bf4d3e
matroska: Assume master with unknown size has ended if a valid parent is found
...
Major refactor of ebml code generator, now decoupled from fq code
Cleanup element descriptions a bit, a bit shorter and less clutter
Cleanup old comments
2023-02-09 20:10:47 +01:00
Mattias Wadman
0d14d7b40f
matroska: Handle unknown size for non-master types a bit better
...
For non-master types assume size to be rest of file
Still does not follow the ebml spec
2023-02-08 12:14:40 +01:00
Mattias Wadman
b60aceca9e
matroska: Add decode_samples option
...
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
2023-02-08 11:31:41 +01:00
Mattias Wadman
dc79a73b72
interp,json: Move error handling to colorjson
...
Cancel error from ValueFn etc will be return by Marshal instead
2023-02-07 16:57:54 +01:00
Mattias Wadman
5c8e115106
colorjson: Refactor to option struct
2023-02-07 16:18:32 +01:00
Mattias Wadman
62e2cef5c2
tcp_segment: Decode standard options and rename maxseg to mss
2023-01-31 12:42:16 +01:00
Mattias Wadman
1eb5e502af
tcp: Ignore TCP option check for now as it seems unreliable in dumps
...
For example MSS can be to small in local dumps
2023-01-28 20:49:20 +01:00
Mattias Wadman
a2cdb3d6c9
Merge pull request #558 from wader/mp4-udta-with-no-length
...
mp4: udta: Handle box with value rest of box
2023-01-25 16:34:58 +01:00
Mattias Wadman
6340365841
mp4: udta: Handle box with value rest of box
...
Try distinguish by probing length field. Should probably be improved, what does ffmpeg do?
Regression from c3e3b3e90d#553
2023-01-25 16:24:28 +01:00
Mattias Wadman
0d1fdb5edf
Merge pull request #557 from wader/mp3-max-unknown-opt
...
mp3: Add max_unknown option to fail decode if too much unknown bits
2023-01-25 16:00:49 +01:00
Mattias Wadman
2d82c05f64
mp3: Add max_unknown option to fail decode if too much unknown bits
...
Hopefully help fix even more miss-detections
2023-01-25 15:27:31 +01:00
Mattias Wadman
b04a650bae
flac_picture,mpeg: Fix trailing ")" typo in map sym and description
2023-01-25 14:01:34 +01:00
Mattias Wadman
d645e71008
Merge pull request #554 from wader/mp4-ftyp-qt-minor-desc
...
mp4: Decode qt minor verison as YYYY.MM description
2023-01-16 12:23:42 +01:00
Mattias Wadman
f386a5158e
mp4: Decode qt minor verison as YYYY.MM description
2023-01-16 12:13:19 +01:00
Mattias Wadman
c3e3b3e90d
mp4: Decode udta metadata boxes without meta box
2023-01-16 12:03:53 +01:00
Mattias Wadman
c2795a7c2f
Merge pull request #552 from wader/mp4-tkhd-flags
...
mp4: Decode tkhd flags
2023-01-13 18:02:53 +01:00
Mattias Wadman
3555dc6721
mp4: Decode tkhd flags
2023-01-13 17:49:03 +01:00
Mattias Wadman
c49012dbca
mp4: sgpd,sbgp: Change grouping_type to a string
...
Is defined as int32 but is a string in practive it seems
2023-01-05 19:18:48 +01:00
Mattias Wadman
8b49b42fa3
interp: Wrap Binary in decodeValue to fix prompt issue with bits/bytes format
2023-01-04 15:30:05 +01:00
Pavel Safonov
bffa00835d
postgres: fix error in tests
2022-12-26 14:47:23 +03:00
Pavel Safonov
97bbc22a1d
postgres: add page arg in pg_btree, change args names in pg_heap
2022-12-22 11:22:02 +03:00
Mattias Wadman
e3ae1440c9
interp: Rename to/from<format> functions to to_/from_<format>
...
Feels less cluttered, easier to read and more consistent.
Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.
Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
8f39ef6335
bplist: Harmonize ns_keyed_archive jq style a bit
...
Let root arg be a lambda so torepr can be done once
2022-12-20 00:56:19 +01:00
Mattias Wadman
6f354bb350
Merge pull request #527 from dgmcdona/apple_package
...
decode: add ns_keyed_archiver, restructure apple decoders into package
2022-12-19 22:47:29 +01:00
