1
1
mirror of https://github.com/wader/fq.git synced 2024-12-27 15:42:07 +03:00
Commit Graph

263 Commits

Author SHA1 Message Date
Mattias Wadman
97c952b3f1 doc: Add some more examples 2023-04-14 10:19:33 +02:00
Mattias Wadman
427ce78df6 interp: Add --value-output/-V option to do tovalue before output
Idea is to skip display so that JSON is outputted instead of showing tree and hexdump etc.
2023-04-13 11:29:16 +02:00
Mattias Wadman
dc4a82eeed aiff: Add basic decoder 2023-03-09 15:16:52 +01:00
רטו/רעטאָ/רֵיטוֹ • Reto
87e5bb142d
fix typo 2023-03-09 07:01:52 +01:00
Mattias Wadman
cc52a4419d id3v2: Decode subframes for CTOC and add struct for headers 2023-03-05 19:29:22 +01:00
Mattias Wadman
9852f56b74 tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
  and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.

What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet

Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.

Will be base for decoding http2 and other TLS based on protocols.

Fixes #587
2023-03-05 13:52:12 +01:00
Mattias Wadman
3e0ebafa6d doc: Run make doc 2023-02-26 21:41:46 +01:00
Mattias Wadman
73db6587a0 interp: Exit with error if -o name=@path fails to be read, also document 2023-02-26 21:18:36 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
b60aceca9e matroska: Add decode_samples option
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
2023-02-08 11:31:41 +01:00
Mattias Wadman
2d82c05f64 mp3: Add max_unknown option to fail decode if too much unknown bits
Hopefully help fix even more miss-detections
2023-01-25 15:27:31 +01:00
Mattias Wadman
ca27e4266e doc: Add _parent for decode values and clenaup doc a bit 2023-01-03 11:21:09 +01:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
David McDonald
22064f50d6 doc: remake 2022-12-19 15:27:48 -06:00
Mattias Wadman
1dea40e6af mp4,doc: Add JSON box tree example and reorder a bit 2022-12-15 14:15:31 +01:00
Mattias Wadman
9e5a072e50 mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders 2022-12-15 12:10:06 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
48522e3cb8 mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one 2022-12-14 11:34:53 +01:00
David McDonald
4aad2fdeea doc: remake 2022-12-05 22:03:26 -06:00
David McDonald
34c560c2f5 Merge branch 'master' of https://github.com/wader/fq into macos_bookmark 2022-12-05 22:01:44 -06:00
Mattias Wadman
2fc16ae22a doc: Add some padding margin to formats table to make it less likely to cause git conflicts 2022-12-05 12:25:00 +01:00
Mattias Wadman
2c4862c9fb
Merge pull request #498 from bitbears-dev/master
support tzif (time zone information format)
2022-12-04 13:21:19 +01:00
@0xb17bea125
c4e7fc7919 tzif: moved document to tzif.md 2022-12-04 21:04:13 +09:00
@0xb17bea125
2ee6360b97 support tzif (time zone information format) 2022-12-04 21:02:39 +09:00
Mattias Wadman
42debe5871 dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg
Update format dev docs to only generate README.md and doc/foramts.md to not
cause formats.svg conflicts.
2022-12-04 12:37:10 +01:00
David McDonald
b872b1a337 doc: remake 2022-12-03 17:36:56 -06:00
Mattias Wadman
6fc84a885c doc,dev: Add more usage and dev tips 2022-12-03 19:00:47 +01:00
David McDonald
f5e25fca26 updated docs 2022-12-01 23:31:57 -06:00
David McDonald
44c91d8296 tweaks apple_bookmark markdown documentation 2022-12-01 23:31:57 -06:00
David McDonald
4a28e44f44 changes decoder package name from bookmark to apple_bookmark 2022-12-01 23:31:57 -06:00
David McDonald
0a72635a78 remade documentation 2022-12-01 23:31:57 -06:00
David McDonald
34db9d7fea regenerated docs, added tests, fixed torepr 2022-12-01 23:31:57 -06:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
8d69f1fb23 interp: Change default bits_format=string
I think this is more intuitive but might in some case cause very large JSON output
but maybe that less common or expected. In does cases i think you either want to
use some other bits_format (md5, truncate, etc) or you delete/transform the jq value
before turn it into JSON.

Strings in gojq are binary safe so you can use to hold raw bytes. But note that
convert the binary into JSON is lossy, same as the JSON standard.

Add bits_format option documentation.
2022-12-01 17:49:34 +01:00
Mattias Wadman
ef2d5232ba mp4: Add png mapping
Also support samples decode for tracks without hdlr box
2022-11-25 19:49:18 +01:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
0cf46e1196 wav,avi,avc_au: Add avi decoder and refactor wav decoder
avi and wav now share  common riff decode code
avc_au decoder can now handle annexb format
2022-11-19 13:11:27 +01:00
Mattias Wadman
7b6492eed0 Improve README.md a bit, one more demo and move up usage 2022-11-17 11:10:08 +01:00
Mattias Wadman
c3a0686c9e mp3_frame_tags: Refactor and rename xing format to mp3_frame_tags
Make more sense to use a more general name when there is both info and xing
and possibly vbri in the future. Also both xing and info seems to have extension
fields

Remove some old tests that are not really needed anymore.
2022-11-16 17:32:03 +01:00
Mattias Wadman
4bfd9d8119 doc: Add link to nttw6 presentation video and slides
Also clarify install instructions a bit
2022-10-30 13:28:46 +01:00
Mattias Wadman
2a86d323b4 doc,rtmp,pcap,markdown: Add more examples 2022-10-09 19:18:14 +02:00
Mattias Wadman
09ea08f3aa
Merge pull request #427 from dgmcdona/bplist
Adds support for Apple Binary Plist, version 00
2022-10-04 14:25:53 +02:00
Mattias Wadman
0fdc03a4f1 doc: Fix some incorrect example prompts 2022-09-27 13:42:31 +02:00
Mattias Wadman
00ee10a1a1 interp: Make to{actual,sym} behave similar to tovalue
Before if you did toactual it behaved differently and even gave error for
a scalar that only had actual value set. Now both to{actal,sym} works
similar to tovalue but will force if actual or sym value should be used.
2022-09-25 17:53:12 +02:00
David McDonald
bcccde2358 Fixes and embeds documentation 2022-09-24 12:57:11 -05:00
David McDonald
5f61994087 adds function for decoding fixed sized arrays 2022-09-23 20:50:41 -05:00
David McDonald
a77cec921f Added documentation and tests, fixed bad date parsing 2022-09-23 20:23:23 -05:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
725ab1b17b doc,html,xml: Add more documentation and examples 2022-09-20 17:35:27 +02:00
Mattias Wadman
fe64530e11 csv: Add tsv and header example
Also fix string esacpe in fuzz options parser
2022-09-15 01:16:45 +02:00