1
1
mirror of https://github.com/wader/fq.git synced 2024-11-23 00:57:15 +03:00
Commit Graph

285 Commits

Author SHA1 Message Date
Mattias Wadman
d703321a7a avi: Add extended chunks support and option
This is used for >1gb files. Disable decode will speed up
deocde a lot but will probably also produce some gaps as same
part of the movi chunks will not be reference by the indx index.
2023-10-20 15:55:19 +02:00
Mattias Wadman
051a70bd4b interp: Change bit ranges to use exclusive end
All other ranges and slicing uses exclusive end so i think it make sense
to make it consistent.

Update docs and add additional example for non-byte-aligned field.

Also fixes issue showing zero bit ranges as start-NA.
2023-10-20 15:37:26 +02:00
fiatjaf
976a7564c7
opentimestamps: one last make doc. 2023-09-26 15:42:02 -03:00
fiatjaf
5e7c01a013
opentimestamps: address comments and improve things. 2023-09-26 12:35:18 -03:00
Mattias Wadman
406f3926e9 doc: Move up and update differences jq section a bit 2023-08-22 13:20:50 +02:00
Mattias Wadman
f15f9bc1a4 doc,moc3,caff: Add author and regenerate docs 2023-08-21 10:04:37 +02:00
Ronsor
cc58c4b81d caff: update doc/formats.md 2023-08-18 11:04:57 -07:00
Mattias Wadman
d5345f0b8a cli: Rename --null/nul-output to --raw-output0
Remove -0 short arg, now it means the expression "-0".
This is to be in sync with jq 1.7 https://github.com/jqlang/jq/pull/2684

Correct and clarify that NUL and new lines are outputted after and not between each output.

Still missing is to abort on output containing zero when using --raw-output0
2023-08-08 16:33:39 +02:00
Mattias Wadman
44f0060205 dev,jq: Reformat jq code to look more the same 2023-07-27 13:21:04 +02:00
Mattias Wadman
23b9eeab1f luajit: make doc 2023-06-22 23:54:49 +02:00
Mattias Wadman
93423d0da6
Merge pull request #673 from wader/bit-format-hex
interp: Add hex bits format
2023-05-15 17:49:15 +02:00
Mattias Wadman
8a468f45df interp: Add hex bits format
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d

Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Kian-Meng Ang
dd4fa26867 doc: fix typos
Found via `codespell -S format -L bu,ue,trys,nd,tbe,te,trun,actuall`
2023-05-14 21:45:15 +08:00
Mattias Wadman
e2eb667091 html: Add to probe group
As decoder now can know they are decoding as part of probing we can now
use some heuristics to see if we should decode as html.
The reason heuristics is needed is that x/html parser will alwaus succeed.

Add lazyre package to help delay compile of RE and make it concurrency safe.
2023-05-11 19:07:18 +02:00
Mattias Wadman
88be3a7f9f doc: Hopefully fix svg fixed font issue
Regenerate with new ansisvg that uses Courier with monospace as fallback

Related to #655
2023-05-08 15:27:00 +02:00
Pavel Safonov
e6a9cdbecf postgres: doc 2023-05-06 08:54:56 +03:00
Pavel Safonov
7a89234b13 postgres: update doc 2023-05-04 08:34:32 +03:00
Mattias Wadman
ddd7b0e439
Merge pull request #650 from matthewdale/fix-bson
bson: support all non-deprecated types and fix int/uint bugs
2023-05-02 09:33:25 +02:00
Matt Dale
40630d39b2
bson: fix doc formatting and add author info 2023-05-01 18:41:52 -07:00
Mattias Wadman
af68511acf dev,doc Clarify some dev docs and rename launch.json to be a template 2023-05-01 17:39:43 +02:00
Matt Dale
2017ff8766
bson: support all non-deprecated types and fix int/uint bugs 2023-04-30 10:28:42 -07:00
Mattias Wadman
c51271399c interp: Add skip_gaps option for tovalue/-V
Skips gap fields in struct and arrays.

Gaps fields are bit ranges that a decoder did not add any fields for.
Note that skipping gaps in arrays will affect indexes.
2023-04-30 13:04:39 +02:00
Mattias Wadman
97c952b3f1 doc: Add some more examples 2023-04-14 10:19:33 +02:00
Mattias Wadman
427ce78df6 interp: Add --value-output/-V option to do tovalue before output
Idea is to skip display so that JSON is outputted instead of showing tree and hexdump etc.
2023-04-13 11:29:16 +02:00
Mattias Wadman
dc4a82eeed aiff: Add basic decoder 2023-03-09 15:16:52 +01:00
רטו/רעטאָ/רֵיטוֹ • Reto
87e5bb142d
fix typo 2023-03-09 07:01:52 +01:00
Mattias Wadman
cc52a4419d id3v2: Decode subframes for CTOC and add struct for headers 2023-03-05 19:29:22 +01:00
Mattias Wadman
9852f56b74 tls: Add TLS 1.0, 1.1, 1.2 decode and decryption
What it can do:
- Decodes records and most standard messages and extensions.
- Decryptes records and reassemples application data stream if a keylog is provided
  and the cipher suite is supported.
- Supports most recommended and used ciphers and a bunch of older ones.

What it can't do:
- SSL v3 maybe supported, is similar to TLS 1.0, not tested.
- Decryption and renegotiation/cipher change.
- Record defragmentation not supported, seems rare over TCP.
- TLS 1.3
- SSL v2 but v2 compat header is supported.
- Some key exchange messages not decoded yet

Decryption code is heavly based on golang crypto/tls and zmap/zcrypto.

Will be base for decoding http2 and other TLS based on protocols.

Fixes #587
2023-03-05 13:52:12 +01:00
Mattias Wadman
3e0ebafa6d doc: Run make doc 2023-02-26 21:41:46 +01:00
Mattias Wadman
73db6587a0 interp: Exit with error if -o name=@path fails to be read, also document 2023-02-26 21:18:36 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
b60aceca9e matroska: Add decode_samples option
Also change the option help a bit, maybe medid and support is confusing, we
might evetually decode non-audio/video things.
2023-02-08 11:31:41 +01:00
Mattias Wadman
2d82c05f64 mp3: Add max_unknown option to fail decode if too much unknown bits
Hopefully help fix even more miss-detections
2023-01-25 15:27:31 +01:00
Mattias Wadman
ca27e4266e doc: Add _parent for decode values and clenaup doc a bit 2023-01-03 11:21:09 +01:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
David McDonald
22064f50d6 doc: remake 2022-12-19 15:27:48 -06:00
Mattias Wadman
1dea40e6af mp4,doc: Add JSON box tree example and reorder a bit 2022-12-15 14:15:31 +01:00
Mattias Wadman
9e5a072e50 mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders 2022-12-15 12:10:06 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
48522e3cb8 mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one 2022-12-14 11:34:53 +01:00
David McDonald
4aad2fdeea doc: remake 2022-12-05 22:03:26 -06:00
David McDonald
34c560c2f5 Merge branch 'master' of https://github.com/wader/fq into macos_bookmark 2022-12-05 22:01:44 -06:00
Mattias Wadman
2fc16ae22a doc: Add some padding margin to formats table to make it less likely to cause git conflicts 2022-12-05 12:25:00 +01:00
Mattias Wadman
2c4862c9fb
Merge pull request #498 from bitbears-dev/master
support tzif (time zone information format)
2022-12-04 13:21:19 +01:00
@0xb17bea125
c4e7fc7919 tzif: moved document to tzif.md 2022-12-04 21:04:13 +09:00
@0xb17bea125
2ee6360b97 support tzif (time zone information format) 2022-12-04 21:02:39 +09:00
Mattias Wadman
42debe5871 dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg
Update format dev docs to only generate README.md and doc/foramts.md to not
cause formats.svg conflicts.
2022-12-04 12:37:10 +01:00
David McDonald
b872b1a337 doc: remake 2022-12-03 17:36:56 -06:00
Mattias Wadman
6fc84a885c doc,dev: Add more usage and dev tips 2022-12-03 19:00:47 +01:00
David McDonald
f5e25fca26 updated docs 2022-12-01 23:31:57 -06:00