1
1
mirror of https://github.com/wader/fq.git synced 2024-12-23 13:22:58 +03:00
Commit Graph

24 Commits

Author SHA1 Message Date
Mattias Wadman
56edb59e83 toml,xml: Fail fast on invalid content
encoding/xml and github.com/BurntSushi/toml both reads a lot before detecting
that it can't decode. Now we instead read one UTF-8 and make sure it's valid
xml or toml.

Should speed up probing

Related to #586 bigzero-zip.zip
2023-02-22 16:23:21 +01:00
Mattias Wadman
8e0dde03d0 decode: Support multiple format args and some rename and refactor
This will allow passing both cli options and format options to sub decoder.
Ex: pass keylog option to a tls decoder when decoding a pcap.
Ex: pass decode options to a format inside a http body inside a pcap.

Add ArgAs method to lookup argument based on type. This also makes the format
decode function have same signature as sub decoders in the decode API.

This change decode.Format a bit:
DecodeFn is now just func(d *D) any
DecodeInArg renamed to DefaultInArg
2023-02-18 21:38:51 +01:00
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
42d9f2c27d interp,help: Properly count line length when breaking on whole words
Whitespace between words were not counted
2022-11-29 18:59:32 +01:00
Mattias Wadman
c93301fc70 raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binary
raw format was a hack to skip decoding to be able to get a binary using tobyte etc.
Now you can do fq -d bytes ... instead of fq -d raw 'tobytes | ...'
2022-11-20 19:51:12 +01:00
Mattias Wadman
0fdc03a4f1 doc: Fix some incorrect example prompts 2022-09-27 13:42:31 +02:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
725ab1b17b doc,html,xml: Add more documentation and examples 2022-09-20 17:35:27 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
8016352b3b html: Handle html with leading doctype etc 2022-09-01 22:58:48 +02:00
Mattias Wadman
226a9a3e08 generics: Use more from x/exp 2022-08-31 10:50:56 +02:00
Mattias Wadman
7d86534348 sortex: Package with type safe sort helpers 2022-08-30 11:02:57 +02:00
Mattias Wadman
095e1161b7 xml: Switch from "-" to "@" as attribute prefix and make it an option
Seems to be more common.
2022-08-25 17:15:23 +02:00
Mattias Wadman
716323cea1 xml: Even more namespace fixes
fromXML*: Collect namespaces before resolving to not relay on attribute order.
toXML*: Sort attributes, prefer non-namescaped first.
2022-08-25 15:45:09 +02:00
Mattias Wadman
f24d685a5b xml: Keep track of default namespace and skip it element names
Refactor element name handling a bit, return it instead, feels nicer.
2022-08-21 10:11:59 +02:00
Mattias Wadman
71900c2a9d xml: Correctly sort if one #seq is found and allow negative seq numbers 2022-08-18 23:08:31 +02:00
Mattias Wadman
3623eac365 yaml: Error on trailing yaml/json
Turns our yaml is a superset of json
2022-08-15 19:14:25 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
9a5fcc89f1 xml: Allow trailing <?procinstr?>
Also more context in error messages and refactor trim function
2022-08-11 23:48:29 +02:00
Mattias Wadman
f4e0137244 xml,html: Always include attrs and children in array mode
Was hard to use when you dont know what indexes things will have. They are sill optioal for toxml
2022-08-04 11:44:59 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00