1
1
mirror of https://github.com/wader/fq.git synced 2024-12-23 13:22:58 +03:00
Commit Graph

27 Commits

Author SHA1 Message Date
Mattias Wadman
e3ae1440c9 interp: Rename to/from<format> functions to to_/from_<format>
Feels less cluttered, easier to read and more consistent.

Still keep tovalue, tobytes etc that are more basic functions this
only renamed format related functions.
Also there is an exceptin for to/fromjson as it comes from jq.

Also fixes lots of spelling errors while reading thru.
2022-12-21 17:48:39 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
e77f776999 decode,interp: Rename unknown gap fields from "unknown#" to "gap#"
Think it makes it clearer and also less likely to collide with a field
name a deocder wants to use.
2022-12-01 20:43:30 +01:00
Mattias Wadman
8d69f1fb23 interp: Change default bits_format=string
I think this is more intuitive but might in some case cause very large JSON output
but maybe that less common or expected. In does cases i think you either want to
use some other bits_format (md5, truncate, etc) or you delete/transform the jq value
before turn it into JSON.

Strings in gojq are binary safe so you can use to hold raw bytes. But note that
convert the binary into JSON is lossy, same as the JSON standard.

Add bits_format option documentation.
2022-12-01 17:49:34 +01:00
Mattias Wadman
a7a101ca67 doc,help: Nicer format help and move help tests into each format
Also add authors to avro and macho decoders

Generated with:
for i in $(go run . -r -n '_registry | [([([.files[][].name | split(".")[0]] | unique[]), (.formats[].name)] | count[] | select(.[1] == 2)[0]), (.formats[] | select(.decode_in_arg).name)] | unique[]'); do DIR=$(dirname $(find format -name $i.go)); echo "$ fq -h $i" > "$DIR/testdata/help_$i.fqtest" ; done
2022-09-22 23:07:58 +02:00
Mattias Wadman
abd19ed89b doc: Fix format sections a bit 2022-09-11 09:55:56 +02:00
Mattias Wadman
87b2c6c10c help,doc: Use markdown for format documentation again
Markdown is used as is in online documentation and in cli the markdown decoder
is used to decode and the some jq code massages it into something cli friendly.

Was just too much of a mess to have doc in jq.
2022-09-11 00:53:29 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
a6a97136b7 decode: Cleanup Try<f>/<f> pairs
Decode API design is that Try* returns error non-Try panics.
Also rename Must* as they should panic and introduce some new Try* functions
that were missing.
2022-06-30 12:13:46 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
27e761570a format: Simplify torepr, no need for _f function 2022-03-08 17:09:48 +01:00
Mattias Wadman
0829c167cc asn1_ber: Add decoder
Basic support, can also do CER and DER but without any extra validation.
No schema support.

Redo format doc.md usage a bit, now format/<dir>/<format>.md instead.

Related to #20
2022-02-06 22:13:03 +01:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
2022-02-04 21:41:53 +01:00
Mattias Wadman
2ab395a03b protobuf: Add note about sub message decoding
Fix and cleanup some other doc also
2022-02-01 22:22:19 +01:00
Mattias Wadman
b66e3287f2
Merge pull request #112 from wader/interp-cleanup-d-alises
interp: Cleanup display aliases, now: d, da, dd, dv, ddv
2022-01-29 12:28:46 +01:00
Mattias Wadman
a8664ed5ba doc: Add per format documentation 2022-01-29 12:01:36 +01:00
Mattias Wadman
fc0aacb654 interp: Cleanup display aliases, now: d, da, dd, dv, ddv
Think it makes sense to have them all start with d.
Also f is often used as function argument name.
2022-01-28 18:25:38 +01:00
Mattias Wadman
548a065388 decode,interp: Finish up nil value support
Update msgpack, bson and cbor to use it.
2022-01-17 10:49:46 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra
Think it makes more sense
2022-01-15 19:00:42 +01:00
Mattias Wadman
1383b411ae decode,interp: Add arbitrary large integer support (BigInt)
Was already handled in fq in various places as gojq uses them

Update msgpack to support negative integers that can't represented as int64
Rename read try* number functions to make them more explicit
2022-01-15 19:00:42 +01:00
Mattias Wadman
0b0f28e966 cbor: Add decoder
Does not decode sematic tag types
Also fixes broken float16 support

Fixes #71
2022-01-12 17:56:03 +01:00