1
1
mirror of https://github.com/wader/fq.git synced 2024-11-23 09:56:07 +03:00
Commit Graph

100 Commits

Author SHA1 Message Date
bump
888b0be069 Update github-golangci-lint to 1.56.0 from 1.55.2
Release notes https://github.com/golangci/golangci-lint/releases/tag/v1.56.0

Remove some nolint:errorlint not needed anymore
2024-02-07 17:31:49 +01:00
Mattias Wadman
69ae76593c interp: Fix crash when using line_bytes=0 2024-02-06 16:33:50 +01:00
Mattias Wadman
23ae4d978b decode,interp: Make synthetic values more visible and not act as decode values 2023-10-07 17:40:16 +02:00
Mattias Wadman
8a468f45df interp: Add hex bits format
$ cat random.bin.gz | fq -rV -o bits_format=hex .uncompressed
f6f2074cf77d449d

Also made unknown bit formats an error.
2023-05-15 17:31:38 +02:00
Mattias Wadman
c51271399c interp: Add skip_gaps option for tovalue/-V
Skips gap fields in struct and arrays.

Gaps fields are bit ranges that a decoder did not add any fields for.
Note that skipping gaps in arrays will affect indexes.
2023-04-30 13:04:39 +02:00
Mattias Wadman
3dd2c61d3c interp: Fix input completion regression in sub-REPLs
readline Config was used to pass completer function per readline call,
was changed in #612 and caused regression. Now use our own member in
stdOS to pass it instead.

Add a test but test script completer is implemented differently.
2023-03-10 13:57:21 +01:00
Mattias Wadman
dc79a73b72 interp,json: Move error handling to colorjson
Cancel error from ValueFn etc will be return by Marshal instead
2023-02-07 16:57:54 +01:00
Mattias Wadman
5c8e115106 colorjson: Refactor to option struct 2023-02-07 16:18:32 +01:00
Mattias Wadman
9b81d4d3ab decode: More type safe API and split scalar into multiple types
Preparation to make decoder use less memory and API more type safe.
Now each scalar type has it's own struct type so it can store different
things and enables to have a scalar interface.
Also own types will enable experimenting with decode DLS designs like
using chained methods that are type aware.
2022-12-14 16:23:58 +01:00
Mattias Wadman
2606916753 yaml,fuzz: gojq.Normalize value to fix type panic 2022-10-11 19:22:49 +02:00
Mattias Wadman
0cd846a18c *extra: Rename <pkg>extra to just <pkg>ex and refactor to use generics 2022-08-12 16:29:47 +02:00
Mattias Wadman
3c21b058e4 lint: Fix ioutil deprecation, reformat for new doc standard 2022-08-05 00:22:32 +02:00
Mattias Wadman
9e447c9af1 interp: Use RegisterFS instead of format files 2022-07-27 13:22:59 +02:00
Mattias Wadman
cae288e6be format,intepr: Refactor json, yaml, etc into formats also move out related functions
json, yaml, toml, xml, html, csv are now normal formats and most of them also particiate
in probing (not html and csv).

Also fixes a bunch of bugs in to/fromxml, to/fromjq etc.
2022-07-23 21:48:45 +02:00
Mattias Wadman
342612eb7e dev: Cleanup linters and fix some unused args 2022-07-19 18:33:50 +02:00
Mattias Wadman
9b2e474ea8 gojqextra: Simplify function type helpers
No need for env to be a function
2022-07-16 21:55:47 +02:00
Mattias Wadman
1ddea1ada3 interp,format: Refactor registry usage and use function helpers
Move registry to interp and add support for functions and filesystems.
This will be used later for allow formats to add own functions and fq code.

Add gojqextra function helpers to have more comfortable API to add functions.
Takes care of argument type casting and JQValue:s and some more things.

Refactor interp package to use new function helper and registry. Probably
fixes a bunch of JQValue bugs and other type errors.

Refactor out some mpeg nal things to mpeg format.

Refactor interp jq code into display.q and init.jq.

Remove undocumented aes_ctr funciton, was a test. Hopefully will add more crypto things laster.
2022-07-16 19:24:13 +02:00
Mattias Wadman
c7701851b0 interp: Extract to/from map/struct to own package 2022-06-20 21:24:59 +02:00
Mattias Wadman
32361deefe gojqextra: Cleanup gojq type cast code 2022-06-20 18:02:35 +02:00
Mattias Wadman
a64c28d927 dump: Skip JQValueEx if there are not options
I used to tojson and would crash in binary values are there were
not optins deciding how to format the strinh. Now just treat as utf8 string.
2022-06-07 16:39:45 +02:00
Mattias Wadman
6b0880002d interp: Cast jq value to go value properly for encoding functions
Some encoding fuctions accept binary used string as input type, should be any.
Add cast helper functions, hopefully can be useful in future for even nicer
function bindings api.
2022-05-30 18:27:12 +02:00
Mattias Wadman
3b717c3ba4 interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth

Add tojson, same as in jq but also has {indent:number} options

Add toyaml/fromyaml for YAML

Add totoml/fromtoml for TOML

Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)

Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character

Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)

Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex

Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512

Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-28 16:31:20 +02:00
Mattias Wadman
9a7ce14886 gojq: Update rebased fq fork
Use new gojq.TypeOf and gojq.Preview
2022-05-23 21:33:12 +02:00
Mattias Wadman
e9d9f8aef9 fq: Use go 1.18
Rename s/interface{}/any/g
Preparation for using generics in decode API and native jq funcations etc
Remove some unused linter ignores as linter has been fixed
2022-05-20 15:23:16 +02:00
Mattias Wadman
8dde3ef5b1 interp: Fix crash when including relatve path when no search paths are set 2022-05-17 18:05:10 +02:00
Mattias Wadman
f4480c6fe5 decode,interp: Support for format specific options
interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq
2022-05-01 17:08:30 +02:00
Mattias Wadman
dabad85080 interp: Proper display column truncate
Also speed up by using less string allocs
2022-04-16 18:48:21 +02:00
Mattias Wadman
6f03471d15 interp: Paths with a array as root was missing start dot 2022-04-08 16:47:40 +02:00
Mattias Wadman
41551de331 interp,repl: Improved eval and output interrupt
repl now rewrites query to do implicit display inside the sub eval.
This makes it possible to interrupt eval and output in a better and faster way.

Make JSON encoder fail early on errors.

Add more interrupt tests.
2022-03-11 16:22:30 +01:00
Mattias Wadman
dff7e7dadb interp: Cleanup binary regexp overloading and add explode
Add explode for binary that explode into an array of the binary unit tobits
Remove scan_toend as it feels a bit weird and one can use tobytes | .[match(...):] instead
Some general cleanup also
2022-03-09 20:17:12 +01:00
Mattias Wadman
33efb02af0 interp,repl: Add path and value colors to prompt 2022-03-03 18:25:19 +01:00
Mattias Wadman
fe8183b500 interp: Color parse in jq 2022-03-03 17:55:24 +01:00
Mattias Wadman
0a043f9096 repl,interp: Refactor repl and slurp
Now repl, slurp and help implemented using same query rewrite.
Include filename context in error if possible.
Add spew function that does opposite of slurp.
Start of help infra, not done or documented yet.
Show error pointer on parse error.
Rename internal eval to _eval and make eval be wrapper that
does rewrite and has various eror handling etc.
Nicer repl, slupr and help errors.
2022-03-01 18:31:00 +01:00
Mattias Wadman
03f450f82e interp: Skip redundant includes
Speeds up interp quite a bit.
Also good as i've start to use more includes to make code easier to follow
where thigns come from and also makes jq-lsp happier.

Also rename evalContext to evalInstace to make it less confused with context.
2022-02-24 12:44:26 +01:00
Mattias Wadman
9dc59e5d39 interp: Move _is_decode_value to jq
Make _extype work like type
2022-02-15 22:20:32 +01:00
Mattias Wadman
af053811cf repl,interp: Make stdio work during completion
Unbreaks completion using options
2022-02-11 20:04:07 +01:00
Mattias Wadman
48a19cb82c interp: Add paste function to allow pasting text into REPL etc
Also refactor readline and eval args into option struct and partinally start
addressing some side effects during completion.
2022-02-11 17:59:06 +01:00
Mattias Wadman
77ab667bfa interp: Use absolute path in errors
Make it easier to find parse error in init.jq etc
2022-02-11 16:34:20 +01:00
Mattias Wadman
06245d1295 binary,decode,doc: Rename buffer to binary and add some documentation
Rename buffer to binary. Still some work left what to call buffer/binary in decode code.
Document decode value and binary type
Fix proper unit padding for tobytes and add still undocumenated extra padding argument.
Add some additional binary tests
2022-02-08 22:20:28 +01:00
Mattias Wadman
7c5215347d bitio,decode: Refactor bitio usage and make buffer slicing more correct
Remove bitio.Buffer layer. bitio.Buffer was a kitchen sink layer with helpers
now it's just a buffer and most functions have been moved to decode instead.

bitio package now only have primitive types and functions simialar to standard
library io and bytes packages.

Make nearly eveything internally use bitio.Bit* interfaces so that slicing work
correctly this will also make it possible to start experimenting with more
complicated silcing helpers, ex things like:
breplace(.header.bitrate; 123) to get a new buffer with bitrate changed.
2022-02-04 21:41:53 +01:00
Mattias Wadman
ca68e6a1e4 interp: Add Platform() method to OS interface
Felt bad that main should have to know about it
2022-01-29 11:31:36 +01:00
Mattias Wadman
233d86a384 fq: Add arch and os to --version 2022-01-26 18:09:59 +01:00
Mattias Wadman
898dfec1d1 lint: Fix typeassert and case exhaustive warnings 2022-01-25 17:23:41 +01:00
Mattias Wadman
edad481878 num,mathextra: Rename num package to mathextra
Think it makes more sense
2022-01-15 19:00:42 +01:00
Mattias Wadman
e365f22084 interp: Cleanup stdio usage and functions 2022-01-05 21:41:11 +01:00
Mattias Wadman
a41f0d4b5f windows: Correct @builtin include path join
Failed because go embed paths always uses /
Switch to always use forward slash paths internally

Related to #21
2022-01-04 22:10:53 +01:00
Mattias Wadman
55b1d5c3e7 interp: Move _registry to decode
More related to decode than interp. Maybe should have a format.go?
2021-12-25 16:29:45 +01:00
Mattias Wadman
fa350c6165 interp: Move display to jq 2021-12-06 21:08:40 +01:00
Mattias Wadman
4558192e7c interp: Cleanup buffer code and implement ExtType() 2021-12-06 21:08:40 +01:00
Mattias Wadman
9aec91a675 interp: match: Fix issue with regexp meta characters when matching using a buffer 2021-12-04 17:24:43 +01:00