mirror of
https://github.com/wader/fq.git
synced 2024-10-27 04:09:37 +03:00
doc: Reorganize and cleanup function descriptions
This commit is contained in:
parent
ff65812941
commit
40f38a5558
245
doc/usage.md
245
doc/usage.md
@ -406,8 +406,6 @@ There is also `tobitsrange` and `tobytesrange` which does the same thing but wil
|
|||||||
|
|
||||||
Both `.[index]` and `.[start:end]` support negative indices to index from end.
|
Both `.[index]` and `.[start:end]` support negative indices to index from end.
|
||||||
|
|
||||||
TODO: tobytesrange, padding
|
|
||||||
|
|
||||||
#### Binary array
|
#### Binary array
|
||||||
|
|
||||||
Is an array of numbers, strings, binaries or other nested binary arrays. When used as input to `tobits`/`tobytes` the following rules are used:
|
Is an array of numbers, strings, binaries or other nested binary arrays. When used as input to `tobits`/`tobytes` the following rules are used:
|
||||||
@ -428,92 +426,175 @@ Some examples:
|
|||||||
|
|
||||||
`[(.a | tobytes[-10:]), 255, (.b | tobits[:10])] | tobytes` the concatenation of the last 10 bytes of `.a`, a byte with value 255 and the first 10 bits of `.b`.
|
`[(.a | tobytes[-10:]), 255, (.b | tobits[:10])] | tobytes` the concatenation of the last 10 bytes of `.a`, a byte with value 255 and the first 10 bits of `.b`.
|
||||||
|
|
||||||
The difference between `tobits` and `tobytes` is
|
|
||||||
|
|
||||||
TODO: padding and alignment
|
|
||||||
|
|
||||||
## Functions
|
## Functions
|
||||||
|
|
||||||
All decode functions are available in two forms, just `<format>` (like `mp3`) that returns a decode value on error and `from_<format>` which throws error on decode error.
|
fq has all the same standard library jq functions and in addition some new ones.
|
||||||
|
|
||||||
|
### Additional generic functions
|
||||||
|
|
||||||
|
#### `grep_by(f)`
|
||||||
|
Recursively select using a filter. Ex: `grep_by(. > 180 and . < 200)`, `first(grep_by(format == "id3v2"))`.
|
||||||
|
|
||||||
|
#### `group`
|
||||||
|
Group values, same as `group_by(.)`.
|
||||||
|
|
||||||
|
#### `streaks`, `streaks_by(f)`
|
||||||
|
Like `group` but groups streaks based on condition.
|
||||||
|
|
||||||
|
#### `count`, `count_by(f)`
|
||||||
|
Like `group` but counts groups lengths based on condition.
|
||||||
|
|
||||||
|
#### `delta`, `delta_by(f)`
|
||||||
|
Array with difference between consecutive. `delta` is same as `delta_by(.b - .a)`.
|
||||||
|
|
||||||
|
#### `chunk($size)`
|
||||||
|
Split array or string into `$size` length chunks. Last chunk might be shorter.
|
||||||
|
|
||||||
|
#### `path_to_expr`
|
||||||
|
Converts a path value `["key", 1]` to a string `".key[1]"`.
|
||||||
|
|
||||||
|
#### `expr_to_path`
|
||||||
|
Converts from a string `".key[1]"` to path value `["key", 1]`.
|
||||||
|
|
||||||
|
#### `diff($a; $b)`
|
||||||
|
Produce a diff between `$a` and `$b`. Differences are represented as a object `{a: <value from a>, b: <value from b>}`.
|
||||||
|
|
||||||
|
#### `band`, `bor`, `bxor`, `bsl`, `bsr`, `bnot`.
|
||||||
|
Bitwise functions. Works the same as jq math functions. Functions with no arguments like `1 | bnot` uses only input, functions with more than one argument ignores input, `bsl(1; 3)`.
|
||||||
|
|
||||||
|
### `repl`/`repl($opts)`
|
||||||
|
Nested REPL. Must be last in a pipeline. `1 | repl`, can "slurp" outputs. Ex: `1, 2, 3 | repl`, `[1,2,3] | repl({compact: true})`.
|
||||||
|
|
||||||
|
### `slurp("<name>")`
|
||||||
|
Slurp outputs and save them to `$name`. Must be last in the pipeline. Will be available as a global array `$name`. Ex `1,2,3 | slurp("a")`, `$a[]` same as `spew("a")`.
|
||||||
|
|
||||||
|
### `spew`/`spew("<name>")`
|
||||||
|
Output previously slurped values.
|
||||||
|
|
||||||
|
### `spew`
|
||||||
|
Outputs all slurps as an object. `spew("<name>")` outputs one slurp. Ex: `spew("a")`.
|
||||||
|
|
||||||
|
### `paste`
|
||||||
|
Read string from stdin until ^D. Useful for pasting text. Ex: `paste | from_pem | asn1_ber | repl` read from stdin then decode and start a new sub-REPL with result.
|
||||||
|
|
||||||
|
### Format decode functions
|
||||||
|
|
||||||
|
Format decode functions are available in two forms, just `mp3` or `mp3($opts)` that returns a decode value even on error and `from_mp3` or `from_mp3($opts)` which throws error on decode error.
|
||||||
|
|
||||||
|
The the only general format option currently is `force` to ignore decoder asserts.
|
||||||
|
For example to decode as mp3 and ignore assets do `mp3({force: true})` or `decode("mp3"; {force: true})`. From command line you can either do `fq -d mp3 -o force=true . file.mp3` or `fq -d bytes 'mp3({force: true})' file.mp3`.
|
||||||
|
|
||||||
|
Some formats has own options that can be specificed as part of `$opts` or as `-o name=value`. Too see options for a format do `fq -h mp3` or `help(mp3)` in a REPL. From command line you can either do `fq -d mp3 -o max_sync_seek=100 . file.mp3` or `fq -d bytes 'mp3({max_sync_seek: 100})' file.mp3`.
|
||||||
|
|
||||||
|
#### `decode`, `decode("<format>")`, `decode("<format>"; $opts)`
|
||||||
|
Decode format.
|
||||||
|
|
||||||
|
#### `probe`, `probe($opts)`
|
||||||
|
Probe and decode format.
|
||||||
|
|
||||||
|
#### `<format>`, `<format>($opts)`
|
||||||
|
Same as `decode("<format>")` and `decode("<format>"; $opts)`. Decode as format and return decode value even on decode error.
|
||||||
|
|
||||||
|
#### `from_<format>`, `from_<format>($opts)`
|
||||||
|
Same as `decode("<format>")` and `decode("<format>"; $opts)` decode as format but throw error on decode error.
|
||||||
|
|
||||||
Note that jq sometimes uses the notation `name/0`, `name/1` etc in error messages and documentation which means `<function-name>/<arity>`. Same function names with different arity are treated as separate functions, but are usually related in some way in practice.
|
Note that jq sometimes uses the notation `name/0`, `name/1` etc in error messages and documentation which means `<function-name>/<arity>`. Same function names with different arity are treated as separate functions, but are usually related in some way in practice.
|
||||||
|
|
||||||
### Function added in fq
|
#### `print`, `println`, `printerr`, `printerrln`
|
||||||
|
Print string or if not a string compact JSON value to stdout or stderr.
|
||||||
|
|
||||||
- All standard library functions from jq
|
#### `root`
|
||||||
- Adds a few new general functions:
|
Root decode value for decode value.
|
||||||
- `print`, `println`, `printerr`, `printerrln` prints to stdout and stderr.
|
|
||||||
- `group` group values, same as `group_by(.)`.
|
#### `buffer_root`
|
||||||
- `streaks`, `streaks_by(f)` like `group` but groups streaks based on condition.
|
Root decode value of buffer for decode value.
|
||||||
- `count`, `count_by(f)` like `group` but counts groups lengths.
|
|
||||||
- `debug(f)` like `debug` but uses arg to produce a debug message. `{a: 123} | debug({a}) | ...`.
|
#### `format_root`
|
||||||
- `path_to_expr` from `["key", 1]` to `".key[1]"`.
|
Root decode value of format for decode value.
|
||||||
- `expr_to_path` from `".key[1]"` to `["key", 1]`.
|
|
||||||
- `diff($a; $b)` produce diff object between two values.
|
#### `parent`
|
||||||
- `delta`, `delta_by(f)`, array with difference between all consecutive pairs.
|
Parent decode value for decode value.
|
||||||
- `chunk(f)`, split array or string into even chunks
|
|
||||||
- Bitwise functions `band`, `bor`, `bxor`, `bsl`, `bsr` and `bnot`. Works the same as jq math functions,
|
#### `parents`
|
||||||
unary uses input and if more than one argument all as arguments ignoring the input. Ex: `1 | bnot` `bsl(1; 3)`
|
Outputs all parent decode values from decode value.
|
||||||
- Adds some decode value specific functions:
|
|
||||||
- `root` tree root for value
|
#### `topath`
|
||||||
- `buffer_root` root value of buffer for value
|
Path for decode value. Use `path_to_expr` to get a string representation.
|
||||||
- `format_root` root value of format for value
|
|
||||||
- `parent` parent value
|
#### `tovalue`, `tovalue($opts)`
|
||||||
- `parents` output parents of value
|
Symbolic, if available, or actual value for decode value.
|
||||||
- `topath` path of value. Use `path_to_expr` to get a string representation.
|
|
||||||
- `tovalue`, `tovalue($opts)` symbolic value if available otherwise actual value
|
#### `toactual`, `toactual($opts)`
|
||||||
- `toactual`, `toactual($opts)` actual value (usually the decoded value)
|
Actual value for decode value.
|
||||||
- `tosym`, `tosym($opts)` symbolic value (mapped etc)
|
|
||||||
- `todescription` description of value
|
#### `tosym`, `tosym($opts)`
|
||||||
- `torepr` converts decode value into what it represents. For example convert msgpack decode value
|
Symbolic value for decode value.
|
||||||
into a value representing its JSON representation.
|
|
||||||
- All regexp functions work with binary as input and pattern argument with these differences
|
#### `todescription`
|
||||||
compared to when using string input:
|
Description for decode value.
|
||||||
- All offset and length will be in bytes.
|
|
||||||
- For `capture` the `.string` value is a binary.
|
#### `torepr`
|
||||||
- If pattern is a binary it will be matched literally and not as a regexp.
|
Converts decode value into what it represents. For example converts msgpack decode value into a value representing its JSON representation.
|
||||||
- If pattern is a binary or flags include "b" each input byte will be read as separate code points
|
|
||||||
- String functions are not overloaded to support binary for now as some of them might have behaviors that might be confusing.
|
### Display functions
|
||||||
- `explode` is overloaded to work with binary. Will explode into array of the unit of the binary.
|
|
||||||
end of binary.
|
Display shows hexdump, ASCII and tree column dump for decode values and jq value for other types.
|
||||||
instead of possibly multi-byte UTF-8 codepoints. This allows to match raw bytes. Ex: `match("\u00ff"; "b")`
|
|
||||||
will match the byte `0xff` and not the UTF-8 encoded codepoint for 255, `match("[^\u00ff]"; "b")` will match
|
#### `d`/`d($opts)`
|
||||||
all non-`0xff` bytes.
|
display value and truncate long arrays and binaries.
|
||||||
- `grep` functions take 1 or 2 arguments. First is a scalar to match, where a string is
|
|
||||||
treated as a regexp. A binary will match exact bytes. Second argument are regexp
|
#### `da`/`da($opts)`
|
||||||
flags with addition that "b" will treat each byte in the input binary as a code point, this
|
Display value and don't truncate arrays.
|
||||||
makes it possible to match exact bytes.
|
|
||||||
- `grep($v)`, `grep($v; $flags)` recursively match value and binary
|
#### `dd`/`dd($opts)`
|
||||||
- `vgrep($v)`, `vgrep($v; $flags)` recursively match value
|
Display value and don't truncate arrays or binaries.
|
||||||
- `bgrep($v)`, `bgrep($v; $flags)` recursively match binary
|
|
||||||
- `fgrep($v)`, `fgrep($v; $flags)` recursively match field name
|
#### `dv`/`dv($opts)`
|
||||||
- `grep_by(f)` recursively match using a filter. Ex: `grep_by(. > 180 and . < 200)`, `first(grep_by(format == "id3v2"))`.
|
Verbosely display value and don't truncate arrays but truncate binaries.
|
||||||
- Binary:
|
|
||||||
- `tobits` - Transform input to binary with bit as unit, does not preserve source range, will start at zero.
|
#### `ddv`/`ddv($opts)`
|
||||||
- `tobitsrange` - Transform input to binary with bit as unit, preserves source range if possible.
|
Verbosely display value and don't truncate arrays or binaries.
|
||||||
- `tobytes` - Transform input to binary with byte as unit, does not preserve source range, will start at zero.
|
|
||||||
- `tobytesrange` - Transform input binary with byte as unit, preserves source range if possible.
|
#### `hd`/`hexdump`
|
||||||
- `.[start:end]`, `.[:end]`, `.[start:]` - Slice binary from start to end preserve source range.
|
Hexdump value.
|
||||||
- `open` open file for reading
|
|
||||||
- All decode functions take an optional option argument. The only option currently is `force` to ignore decoder asserts.
|
### Binary values
|
||||||
For example to decode as mp3 and ignore assets do `mp3({force: true})` or `decode("mp3"; {force: true})`, from command line
|
|
||||||
you currently have to do `fq -d bytes 'mp3({force: true})' file`.
|
Binary values represents raw bits or bytes. When used in standard jq expressions they will behave as strings (UTF-8) with some exceptions listed below.
|
||||||
- `decode`, `decode("<format>")`, `decode("<format>"; $opts)` decode format
|
|
||||||
- `probe`, `probe($opts)` probe and decode format
|
- All regexp functions work with binary as input and pattern argument with these differences
|
||||||
- `mp3`, `mp3($opts)`, ..., `<format>`, `<format>($opts)` same as `decode("<format>")`, `decode("<format>"; $opts)` decode as format and return decode value even on decode error.
|
compared to when using string input:
|
||||||
- `from_mp3`, `from_mp3($opts)`, ..., `from_<format>`, `from_<format>($opts)` same as `decode("<format>")`, `decode("<format>"; $opts)` decode as format but throw error on decode error.
|
- All offset and length will be in bytes.
|
||||||
- Display shows hexdump/ASCII/tree for decode values and jq value for other types.
|
- For `capture` the `.string` value is a binary.
|
||||||
- `d`/`d($opts)` display value and truncate long arrays and binaries
|
- If pattern is a binary it will be matched literally and not as a regexp.
|
||||||
- `da`/`da($opts)` display value and don't truncate arrays
|
- If pattern is a binary or flags include "b" each input byte will be read as separate code points
|
||||||
- `dd`/`dd($opts)` display value and don't truncate arrays or binaries
|
- `explode` is overloaded to work with binary. Will explode into array of the unit of the binary.
|
||||||
- `dv`/`dv($opts)` verbosely display value and don't truncate arrays but truncate binaries
|
- `.[start:end]`, `.[:end]`, `.[start:]` - Slice binary from start to end preserve source range.
|
||||||
- `ddv`/`ddv($opts)` verbosely display value and don't truncate arrays or binaries
|
|
||||||
- `hd`/`hexdump` hexdump value
|
#### `grep($v)`, `grep($v; $flags)`, `vgrep($v)`, `vgrep($v; $flags)`, `bgrep($v)`, `bgrep($v; $flags)`
|
||||||
- `repl`/`repl($opts)` nested REPL, must be last in a pipeline. `1 | repl`, can "slurp" outputs. Ex: `1, 2, 3 | repl`, `[1,2,3] | repl({compact: true})`.
|
Recursively match `$v`.
|
||||||
- `slurp("<name>")` slurp outputs and save them to `$name`, must be last in the pipeline. Will be available as a global array `$name`. Ex `1,2,3 | slurp("a")`, `$a[]` same as `spew("a")`.
|
|
||||||
- `spew`/`spew("<name>")` output previously slurped values. `spew` outputs all slurps as an object, `spew("<name>")` outputs one slurp. Ex: `spew("a")`.
|
`$v` is a scalar to match, where a string is treated as a regexp. A binary will match exact bytes.
|
||||||
- `paste` read string from stdin until ^D. Useful for pasting text.
|
`$flags` argument are regexp flags with additional flag "b" that will treat each byte in the input binary
|
||||||
- Ex: `paste | from_pem | asn1_ber | repl` read from stdin then decode and start a new sub-REPL with result.
|
as a code point. This makes it possible to match exact bytes.
|
||||||
|
|
||||||
|
#### `fgrep($v)`, `fgrep($v; $flags)`
|
||||||
|
Recursively match field name in for decode value.
|
||||||
|
|
||||||
|
#### `tobits`
|
||||||
|
Transform input to binary with bit as unit and don't preserve source range.
|
||||||
|
|
||||||
|
#### `tobitsrange`
|
||||||
|
Transform input to binary with bit as unit and preserve source range.
|
||||||
|
|
||||||
|
#### `tobytes`
|
||||||
|
Transform input to binary with byte as unit and don't preserve source range.
|
||||||
|
|
||||||
|
#### `tobytesrange`
|
||||||
|
Transform input to binary with byte as unit and preserve source range.
|
||||||
|
|
||||||
|
#### `open`
|
||||||
|
Open file for reading.
|
||||||
|
|
||||||
### Naming inconsistencies
|
### Naming inconsistencies
|
||||||
|
|
||||||
|
Loading…
Reference in New Issue
Block a user