mirror of
https://github.com/wader/fq.git
synced 2024-12-23 05:13:30 +03:00
doc: Use f($a; $b) instead of jq f/2 notation
Also some general polish
This commit is contained in:
Mattias Wadman
parent
aea48847c5
commit
c52a1a2364
@ -18,12 +18,13 @@ That also means that there is a great opportunity to help out!
|
||||
|
||||
### Goals
|
||||
|
||||
- Make binary formats accessible and queryable.
|
||||
- Make binaries accessible, queryable and sliceable.
|
||||
- Nested formats and bit-oriented decoding.
|
||||
- Quick and comfortable CLI tool.
|
||||
- Bits and bytes transformations.
|
||||
|
||||
### Hopes
|
||||
|
||||
- Make it useful enough that people want to help improve it.
|
||||
- Inspire people to create similar tools.
|
||||
|
||||
|
||||
180
doc/usage.md
180
doc/usage.md
@ -1,44 +1,59 @@
|
||||
## Basic usage
|
||||
|
||||
fq tries to behave the same way as jq as much as possible, so you can do:
|
||||
|
||||
```sh
|
||||
fq . file
|
||||
fq < file
|
||||
file | fq
|
||||
cat file | fq
|
||||
fq . < file
|
||||
fq . *.png *.mp3
|
||||
fq '.frames[0]' file.mp3
|
||||
fq '.frames[0]' *.mp3
|
||||
```
|
||||
|
||||
Common usages:
|
||||
|
||||
```sh
|
||||
# recursively display decode tree but truncate long arrays
|
||||
fq d file
|
||||
# same as
|
||||
fq display file
|
||||
|
||||
# display all or more bytes for each value
|
||||
# display all bytes for each value
|
||||
fq 'd({display_bytes: 0})' file
|
||||
# display 200 bytes for each value
|
||||
fq 'd({display_bytes: 200})' file
|
||||
|
||||
# recursively display decode tree
|
||||
fq f file
|
||||
# same as
|
||||
fq full file
|
||||
|
||||
# recursively verbosely display decode tree
|
||||
# recursively and verbosely display decode tree
|
||||
fq v file
|
||||
# same as
|
||||
fq verbose file
|
||||
|
||||
# JSON for whole file
|
||||
# JSON repersenation for whole file
|
||||
fq tovalue file
|
||||
|
||||
# recursively look for decode value roots for a format
|
||||
fq '.. | select(format=="jpeg")' file
|
||||
# can also use grep_by
|
||||
fq 'grep_by(format=="jpeg")' file
|
||||
|
||||
# recursively look for first decode value root for a format
|
||||
fq 'first(.. | select(format=="jpeg"))' file
|
||||
fq 'first(grep_by(format=="jpeg"))' file
|
||||
|
||||
# recursively look for objects fullfilling condition
|
||||
fq '.. | select(.type=="trak")?' file
|
||||
fq 'grep_by(.type=="trak")' file
|
||||
|
||||
# grep whole tree
|
||||
fq 'grep("^prefix")' file
|
||||
fq 'grep(123)' file
|
||||
fq 'grep_by(. >= 100 and . =< 100)' file
|
||||
```
|
||||
|
||||
## Interactive REPL
|
||||
@ -46,21 +61,20 @@ fq '.. | select(.type=="trak")?' file
|
||||
The interactive [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop)
|
||||
has auto completion and nested REPL support:
|
||||
|
||||
```
|
||||
```sh
|
||||
# start REPL with null input
|
||||
$ fq -i
|
||||
null>
|
||||
# same as
|
||||
$ fq -ni
|
||||
null>
|
||||
|
||||
# in the REPL you will see a prompt indicating current input and you can type jq expression to evaluate.
|
||||
|
||||
# start REPL with one file as input
|
||||
$ fq -i . doc/file.mp3
|
||||
mp3>
|
||||
```
|
||||
|
||||
In the REPL you will see a prompt indicating current input and you can type jq expression to evaluate.
|
||||
|
||||
```
|
||||
$ fq -i . doc/file.mp3
|
||||
# basic arithmetics and jq expressions
|
||||
mp3> 1+1
|
||||
@ -180,6 +194,10 @@ $ fq -rn '[inputs | [input_filename, first(.chunks[] | select(.type=="IHDR") | .
|
||||
$ fq '.. | select(scalars and in_bytes_range(0x123))' file
|
||||
```
|
||||
|
||||
## Support formats
|
||||
|
||||
See [formats](formats.md)
|
||||
|
||||
## The jq language
|
||||
|
||||
fq is based on the [jq language](https://stedolan.github.io/jq/) and for basic usage its syntax
|
||||
@ -199,9 +217,74 @@ and `,` as output separator.
|
||||
To call a function `f` with two arguments use `f(1; 2)`. If you do `f(1, 2)` you pass a single
|
||||
argument `1, 2` (a lambda expression that output `1` and then output `2`) to `f`.
|
||||
|
||||
## Support formats
|
||||
## Functions
|
||||
|
||||
See [formats](formats.md)
|
||||
- All standard library functions from jq
|
||||
- Adds a few new general functions:
|
||||
- `print`, `println`, `printerr`, `printerrln` prints to stdout and stderr.
|
||||
- `streaks`, `streaks_by(f)` like `group` but groups streaks based on condition.
|
||||
- `count`, `count_by(f)` like `group` but counts groups lengths.
|
||||
- `debug(f)` like `debug` but uses arg to produce debug message. `{a: 123} | debug({a}) | ...`.
|
||||
- `path_to_expr` from `["key", 1]` to `".key[1]"`.
|
||||
- `expr_to_path` from `".key[1]"` to `["key", 1]`.
|
||||
- `diff($a; $b)` produce diff object between two values.
|
||||
- `delta`, `delta_by(f)`, array with difference between all consecutive pairs.
|
||||
- `chunk(f)`, split array or string into even chunks
|
||||
- Bitwise functions `band`, `bor`, `bxor`, `bsl`, `bsr` and `bnot`. Works the same as jq math functions,
|
||||
unary uses input and if more than one argument all as arguments ignoring the input. Ex: `1 | bnot` `bsl(1; 3)`
|
||||
- Adds some decode value specific functions:
|
||||
- `root` tree root for value
|
||||
- `buffer_root` root value of buffer for value
|
||||
- `format_root` root value of format for value
|
||||
- `parent` parent value
|
||||
- `parents` output parents of value
|
||||
- `topath` path of value. Use `path_to_expr` to get a string representation.
|
||||
- `tovalue`, `tovalue($opts)` symbolic value if available otherwise actual value
|
||||
- `toactual` actual value (decoded etc)
|
||||
- `tosym` symbolic value (mapped etc)
|
||||
- `todescription` description of value
|
||||
- `torepr` convert decode value into what it reptresents. For example convert msgpack decode value
|
||||
into a value representing its JSON representation.
|
||||
- All regexp functions work with buffers as input and pattern argument with these differences
|
||||
from the string versions:
|
||||
- All offset and length will be in bytes.
|
||||
- For `capture` the `.string` value is a buffer.
|
||||
- If pattern is a buffer it will be matched literally and not as a regexp.
|
||||
- If pattern is a buffer or flags include "b" each input byte will be read as separate code points
|
||||
- `scan_toend($v)`, `scan_toend($v; $flags)` works the same as `scan` but output buffer are from start of match to
|
||||
end of buffer.
|
||||
instead of possibly multi-byte UTF-8 codepoints. This allows to match raw bytes. Ex: `match("\u00ff"; "b")`
|
||||
will match the byte `0xff` and not the UTF-8 encoded codepoint for 255, `match("[^\u00ff]"; "b")` will match
|
||||
all non-`0xff` bytes.
|
||||
- `grep` functions take 1 or 2 arguments. First is a scalar to match, where a string is
|
||||
treated as a regexp. A buffer scalar will be matches exact bytes. Second argument are regexp
|
||||
flags with addition that "b" will treat each byte in the input buffer as a code point, this
|
||||
makes it possible to match exact bytes.
|
||||
- `grep($v)`, `grep($v; $flags)` recursively match value and buffer
|
||||
- `vgrep($v)`, `vgrep($v; $flags)` recursively match value
|
||||
- `bgrep($v)`, `bgrep($v; $flags)` recursively match buffer
|
||||
- `fgrep($v)`, `fgrep($v; $flags)` recursively match field name
|
||||
- `grep_by(f)` recursively match using a filter. Ex: `grep_by(. > 180 and . < 200)`, `first(grep_by(format == "id3v2"))`.
|
||||
- Buffers:
|
||||
- `tobits` - Transform input into a bits buffer not preserving source range, will start at zero.
|
||||
- `tobitsrange` - Transform input into a bits buffer preserving source range if possible.
|
||||
- `tobytes` - Transform input into a bytes buffer not preserving source range, will start at zero.
|
||||
- `tobytesrange` - Transform input into a byte buffer preserving source range if possible.
|
||||
- `buffer[start:end]`, `buffer[:end]`, `buffer[start:]` - Create a sub buffer from start to end in buffer units preserving source range.
|
||||
- `open` open file for reading
|
||||
- All decode function takes a optional option argument. The only option currently is `force` to ignore decoder asserts.
|
||||
For example to decode as mp3 and ignore assets do `mp3({force: true})` or `decode("mp3"; {force: true})`, from command line
|
||||
you currently have to do `fq -d raw 'mp3({force: true})' file`.
|
||||
- `decode`, `decode($format)`, `decode($format; $opts)` decode format
|
||||
- `probe`, `probe($opts)` probe and decode format
|
||||
- `mp3`, `mp3($opts)`, ..., `<name>`, `<name>($opts)` same as `decode(<name>)($opts)`, `decode($format; $opts)` decode as format
|
||||
- Display shows hexdump/ASCII/tree for decode values and JSON for other values.
|
||||
- `d`/`display` display value and truncate long arrays
|
||||
- `f`/`full` display value and don't truncate arrays
|
||||
- `v`/`verbose` display value verbosely and don't truncate array
|
||||
- `p`/`preview` show preview of field tree
|
||||
- `hd`/`hexdump` hexdump value
|
||||
- `repl` nested REPL, must be last in a pipeline. `1 | repl`, can "slurp" multiple outputs `1, 2, 3 | repl`.
|
||||
|
||||
## Arguments
|
||||
|
||||
@ -227,9 +310,9 @@ Usage: fq [OPTIONS] [--] [EXPR] [FILE...]
|
||||
--include-path,-L PATH Include search path
|
||||
--join-output,-j No newline between outputs
|
||||
--monochrome-output,-M Force monochrome output
|
||||
--null-input,-n Null input (use input/0 and inputs/0 to read input)
|
||||
--null-input,-n Null input (use input and inputs to read input)
|
||||
--null-output,-0 Null byte between outputs
|
||||
--option,-o KEY=VALUE Set option, eg: color=true (use options/0 to see all options)
|
||||
--option,-o KEY=VALUE Set option, eg: color=true (use options to see all options)
|
||||
--raw-file NAME PATH Set variable $NAME to string content of file
|
||||
--raw-input,-R Read raw input strings (don't decode)
|
||||
--raw-output,-r Raw string output (without quotes)
|
||||
@ -271,75 +354,6 @@ notable is support for arbitrary-precision integers.
|
||||
- There can be keys hidden from `keys` and `[]`.
|
||||
- Some values are readonly and can't be updated.
|
||||
|
||||
## Functions
|
||||
|
||||
- All standard library functions from jq
|
||||
- Adds a few new general functions:
|
||||
- `print/0`, `println/0`, `printerr/0`, `printerrln` prints to stdout and stderr.
|
||||
- `streaks/0`, `streaks_by/1` like `group` but groups streaks based on condition.
|
||||
- `count/0`, `count_by/1` like `group` but counts groups lengths.
|
||||
- `debug/1` like `debug/0` but uses arg to produce debug message. `{a: 123} | debug({a}) | ...`.
|
||||
- `path_to_expr/0` from `["key", 1]` to `".key[1]"`.
|
||||
- `expr_to_path/0` from `".key[1]"` to `["key", 1]`.
|
||||
- `diff/2` produce diff object between two values.
|
||||
- `delta/0`, `delta_by/1`, array with difference between all consecutive pairs.
|
||||
- `chunk/1`, split array or string into even chunks
|
||||
- Bitwise functions `band`, `bor`, `bxor`, `bsl`, `bsr` and `bnot`. Works the same as jq math functions,
|
||||
unary uses input and if more than one argument all as arguments ignoring the input. Ex: `1 | bnot` `bsl(1; 3)`
|
||||
- Adds some decode value specific functions:
|
||||
- `root/0` tree root for value
|
||||
- `buffer_root/0` root value of buffer for value
|
||||
- `format_root/0` root value of format for value
|
||||
- `parent/0` parent value
|
||||
- `parents/0` output parents of value
|
||||
- `topath/0` path of value. Use `path_to_expr` to get a string representation.
|
||||
- `tovalue/0`, `tovalue/1` symbolic value if available otherwise actual value
|
||||
- `toactual/0` actual value (decoded etc)
|
||||
- `tosym/0` symbolic value (mapped etc)
|
||||
- `todescription/0` description of value
|
||||
- `torepr/0` convert decode value into what it reptresents. For example convert msgpack decode value
|
||||
into a value representing its JSON representation.
|
||||
- All regexp functions work with buffers as input and pattern argument with these differences
|
||||
from the string versions:
|
||||
- All offset and length will be in bytes.
|
||||
- For `capture` the `.string` value is a buffer.
|
||||
- If pattern is a buffer it will be matched literally and not as a regexp.
|
||||
- If pattern is a buffer or flags include "b" each input byte will be read as separate code points
|
||||
- `scan_toend/1`, `scan_toend/2` works the same as `scan` but output buffer are from start of match to
|
||||
end of buffer.
|
||||
instead of possibly multi-byte UTF-8 codepoints. This allows to match raw bytes. Ex: `match("\u00ff"; "b")`
|
||||
will match the byte `0xff` and not the UTF-8 encoded codepoint for 255, `match("[^\u00ff]"; "b")` will match
|
||||
all non-`0xff` bytes.
|
||||
- `grep` functions take 1 or 2 arguments. First is a scalar to match, where a string is
|
||||
treated as a regexp. A buffer scalar will be matches exact bytes. Second argument are regexp
|
||||
flags with addition that "b" will treat each byte in the input buffer as a code point, this
|
||||
makes it possible to match exact bytes.
|
||||
- `grep/1`, `grep/2` recursively match value and buffer
|
||||
- `vgrep/1`, `vgrep/2` recursively match value
|
||||
- `bgrep/1`, `bgrep/2` recursively match buffer
|
||||
- `fgrep/1`, `fgrep/2` recursively match field name
|
||||
- `grep_by/1` recursively match using a filter. Ex: `grep_by(. > 180 and . < 200)`, `first(grep_by(format == "id3v2"))`.
|
||||
- Buffers:
|
||||
- `tobits/0` - Transform input into a bits buffer not preserving source range, will start at zero.
|
||||
- `tobitsrange/0` - Transform input into a bits buffer preserving source range if possible.
|
||||
- `tobytes/0` - Transform input into a bytes buffer not preserving source range, will start at zero.
|
||||
- `tobytesrange/0` - Transform input into a byte buffer preserving source range if possible.
|
||||
- `buffer[start:end]`, `buffer[:end]`, `buffer[start:]` - Create a sub buffer from start to end in buffer units preserving source range.
|
||||
- `open` open file for reading
|
||||
- All decode function takes a optional option argument. The only option currently is `force` to ignore decoder asserts.
|
||||
For example to decode as mp3 and ignore assets do `mp3({force: true})` or `decode("mp3"; {force: true})`, from command line
|
||||
you currently have to do `fq -d raw 'mp3({force: true})' file`.
|
||||
- `decode/0`, `decode/1`, `decode/2` decode format
|
||||
- `probe/0`, `probe/1` probe and decode format
|
||||
- `mp3/0`, `mp3/1`, ..., `<name>/0`, `<name>/1` same as `decode(<name>)/1`, `decode(<name>; <opts>)/2` decode as format
|
||||
- Display shows hexdump/ASCII/tree for decode values and JSON for other values.
|
||||
- `d/0`/`display/0` display value and truncate long arrays
|
||||
- `f/0`/`full/0` display value and don't truncate arrays
|
||||
- `v/0`/`verbose/0` display value verbosely and don't truncate array
|
||||
- `p/0`/`preview/0` show preview of field tree
|
||||
- `hd/0`/`hexdump/0` hexdump value
|
||||
- `repl/0` nested REPL, must be last in a pipeline. `1 | repl`, can "slurp" multiple outputs `1, 2, 3 | repl`.
|
||||
|
||||
## Decoded values
|
||||
|
||||
When you decode something you will get a decode value. A decode values work like
|
||||
|
||||
@ -214,7 +214,7 @@ def _opt_cli_opts:
|
||||
"null_input": {
|
||||
short: "-n",
|
||||
long: "--null-input",
|
||||
description: "Null input (use input/0 and inputs/0 to read input)",
|
||||
description: "Null input (use input and inputs functions to read input)",
|
||||
bool: true
|
||||
},
|
||||
"monochrome_output": {
|
||||
@ -226,7 +226,7 @@ def _opt_cli_opts:
|
||||
"option": {
|
||||
short: "-o",
|
||||
long: "--option",
|
||||
description: "Set option, eg: color=true (use options/0 to see all options)",
|
||||
description: "Set option, eg: color=true (use options function to see all options)",
|
||||
object: "KEY=VALUE",
|
||||
},
|
||||
"string_input": {
|
||||
|
||||
4
pkg/interp/testdata/args.fqtest
vendored
4
pkg/interp/testdata/args.fqtest
vendored
@ -21,9 +21,9 @@ Usage: fq [OPTIONS] [--] [EXPR] [FILE...]
|
||||
--include-path,-L PATH Include search path
|
||||
--join-output,-j No newline between outputs
|
||||
--monochrome-output,-M Force monochrome output
|
||||
--null-input,-n Null input (use input/0 and inputs/0 to read input)
|
||||
--null-input,-n Null input (use input and inputs functions to read input)
|
||||
--null-output,-0 Null byte between outputs
|
||||
--option,-o KEY=VALUE Set option, eg: color=true (use options/0 to see all options)
|
||||
--option,-o KEY=VALUE Set option, eg: color=true (use options function to see all options)
|
||||
--raw-file NAME PATH Set variable $NAME to string content of file
|
||||
--raw-input,-R Read raw input strings (don't decode)
|
||||
--raw-output,-r Raw string output (without quotes)
|
||||
|
||||
Loading…
Reference in New Issue
Block a user