2021-09-24 16:41:23 +03:00
|
|
|
|
## Basic usage
|
2021-09-18 20:27:46 +03:00
|
|
|
|
|
2021-09-19 01:51:15 +03:00
|
|
|
|
fq tries to behave the same way as jq as much as possible, so you can do:
|
|
|
|
|
```
|
|
|
|
|
fq . file.mp3
|
|
|
|
|
fq < file.mp3
|
|
|
|
|
fq . < file.mp3
|
2021-09-18 20:27:46 +03:00
|
|
|
|
fq . *.png *.jpg
|
2021-09-19 01:51:15 +03:00
|
|
|
|
fq '.frames[0]' file.mp3
|
2021-09-18 20:27:46 +03:00
|
|
|
|
```
|
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
## Interactive REPL
|
2021-09-18 20:27:46 +03:00
|
|
|
|
|
2021-09-19 01:51:15 +03:00
|
|
|
|
fq has an interactive [REPL](https://en.wikipedia.org/wiki/Read%E2%80%93eval%E2%80%93print_loop)
|
|
|
|
|
with auto completion and nested REPL support:
|
2021-09-18 20:27:46 +03:00
|
|
|
|
|
2021-09-19 01:51:15 +03:00
|
|
|
|
```
|
2021-09-18 20:27:46 +03:00
|
|
|
|
# start REPL with null input
|
2021-10-29 19:46:19 +03:00
|
|
|
|
$ fq -i
|
|
|
|
|
null>
|
2021-09-18 20:27:46 +03:00
|
|
|
|
# same as
|
2021-10-29 19:46:19 +03:00
|
|
|
|
$ fq -ni
|
|
|
|
|
null>
|
2021-09-18 20:27:46 +03:00
|
|
|
|
# start REPL with one file as input
|
2021-10-29 19:46:19 +03:00
|
|
|
|
$ fq -i . file.mp3
|
|
|
|
|
mp3>
|
2021-09-18 20:27:46 +03:00
|
|
|
|
```
|
|
|
|
|
|
|
|
|
|
In the REPL you will see a prompt indicating current input and you can type jq expression to evaluate.
|
|
|
|
|
|
2021-09-19 01:51:15 +03:00
|
|
|
|
```
|
2021-10-29 19:46:19 +03:00
|
|
|
|
$ fq -i . file.mp3
|
2021-09-18 20:27:46 +03:00
|
|
|
|
# basic arithmetics
|
|
|
|
|
mp3> 1+1
|
|
|
|
|
2
|
2021-10-29 19:46:19 +03:00
|
|
|
|
# "." is the identity function which just returns current input, the mp3 file.
|
2021-09-18 20:27:46 +03:00
|
|
|
|
mp3> .
|
|
|
|
|
# access the first frame in the mp3 file
|
|
|
|
|
mp3> .frames[0]
|
2021-10-29 19:46:19 +03:00
|
|
|
|
# start a new nested REPL with first frame as input
|
2021-09-18 20:27:46 +03:00
|
|
|
|
mp3> .frames[0] | repl
|
2021-09-24 16:41:23 +03:00
|
|
|
|
# prompt shows "path" to current input and that it's an mp3_frame.
|
2021-10-29 19:46:19 +03:00
|
|
|
|
# Ctrl-D to exit REPL
|
2021-09-24 16:41:23 +03:00
|
|
|
|
> .frames[0] mp3_frame> ^D
|
2021-10-29 19:46:19 +03:00
|
|
|
|
# Ctrl-D to exit to shell
|
2021-09-24 16:41:23 +03:00
|
|
|
|
mp3> ^D
|
|
|
|
|
$
|
2021-09-18 20:27:46 +03:00
|
|
|
|
```
|
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
Use Ctrl-D to exits, Ctrl-C to interrupt current evaluation.
|
|
|
|
|
|
|
|
|
|
## The jq langauge
|
|
|
|
|
|
|
|
|
|
fq is based on the [jq language](https://stedolan.github.io/jq/) and for basic usage its syntax
|
|
|
|
|
is similar to how object and array access looks in JavaScript or JSON path, `.food[10]` etc.
|
|
|
|
|
|
|
|
|
|
To get the most out of fq it's recommended to learn more about jq, here are some good starting points:
|
|
|
|
|
|
|
|
|
|
- [jq manual](https://stedolan.github.io/jq/manual/)
|
|
|
|
|
- jq wiki pages
|
|
|
|
|
[jq Language Description](https://github.com/stedolan/jq/wiki/jq-Language-Description),
|
|
|
|
|
[jq wiki page Cookbook](https://github.com/stedolan/jq/wiki/Cookbook),
|
|
|
|
|
[FAQ](https://github.com/stedolan/jq/wiki/FAQ) and
|
|
|
|
|
[Pitfalls](https://github.com/stedolan/jq/wiki/How-to:-Avoid-Pitfalls)
|
|
|
|
|
|
|
|
|
|
The most common beginner gotcha is probably jq's use of `;` and `,`. jq uses `;` as argument separator
|
|
|
|
|
and `,` as output separator.
|
|
|
|
|
To call a function `f` with two arguments use `f(1; 2)`. If you do `f(1, 2)` you pass a single
|
|
|
|
|
argument `1, 2` (a lambda expression that output `1` and then output `2`) to `f`.
|
2021-09-18 20:27:46 +03:00
|
|
|
|
|
2021-09-19 11:27:56 +03:00
|
|
|
|
## Support formats
|
|
|
|
|
|
|
|
|
|
See [formats](formats.md)
|
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
## Arguments
|
|
|
|
|
|
|
|
|
|
TODO: examples, stdin/stdout
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
|
|
|
|
<pre sh>
|
|
|
|
|
$ fq -h
|
2021-08-21 19:52:13 +03:00
|
|
|
|
fq - jq for files
|
2021-09-19 02:18:22 +03:00
|
|
|
|
Tool, language and format decoders for exploring binary data.
|
2021-08-12 21:07:34 +03:00
|
|
|
|
For more information see https://github.com/wader/fq
|
|
|
|
|
|
|
|
|
|
Usage: fq [OPTIONS] [--] [EXPR] [FILE...]
|
2021-10-30 19:35:21 +03:00
|
|
|
|
|
2021-09-07 01:26:24 +03:00
|
|
|
|
--arg NAME VALUE Set variable $NAME to string VALUE
|
|
|
|
|
--argjson NAME JSON Set variable $NAME to JSON
|
2021-09-19 02:18:22 +03:00
|
|
|
|
--color-output,-C Force color output
|
2021-09-07 01:26:24 +03:00
|
|
|
|
--compact-output,-c Compact output
|
|
|
|
|
--decode,-d NAME Decode format (probe)
|
|
|
|
|
--decode-file NAME PATH Set variable $NAME to decode of file
|
|
|
|
|
--formats Show supported formats
|
|
|
|
|
--from-file,-f PATH Read EXPR from file
|
|
|
|
|
--help,-h Show help
|
|
|
|
|
--include-path,-L PATH Include search path
|
|
|
|
|
--join-output,-j No newline between outputs
|
2021-09-19 02:18:22 +03:00
|
|
|
|
--monochrome-output,-M Force monochrome output
|
2021-09-07 01:26:24 +03:00
|
|
|
|
--null-input,-n Null input (use input/0 and inputs/0 to read input)
|
|
|
|
|
--null-output,-0 Null byte between outputs
|
2021-10-18 01:59:35 +03:00
|
|
|
|
--option,-o KEY=VALUE Set option, eg: color=true (use options/0 to see all options)
|
2021-11-01 14:02:06 +03:00
|
|
|
|
--raw-file NAME PATH Set variable $NAME to string content of file
|
2021-09-07 01:26:24 +03:00
|
|
|
|
--raw-input,-R Read raw input strings (don't decode)
|
|
|
|
|
--raw-output,-r Raw string output (without quotes)
|
|
|
|
|
--repl,-i Interactive REPL
|
|
|
|
|
--slurp,-s Read (slurp) all inputs into an array
|
2021-09-20 18:05:24 +03:00
|
|
|
|
--version,-v Show version
|
2020-06-08 03:29:51 +03:00
|
|
|
|
</pre>
|
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
## Use as script interpreter
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
fq can be used as a scrip interpreter:
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
`mp3_duration.jq`:
|
|
|
|
|
```jq
|
|
|
|
|
#!/usr/bin/env fq -d mp3 -rf
|
|
|
|
|
[.frames[].header | .sample_count / .sample_rate] | add
|
|
|
|
|
```
|
2021-09-14 13:56:09 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
## Differences to jq
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
|
|
|
|
- [gojq's differences to jq](https://github.com/itchyny/gojq#difference-to-jq),
|
|
|
|
|
notable is support for arbitrary-precision integers.
|
2021-09-24 16:41:23 +03:00
|
|
|
|
- Supports hexdecimal `0xab`, octal `0o77` and binary `0b101` integer literals.
|
|
|
|
|
- Has bitwise operations, `band`, `bor`, `bxor`, `bsl`, `bsr`, `bnot`.
|
|
|
|
|
- Try include `include "file?";` that don't fail if file is missing.
|
|
|
|
|
- Some values can act as a object with keys even when it's an array, number etc.
|
2020-06-08 03:29:51 +03:00
|
|
|
|
- There can be keys hidden from `keys` and `[]`. Used for, `_format`, `_bytes` etc.
|
2021-09-24 16:41:23 +03:00
|
|
|
|
- Some values are readonly and can't be updated.
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
## Functions
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
|
|
|
|
- All standard library functions from jq
|
2021-09-28 00:11:18 +03:00
|
|
|
|
- Adds a few new general functions:
|
2021-09-04 02:43:56 +03:00
|
|
|
|
- `streaks/0`, `streaks_by/1` like `group` but groups streaks based on condition.
|
|
|
|
|
- `count`, `count_by/1` like `group` but counts groups lengths.
|
2021-08-18 17:49:56 +03:00
|
|
|
|
- `debug/1` like `debug/0` but uses arg to produce debug message. `{a: 123} | debug({a}) | ...`.
|
|
|
|
|
- `path_to_expr` from `["key", 1]` to `".key[1]"`.
|
|
|
|
|
- `expr_to_path` from `".key[1]"` to `["key", 1]`.
|
|
|
|
|
- `diff/2` produce diff object between two values.
|
2021-09-04 02:43:56 +03:00
|
|
|
|
- `delta`, `delta_by/1`, array with difference between all consecutive pairs.
|
2021-09-04 12:38:49 +03:00
|
|
|
|
- `chunk/1`, split array or string into even chunks
|
2021-09-28 00:11:18 +03:00
|
|
|
|
- Adds some decode value specific functions:
|
|
|
|
|
- `root/0` return tree root for value
|
|
|
|
|
- `buffer_root/0` return root value of buffer for value
|
|
|
|
|
- `format_root/0` return root value of format for value
|
|
|
|
|
- `parent/0` return parent value
|
|
|
|
|
- `parents/0` output parents of value
|
2021-10-20 22:16:13 +03:00
|
|
|
|
- All `match` and `grep` functions take 1 or 2 arguments. First is a scalar to match, where a string is
|
2021-10-15 19:19:59 +03:00
|
|
|
|
treated as a regexp. A buffer scalar will be matches exact bytes. Second argument are regexp
|
|
|
|
|
flags with addition that "b" will treat each byte in the input buffer as a code point, this
|
2021-10-29 19:46:19 +03:00
|
|
|
|
makes it possible to match exact bytes, ex: `match("\u00ff"; "b")` will match the byte `0xff` and not
|
2021-10-15 19:19:59 +03:00
|
|
|
|
the UTF-8 encoded codepoint for 255.
|
2021-10-20 22:16:13 +03:00
|
|
|
|
- `match/1`, `match/2` overloaded to support buffers. Match in buffer and output match buffers
|
2021-10-17 02:26:30 +03:00
|
|
|
|
- `grep/1`, `grep/2` recursively match value and buffer
|
|
|
|
|
- `vgrep/1`, `vgrep/2` recursively match value
|
|
|
|
|
- `bgrep/1`, `bgrep/2` recursively match buffer
|
|
|
|
|
- `fgrep/1`, `fgrep/2` recursively match field name
|
2021-10-15 19:19:59 +03:00
|
|
|
|
- Buffers:
|
|
|
|
|
- `tobits` - Transform input into a bits buffer not preserving source range, will start at zero.
|
|
|
|
|
- `tobitsrange` - Transform input into a bits buffer preserving source range if possible.
|
|
|
|
|
- `tobytes` - Transform input into a bytes buffer not preserving source range, will start at zero.
|
|
|
|
|
- `tobytesrange` - Transform input into a byte buffer preserving source range if possible.
|
|
|
|
|
- `buffer[start:end]`, `buffer[:end]`, `buffer[start:]` - Create a sub buffer from start to end in buffer units preserving source range.
|
2020-06-08 03:29:51 +03:00
|
|
|
|
- `open` open file for reading
|
2021-09-04 02:43:56 +03:00
|
|
|
|
- `probe` or `decode` probe format and decode
|
|
|
|
|
- `mp3`, `matroska`, ..., `<name>`, `decode([name])` force decode as format
|
2021-08-30 13:51:42 +03:00
|
|
|
|
- `d`/`display` display value and truncate long arrays
|
|
|
|
|
- `f`/`full` display value and don't truncate arrays
|
|
|
|
|
- `v`/`verbose` display value verbosely and don't truncate array
|
2020-06-08 03:29:51 +03:00
|
|
|
|
- `p`/`preview` show preview of field tree
|
|
|
|
|
- `hd`/`hexdump` hexdump value
|
2021-09-24 16:41:23 +03:00
|
|
|
|
- `repl` nested REPL, must be last in a pipeline. `1 | repl`, can "slurp" multiple outputs `1, 2, 3 | repl`.
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
## Decoded values (TODO: better name?)
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
When you decode something you will get a decode value. A decode values work like
|
|
|
|
|
normal jq values but has special abilities and is used to represent a tree structure of the decoded
|
2020-06-08 03:29:51 +03:00
|
|
|
|
binary data. Each value always has a name, type and a bit range.
|
|
|
|
|
|
|
|
|
|
A value has these special keys:
|
|
|
|
|
|
|
|
|
|
- `_name` name of value
|
|
|
|
|
- `_value` jq value of value
|
|
|
|
|
- `_start` bit range start
|
|
|
|
|
- `_stop` bit range stop
|
|
|
|
|
- `_len` bit range length (TODO: rename)
|
|
|
|
|
- `_bits` bits in range as a binary
|
|
|
|
|
- `_bytes` bits in range as binary using byte units
|
|
|
|
|
- `_path` jq path to value
|
|
|
|
|
- `_unknown` value is un-decoded gap
|
|
|
|
|
- `_symbol` symbolic string representation of value (optional)
|
|
|
|
|
- `_description` longer description of value (optional)
|
|
|
|
|
- `_format` name of decoded format (optional)
|
|
|
|
|
- `_error` error message (optional)
|
|
|
|
|
|
|
|
|
|
- TODO: unknown gaps
|
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
## Binary and IO lists
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
|
|
|
|
- TODO: similar to erlang io lists, [], binary, string (utf8) and numbers
|
|
|
|
|
|
|
|
|
|
## Configuration
|
|
|
|
|
|
|
|
|
|
To add own functions you can use `init.fq` that will be read from
|
|
|
|
|
- `$HOME/Library/Application Support/fq/init.jq` on macOS
|
|
|
|
|
- `$HOME/.config/fq/init.jq` on Linux, BSD etc
|
|
|
|
|
- `%AppData%\fq\init.jq` on Windows (TODO: not tested)
|
|
|
|
|
|
|
|
|
|
## Own decoders and use as library
|
|
|
|
|
|
|
|
|
|
TODO
|
|
|
|
|
|
|
|
|
|
|
2021-09-07 02:38:52 +03:00
|
|
|
|
## Known issues and useful tricks
|
2021-09-06 02:24:51 +03:00
|
|
|
|
|
2021-09-07 02:38:52 +03:00
|
|
|
|
### Run interactive mode with no input
|
2021-09-06 02:24:51 +03:00
|
|
|
|
```sh
|
|
|
|
|
fq -i
|
|
|
|
|
null>
|
|
|
|
|
```
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
### `select` fails with `expected an ... but got: ...`
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
Try add `select(...)?` to catch and ignore type errors in the select expression.
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-07 02:38:52 +03:00
|
|
|
|
### Manual decode
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
|
|
|
|
Sometimes fq fails to decode or you know there is valid data buried inside some binary or maybe
|
|
|
|
|
you know the format of some unknown value. Then you can decode manually.
|
|
|
|
|
|
|
|
|
|
<pre>
|
|
|
|
|
# try decode a `mp3_frame` that failed to decode
|
2021-10-29 19:46:19 +03:00
|
|
|
|
$ fq -d mp3 '.unknown0 | mp3_frame' file.mp3
|
2020-06-08 03:29:51 +03:00
|
|
|
|
# skip first 10 bytes then decode as `mp3_frame`
|
2021-10-29 19:46:19 +03:00
|
|
|
|
$ fq -d raw 'tobytes[10:] | mp3_frame' file.mp3
|
2020-06-08 03:29:51 +03:00
|
|
|
|
</pre>
|
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
### Use `.` as input and in a positional argument
|
2021-09-04 12:52:40 +03:00
|
|
|
|
|
2021-09-24 16:41:23 +03:00
|
|
|
|
The expression `.a | f(.b)` might not work as expected. `.` is `.a` when evaluating the arguments so
|
|
|
|
|
the positional argument will end up being `.a.b`. Instead do `. as $c | .a | f($c.b)`.
|
2021-09-04 12:52:40 +03:00
|
|
|
|
|
2021-09-07 02:38:52 +03:00
|
|
|
|
### Building array is slow
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-06 02:24:51 +03:00
|
|
|
|
Try to use `map` or `foreach` to avoid rebuilding the whole array for each append.
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
2021-09-07 02:38:52 +03:00
|
|
|
|
### Use `print` and `println` to produce more friendly compact output
|
2020-06-08 03:29:51 +03:00
|
|
|
|
|
|
|
|
|
```
|
|
|
|
|
> [[0,"a"],[1,"b"]]
|
|
|
|
|
[
|
|
|
|
|
[
|
|
|
|
|
0,
|
|
|
|
|
"a"
|
|
|
|
|
],
|
|
|
|
|
[
|
|
|
|
|
1,
|
|
|
|
|
"b"
|
|
|
|
|
]
|
|
|
|
|
]
|
|
|
|
|
> [[0,"a"],[1,"b"]] | .[] | "\(.[0]): \(.[1])" | println
|
|
|
|
|
0: a
|
|
|
|
|
1: b
|
|
|
|
|
```
|
|
|
|
|
|
2021-09-06 02:24:51 +03:00
|
|
|
|
### `repl` argument using function or variable causes `variable not defined`
|
|
|
|
|
|
|
|
|
|
`true as $verbose | repl({verbose: $verbose})` will currently fail as `repl` is
|
|
|
|
|
implemented by rewriting the query to `map(true as $verbose | .) | repl({verbose: $verbose})`.
|
2021-09-07 02:38:52 +03:00
|
|
|
|
|
|
|
|
|
### `error` produces no output
|
|
|
|
|
|
|
|
|
|
`null | error` behaves as `empty`.
|