### Arguments 
$ fq -h 
fq - jq for files
Tool, language and decoders for exploring binary data.
For more information see https://github.com/wader/fq

Usage: fq [OPTIONS] [--] [EXPR] [FILE...]
--arg NAME VALUE         Set variable $NAME to string VALUE
--argjson NAME JSON      Set variable $NAME to JSON
--compact-output,-c      Compact output
--decode,-d NAME         Decode format (probe)
--decode-file NAME PATH  Set variable $NAME to decode of file
--formats                Show supported formats
--from-file,-f PATH      Read EXPR from file
--help,-h                Show help
--include-path,-L PATH   Include search path
--join-output,-j         No newline between outputs
--null-input,-n          Null input (use input/0 and inputs/0 to read input)
--null-output,-0         Null byte between outputs
--option,-o KEY=VALUE    Set option, eg: color=true
--options                Show all options
--raw-input,-R           Read raw input strings (don't decode)
--raw-output,-r          Raw string output (without quotes)
--rawfile NAME PATH      Set variable $NAME to string content of file
--repl,-i                Interactive REPL
--slurp,-s               Read (slurp) all inputs into an array
--version,-v             Show version (dev)
- TODO: null input - TODO: expressions ### Running - TODO: stdin/stdout ### Interactive REPL - TODO: tab completion, ctrl-d, ctrl-d, help - TODO: nested, nested with generator ### Script - TODO: #! ## Langauge fq is based on the [jq language](https://stedolan.github.io/jq/) and for basic usage its syntax is similar to how object and array access looks in JavaScript or JSON path, `.food[10]` etc. To get the most out of fq it's recommended to learn more about jq, here are some good starting points: - [jq manual](https://stedolan.github.io/jq/manual/) - [jq Cookbook](https://github.com/stedolan/jq/wiki/Cookbook), [FAQ](https://github.com/stedolan/jq/wiki/FAQ), [Pitfalls](https://github.com/stedolan/jq/wiki/How-to:-Avoid-Pitfalls) The most common beginner gotcha is probably jq's use of `;` and `,`. jq uses `;` as argument separator and `,` as output separator. To call `f` with two arguments use `f(a; b)`. If you do `f(a, b)` you pass a single argument `a, b` to `f`. ### Differences to jq - [gojq's differences to jq](https://github.com/itchyny/gojq#difference-to-jq), notable is support for arbitrary-precision integers. - Supports hexdecimal `0xab`, octal `0o77` and binary `0b101` integer literals - Has bitwise operations, `band`, `bor`, `bxor`, `bsl`, `bsr`, `bnot` - Try include `include "file?";` that don't fail if file is missing - Possible for a value to act as a object with keys even when it's an array, number etc. - There can be keys hidden from `keys` and `[]`. Used for, `_format`, `_bytes` etc. - Some values do not support to be updated ### Functions - All standard library functions from jq - `streaks/0`, `streaks_by/1` like `group` but groups streaks based on condition. - `count`, `count_by/1` like `group` but counts groups lengths. - `debug/1` like `debug/0` but uses arg to produce debug message. `{a: 123} | debug({a}) | ...`. - `path_to_expr` from `["key", 1]` to `".key[1]"`. - `expr_to_path` from `".key[1]"` to `["key", 1]`. - `diff/2` produce diff object between two values. - `delta`, `delta_by/1`, array with difference between all consecutive pairs. - `chunk/1`, split array or string into even chunks - `open` open file for reading - `probe` or `decode` probe format and decode - `mp3`, `matroska`, ..., ``, `decode([name])` force decode as format - `d`/`display` display value and truncate long arrays - `f`/`full` display value and don't truncate arrays - `v`/`verbose` display value verbosely and don't truncate array - `p`/`preview` show preview of field tree - `hd`/`hexdump` hexdump value - `repl` nested REPL, must be last in a pipeline. `1 | repl` or "slurp" multiple outputs `1, 2, 3 | repl`. ### Decoded values (TODO: better name?) When you decode something successfully in fq you will get a value. A value work a bit like jq object with special abilities and is used to represent a tree structure of the decoded binary data. Each value always has a name, type and a bit range. A value has these special keys: - `_name` name of value - `_value` jq value of value - `_start` bit range start - `_stop` bit range stop - `_len` bit range length (TODO: rename) - `_bits` bits in range as a binary - `_bytes` bits in range as binary using byte units - `_path` jq path to value - `_unknown` value is un-decoded gap - `_symbol` symbolic string representation of value (optional) - `_description` longer description of value (optional) - `_format` name of decoded format (optional) - `_error` error message (optional) - TODO: unknown gaps ### Binary and IO lists - TODO: similar to erlang io lists, [], binary, string (utf8) and numbers ## Configuration To add own functions you can use `init.fq` that will be read from - `$HOME/Library/Application Support/fq/init.jq` on macOS - `$HOME/.config/fq/init.jq` on Linux, BSD etc - `%AppData%\fq\init.jq` on Windows (TODO: not tested) ## Decoders [./formats_table.jq]: sh-start |Name |Description |Uses| |- |- |-| |`aac_frame` |Advanced Audio Coding frame || |`adts` |Audio Data Transport Stream |`adts_frame`| |`adts_frame` |Audio Data Transport Stream frame |`aac_frame`| |`apev2` |APEv2 metadata tag |`image`| |`av1_ccr` |AV1 Codec Configuration Record || |`av1_frame` |AV1 frame |`av1_obu`| |`av1_obu` |AV1 Open Bitstream Unit || |`avc_annexb` |H.264/AVC Annex B |`avc_nalu`| |`avc_au` |H.264/AVC Access Unit |`avc_nalu`| |`avc_dcr` |H.264/AVC Decoder Configuration Record |`avc_nalu`| |`avc_nalu` |H.264/AVC Network Access Layer Unit |`avc_sps` `avc_pps` `avc_sei`| |`avc_pps` |H.264/AVC Picture Parameter Set || |`avc_sei` |H.264/AVC Supplemental Enhancement Information || |`avc_sps` |H.264/AVC Sequence Parameter Set || |`bzip2` |bzip2 compression |`probe`| |`dns` |DNS packet || |`elf` |Executable and Linkable Format || |`exif` |Exchangeable Image File Format || |`flac` |Free Lossless Audio Codec file |`flac_metadatablock` `flac_frame`| |`flac_frame` |FLAC frame || |`flac_metadatablock` |FLAC metadatablock |`flac_picture` `vorbis_comment`| |`flac_picture` |FLAC metadatablock picture |`image`| |`gif` |Graphics Interchange Format || |`gzip` |gzip compression |`probe`| |`hevc_annexb` |H.265/HEVC Annex B |`hevc_nalu`| |`hevc_au` |H.265/HEVC Access Unit |`hevc_nalu`| |`hevc_dcr` |H.265/HEVC Decoder Configuration Record |`hevc_nalu`| |`hevc_nalu` |H.265/HEVC Network Access Layer Unit || |`icc_profile` |International Color Consortium profile || |`id3v1` |ID3v1 metadata || |`id3v11` |ID3v1.1 metadata || |`id3v2` |ID3v2 metadata |`image`| |`jpeg` |Joint Photographic Experts Group file |`exif` `icc_profile`| |`json` |JSON || |`matroska` |Matroska file |`aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablock` `hevc_au` `hevc_dcr` `image` `mp3_frame` `mpeg_asc` `mpeg_pes_packet` `mpeg_spu` `opus_packet` `vorbis_packet` `vp8_frame` `vp9_cfm` `vp9_frame`| |`mp3` |MP3 file |`id3v2` `id3v1` `id3v11` `apev2` `mp3_frame`| |`mp3_frame` |MPEG audio layer 3 frame |`xing`| |`mp4` |MPEG-4 file and similar |`aac_frame` `av1_ccr` `av1_frame` `flac_frame` `flac_metadatablock` `id3v2` `image` `jpeg` `mp3_frame` `avc_au` `avc_dcr` `mpeg_es` `hevc_au` `hevc_dcr` `mpeg_pes_packet` `opus_packet` `protobuf_widevine` `pssh_playready` `vorbis_packet` `vp9_frame` `vpx_ccr`| |`mpeg_asc` |MPEG-4 Audio Specific Config || |`mpeg_es` |MPEG Elementary Stream |`mpeg_asc` `vorbis_packet`| |`mpeg_pes` |MPEG Packetized elementary stream |`mpeg_pes_packet` `mpeg_spu`| |`mpeg_pes_packet` |MPEG Packetized elementary stream packet || |`mpeg_spu` |Sub Picture Unit (DVD subtitle) || |`mpeg_ts` |MPEG Transport Stream || |`ogg` |OGG file |`ogg_page` `vorbis_packet` `opus_packet`| |`ogg_page` |OGG page || |`opus_packet` |Opus packet |`vorbis_comment`| |`png` |Portable Network Graphics file |`icc_profile` `exif`| |`protobuf` |Protobuf || |`protobuf_widevine` |Widevine protobuf |`protobuf`| |`pssh_playready` |PlayReady PSSH || |`raw` |Raw bits || |`tar` |Tar archive |`probe`| |`tiff` |Tag Image File Format |`icc_profile`| |`vorbis_comment` |Vorbis comment |`flac_picture`| |`vorbis_packet` |Vorbis packet |`vorbis_comment`| |`vp8_frame` |VP8 frame || |`vp9_cfm` |VP9 Codec Feature Metadata || |`vp9_frame` |VP9 frame || |`vpx_ccr` |VPX Codec Configuration Record || |`wav` |WAV file |`id3v2` `id3v1` `id3v11`| |`webp` |WebP image |`vp8_frame`| |`xing` |Xing header || |`image` |Group |`gif` `jpeg` `png` `tiff` `webp`| |`probe` |Group |`adts` `bzip2` `elf` `flac` `gif` `gzip` `jpeg` `json` `matroska` `mp3` `mp4` `mpeg_ts` `ogg` `png` `tar` `tiff` `wav` `webp`| [#]: sh-end TODO: format graph? ## Own decoders and use as library TODO ### Known issues and useful tricks #### Run interactive mode with no input ```sh fq -i null> ``` #### `.. | select(...)` fails with `expected an ... but got: ...` Try add `select(...)?` to catch type errors in the select expression. #### Manual decode Sometimes fq fails to decode or you know there is valid data buried inside some binary or maybe you know the format of some unknown value. Then you can decode manually. 
# try decode a `mp3_frame` that failed to decode
$ fq file.mp3 .unknown0 mp3_frame
# skip first 10 bytes then decode as `mp3_frame`
$ fq file.mp3 .unknown0._bytes[10:] mp3_frame
### Pass `.` as input and argument This won't work as expected `.a | f(.b)` as `.` is `.a` when evaluating the arguments. Instead do `. as $c | .a | f($c.b)`. #### Building array is slow Try to use `map` or `foreach` to avoid rebuilding the whole array for each append. #### Use `print` and `println` to produce more friendly compact output ``` > [[0,"a"],[1,"b"]] [ [ 0, "a" ], [ 1, "b" ] ] > [[0,"a"],[1,"b"]] | .[] | "\(.[0]): \(.[1])" | println 0: a 1: b ``` ### `repl` argument using function or variable causes `variable not defined` `true as $verbose | repl({verbose: $verbose})` will currently fail as `repl` is implemented by rewriting the query to `map(true as $verbose | .) | repl({verbose: $verbose})`.