fq/doc/usage.md

Arguments

$ fq -h 
fq - jq for files
Tool, language and decoders for exploring binary data.
For more information see https://github.com/wader/fq

Usage: fq [OPTIONS] [--] [EXPR] [FILE...]
--arg NAME VALUE         Set variable $NAME to string VALUE
--argjson NAME JSON      Set variable $NAME to JSON
--compact-output,-c      Compact output
--decode,-d NAME         Decode format (probe)
--decode-file NAME PATH  Set variable $NAME to decode of file
--formats                Show supported formats
--from-file,-f PATH      Read EXPR from file
--help,-h                Show help
--include-path,-L PATH   Include search path
--join-output,-j         No newline between outputs
--null-input,-n          Null input (use input/0 and inputs/0 to read input)
--null-output,-0         Null byte between outputs
--option,-o KEY=VALUE    Set option, eg: color=true
--options                Show all options
--raw-input,-R           Read raw input strings (don't decode)
--raw-output,-r          Raw string output (without quotes)
--rawfile NAME PATH      Set variable $NAME to string content of file
--repl,-i                Interactive REPL
--slurp,-s               Read (slurp) all inputs into an array
--version,-v             Show version (dev)

• TODO: null input
• TODO: expressions

Running

• TODO: stdin/stdout

Interactive REPL

• TODO: tab completion, ctrl-d, ctrl-d, help
• TODO: nested, nested with generator

Script

• TODO: #!

Langauge

fq is based on the jq language and for basic usage its syntax is similar to how object and array access looks in JavaScript or JSON path, .food[10] etc.

To get the most out of fq it's recommended to learn more about jq, here are some good starting points:

• jq manual
• jq Cookbook, FAQ, Pitfalls

The most common beginner gotcha is probably jq's use of ; and ,. jq uses ; as argument separator and , as output separator. To call f with two arguments use f(a; b). If you do f(a, b) you pass a single argument a, b to f.

Differences to jq

• gojq's differences to jq, notable is support for arbitrary-precision integers.
• Supports hexdecimal 0xab, octal 0o77 and binary 0b101 integer literals
• Has bitwise operations, band, bor, bxor, bsl, bsr, bnot
• Try include include "file?"; that don't fail if file is missing
• Possible for a value to act as a object with keys even when it's an array, number etc.
• There can be keys hidden from keys and []. Used for, _format, _bytes etc.
• Some values do not support to be updated

Functions

• All standard library functions from jq
  • streaks/0, streaks_by/1 like group but groups streaks based on condition.
  • count, count_by/1 like group but counts groups lengths.
  • debug/1 like debug/0 but uses arg to produce debug message. {a: 123} | debug({a}) | ....
  • path_to_expr from ["key", 1] to ".key[1]".
  • expr_to_path from ".key[1]" to ["key", 1].
  • diff/2 produce diff object between two values.
  • delta, delta_by/1, array with difference between all consecutive pairs.
  • chunk/1, split array or string into even chunks
• open open file for reading
• probe or decode probe format and decode
• mp3, matroska, ..., <name>, decode([name]) force decode as format
• d/display display value and truncate long arrays
• f/full display value and don't truncate arrays
• v/verbose display value verbosely and don't truncate array
• p/preview show preview of field tree
• hd/hexdump hexdump value
• repl nested REPL, must be last in a pipeline. 1 | repl or "slurp" multiple outputs 1, 2, 3 | repl.

Decoded values (TODO: better name?)

When you decode something successfully in fq you will get a value. A value work a bit like jq object with special abilities and is used to represent a tree structure of the decoded binary data. Each value always has a name, type and a bit range.

A value has these special keys:

• _name name of value

• _value jq value of value

• _start bit range start

• _stop bit range stop

• _len bit range length (TODO: rename)

• _bits bits in range as a binary

• _bytes bits in range as binary using byte units

• _path jq path to value

• _unknown value is un-decoded gap

• _symbol symbolic string representation of value (optional)

• _description longer description of value (optional)

• _format name of decoded format (optional)

• _error error message (optional)

• TODO: unknown gaps

Binary and IO lists

• TODO: similar to erlang io lists, [], binary, string (utf8) and numbers

Configuration

To add own functions you can use init.fq that will be read from

• $HOME/Library/Application Support/fq/init.jq on macOS
• $HOME/.config/fq/init.jq on Linux, BSD etc
• %AppData%\fq\init.jq on Windows (TODO: not tested)

Decoders

Name	Description	Uses
aac_frame	Advanced Audio Coding frame
adts	Audio Data Transport Stream	adts_frame
adts_frame	Audio Data Transport Stream frame	aac_frame
apev2	APEv2 metadata tag	image
av1_ccr	AV1 Codec Configuration Record
av1_frame	AV1 frame	av1_obu
av1_obu	AV1 Open Bitstream Unit
avc_annexb	H.264/AVC Annex B	avc_nalu
avc_au	H.264/AVC Access Unit	avc_nalu
avc_dcr	H.264/AVC Decoder Configuration Record	avc_nalu
avc_nalu	H.264/AVC Network Access Layer Unit	avc_sps avc_pps avc_sei
avc_pps	H.264/AVC Picture Parameter Set
avc_sei	H.264/AVC Supplemental Enhancement Information
avc_sps	H.264/AVC Sequence Parameter Set
bzip2	bzip2 compression	probe
dns	DNS packet
elf	Executable and Linkable Format
exif	Exchangeable Image File Format
flac	Free Lossless Audio Codec file	flac_metadatablock flac_frame
flac_frame	FLAC frame
flac_metadatablock	FLAC metadatablock	flac_picture vorbis_comment
flac_picture	FLAC metadatablock picture	image
gif	Graphics Interchange Format
gzip	gzip compression	probe
hevc_annexb	H.265/HEVC Annex B	hevc_nalu
hevc_au	H.265/HEVC Access Unit	hevc_nalu
hevc_dcr	H.265/HEVC Decoder Configuration Record	hevc_nalu
hevc_nalu	H.265/HEVC Network Access Layer Unit
icc_profile	International Color Consortium profile
id3v1	ID3v1 metadata
id3v11	ID3v1.1 metadata
id3v2	ID3v2 metadata	image
jpeg	Joint Photographic Experts Group file	exif icc_profile
json	JSON
matroska	Matroska file	aac_frame av1_ccr av1_frame avc_au avc_dcr flac_frame flac_metadatablock hevc_au hevc_dcr image mp3_frame mpeg_asc mpeg_pes_packet mpeg_spu opus_packet vorbis_packet vp8_frame vp9_cfm vp9_frame
mp3	MP3 file	id3v2 id3v1 id3v11 apev2 mp3_frame
mp3_frame	MPEG audio layer 3 frame	xing
mp4	MPEG-4 file and similar	aac_frame av1_ccr av1_frame flac_frame flac_metadatablock id3v2 image jpeg mp3_frame avc_au avc_dcr mpeg_es hevc_au hevc_dcr mpeg_pes_packet opus_packet protobuf_widevine pssh_playready vorbis_packet vp9_frame vpx_ccr
mpeg_asc	MPEG-4 Audio Specific Config
mpeg_es	MPEG Elementary Stream	mpeg_asc vorbis_packet
mpeg_pes	MPEG Packetized elementary stream	mpeg_pes_packet mpeg_spu
mpeg_pes_packet	MPEG Packetized elementary stream packet
mpeg_spu	Sub Picture Unit (DVD subtitle)
mpeg_ts	MPEG Transport Stream
ogg	OGG file	ogg_page vorbis_packet opus_packet
ogg_page	OGG page
opus_packet	Opus packet	vorbis_comment
png	Portable Network Graphics file	icc_profile exif
protobuf	Protobuf
protobuf_widevine	Widevine protobuf	protobuf
pssh_playready	PlayReady PSSH
raw	Raw bits
tar	Tar archive	probe
tiff	Tag Image File Format	icc_profile
vorbis_comment	Vorbis comment	flac_picture
vorbis_packet	Vorbis packet	vorbis_comment
vp8_frame	VP8 frame
vp9_cfm	VP9 Codec Feature Metadata
vp9_frame	VP9 frame
vpx_ccr	VPX Codec Configuration Record
wav	WAV file	id3v2 id3v1 id3v11
webp	WebP image	vp8_frame
xing	Xing header
image	Group	gif jpeg png tiff webp
probe	Group	adts bzip2 elf flac gif gzip jpeg json matroska mp3 mp4 mpeg_ts ogg png tar tiff wav webp

TODO: format graph?

Own decoders and use as library

TODO

Known issues and useful tricks

Run interactive mode with no input

fq -i
null>

.. | select(...) fails with expected an ... but got: ...

Try add select(...)? to catch type errors in the select expression.

Manual decode

Sometimes fq fails to decode or you know there is valid data buried inside some binary or maybe you know the format of some unknown value. Then you can decode manually.

# try decode a `mp3_frame` that failed to decode
$ fq file.mp3 .unknown0 mp3_frame
# skip first 10 bytes then decode as `mp3_frame`
$ fq file.mp3 .unknown0._bytes[10:] mp3_frame

Pass . as input and argument

This won't work as expected .a | f(.b) as . is .a when evaluating the arguments. Instead do . as $c | .a | f($c.b).

Building array is slow

Try to use map or foreach to avoid rebuilding the whole array for each append.

Use print and println to produce more friendly compact output

> [[0,"a"],[1,"b"]]
[
  [
    0,
    "a"
  ],
  [
    1,
    "b"
  ]
]
> [[0,"a"],[1,"b"]] | .[] | "\(.[0]): \(.[1])" | println
0: a
1: b

repl argument using function or variable causes variable not defined

true as $verbose | repl({verbose: $verbose}) will currently fail as repl is implemented by rewriting the query to map(true as $verbose | .) | repl({verbose: $verbose}).

error produces no output

null | error behaves as empty.