14 KiB
Arguments
$ fq -h fq - jq for files Tool, language and decoders for exploring binary data. For more information see https://github.com/wader/fq Usage: fq [OPTIONS] [--] [EXPR] [FILE...] --arg NAME VALUE Set variable $NAME to string VALUE --argjson NAME JSON Set variable $NAME to JSON --compact-output,-c Compact output --decode,-d NAME Decode format (probe) --decode-file NAME PATH Set variable $NAME to decode of file --formats Show supported formats --from-file,-f PATH Read EXPR from file --help,-h Show help --include-path,-L PATH Include search path --join-output,-j No newline between outputs --null-input,-n Null input (use input/0 and inputs/0 to read input) --null-output,-0 Null byte between outputs --option,-o KEY=VALUE Set option, eg: color=true --options Show all options --raw-input,-R Read raw input strings (don't decode) --raw-output,-r Raw string output (without quotes) --rawfile NAME PATH Set variable $NAME to string content of file --repl,-i Interactive REPL --slurp,-s Read (slurp) all inputs into an array --version,-v Show version (dev)
- TODO: null input
- TODO: expressions
Running
- TODO: stdin/stdout
Interactive REPL
- TODO: tab completion, ctrl-d, ctrl-d, help
- TODO: nested, nested with generator
Script
- TODO: #!
Langauge
fq is based on the jq language and for basic usage its syntax
is similar to how object and array access looks in JavaScript or JSON path, .food[10]
etc.
To get the most out of fq it's recommended to learn more about jq, here are some good starting points:
The most common beginner gotcha is probably jq's use of ;
and ,
. jq uses ;
as argument separator
and ,
as output separator.
To call f
with two arguments use f(a; b)
. If you do f(a, b)
you pass a single
argument a, b
to f
.
Differences to jq
- gojq's differences to jq, notable is support for arbitrary-precision integers.
- Supports hexdecimal
0xab
, octal0o77
and binary0b101
integer literals - Has bitwise operations,
band
,bor
,bxor
,bsl
,bsr
,bnot
- Try include
include "file?";
that don't fail if file is missing - Possible for a value to act as a object with keys even when it's an array, number etc.
- There can be keys hidden from
keys
and[]
. Used for,_format
,_bytes
etc. - Some values do not support to be updated
Functions
- All standard library functions from jq
streaks/0
,streaks_by/1
likegroup
but groups streaks based on condition.count
,count_by/1
likegroup
but counts groups lengths.debug/1
likedebug/0
but uses arg to produce debug message.{a: 123} | debug({a}) | ...
.path_to_expr
from["key", 1]
to".key[1]"
.expr_to_path
from".key[1]"
to["key", 1]
.diff/2
produce diff object between two values.delta
,delta_by/1
, array with difference between all consecutive pairs.chunk/1
, split array or string into even chunks
open
open file for readingprobe
ordecode
probe format and decodemp3
,matroska
, ...,<name>
,decode([name])
force decode as formatd
/display
display value and truncate long arraysf
/full
display value and don't truncate arraysv
/verbose
display value verbosely and don't truncate arrayp
/preview
show preview of field treehd
/hexdump
hexdump valuerepl
nested REPL, must be last in a pipeline.1 | repl
or "slurp" multiple outputs1, 2, 3 | repl
.
Decoded values (TODO: better name?)
When you decode something successfully in fq you will get a value. A value work a bit like jq object with special abilities and is used to represent a tree structure of the decoded binary data. Each value always has a name, type and a bit range.
A value has these special keys:
-
_name
name of value -
_value
jq value of value -
_start
bit range start -
_stop
bit range stop -
_len
bit range length (TODO: rename) -
_bits
bits in range as a binary -
_bytes
bits in range as binary using byte units -
_path
jq path to value -
_unknown
value is un-decoded gap -
_symbol
symbolic string representation of value (optional) -
_description
longer description of value (optional) -
_format
name of decoded format (optional) -
_error
error message (optional) -
TODO: unknown gaps
Binary and IO lists
- TODO: similar to erlang io lists, [], binary, string (utf8) and numbers
Configuration
To add own functions you can use init.fq
that will be read from
$HOME/Library/Application Support/fq/init.jq
on macOS$HOME/.config/fq/init.jq
on Linux, BSD etc%AppData%\fq\init.jq
on Windows (TODO: not tested)
Decoders
Name | Description | Uses |
---|---|---|
aac_frame |
Advanced Audio Coding frame | |
adts |
Audio Data Transport Stream | adts_frame |
adts_frame |
Audio Data Transport Stream frame | aac_frame |
apev2 |
APEv2 metadata tag | image |
av1_ccr |
AV1 Codec Configuration Record | |
av1_frame |
AV1 frame | av1_obu |
av1_obu |
AV1 Open Bitstream Unit | |
avc_annexb |
H.264/AVC Annex B | avc_nalu |
avc_au |
H.264/AVC Access Unit | avc_nalu |
avc_dcr |
H.264/AVC Decoder Configuration Record | avc_nalu |
avc_nalu |
H.264/AVC Network Access Layer Unit | avc_sps avc_pps avc_sei |
avc_pps |
H.264/AVC Picture Parameter Set | |
avc_sei |
H.264/AVC Supplemental Enhancement Information | |
avc_sps |
H.264/AVC Sequence Parameter Set | |
bzip2 |
bzip2 compression | probe |
dns |
DNS packet | |
elf |
Executable and Linkable Format | |
exif |
Exchangeable Image File Format | |
flac |
Free Lossless Audio Codec file | flac_metadatablock flac_frame |
flac_frame |
FLAC frame | |
flac_metadatablock |
FLAC metadatablock | flac_picture vorbis_comment |
flac_picture |
FLAC metadatablock picture | image |
gif |
Graphics Interchange Format | |
gzip |
gzip compression | probe |
hevc_annexb |
H.265/HEVC Annex B | hevc_nalu |
hevc_au |
H.265/HEVC Access Unit | hevc_nalu |
hevc_dcr |
H.265/HEVC Decoder Configuration Record | hevc_nalu |
hevc_nalu |
H.265/HEVC Network Access Layer Unit | |
icc_profile |
International Color Consortium profile | |
id3v1 |
ID3v1 metadata | |
id3v11 |
ID3v1.1 metadata | |
id3v2 |
ID3v2 metadata | image |
jpeg |
Joint Photographic Experts Group file | exif icc_profile |
json |
JSON | |
matroska |
Matroska file | aac_frame av1_ccr av1_frame avc_au avc_dcr flac_frame flac_metadatablock hevc_au hevc_dcr image mp3_frame mpeg_asc mpeg_pes_packet mpeg_spu opus_packet vorbis_packet vp8_frame vp9_cfm vp9_frame |
mp3 |
MP3 file | id3v2 id3v1 id3v11 apev2 mp3_frame |
mp3_frame |
MPEG audio layer 3 frame | xing |
mp4 |
MPEG-4 file and similar | aac_frame av1_ccr av1_frame flac_frame flac_metadatablock id3v2 image jpeg mp3_frame avc_au avc_dcr mpeg_es hevc_au hevc_dcr mpeg_pes_packet opus_packet protobuf_widevine pssh_playready vorbis_packet vp9_frame vpx_ccr |
mpeg_asc |
MPEG-4 Audio Specific Config | |
mpeg_es |
MPEG Elementary Stream | mpeg_asc vorbis_packet |
mpeg_pes |
MPEG Packetized elementary stream | mpeg_pes_packet mpeg_spu |
mpeg_pes_packet |
MPEG Packetized elementary stream packet | |
mpeg_spu |
Sub Picture Unit (DVD subtitle) | |
mpeg_ts |
MPEG Transport Stream | |
ogg |
OGG file | ogg_page vorbis_packet opus_packet |
ogg_page |
OGG page | |
opus_packet |
Opus packet | vorbis_comment |
png |
Portable Network Graphics file | icc_profile exif |
protobuf |
Protobuf | |
protobuf_widevine |
Widevine protobuf | protobuf |
pssh_playready |
PlayReady PSSH | |
raw |
Raw bits | |
tar |
Tar archive | probe |
tiff |
Tag Image File Format | icc_profile |
vorbis_comment |
Vorbis comment | flac_picture |
vorbis_packet |
Vorbis packet | vorbis_comment |
vp8_frame |
VP8 frame | |
vp9_cfm |
VP9 Codec Feature Metadata | |
vp9_frame |
VP9 frame | |
vpx_ccr |
VPX Codec Configuration Record | |
wav |
WAV file | id3v2 id3v1 id3v11 |
webp |
WebP image | vp8_frame |
xing |
Xing header | |
image |
Group | gif jpeg png tiff webp |
probe |
Group | adts bzip2 elf flac gif gzip jpeg json matroska mp3 mp4 mpeg_ts ogg png tar tiff wav webp |
TODO: format graph?
Own decoders and use as library
TODO
Known issues and useful tricks
Run interactive mode with no input
fq -i
null>
.. | select(...)
fails with expected an ... but got: ...
Try add select(...)?
to catch type errors in the select expression.
Manual decode
Sometimes fq fails to decode or you know there is valid data buried inside some binary or maybe you know the format of some unknown value. Then you can decode manually.
# try decode a `mp3_frame` that failed to decode $ fq file.mp3 .unknown0 mp3_frame # skip first 10 bytes then decode as `mp3_frame` $ fq file.mp3 .unknown0._bytes[10:] mp3_frame
Pass .
as input and argument
This won't work as expected .a | f(.b)
as .
is .a
when evaluating the arguments.
Instead do . as $c | .a | f($c.b)
.
Building array is slow
Try to use map
or foreach
to avoid rebuilding the whole array for each append.
Use print
and println
to produce more friendly compact output
> [[0,"a"],[1,"b"]]
[
[
0,
"a"
],
[
1,
"b"
]
]
> [[0,"a"],[1,"b"]] | .[] | "\(.[0]): \(.[1])" | println
0: a
1: b
repl
argument using function or variable causes variable not defined
true as $verbose | repl({verbose: $verbose})
will currently fail as repl
is
implemented by rewriting the query to map(true as $verbose | .) | repl({verbose: $verbose})
.