2020-06-08 03:29:51 +03:00
# fq
2022-01-15 21:26:45 +03:00
Tool, language and decoders for working with binary data.
2020-06-08 03:29:51 +03:00
2021-10-10 03:17:29 +03:00
![fq demo ](doc/demo.svg )
2020-06-08 03:29:51 +03:00
2022-11-17 13:10:08 +03:00
Basic usage is `fq . file` or `fq d file` .
For details see [usage.md ](doc/usage.md ).
### Background
2022-05-23 22:54:24 +03:00
fq is inspired by the well known jq tool and language and allows you to work with binary formats the same way you would using jq. In addition it can present data like a hex viewer, transform, slice and concatenate binary data. It also supports nested formats and has an interactive REPL with auto-completion.
2021-11-22 16:16:59 +03:00
2022-11-17 13:10:08 +03:00
It was originally designed to query, inspect and debug media codecs and containers like mp4, flac, mp3, jpeg. But has since then been extended to support a variety of formats like executables, packet captures (with TCP reassembly) and serialization formats like JSON, YAML, XML, ASN1 BER, Avro, CBOR, protobuf. In addition it also has functions to work with URL:s, convert to/from hex, number bases, search for things etc.
2021-12-29 01:13:23 +03:00
2022-05-23 22:54:24 +03:00
In summary it aims to be jq, hexdump, dd and gdb for files combined into one.
2021-11-22 16:16:59 +03:00
2022-07-23 23:50:33 +03:00
**NOTE:** fq is still early in development so things might change, be broken or do not make sense. That also means that there is a great opportunity to help out!
2022-02-16 20:45:18 +03:00
2022-01-15 21:26:45 +03:00
### Goals
2021-09-18 20:27:46 +03:00
2022-01-27 11:55:45 +03:00
- Make binaries accessible, queryable and sliceable.
2021-09-19 01:51:15 +03:00
- Nested formats and bit-oriented decoding.
2021-10-31 17:25:45 +03:00
- Quick and comfortable CLI tool.
2022-01-15 21:26:45 +03:00
- Bits and bytes transformations.
2021-09-18 20:27:46 +03:00
2022-01-15 21:26:45 +03:00
### Hopes
2022-01-27 11:55:45 +03:00
2021-12-29 01:13:23 +03:00
- Make it useful enough that people want to help improve it.
- Inspire people to create similar tools.
2022-01-22 13:49:31 +03:00
### Supported formats
2020-06-08 03:29:51 +03:00
2021-12-09 19:15:21 +03:00
[fq -rn -L doc 'include "formats"; formats_list']: sh-start
2020-06-08 03:29:51 +03:00
2021-12-09 19:15:21 +03:00
[aac_frame ](doc/formats.md#aac_frame ),
2022-01-12 20:35:28 +03:00
adts,
adts_frame,
2022-03-29 23:41:11 +03:00
amf0,
2022-01-12 20:35:28 +03:00
apev2,
2022-11-28 01:43:19 +03:00
[apple_bookmark ](doc/formats.md#apple_bookmark ),
2022-01-12 20:35:28 +03:00
ar,
2022-02-19 03:33:45 +03:00
[asn1_ber ](doc/formats.md#asn1_ber ),
2022-01-12 20:35:28 +03:00
av1_ccr,
av1_frame,
av1_obu,
avc_annexb,
2021-12-09 19:15:21 +03:00
[avc_au ](doc/formats.md#avc_au ),
2022-01-12 20:35:28 +03:00
avc_dcr,
avc_nalu,
avc_pps,
avc_sei,
avc_sps,
2022-10-29 20:23:50 +03:00
[avi ](doc/formats.md#avi ),
2022-02-19 03:33:45 +03:00
[avro_ocf ](doc/formats.md#avro_ocf ),
2021-12-09 19:15:21 +03:00
[bencode ](doc/formats.md#bencode ),
2022-04-17 11:33:01 +03:00
bitcoin_blkdat,
2022-09-10 19:28:54 +03:00
[bitcoin_block ](doc/formats.md#bitcoin_block ),
2022-04-17 11:33:01 +03:00
bitcoin_script,
bitcoin_transaction,
2022-11-20 21:22:16 +03:00
[bits ](doc/formats.md#bits ),
2022-09-24 20:57:11 +03:00
[bplist ](doc/formats.md#bplist ),
2022-01-12 20:35:28 +03:00
bsd_loopback_frame,
2022-02-19 03:33:45 +03:00
[bson ](doc/formats.md#bson ),
2022-11-20 21:22:16 +03:00
[bytes ](doc/formats.md#bytes ),
2022-01-12 20:35:28 +03:00
bzip2,
2022-02-19 03:33:45 +03:00
[cbor ](doc/formats.md#cbor ),
2022-06-01 17:55:55 +03:00
[csv ](doc/formats.md#csv ),
2022-01-12 20:35:28 +03:00
dns,
dns_tcp,
elf,
ether8023_frame,
exif,
2022-07-07 18:00:25 +03:00
fairplay_spc,
2022-01-12 20:35:28 +03:00
flac,
2021-12-09 19:15:21 +03:00
[flac_frame ](doc/formats.md#flac_frame ),
2022-01-12 20:35:28 +03:00
flac_metadatablock,
flac_metadatablocks,
flac_picture,
flac_streaminfo,
gif,
gzip,
hevc_annexb,
2021-12-09 19:15:21 +03:00
[hevc_au ](doc/formats.md#hevc_au ),
2022-01-12 20:35:28 +03:00
hevc_dcr,
hevc_nalu,
2022-03-21 19:21:58 +03:00
hevc_pps,
hevc_sps,
hevc_vps,
2022-06-01 17:55:55 +03:00
[html ](doc/formats.md#html ),
2022-01-12 20:35:28 +03:00
icc_profile,
icmp,
2022-04-01 17:31:55 +03:00
icmpv6,
2022-01-12 20:35:28 +03:00
id3v1,
id3v11,
id3v2,
ipv4_packet,
2022-04-01 17:31:55 +03:00
ipv6_packet,
2022-01-12 20:35:28 +03:00
jpeg,
json,
2022-08-15 13:57:02 +03:00
jsonl,
2022-02-28 12:28:21 +03:00
[macho ](doc/formats.md#macho ),
2022-08-01 19:42:24 +03:00
macho_fat,
2022-10-09 20:05:30 +03:00
[markdown ](doc/formats.md#markdown ),
2022-02-19 03:33:45 +03:00
[matroska ](doc/formats.md#matroska ),
2021-12-09 19:15:21 +03:00
[mp3 ](doc/formats.md#mp3 ),
2022-01-12 20:35:28 +03:00
mp3_frame,
2022-12-15 13:56:50 +03:00
mp3_frame_vbri,
mp3_frame_xing,
2022-02-19 03:33:45 +03:00
[mp4 ](doc/formats.md#mp4 ),
2022-01-12 20:35:28 +03:00
mpeg_asc,
mpeg_es,
mpeg_pes,
mpeg_pes_packet,
mpeg_spu,
mpeg_ts,
2022-02-19 03:33:45 +03:00
[msgpack ](doc/formats.md#msgpack ),
2022-01-12 20:35:28 +03:00
ogg,
ogg_page,
opus_packet,
2022-10-09 20:05:30 +03:00
[pcap ](doc/formats.md#pcap ),
2022-01-12 20:35:28 +03:00
pcapng,
png,
2022-08-24 20:08:41 +03:00
prores_frame,
2022-02-19 03:33:45 +03:00
[protobuf ](doc/formats.md#protobuf ),
2022-01-12 20:35:28 +03:00
protobuf_widevine,
pssh_playready,
2022-03-29 23:41:11 +03:00
[rtmp ](doc/formats.md#rtmp ),
2022-01-12 20:35:28 +03:00
sll2_packet,
sll_packet,
tar,
tcp_segment,
tiff,
2022-06-01 17:55:55 +03:00
toml,
2022-12-01 15:44:57 +03:00
[tzif ](doc/formats.md#tzif ),
2022-01-12 20:35:28 +03:00
udp_datagram,
vorbis_comment,
vorbis_packet,
vp8_frame,
vp9_cfm,
vp9_frame,
vpx_ccr,
2022-09-12 12:47:16 +03:00
[wasm ](doc/formats.md#wasm ),
2022-01-12 20:35:28 +03:00
wav,
webp,
2022-06-01 17:55:55 +03:00
[xml ](doc/formats.md#xml ),
yaml,
2022-05-26 13:39:14 +03:00
[zip ](doc/formats.md#zip )
2020-06-08 03:29:51 +03:00
[#]: sh-end
2022-07-23 23:50:33 +03:00
It can also work with some common text formats like URL:s, hex, base64, PEM etc and for some serialization formats like XML, YAML etc it can transform both from and to jq values.
interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth
Add tojson, same as in jq but also has {indent:number} options
Add toyaml/fromyaml for YAML
Add totoml/fromtoml for TOML
Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)
Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character
Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)
Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex
Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512
Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-09 18:50:28 +03:00
For details see [formats.md ](doc/formats.md ) and [usage.md ](doc/usage.md ).
2020-06-08 03:29:51 +03:00
2022-02-03 02:20:50 +03:00
## Presentations
2023-02-06 18:50:04 +03:00
- "fq - jq for binary formats" at [FOSDEM 2023 ](https://fosdem.org/2023/ ) - [video & slides ](https://fosdem.org/2023/schedule/event/bintools_fq/ )
2022-10-30 15:28:03 +03:00
- "fq - jq for binary formats" at [No time to wait 6 ](https://mediaarea.net/NoTimeToWait6 ) - [video ](https://www.youtube.com/watch?v=-Pwt5KL-xRs&t=1450s ) - [slides ](doc/presentations/nttw6/fq-nttw6-slides.pdf )
2022-05-23 22:54:24 +03:00
- "fq - jq for binary formats" at [Binary Tools Summit 2022 ](https://binary-tools.net/summit.html ) - [video ](https://www.youtube.com/watch?v=GJOq_b0eb-s&list=PLTj8twuHdQz-JcX7k6eOwyVPDB8CyfZc8&index=1 ) - [slides ](doc/presentations/bts2022/fq-bts2022-v1.pdf )
2022-02-03 02:20:50 +03:00
2022-01-22 13:49:31 +03:00
## Install
2022-10-30 15:28:03 +03:00
Use one of the methods listed below or download a pre-built [release ](https://github.com/wader/fq/releases ) for macOS, Linux or Windows. Unarchive it and move the executable to `PATH` etc.
2022-01-22 13:49:31 +03:00
2022-12-21 15:59:54 +03:00
On macOS if you don't install using one of the method below you might have to manually allow the binary to run. This can be done by trying to run the binary, ignore the warning and then go into security preference and allow it. Same can be done with this command:
2022-01-22 13:49:31 +03:00
```sh
xattr -d com.apple.quarantine fq & & spctl --add fq
```
2022-10-30 15:28:03 +03:00
### Homebrew (macOS)
2022-01-22 13:49:31 +03:00
```sh
brew install wader/tap/fq
```
2022-05-02 13:57:31 +03:00
### MacPorts
On macOS, `fq` can also be installed via [MacPorts ](https://www.macports.org ). More details [here ](https://ports.macports.org/port/fq/ ).
```sh
sudo port install fq
```
2022-02-10 13:57:41 +03:00
### Windows
`fq` can be installed via [scoop ](https://scoop.sh/ ).
```powershell
scoop install fq
```
2022-01-22 13:49:31 +03:00
### Arch Linux
`fq` can be installed from the [community repository ](https://archlinux.org/packages/community/x86_64/fq/ ) using [pacman ](https://wiki.archlinux.org/title/Pacman ):
```sh
pacman -S fq
```
You can also build and install the development (VCS) package using an [AUR helper ](https://wiki.archlinux.org/index.php/AUR_helpers ):
```sh
paru -S fq-git
```
### Nix
```sh
nix-shell -p fq
```
### FreeBSD
Use the [fq ](https://cgit.freebsd.org/ports/tree/misc/fq ) port.
### Alpine
Currently in edge testing but should work fine in stable also.
```
apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing fq
```
### Build from source
2022-10-30 15:28:03 +03:00
Make sure you have [go ](https://go.dev ) 1.18 or later installed.
2022-01-22 13:49:31 +03:00
2022-10-30 15:28:03 +03:00
To install directly from git repository (no clone needed) do:
2022-01-22 13:49:31 +03:00
```sh
# build and install latest release
go install github.com/wader/fq@latest
2022-10-30 15:28:03 +03:00
# build and install latest master
2022-01-22 13:49:31 +03:00
go install github.com/wader/fq@master
# copy binary to $PATH if needed
cp "$(go env GOPATH)/bin/fq" /usr/local/bin
```
2022-10-30 15:28:03 +03:00
To build, run and test from source:
2022-01-22 13:49:31 +03:00
```sh
2022-12-21 15:59:54 +03:00
# build and run
2022-10-30 15:28:03 +03:00
go run .
2023-01-03 13:11:12 +03:00
# build and run with arguments
go run . -d mp3 . file.mp3
2022-10-30 15:28:03 +03:00
# just build
go build -o fq .
2022-01-22 13:49:31 +03:00
# run all tests and build binary
make test fq
```
2020-06-08 03:29:51 +03:00
## TODO and ideas
2021-11-17 12:20:51 +03:00
See [TODO.md ](doc/TODO.md )
2020-06-08 03:29:51 +03:00
2022-02-16 20:59:46 +03:00
## Development and adding a new decoder
2020-06-08 03:29:51 +03:00
2021-11-17 12:20:51 +03:00
See [dev.md ](doc/dev.md )
2020-06-08 03:29:51 +03:00
## Thanks and related projects
2021-11-19 02:23:04 +03:00
This project would not have been possible without [itchyny ](https://github.com/itchyny )'s
2021-11-29 15:34:45 +03:00
jq implementation [gojq ](https://github.com/itchyny/gojq ). I also want to thank
2020-06-08 03:29:51 +03:00
[HexFiend ](https://github.com/HexFiend/HexFiend ) for inspiration and ideas and [stedolan ](https://github.com/stedolan )
for inventing the [jq ](https://github.com/stedolan/jq ) language.
2022-12-19 13:40:13 +03:00
### Similar or related works
#### Tools
2022-01-12 20:28:02 +03:00
2022-06-21 19:10:23 +03:00
- [HexFiend ](https://github.com/HexFiend/HexFiend ) Hex editor for macOS with format template support.
- [binspector ](https://github.com/binspector/binspector ) Binary format analysis tool with query langauge and REPL.
- [kaitai ](https://kaitai.io ) Declarative binary format parsing.
- [Wireshark ](https://www.wireshark.org ) Decodes network traffic (tip: `tshark -T json` ).
- [MediaInfo ](https://mediaarea.net/en/MediaInfo ) Analyze media files (tip `mediainfo --Output=JSON` and `mediainfo --Details=1` ).
- [GNU poke ](https://www.jemarch.net/poke ) The extensible editor for structured binary data.
- [ffmpeg/ffprobe ](https://ffmpeg.org ) Powerful media libraries and tools.
- [hexdump ](https://git.kernel.org/pub/scm/utils/util-linux/util-linux.git/tree/text-utils/hexdump.c ) Hex viewer tool.
- [hex ](https://git.janouch.name/p/hex ) Interactive hex viewer with format support via lua.
- [hachoir ](https://github.com/vstinner/hachoir ) General python library for working binary data.
- [scapy ](https://scapy.net ) Decode/Encode formats, focus on network protocols.
2022-02-10 14:38:42 +03:00
2022-12-19 13:40:13 +03:00
#### Projects and Standards
- [Let's Solve the File Format Problem ](http://fileformats.archiveteam.org )
- [PRONOM ](https://www.nationalarchives.gov.uk/PRONOM/ ) file format registry
- [Sustainability of Digital Formats ](https://www.loc.gov/preservation/digital/formats/ ) at Library of Congress
- [Data Format Description Language ](https://en.wikipedia.org/wiki/Data_Format_Description_Language )
2022-02-10 14:38:42 +03:00
## License
`fq` is distributed under the terms of the MIT License.
See the [LICENSE ](LICENSE ) file for license details.
Licenses of direct dependencies:
- Forked version of gojq https://github.com/itchyny/gojq/blob/main/LICENSE (MIT)
- Forked version of readline https://github.com/chzyer/readline/blob/master/LICENSE (MIT)
interp: Add to/from<encoding> for some common serialzations, encodings and hashes
Add toxml/fromxml for XML encoding, mighe be lossy on ordering
fromxml has {seq:bool} to add #seq attributes to improve ordering
toxml has {indent:number} to choose space indent depth
Add tojson, same as in jq but also has {indent:number} options
Add toyaml/fromyaml for YAML
Add totoml/fromtoml for TOML
Add tojq/fromjq for jq-flavored JSON (optional quotes for keys, comments and trailing commas support)
Add tocsv/fromcsv for CSV
formcvs takes {comma:string, comment:string} for custom separtor and comment character
Rename/split hex into tohex/fromhex
Rename/split base64 into tobase64/frombase64
tobase64/frombase64 takes {encoding:string} option for base64 flavour (std, url, rawstd, rawurl)
Add to/from<format> urlpath, urlquery, url, xmlentities, base64, hex
Add to<hash> md4, md5, sha1, sha256, sha512, sha3_224, sha3_256, sha3_384, sha3_512
Add to/from<encoding> iso8859-1, utf8, utf16, utf16le, utf16be
2022-05-09 18:50:28 +03:00
- github.com/BurntSushi/toml https://github.com/BurntSushi/toml/blob/master/COPYING (MIT)
2022-07-29 17:34:27 +03:00
- github.com/creasty/defaults https://github.com/creasty/defaults/blob/master/LICENSE (MIT)
2022-10-14 13:13:32 +03:00
- github.com/gomarkdown/markdown https://github.com/gomarkdown/markdown/blob/master/LICENSE.txt (BSD)
- github.com/gopacket/gopacket https://github.com/gopacket/gopacket/blob/master/LICENSE (BSD)
- github.com/mitchellh/copystructure https://github.com/mitchellh/copystructure/blob/master/LICENSE (MIT)
- github.com/mitchellh/mapstructure https://github.com/mitchellh/mapstructure/blob/master/LICENSE (MIT)
- github.com/pmezard/go-difflib https://github.com/pmezard/go-difflib/blob/master/LICENSE (BSD)
- golang/snappy https://github.com/golang/snappy/blob/master/LICENSE (BSD)
- golang/x/* https://github.com/golang/text/blob/master/LICENSE (BSD)
- gopkg.in/yaml.v3 https://github.com/go-yaml/yaml/blob/v3/LICENSE (MIT)