|
||
---|---|---|
.github | ||
.vscode | ||
dev | ||
doc | ||
format | ||
internal | ||
pkg | ||
.gitattributes | ||
.gitignore | ||
.golangci.yml | ||
.goreleaser.yml | ||
Bumpfile | ||
CHANGES.md | ||
Dockerfile | ||
fq.go | ||
go.mod | ||
go.sum | ||
LICENSE | ||
Makefile | ||
README.md |
fq
Tool, language and decoders for working with binary data.
Basic usage is fq . file
or fq d file
.
For details see usage.md.
Background
fq is inspired by the well known jq tool and language that allows you to work with binary formats the same way you would using jq. In addition it can present data like a hex viewer, transform, slice and concatenate binary data. It also supports nested formats and has an interactive REPL with auto-completion.
It was originally designed to query, inspect and debug media codecs and containers like mp4, flac, mp3, jpeg. But since then it has been extended to support a variety of formats like executables, packet captures (with TCP reassembly) and serialization formats like JSON, YAML, XML, ASN1 BER, Avro, CBOR, protobuf. In addition it also has functions to work with URLs, convert to/from hex, number bases, search for things etc.
In summary it aims to be jq, hexdump, dd and gdb for files combined into one.
NOTE: fq is still early in development so things might change, be broken or do not make sense. That also means that there is a great opportunity to help out!
Goals
- Make binaries accessible, queryable and sliceable.
- Nested formats and bit-oriented decoding.
- Quick and comfortable CLI tool.
- Bits and bytes transformations.
Hopes
- Make it useful enough that people want to help improve it.
- Inspire people to create similar tools.
Supported formats
aac_frame, adts, adts_frame, aiff, amf0, apev2, apple_bookmark, ar, asn1_ber, av1_ccr, av1_frame, av1_obu, avc_annexb, avc_au, avc_dcr, avc_nalu, avc_pps, avc_sei, avc_sps, avi, avro_ocf, bencode, bitcoin_blkdat, bitcoin_block, bitcoin_script, bitcoin_transaction, bits, bplist, bsd_loopback_frame, bson, bytes, bzip2, caff, cbor, csv, dns, dns_tcp, elf, ether8023_frame, exif, fairplay_spc, fit, flac, flac_frame, flac_metadatablock, flac_metadatablocks, flac_picture, flac_streaminfo, gif, gzip, hevc_annexb, hevc_au, hevc_dcr, hevc_nalu, hevc_pps, hevc_sps, hevc_vps, html, icc_profile, icmp, icmpv6, id3v1, id3v11, id3v2, ipv4_packet, ipv6_packet, jp2c, jpeg, json, jsonl, leveldb_descriptor, leveldb_log, leveldb_table, luajit, macho, macho_fat, markdown, matroska, moc3, mp3, mp3_frame, mp3_frame_vbri, mp3_frame_xing, mp4, mpeg_asc, mpeg_es, mpeg_pes, mpeg_pes_packet, mpeg_spu, mpeg_ts, msgpack, nes, ogg, ogg_page, opentimestamps, opus_packet, pcap, pcapng, pg_btree, pg_control, pg_heap, png, prores_frame, protobuf, protobuf_widevine, pssh_playready, rtmp, sll2_packet, sll_packet, tar, tcp_segment, tiff, tls, toml, tzif, udp_datagram, vorbis_comment, vorbis_packet, vp8_frame, vp9_cfm, vp9_frame, vpx_ccr, wasm, wav, webp, xml, yaml, zip
It can also work with some common text formats like URLs, hex, base64, PEM etc and for some serialization formats like XML, YAML, etc. it can transform both from and to jq values.
For details see formats.md and usage.md.
Presentations
- "fq - jq for binary formats" at FOSDEM 2023 - video & slides
- "fq - jq for binary formats" at No time to wait 6 - video - slides
- "fq - jq for binary formats" at Binary Tools Summit 2022 - video - slides
Install
Use one of the methods listed below or download a pre-built release for macOS, Linux or Windows. Unarchive it and move the executable to PATH
etc.
On macOS if you don't install using one of the method below then you might have to manually allow the binary to run. This can be done by trying to run the binary, ignore the warning and then go into security preference and allow it. Same can be done with this command:
xattr -d com.apple.quarantine fq && spctl --add fq
Homebrew (macOS)
brew install wader/tap/fq
MacPorts
On macOS, fq
can also be installed via MacPorts. More details here.
sudo port install fq
Windows
fq
can be installed via scoop.
scoop install fq
Arch Linux
fq
can be installed from the extra repository using pacman:
pacman -S fq
You can also build and install the development (VCS) package using an AUR helper:
paru -S fq-git
Nix
nix-shell -p fq
FreeBSD
Use the fq port.
Alpine
Currently in edge testing but should work fine in stable also.
apk add -X http://dl-cdn.alpinelinux.org/alpine/edge/testing fq
Build from source
Make sure you have go 1.20 or later installed.
To install directly from git repository (no clone needed):
# build and install latest release
go install github.com/wader/fq@latest
# build and install latest master
go install github.com/wader/fq@master
# copy binary to $PATH if needed
cp "$(go env GOPATH)/bin/fq" /usr/local/bin
To build, run and test from source:
# build and run
go run .
# build and run with arguments
go run . -d mp3 . file.mp3
# just build
go build -o fq .
# run all tests and build binary
make test fq
TODO and ideas
See TODO.md
Development and adding a new decoder
See dev.md
Thanks and related projects
This project would not have been possible without itchyny's jq implementation gojq. I also want to thank HexFiend for inspiration and ideas and stedolan for inventing the jq language.
Similar or related works
Tools
- HexFiend - Hex editor for macOS with format template support.
- ImHex - A Hex Editor for Reverse Engineers.
- binspector - Binary format analysis tool with query language and REPL.
- kaitai - Declarative binary format parsing.
- Wireshark - Decodes network traffic (tip:
tshark -T json
). - MediaInfo - Analyze media files (tip
mediainfo --Output=JSON
andmediainfo --Details=1
). - GNU poke - The extensible editor for structured binary data.
- ffmpeg/ffprobe - Powerful media libraries and tools.
- hexdump - Hex viewer tool.
- hex - Interactive hex viewer with format support via lua.
- hachoir - General python library for working binary data.
- scapy - Decode/Encode formats, focus on network protocols.
Projects and Standards
- Let's Solve the File Format Problem.
- PRONOM file format registry.
- Sustainability of Digital Formats at Library of Congress.
- Data Format Description Language (DFDL).
License
fq
is distributed under the terms of the MIT License.
See the LICENSE file for license details.
Licenses of direct dependencies:
- Forked version of gojq - https://github.com/itchyny/gojq/blob/main/LICENSE (MIT)
- github.com/ergochat/readline - https://github.com/ergochat/readline/blob/master/LICENSE (MIT)
- github.com/BurntSushi/toml - https://github.com/BurntSushi/toml/blob/master/COPYING (MIT)
- github.com/creasty/defaults - https://github.com/creasty/defaults/blob/master/LICENSE (MIT)
- github.com/gomarkdown/markdown - https://github.com/gomarkdown/markdown/blob/master/LICENSE.txt (BSD)
- github.com/gopacket/gopacket - https://github.com/gopacket/gopacket/blob/master/LICENSE (BSD)
- github.com/mitchellh/copystructure - https://github.com/mitchellh/copystructure/blob/master/LICENSE (MIT)
- github.com/mitchellh/mapstructure - https://github.com/mitchellh/mapstructure/blob/master/LICENSE (MIT)
- github.com/pmezard/go-difflib - https://github.com/pmezard/go-difflib/blob/master/LICENSE (BSD)
- golang/snappy - https://github.com/golang/snappy/blob/master/LICENSE (BSD)
- golang/x/* - https://github.com/golang/text/blob/master/LICENSE (BSD)
- gopkg.in/yaml.v3 - https://github.com/go-yaml/yaml/blob/v3/LICENSE (MIT)
- Parts of go crypto/tls and github.com/zmap/zcrypto - https://github.com/zmap/zcrypto/blob/master/LICENSE (Apache)