fq/doc/formats.md

Supported formats

Name	Description	Dependencies
aac_frame	Advanced Audio Coding frame
adts	Audio Data Transport Stream	adts_frame
adts_frame	Audio Data Transport Stream frame	aac_frame
amf0	Action Message Format 0
apev2	APEv2 metadata tag	image
ar	Unix archive	probe
asn1_ber	ASN1 Basic Encoding Rules (also CER and DER)
av1_ccr	AV1 Codec Configuration Record
av1_frame	AV1 frame	av1_obu
av1_obu	AV1 Open Bitstream Unit
avc_annexb	H.264/AVC Annex B	avc_nalu
avc_au	H.264/AVC Access Unit	avc_nalu
avc_dcr	H.264/AVC Decoder Configuration Record	avc_nalu
avc_nalu	H.264/AVC Network Access Layer Unit	avc_sps avc_pps avc_sei
avc_pps	H.264/AVC Picture Parameter Set
avc_sei	H.264/AVC Supplemental Enhancement Information
avc_sps	H.264/AVC Sequence Parameter Set
avro_ocf	Avro object container file
bencode	BitTorrent bencoding
bsd_loopback_frame	BSD loopback frame	inet_packet
bson	Binary JSON
bzip2	bzip2 compression	probe
cbor	Concise Binary Object Representation
dns	DNS packet
dns_tcp	DNS packet (TCP)
elf	Executable and Linkable Format
ether8023_frame	Ethernet 802.3 frame	inet_packet
exif	Exchangeable Image File Format
flac	Free Lossless Audio Codec file	flac_metadatablocks flac_frame
flac_frame	FLAC frame
flac_metadatablock	FLAC metadatablock	flac_streaminfo flac_picture vorbis_comment
flac_metadatablocks	FLAC metadatablocks	flac_metadatablock
flac_picture	FLAC metadatablock picture	image
flac_streaminfo	FLAC streaminfo
gif	Graphics Interchange Format
gzip	gzip compression	probe
hevc_annexb	H.265/HEVC Annex B	hevc_nalu
hevc_au	H.265/HEVC Access Unit	hevc_nalu
hevc_dcr	H.265/HEVC Decoder Configuration Record	hevc_nalu
hevc_nalu	H.265/HEVC Network Access Layer Unit	hevc_vps hevc_pps hevc_sps
hevc_pps	H.265/HEVC Picture Parameter Set
hevc_sps	H.265/HEVC Sequence Parameter Set
hevc_vps	H.265/HEVC Video Parameter Set
icc_profile	International Color Consortium profile
icmp	Internet Control Message Protocol
icmpv6	Internet Control Message Protocol v6
id3v1	ID3v1 metadata
id3v11	ID3v1.1 metadata
id3v2	ID3v2 metadata	image
ipv4_packet	Internet protocol v4 packet	ip_packet
ipv6_packet	Internet protocol v6 packet	ip_packet
jpeg	Joint Photographic Experts Group file	exif icc_profile
json	JSON
macho	Mach-O macOS executable
matroska	Matroska file	aac_frame av1_ccr av1_frame avc_au avc_dcr flac_frame flac_metadatablocks hevc_au hevc_dcr image mp3_frame mpeg_asc mpeg_pes_packet mpeg_spu opus_packet vorbis_packet vp8_frame vp9_cfm vp9_frame
mp3	MP3 file	id3v2 id3v1 id3v11 apev2 mp3_frame
mp3_frame	MPEG audio layer 3 frame	xing
mp4	MPEG-4 file and similar	aac_frame av1_ccr av1_frame flac_frame flac_metadatablocks id3v2 image jpeg mp3_frame avc_au avc_dcr mpeg_es hevc_au hevc_dcr mpeg_pes_packet opus_packet protobuf_widevine pssh_playready vorbis_packet vp9_frame vpx_ccr icc_profile
mpeg_asc	MPEG-4 Audio Specific Config
mpeg_es	MPEG Elementary Stream	mpeg_asc vorbis_packet
mpeg_pes	MPEG Packetized elementary stream	mpeg_pes_packet mpeg_spu
mpeg_pes_packet	MPEG Packetized elementary stream packet
mpeg_spu	Sub Picture Unit (DVD subtitle)
mpeg_ts	MPEG Transport Stream
msgpack	MessagePack
ogg	OGG file	ogg_page vorbis_packet opus_packet flac_metadatablock flac_frame
ogg_page	OGG page
opus_packet	Opus packet	vorbis_comment
pcap	PCAP packet capture	link_frame tcp_stream ipv4_packet
pcapng	PCAPNG packet capture	link_frame tcp_stream ipv4_packet
png	Portable Network Graphics file	icc_profile exif
protobuf	Protobuf
protobuf_widevine	Widevine protobuf	protobuf
pssh_playready	PlayReady PSSH
raw	Raw bits
rtmp	Real-Time Messaging Protocol	amf0 mpeg_asc
sll2_packet	Linux cooked capture encapsulation v2	inet_packet
sll_packet	Linux cooked capture encapsulation	inet_packet
tar	Tar archive	probe
tcp_segment	Transmission control protocol segment
tiff	Tag Image File Format	icc_profile
udp_datagram	User datagram protocol	udp_payload
vorbis_comment	Vorbis comment	flac_picture
vorbis_packet	Vorbis packet	vorbis_comment
vp8_frame	VP8 frame
vp9_cfm	VP9 Codec Feature Metadata
vp9_frame	VP9 frame
vpx_ccr	VPX Codec Configuration Record
wav	WAV file	id3v2 id3v1 id3v11
webp	WebP image	vp8_frame
xing	Xing header
zip	ZIP archive	probe
image	Group	gif jpeg mp4 png tiff webp
inet_packet	Group	ipv4_packet ipv6_packet
ip_packet	Group	icmp icmpv6 tcp_segment udp_datagram
link_frame	Group	bsd_loopback_frame ether8023_frame sll2_packet sll_packet
probe	Group	adts ar avro_ocf bzip2 elf flac gif gzip jpeg json macho matroska mp3 mp4 mpeg_ts ogg pcap pcapng png tar tiff wav webp zip
tcp_stream	Group	dns rtmp
udp_payload	Group	dns

Format options

Currently the only option is force and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI -o option:

fq -d mp4 -o force=true file.mp4
fq -d raw 'mp4({force: true})' file.mp4

Format details

asn1_ber

Supports decoding BER, CER and DER (X.690).

• Currently no extra validation is done for CER and DER.
• Does not support specifying a schema.
• Supports torepr but without schema all sequences and sets will be arrays.

fq -d asn1_ber torepr file.ber

Functions frompem and topem can help working with PEM format:

fq -d raw 'frompem | asn1_ber | d' cert.pem

If the schema is known and not that complicated it can be reproduced:

fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der

Can be used to decode nested parts:

fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber

References and tools:

• https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf
• https://en.wikipedia.org/wiki/X.690
• https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
• https://lapo.it/asn1js/

avro_ocf

Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.

Capable of handling null, deflate, and snappy codecs for data compression.

Limitations:

• Schema does not support self-referential types, only built-in types.
• Decimal logical types are not supported for decoding, will just be treated as their primitive type

becode

Supports torepr:

fq -d bencode torepr file.torrent

bson

Supports torepr:

fq -d bson torepr file.bson

cbor

Supports torepr:

fq -d cbor torepr file.cbor
fq -d cbor 'torepr.field' file.cbor
fq -d cbor 'torepr | .field' file.cbor
fq -d cbor 'torepr | grep("abc")' file.cbor

macho

Supports decoding vanilla and FAT Mach-O binaries.

Examples

To decode the macOS build of fq:

fq . /path/to/fq

fq '.load_commands[] | select(.cmd=="segment_64")' /path/to/fq

Note you can use -d macho to decode a broken Mach-O binary.

References:

• https://github.com/aidansteele/osx-abi-macho-file-format-reference

matroska

Supports matroska_path:

$ fq 'matroska_path(".Segment.Tracks[0]")' file.mkv
     │00 01 02 03 04 05 06 07 08 09│0123456789│.elements[1].elements[3]{}:
0x122│         16 54 ae 6b         │   .T.k   │  id: "Tracks" (0x1654ae6b) (A Top-Level Element of information with many tracks described.)
     │                             │          │  type: "master" (7)
0x122│                     4d bf   │       M. │  size: 3519
0x122│                           bf│         .│  elements[0:3]:
0x12c│84 cf 8b db a0 ae 01 00 00 00│..........│
0x136│00 00 00 78 d7 81 01 73 c5 88│...x...s..│
*    │until 0xee9.7 (3519)         │          │

$ fq 'first(grep_by(.id == "Tracks")) | matroska_path' test.mkv
".Segment.Tracks"

mp4

Supports mp4_path:

$ fq 'mp4_path(".moov.trak[1]")' file.mp4
     │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[1]{}:
0x4f6│                     00 00 02│       ...│  size: 573
0x500│3d                           │=         │
0x500│   74 72 61 6b               │ trak     │  type: "trak" (Container for an individual track or stream)
0x500│               00 00 00 5c 74│     ...\t│  boxes[0:3]:
0x50a│6b 68 64 00 00 00 03 00 00 00│khd.......│
0x514│00 00 00 00 00 00 00 00 01 00│..........│
*    │until 0x739.7 (565)          │          │

$ fq 'first(grep_by(.type == "trak")) | mp4_path' file.mp4
".moov.trak"

msgpack

Supports torepr:

fq -d msgpack torepr file.msgpack

protobuf

protobuf decoder can be used to decode sub messages:

fq -d protobuf '.fields[6].wire_value | protobuf | d'

rtmp

Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).

Dependency graph