fq/doc/formats.md

Supported formats

Name	Description	Dependencies
aac_frame	Advanced Audio Coding frame
adts	Audio Data Transport Stream	adts_frame
adts_frame	Audio Data Transport Stream frame	aac_frame
amf0	Action Message Format 0
apev2	APEv2 metadata tag	image
ar	Unix archive	probe
asn1_ber	ASN1 BER (basic encoding rules, also CER and DER)
av1_ccr	AV1 Codec Configuration Record
av1_frame	AV1 frame	av1_obu
av1_obu	AV1 Open Bitstream Unit
avc_annexb	H.264/AVC Annex B	avc_nalu
avc_au	H.264/AVC Access Unit	avc_nalu
avc_dcr	H.264/AVC Decoder Configuration Record	avc_nalu
avc_nalu	H.264/AVC Network Access Layer Unit	avc_sps avc_pps avc_sei
avc_pps	H.264/AVC Picture Parameter Set
avc_sei	H.264/AVC Supplemental Enhancement Information
avc_sps	H.264/AVC Sequence Parameter Set
avro_ocf	Avro object container file
bencode	BitTorrent bencoding
bitcoin_blkdat	Bitcoin blk.dat	bitcoin_block
bitcoin_block	Bitcoin block	bitcoin_transaction
bitcoin_script	Bitcoin script
bitcoin_transaction	Bitcoin transaction	bitcoin_script
bsd_loopback_frame	BSD loopback frame	inet_packet
bson	Binary JSON
bzip2	bzip2 compression	probe
cbor	Concise Binary Object Representation
dns	DNS packet
dns_tcp	DNS packet (TCP)
elf	Executable and Linkable Format
ether8023_frame	Ethernet 802.3 frame	inet_packet
exif	Exchangeable Image File Format
fairplay_spc	FairPlay Server Playback Context
flac	Free Lossless Audio Codec file	flac_metadatablocks flac_frame
flac_frame	FLAC frame
flac_metadatablock	FLAC metadatablock	flac_streaminfo flac_picture vorbis_comment
flac_metadatablocks	FLAC metadatablocks	flac_metadatablock
flac_picture	FLAC metadatablock picture	image
flac_streaminfo	FLAC streaminfo
gif	Graphics Interchange Format
gzip	gzip compression	probe
hevc_annexb	H.265/HEVC Annex B	hevc_nalu
hevc_au	H.265/HEVC Access Unit	hevc_nalu
hevc_dcr	H.265/HEVC Decoder Configuration Record	hevc_nalu
hevc_nalu	H.265/HEVC Network Access Layer Unit	hevc_vps hevc_pps hevc_sps
hevc_pps	H.265/HEVC Picture Parameter Set
hevc_sps	H.265/HEVC Sequence Parameter Set
hevc_vps	H.265/HEVC Video Parameter Set
icc_profile	International Color Consortium profile
icmp	Internet Control Message Protocol
icmpv6	Internet Control Message Protocol v6
id3v1	ID3v1 metadata
id3v11	ID3v1.1 metadata
id3v2	ID3v2 metadata	image
ipv4_packet	Internet protocol v4 packet	ip_packet
ipv6_packet	Internet protocol v6 packet	ip_packet
jpeg	Joint Photographic Experts Group file	exif icc_profile
json	JSON
macho	Mach-O macOS executable
matroska	Matroska file	aac_frame av1_ccr av1_frame avc_au avc_dcr flac_frame flac_metadatablocks hevc_au hevc_dcr image mp3_frame mpeg_asc mpeg_pes_packet mpeg_spu opus_packet vorbis_packet vp8_frame vp9_cfm vp9_frame
mp3	MP3 file	id3v2 id3v1 id3v11 apev2 mp3_frame
mp3_frame	MPEG audio layer 3 frame	xing
mp4	ISOBMFF MPEG-4 part 12 and similar	aac_frame av1_ccr av1_frame flac_frame flac_metadatablocks id3v2 image jpeg mp3_frame avc_au avc_dcr mpeg_es hevc_au hevc_dcr mpeg_pes_packet opus_packet protobuf_widevine pssh_playready vorbis_packet vp9_frame vpx_ccr icc_profile
mpeg_asc	MPEG-4 Audio Specific Config
mpeg_es	MPEG Elementary Stream	mpeg_asc vorbis_packet
mpeg_pes	MPEG Packetized elementary stream	mpeg_pes_packet mpeg_spu
mpeg_pes_packet	MPEG Packetized elementary stream packet
mpeg_spu	Sub Picture Unit (DVD subtitle)
mpeg_ts	MPEG Transport Stream
msgpack	MessagePack
ogg	OGG file	ogg_page vorbis_packet opus_packet flac_metadatablock flac_frame
ogg_page	OGG page
opus_packet	Opus packet	vorbis_comment
pcap	PCAP packet capture	link_frame tcp_stream ipv4_packet
pcapng	PCAPNG packet capture	link_frame tcp_stream ipv4_packet
png	Portable Network Graphics file	icc_profile exif
protobuf	Protobuf
protobuf_widevine	Widevine protobuf	protobuf
pssh_playready	PlayReady PSSH
raw	Raw bits
rtmp	Real-Time Messaging Protocol	amf0 mpeg_asc
sll2_packet	Linux cooked capture encapsulation v2	inet_packet
sll_packet	Linux cooked capture encapsulation	inet_packet
tar	Tar archive	probe
tcp_segment	Transmission control protocol segment
tiff	Tag Image File Format	icc_profile
udp_datagram	User datagram protocol	udp_payload
vorbis_comment	Vorbis comment	flac_picture
vorbis_packet	Vorbis packet	vorbis_comment
vp8_frame	VP8 frame
vp9_cfm	VP9 Codec Feature Metadata
vp9_frame	VP9 frame
vpx_ccr	VPX Codec Configuration Record
wav	WAV file	id3v2 id3v1 id3v11
webp	WebP image	vp8_frame
xing	Xing header
zip	ZIP archive	probe
image	Group	gif jpeg mp4 png tiff webp
inet_packet	Group	ipv4_packet ipv6_packet
ip_packet	Group	icmp icmpv6 tcp_segment udp_datagram
link_frame	Group	bsd_loopback_frame ether8023_frame sll2_packet sll_packet
probe	Group	adts ar avro_ocf bitcoin_blkdat bzip2 elf flac gif gzip jpeg json macho matroska mp3 mp4 mpeg_ts ogg pcap pcapng png tar tiff wav webp zip
tcp_stream	Group	dns rtmp
udp_payload	Group	dns

Global format options

Currently the only global option is force and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI -o option:

fq -d mp4 -o force=true file.mp4
fq -d raw 'mp4({force: true})' file.mp4

Format details

aac_frame

Options

Name	Default	Description
object_type	1	Audio object type

Examples

Decode file using aac_frame options

$ fq -d aac_frame -o object_type=1 . file

Decode value as aac_frame

... | aac_frame({object_type:1})

asn1_ber

Supports decoding BER, CER and DER (X.690).

• Currently no extra validation is done for CER and DER.
• Does not support specifying a schema.
• Supports torepr but without schema all sequences and sets will be arrays.

Examples

frompem and topem can be used to work with PEM format

$ fq -d raw 'frompem | asn1_ber | d' cert.pem

Can be used to decode nested parts

$ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber

If schema is known and not complicated it can be reproduced

$ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der

Supports torepr

$ fq -d asn1_ber torepr file

Supports torepr

... | asn1_ber | torepr

References and links

• https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf
• https://en.wikipedia.org/wiki/X.690
• https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
• https://lapo.it/asn1js/

avc_au

Options

Name	Default	Description
length_size	4	Length value size

Examples

Decode file using avc_au options

$ fq -d avc_au -o length_size=4 . file

Decode value as avc_au

... | avc_au({length_size:4})

avro_ocf

Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.

Capable of handling null, deflate, and snappy codecs for data compression.

Limitations:

• Schema does not support self-referential types, only built-in types.
• Decimal logical types are not supported for decoding, will just be treated as their primitive type

References and links

• https://avro.apache.org/docs/current/spec.html#Object+Container+Files

bencode

Examples

Supports torepr

$ fq -d bencode torepr file

Supports torepr

... | bencode | torepr

References and links

• https://wiki.theory.org/BitTorrentSpecification#Bencoding

bson

Examples

BSON as JSON

$ fq -d bson torepr file

Supports torepr

$ fq -d bson torepr file

Supports torepr

... | bson | torepr

References and links

• https://wiki.theory.org/BitTorrentSpecification#Bencoding

cbor

Examples

Supports torepr

$ fq -d cbor torepr file

Supports torepr

... | cbor | torepr

References and links

• https://en.wikipedia.org/wiki/CBOR
• https://www.rfc-editor.org/rfc/rfc8949.html

flac_frame

Options

Name	Default	Description
bits_per_sample	16	Bits per sample

Examples

Decode file using flac_frame options

$ fq -d flac_frame -o bits_per_sample=16 . file

Decode value as flac_frame

... | flac_frame({bits_per_sample:16})

hevc_au

Options

Name	Default	Description
length_size	4	Length value size

Examples

Decode file using hevc_au options

$ fq -d hevc_au -o length_size=4 . file

Decode value as hevc_au

... | hevc_au({length_size:4})

macho

Supports decoding vanilla and FAT Mach-O binaries.

Examples

Select 64bit load segments

$ fq '.load_commands[] | select(.cmd=="segment_64")' file

References and links

• https://github.com/aidansteele/osx-abi-macho-file-format-reference

matroska

Examples

Lookup element decode value using matroska_path

... | matroska_path(".Segment.Tracks[0)"

Return matroska_path string for a box decode value

... | grep_by(.id == "Tracks") | matroska_path

References and links

• https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
• https://matroska.org/technical/specs/index.html
• https://www.matroska.org/technical/basics.html
• https://www.matroska.org/technical/codec_specs.html
• https://wiki.xiph.org/MatroskaOpus

mp3

Options

Name	Default	Description
max_sync_seek	32768	Max byte distance to next sync
max_unique_header_configs	5	Max number of unique frame header configs allowed

Examples

Decode file using mp3 options

$ fq -d mp3 -o max_sync_seek=32768 -o max_unique_header_configs=5 . file

Decode value as mp3

... | mp3({max_sync_seek:32768,max_unique_header_configs:5})

mp4

Support mp4_path

Options

Name	Default	Description
allow_truncated	false	Allow box to be truncated
decode_samples	true	Decode supported media samples

Examples

Lookup box decode value using mp4_path

... | mp4_path(".moov.trak[1]")

Return mp4_path string for a box decode value

... | grep_by(.type == "trak") | mp4_path

Decode file using mp4 options

$ fq -d mp4 -o allow_truncated=false -o decode_samples=true . file

Decode value as mp4

... | mp4({allow_truncated:false,decode_samples:true})

References and links

• ISO/IEC base media file format (MPEG-4 Part 12)
• Quicktime file format

msgpack

Examples

Supports torepr

$ fq -d msgpack torepr file

Supports torepr

... | msgpack | torepr

References and links

• https://github.com/msgpack/msgpack/blob/master/spec.md

protobuf

Examples

Can be used to decode sub messages

$ fq -d protobuf '.fields[6].wire_value | protobuf | d'

References and links

• https://developers.google.com/protocol-buffers/docs/encoding

rtmp

Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).

References and links

• https://rtmp.veriskope.com/docs/spec/
• https://rtmp.veriskope.com/pdf/video_file_format_spec_v10.pdf

zip

Supports ZIP64.

Options

Name	Default	Description
uncompress	true	Uncompress and probe files

Examples

Decode file using zip options

$ fq -d zip -o uncompress=true . file

Decode value as zip

... | zip({uncompress:true})

References and links

• https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT

Dependency graph