fq/formats.md at 8c9de7054630786066629d5cb9486e3e3dea2380

wader/fq

mirror of https://github.com/wader/fq.git synced 2024-12-29 16:42:06 +03:00

Mattias Wadman f4480c6fe5 decode,interp: Support for format specific options

interp: Refactor format help and also include options
interp: Add -o name=@path to load file content as value (not documented yet, might change)
interp,decode: Expose decode out value as _out (might change)
interp: Refactor foramts.jq into format_{decode,func,include}.jq
interp: Refactor torepr into _format_func for generic format function overloading
interp: Refactor -o options parsing to be more generic and collect unknowns options to be used as format options
decode of decode alises
func for format overloaded functions
include for format specific jq functions (also _help, torepr etc)
flac_frame: Add bits_per_sample option
mp3: Add max_unique_header_config and max_sync_seek options
mp4: Add decode_samples and allow_truncate options
avc_au: Has length_size option
hevc_au: Has length_size option
aac_frame: Has object_typee option
doc: Rewrite format doc generation, less hack more jq

2022-05-01 17:08:30 +02:00

21 KiB

Raw Blame History

Supported formats

Name	Description	Dependencies
`aac_frame`	Advanced Audio Coding frame
`adts`	Audio Data Transport Stream	_{adts_frame}
`adts_frame`	Audio Data Transport Stream frame	_{aac_frame}
`amf0`	Action Message Format 0
`apev2`	APEv2 metadata tag	_image
`ar`	Unix archive	_probe
`asn1_ber`	ASN1 BER (basic encoding rules, also CER and DER)
`av1_ccr`	AV1 Codec Configuration Record
`av1_frame`	AV1 frame	_{av1_obu}
`av1_obu`	AV1 Open Bitstream Unit
`avc_annexb`	H.264/AVC Annex B	_{avc_nalu}
`avc_au`	H.264/AVC Access Unit	_{avc_nalu}
`avc_dcr`	H.264/AVC Decoder Configuration Record	_{avc_nalu}
`avc_nalu`	H.264/AVC Network Access Layer Unit	_{avc_sps avc_pps avc_sei}
`avc_pps`	H.264/AVC Picture Parameter Set
`avc_sei`	H.264/AVC Supplemental Enhancement Information
`avc_sps`	H.264/AVC Sequence Parameter Set
`avro_ocf`	Avro object container file
`bencode`	BitTorrent bencoding
`bsd_loopback_frame`	BSD loopback frame	_{inet_packet}
`bson`	Binary JSON
`bzip2`	bzip2 compression	_probe
`cbor`	Concise Binary Object Representation
`dns`	DNS packet
`dns_tcp`	DNS packet (TCP)
`elf`	Executable and Linkable Format
`ether8023_frame`	Ethernet 802.3 frame	_{inet_packet}
`exif`	Exchangeable Image File Format
`flac`	Free Lossless Audio Codec file	_{flac_metadatablocks flac_frame}
`flac_frame`	FLAC frame
`flac_metadatablock`	FLAC metadatablock	_{flac_streaminfo flac_picture vorbis_comment}
`flac_metadatablocks`	FLAC metadatablocks	_{flac_metadatablock}
`flac_picture`	FLAC metadatablock picture	_image
`flac_streaminfo`	FLAC streaminfo
`gif`	Graphics Interchange Format
`gzip`	gzip compression	_probe
`hevc_annexb`	H.265/HEVC Annex B	_{hevc_nalu}
`hevc_au`	H.265/HEVC Access Unit	_{hevc_nalu}
`hevc_dcr`	H.265/HEVC Decoder Configuration Record	_{hevc_nalu}
`hevc_nalu`	H.265/HEVC Network Access Layer Unit	_{hevc_vps hevc_pps hevc_sps}
`hevc_pps`	H.265/HEVC Picture Parameter Set
`hevc_sps`	H.265/HEVC Sequence Parameter Set
`hevc_vps`	H.265/HEVC Video Parameter Set
`icc_profile`	International Color Consortium profile
`icmp`	Internet Control Message Protocol
`icmpv6`	Internet Control Message Protocol v6
`id3v1`	ID3v1 metadata
`id3v11`	ID3v1.1 metadata
`id3v2`	ID3v2 metadata	_image
`ipv4_packet`	Internet protocol v4 packet	_{ip_packet}
`ipv6_packet`	Internet protocol v6 packet	_{ip_packet}
`jpeg`	Joint Photographic Experts Group file	_{exif icc_profile}
`json`	JSON
`macho`	Mach-O macOS executable
`matroska`	Matroska file	_{aac_frame av1_ccr av1_frame avc_au avc_dcr flac_frame flac_metadatablocks hevc_au hevc_dcr image mp3_frame mpeg_asc mpeg_pes_packet mpeg_spu opus_packet vorbis_packet vp8_frame vp9_cfm vp9_frame}
`mp3`	MP3 file	_{id3v2 id3v1 id3v11 apev2 mp3_frame}
`mp3_frame`	MPEG audio layer 3 frame	_xing
`mp4`	ISOBMFF MPEG-4 part 12 and similar	_{aac_frame av1_ccr av1_frame flac_frame flac_metadatablocks id3v2 image jpeg mp3_frame avc_au avc_dcr mpeg_es hevc_au hevc_dcr mpeg_pes_packet opus_packet protobuf_widevine pssh_playready vorbis_packet vp9_frame vpx_ccr icc_profile}
`mpeg_asc`	MPEG-4 Audio Specific Config
`mpeg_es`	MPEG Elementary Stream	_{mpeg_asc vorbis_packet}
`mpeg_pes`	MPEG Packetized elementary stream	_{mpeg_pes_packet mpeg_spu}
`mpeg_pes_packet`	MPEG Packetized elementary stream packet
`mpeg_spu`	Sub Picture Unit (DVD subtitle)
`mpeg_ts`	MPEG Transport Stream
`msgpack`	MessagePack
`ogg`	OGG file	_{ogg_page vorbis_packet opus_packet flac_metadatablock flac_frame}
`ogg_page`	OGG page
`opus_packet`	Opus packet	_{vorbis_comment}
`pcap`	PCAP packet capture	_{link_frame tcp_stream ipv4_packet}
`pcapng`	PCAPNG packet capture	_{link_frame tcp_stream ipv4_packet}
`png`	Portable Network Graphics file	_{icc_profile exif}
`protobuf`	Protobuf
`protobuf_widevine`	Widevine protobuf	_protobuf
`pssh_playready`	PlayReady PSSH
`raw`	Raw bits
`rtmp`	Real-Time Messaging Protocol	_{amf0 mpeg_asc}
`sll2_packet`	Linux cooked capture encapsulation v2	_{inet_packet}
`sll_packet`	Linux cooked capture encapsulation	_{inet_packet}
`tar`	Tar archive	_probe
`tcp_segment`	Transmission control protocol segment
`tiff`	Tag Image File Format	_{icc_profile}
`udp_datagram`	User datagram protocol	_{udp_payload}
`vorbis_comment`	Vorbis comment	_{flac_picture}
`vorbis_packet`	Vorbis packet	_{vorbis_comment}
`vp8_frame`	VP8 frame
`vp9_cfm`	VP9 Codec Feature Metadata
`vp9_frame`	VP9 frame
`vpx_ccr`	VPX Codec Configuration Record
`wav`	WAV file	_{id3v2 id3v1 id3v11}
`webp`	WebP image	_{vp8_frame}
`xing`	Xing header
`zip`	ZIP archive	_probe
`image`	Group	_{gif jpeg mp4 png tiff webp}
`inet_packet`	Group	_{ipv4_packet ipv6_packet}
`ip_packet`	Group	_{icmp icmpv6 tcp_segment udp_datagram}
`link_frame`	Group	_{bsd_loopback_frame ether8023_frame sll2_packet sll_packet}
`probe`	Group	_{adts ar avro_ocf bzip2 elf flac gif gzip jpeg json macho matroska mp3 mp4 mpeg_ts ogg pcap pcapng png tar tiff wav webp zip}
`tcp_stream`	Group	_{dns rtmp}
`udp_payload`	Group	_dns

Global format options

Currently the only global option is force and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI -o option:

fq -d mp4 -o force=true file.mp4
fq -d raw 'mp4({force: true})' file.mp4

Format details

aac_frame

Options

Name	Default	Description
`object_type`	1	Audio object type

Examples

Decode file using options

$ fq -d aac_frame -o object_type=1 file

Decode value as aac_frame

... | aac_frame({object_type: 1})

asn1_ber

Supports decoding BER, CER and DER (X.690).

Currently no extra validation is done for CER and DER.
Does not support specifying a schema.
Supports torepr but without schema all sequences and sets will be arrays.

Examples

frompem and topem can be used to work with PEM format

$ fq -d raw 'frompem | asn1_ber | d' cert.pem

Can be used to decode nested parts

$ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber

If schema is known and not complicated it can be reproduced

$ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der

Supports torepr

$ fq -d asn1_ber torepr file

Supports torepr

... | asn1_ber | torepr

References and links

avc_au

Options

Name	Default	Description
`length_size`	4	Length value size

Examples

Decode file using options

$ fq -d avc_au -o length_size=4 file

Decode value as avc_au

... | avc_au({length_size: 4})

avro_ocf

Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.

Capable of handling null, deflate, and snappy codecs for data compression.

Limitations:

Schema does not support self-referential types, only built-in types.
Decimal logical types are not supported for decoding, will just be treated as their primitive type

References and links

https://avro.apache.org/docs/current/spec.html#Object+Container+Files

bencode

Examples

bencode as JSON

$ fq -d bencode torepr file

Supports torepr

$ fq -d bencode torepr file

Supports torepr

... | bencode | torepr

References and links

https://bsonspec.org/spec.html

bson

Examples

BSON as JSON

$ fq -d bson torepr file

Supports torepr

$ fq -d bson torepr file

Supports torepr

... | bson | torepr

References and links

https://wiki.theory.org/BitTorrentSpecification#Bencoding

cbor

Examples

Supports torepr

$ fq -d cbor torepr file

Supports torepr

... | cbor | torepr

References and links

flac_frame

Options

Name	Default	Description
`bits_per_sample`	16	Bits per sample

Examples

Decode file using options

$ fq -d flac_frame -o bits_per_sample=16 file

Decode value as flac_frame

... | flac_frame({bits_per_sample: 16})

hevc_au

Options

Name	Default	Description
`length_size`	4	Length value size

Examples

Decode file using options

$ fq -d hevc_au -o length_size=4 file

Decode value as hevc_au

... | hevc_au({length_size: 4})

macho

Supports decoding vanilla and FAT Mach-O binaries.

Examples

Select 64bit load segments

$ fq '.load_commands[] | select(.cmd=="segment_64")' file

References and links

https://github.com/aidansteele/osx-abi-macho-file-format-reference

matroska

Examples

Lookup element decode value using matroska_path

... | matroska_path(".Segment.Tracks[0)"

Return matroska_path string for a box decode value

... | grep_by(.id == "Tracks") | matroska_path

References and links

mp3

Options

Name	Default	Description
`max_sync_seek`	32768	Max byte distance to next sync
`max_unique_header_configs`	5	Max number of unique frame header configs allowed

Examples

Decode file using options

$ fq -d mp3 -o max_sync_seek=32768 -o max_unique_header_configs=5 file

Decode value as mp3

... | mp3({max_sync_seek: 32768, max_unique_header_configs: 5})

mp4

Support mp4_path

Options

Name	Default	Description
`allow_truncated`	false	Allow box to be truncated
`decode_samples`	true	Decode supported media samples

Examples

Lookup box decode value using mp4_path

... | mp4_path(".moov.trak[1]")

Return mp4_path string for a box decode value

... | grep_by(.type == "trak") | mp4_path

Decode file using options

$ fq -d mp4 -o allow_truncated=false -o decode_samples=true file

Decode value as mp4

... | mp4({allow_truncated: false, decode_samples: true})

References and links

msgpack

Examples

Supports torepr

$ fq -d msgpack torepr file

Supports torepr

... | msgpack | torepr

References and links

https://github.com/msgpack/msgpack/blob/master/spec.md

protobuf

Examples

Can be used to decode sub messages

$ fq -d protobuf '.fields[6].wire_value | protobuf | d'

References and links

https://developers.google.com/protocol-buffers/docs/encoding

rtmp

Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).

21 KiB Raw Blame History

Supported formats

Global format options

Format details

aac_frame

Options

Examples

asn1_ber

Examples

References and links

avc_au

Options

Examples

avro_ocf

References and links

bencode

Examples

References and links

bson

Examples

References and links

cbor

Examples

References and links

flac_frame

Options

Examples

hevc_au

Options

Examples

macho

Examples

References and links

matroska

Examples

References and links

mp3

Options

Examples

mp4

Options

Examples

References and links

msgpack

Examples

References and links

protobuf

Examples

References and links

rtmp

References and links

Dependency graph

21 KiB

Raw Blame History