2021-09-19 11:27:56 +03:00
## Supported formats
2021-12-09 19:15:21 +03:00
[fq -rn -L . 'include "formats"; formats_table']: sh-start
|Name |Description |Dependencies|
|- |- |-|
|[`aac_frame`](#aac_frame) |Advanced Audio Coding frame |< sub > < / sub > |
|`adts` |Audio Data Transport Stream |< sub > `adts_frame`< / sub > |
|`adts_frame` |Audio Data Transport Stream frame |< sub > `aac_frame`< / sub > |
|`amf0` |Action Message Format 0 |< sub > < / sub > |
|`apev2` |APEv2 metadata tag |< sub > `image`< / sub > |
|`ar` |Unix archive |< sub > `probe`< / sub > |
|[`asn1_ber`](#asn1_ber) |ASN1 BER (basic encoding rules, also CER and DER)|< sub > < / sub > |
|`av1_ccr` |AV1 Codec Configuration Record |< sub > < / sub > |
|`av1_frame` |AV1 frame |< sub > `av1_obu`< / sub > |
|`av1_obu` |AV1 Open Bitstream Unit |< sub > < / sub > |
|`avc_annexb` |H.264/AVC Annex B |< sub > `avc_nalu`< / sub > |
|[`avc_au`](#avc_au) |H.264/AVC Access Unit |< sub > `avc_nalu`< / sub > |
|`avc_dcr` |H.264/AVC Decoder Configuration Record |< sub > `avc_nalu`< / sub > |
|`avc_nalu` |H.264/AVC Network Access Layer Unit |< sub > `avc_sps` `avc_pps` `avc_sei` </ sub > |
|`avc_pps` |H.264/AVC Picture Parameter Set |< sub > < / sub > |
|`avc_sei` |H.264/AVC Supplemental Enhancement Information |< sub > < / sub > |
|`avc_sps` |H.264/AVC Sequence Parameter Set |< sub > < / sub > |
|[`avro_ocf`](#avro_ocf) |Avro object container file |< sub > < / sub > |
|[`bencode`](#bencode) |BitTorrent bencoding |< sub > < / sub > |
|`bsd_loopback_frame` |BSD loopback frame |< sub > `inet_packet`< / sub > |
|[`bson`](#bson) |Binary JSON |< sub > < / sub > |
|`bzip2` |bzip2 compression |< sub > `probe`< / sub > |
|[`cbor`](#cbor) |Concise Binary Object Representation |< sub > < / sub > |
|`dns` |DNS packet |< sub > < / sub > |
|`dns_tcp` |DNS packet (TCP) |< sub > < / sub > |
|`elf` |Executable and Linkable Format |< sub > < / sub > |
|`ether8023_frame` |Ethernet 802.3 frame |< sub > `inet_packet`< / sub > |
|`exif` |Exchangeable Image File Format |< sub > < / sub > |
|`flac` |Free Lossless Audio Codec file |< sub > `flac_metadatablocks` `flac_frame` </ sub > |
|[`flac_frame`](#flac_frame) |FLAC frame |< sub > < / sub > |
|`flac_metadatablock` |FLAC metadatablock |< sub > `flac_streaminfo` `flac_picture` `vorbis_comment` </ sub > |
|`flac_metadatablocks` |FLAC metadatablocks |< sub > `flac_metadatablock`< / sub > |
|`flac_picture` |FLAC metadatablock picture |< sub > `image`< / sub > |
|`flac_streaminfo` |FLAC streaminfo |< sub > < / sub > |
|`gif` |Graphics Interchange Format |< sub > < / sub > |
|`gzip` |gzip compression |< sub > `probe`< / sub > |
|`hevc_annexb` |H.265/HEVC Annex B |< sub > `hevc_nalu`< / sub > |
|[`hevc_au`](#hevc_au) |H.265/HEVC Access Unit |< sub > `hevc_nalu`< / sub > |
|`hevc_dcr` |H.265/HEVC Decoder Configuration Record |< sub > `hevc_nalu`< / sub > |
|`hevc_nalu` |H.265/HEVC Network Access Layer Unit |< sub > `hevc_vps` `hevc_pps` `hevc_sps` </ sub > |
|`hevc_pps` |H.265/HEVC Picture Parameter Set |< sub > < / sub > |
|`hevc_sps` |H.265/HEVC Sequence Parameter Set |< sub > < / sub > |
|`hevc_vps` |H.265/HEVC Video Parameter Set |< sub > < / sub > |
|`icc_profile` |International Color Consortium profile |< sub > < / sub > |
|`icmp` |Internet Control Message Protocol |< sub > < / sub > |
|`icmpv6` |Internet Control Message Protocol v6 |< sub > < / sub > |
|`id3v1` |ID3v1 metadata |< sub > < / sub > |
|`id3v11` |ID3v1.1 metadata |< sub > < / sub > |
|`id3v2` |ID3v2 metadata |< sub > `image`< / sub > |
|`ipv4_packet` |Internet protocol v4 packet |< sub > `ip_packet`< / sub > |
|`ipv6_packet` |Internet protocol v6 packet |< sub > `ip_packet`< / sub > |
|`jpeg` |Joint Photographic Experts Group file |< sub > `exif` `icc_profile` </ sub > |
|`json` |JSON |< sub > < / sub > |
|[`macho`](#macho) |Mach-O macOS executable |< sub > < / sub > |
|[`matroska`](#matroska) |Matroska file |< sub > `aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `image` `mp3_frame` `mpeg_asc` `mpeg_pes_packet` `mpeg_spu` `opus_packet` `vorbis_packet` `vp8_frame` `vp9_cfm` `vp9_frame` </ sub > |
|[`mp3`](#mp3) |MP3 file |< sub > `id3v2` `id3v1` `id3v11` `apev2` `mp3_frame` </ sub > |
|`mp3_frame` |MPEG audio layer 3 frame |< sub > `xing`< / sub > |
|[`mp4`](#mp4) |ISOBMFF MPEG-4 part 12 and similar |< sub > `aac_frame` `av1_ccr` `av1_frame` `flac_frame` `flac_metadatablocks` `id3v2` `image` `jpeg` `mp3_frame` `avc_au` `avc_dcr` `mpeg_es` `hevc_au` `hevc_dcr` `mpeg_pes_packet` `opus_packet` `protobuf_widevine` `pssh_playready` `vorbis_packet` `vp9_frame` `vpx_ccr` `icc_profile` </ sub > |
|`mpeg_asc` |MPEG-4 Audio Specific Config |< sub > < / sub > |
|`mpeg_es` |MPEG Elementary Stream |< sub > `mpeg_asc` `vorbis_packet` </ sub > |
|`mpeg_pes` |MPEG Packetized elementary stream |< sub > `mpeg_pes_packet` `mpeg_spu` </ sub > |
|`mpeg_pes_packet` |MPEG Packetized elementary stream packet |< sub > < / sub > |
|`mpeg_spu` |Sub Picture Unit (DVD subtitle) |< sub > < / sub > |
|`mpeg_ts` |MPEG Transport Stream |< sub > < / sub > |
|[`msgpack`](#msgpack) |MessagePack |< sub > < / sub > |
|`ogg` |OGG file |< sub > `ogg_page` `vorbis_packet` `opus_packet` `flac_metadatablock` `flac_frame` </ sub > |
|`ogg_page` |OGG page |< sub > < / sub > |
|`opus_packet` |Opus packet |< sub > `vorbis_comment`< / sub > |
|`pcap` |PCAP packet capture |< sub > `link_frame` `tcp_stream` `ipv4_packet` </ sub > |
|`pcapng` |PCAPNG packet capture |< sub > `link_frame` `tcp_stream` `ipv4_packet` </ sub > |
|`png` |Portable Network Graphics file |< sub > `icc_profile` `exif` </ sub > |
|[`protobuf`](#protobuf) |Protobuf |< sub > < / sub > |
|`protobuf_widevine` |Widevine protobuf |< sub > `protobuf`< / sub > |
|`pssh_playready` |PlayReady PSSH |< sub > < / sub > |
|`raw` |Raw bits |< sub > < / sub > |
|[`rtmp`](#rtmp) |Real-Time Messaging Protocol |< sub > `amf0` `mpeg_asc` </ sub > |
|`sll2_packet` |Linux cooked capture encapsulation v2 |< sub > `inet_packet`< / sub > |
|`sll_packet` |Linux cooked capture encapsulation |< sub > `inet_packet`< / sub > |
|`tar` |Tar archive |< sub > `probe`< / sub > |
|`tcp_segment` |Transmission control protocol segment |< sub > < / sub > |
|`tiff` |Tag Image File Format |< sub > `icc_profile`< / sub > |
|`udp_datagram` |User datagram protocol |< sub > `udp_payload`< / sub > |
|`vorbis_comment` |Vorbis comment |< sub > `flac_picture`< / sub > |
|`vorbis_packet` |Vorbis packet |< sub > `vorbis_comment`< / sub > |
|`vp8_frame` |VP8 frame |< sub > < / sub > |
|`vp9_cfm` |VP9 Codec Feature Metadata |< sub > < / sub > |
|`vp9_frame` |VP9 frame |< sub > < / sub > |
|`vpx_ccr` |VPX Codec Configuration Record |< sub > < / sub > |
|`wav` |WAV file |< sub > `id3v2` `id3v1` `id3v11` </ sub > |
|`webp` |WebP image |< sub > `vp8_frame`< / sub > |
|`xing` |Xing header |< sub > < / sub > |
|`zip` |ZIP archive |< sub > `probe`< / sub > |
|`image` |Group |< sub > `gif` `jpeg` `mp4` `png` `tiff` `webp` </ sub > |
|`inet_packet` |Group |< sub > `ipv4_packet` `ipv6_packet` </ sub > |
|`ip_packet` |Group |< sub > `icmp` `icmpv6` `tcp_segment` `udp_datagram` </ sub > |
|`link_frame` |Group |< sub > `bsd_loopback_frame` `ether8023_frame` `sll2_packet` `sll_packet` </ sub > |
|`probe` |Group |< sub > `adts` `ar` `avro_ocf` `bzip2` `elf` `flac` `gif` `gzip` `jpeg` `json` `macho` `matroska` `mp3` `mp4` `mpeg_ts` `ogg` `pcap` `pcapng` `png` `tar` `tiff` `wav` `webp` `zip` </ sub > |
|`tcp_stream` |Group |< sub > `dns` `rtmp` </ sub > |
|`udp_payload` |Group |< sub > `dns`< / sub > |
2021-09-19 11:27:56 +03:00
[#]: sh-end
2021-12-09 19:15:21 +03:00
## Global format options
2022-02-05 20:15:18 +03:00
2021-12-09 19:15:21 +03:00
Currently the only global option is `force` and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI `-o` option:
2022-02-05 20:15:18 +03:00
```
fq -d mp4 -o force=true file.mp4
fq -d raw 'mp4({force: true})' file.mp4
```
2022-02-19 03:33:45 +03:00
## Format details
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
[fq -rn -L . 'include "formats"; formats_sections']: sh-start
### aac_frame
#### Options
|Name |Default|Description|
|- |- |-|
|`object_type`|1 |Audio object type|
#### Examples
Decode file using options
```
$ fq -d aac_frame -o object_type=1 file
```
Decode value as aac_frame
```
... | aac_frame({object_type: 1})
```
2022-01-29 14:01:36 +03:00
2022-01-13 20:34:59 +03:00
### asn1_ber
2021-12-09 19:15:21 +03:00
Supports decoding BER, CER and DER (X.690).
2022-01-13 20:34:59 +03:00
2022-02-16 22:13:28 +03:00
- Currently no extra validation is done for CER and DER.
- Does not support specifying a schema.
- Supports `torepr` but without schema all sequences and sets will be arrays.
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
#### Examples
`frompem` and `topem` can be used to work with PEM format
2022-01-13 20:34:59 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d raw 'frompem | asn1_ber | d' cert.pem
2022-01-13 20:34:59 +03:00
```
2021-12-09 19:15:21 +03:00
Can be used to decode nested parts
2022-02-07 19:41:05 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber
2022-02-07 19:41:05 +03:00
```
2021-12-09 19:15:21 +03:00
If schema is known and not complicated it can be reproduced
2022-02-07 19:41:05 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der
2022-02-07 19:41:05 +03:00
```
2022-02-16 22:13:28 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
```
$ fq -d asn1_ber torepr file
```
2022-02-16 22:13:28 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
2022-02-16 22:13:28 +03:00
```
2021-12-09 19:15:21 +03:00
... | asn1_ber | torepr
2022-02-16 22:13:28 +03:00
```
2021-12-09 19:15:21 +03:00
#### References and links
2022-02-16 22:13:28 +03:00
- https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf
- https://en.wikipedia.org/wiki/X.690
- https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
- https://lapo.it/asn1js/
2021-12-09 19:15:21 +03:00
### avc_au
#### Options
|Name |Default|Description|
|- |- |-|
|`length_size`|4 |Length value size|
#### Examples
Decode file using options
```
$ fq -d avc_au -o length_size=4 file
```
Decode value as avc_au
```
... | avc_au({length_size: 4})
```
2022-02-10 05:46:12 +03:00
### avro_ocf
2021-12-09 19:15:21 +03:00
Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.
2022-02-10 05:46:12 +03:00
Capable of handling null, deflate, and snappy codecs for data compression.
Limitations:
- Schema does not support self-referential types, only built-in types.
- Decimal logical types are not supported for decoding, will just be treated as their primitive type
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
#### References and links
- https://avro.apache.org/docs/current/spec.html#Object+Container+Files
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
### bencode
#### Examples
Supports `torepr`
```
$ fq -d bencode torepr file
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
```
... | bencode | torepr
```
#### References and links
- https://bsonspec.org/spec.html
2022-01-29 14:01:36 +03:00
### bson
2021-12-09 19:15:21 +03:00
#### Examples
BSON as JSON
```
$ fq -d bson torepr file
```
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d bson torepr file
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
```
... | bson | torepr
```
#### References and links
- https://wiki.theory.org/BitTorrentSpecification#Bencoding
2022-01-29 14:01:36 +03:00
### cbor
2021-12-09 19:15:21 +03:00
#### Examples
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d cbor torepr file
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
```
... | cbor | torepr
```
2022-02-28 12:28:21 +03:00
2021-12-09 19:15:21 +03:00
#### References and links
- https://en.wikipedia.org/wiki/CBOR
- https://www.rfc-editor.org/rfc/rfc8949.html
### flac_frame
#### Options
|Name |Default|Description|
|- |- |-|
|`bits_per_sample`|16 |Bits per sample|
2022-02-28 12:28:21 +03:00
2022-03-03 11:44:22 +03:00
#### Examples
2022-02-28 12:28:21 +03:00
2021-12-09 19:15:21 +03:00
Decode file using options
```
$ fq -d flac_frame -o bits_per_sample=16 file
```
2022-02-28 12:28:21 +03:00
2021-12-09 19:15:21 +03:00
Decode value as flac_frame
2022-02-28 12:28:21 +03:00
```
2021-12-09 19:15:21 +03:00
... | flac_frame({bits_per_sample: 16})
2022-02-28 12:28:21 +03:00
```
2021-12-09 19:15:21 +03:00
### hevc_au
#### Options
|Name |Default|Description|
|- |- |-|
|`length_size`|4 |Length value size|
#### Examples
Decode file using options
2022-02-28 12:28:21 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d hevc_au -o length_size=4 file
2022-02-28 12:28:21 +03:00
```
2021-12-09 19:15:21 +03:00
Decode value as hevc_au
```
... | hevc_au({length_size: 4})
```
### macho
Supports decoding vanilla and FAT Mach-O binaries.
#### Examples
Select 64bit load segments
```
$ fq '.load_commands[] | select(.cmd=="segment_64")' file
```
#### References and links
2022-03-03 15:37:35 +03:00
2022-02-28 12:28:21 +03:00
- https://github.com/aidansteele/osx-abi-macho-file-format-reference
2022-01-29 14:01:36 +03:00
### matroska
2021-12-09 19:15:21 +03:00
#### Examples
Lookup element decode value using `matroska_path`
```
... | matroska_path(".Segment.Tracks[0)"
```
Return `matroska_path` string for a box decode value
```
... | grep_by(.id == "Tracks") | matroska_path
```
#### References and links
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus
### mp3
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
#### Options
|Name |Default|Description|
|- |- |-|
|`max_sync_seek` |32768 |Max byte distance to next sync|
|`max_unique_header_configs`|5 |Max number of unique frame header configs allowed|
#### Examples
Decode file using options
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d mp3 -o max_sync_seek=32768 -o max_unique_header_configs=5 file
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
Decode value as mp3
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
... | mp3({max_sync_seek: 32768, max_unique_header_configs: 5})
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2022-01-29 14:01:36 +03:00
### mp4
2021-12-09 19:15:21 +03:00
Support `mp4_path`
#### Options
|Name |Default|Description|
|- |- |-|
|`allow_truncated`|false |Allow box to be truncated|
|`decode_samples` |true |Decode supported media samples|
#### Examples
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
Lookup box decode value using `mp4_path`
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
... | mp4_path(".moov.trak[1]")
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
Return `mp4_path` string for a box decode value
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
... | grep_by(.type == "trak") | mp4_path
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
Decode file using options
```
$ fq -d mp4 -o allow_truncated=false -o decode_samples=true file
```
Decode value as mp4
```
... | mp4({allow_truncated: false, decode_samples: true})
```
#### References and links
- [ISO/IEC base media file format (MPEG-4 Part 12) ](https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format )
- [Quicktime file format ](https://developer.apple.com/standards/qtff-2001.pdf )
2022-01-29 14:01:36 +03:00
### msgpack
2021-12-09 19:15:21 +03:00
#### Examples
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
Supports `torepr`
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d msgpack torepr file
```
Supports `torepr`
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
... | msgpack | torepr
```
#### References and links
- https://github.com/msgpack/msgpack/blob/master/spec.md
2022-01-13 20:34:59 +03:00
2022-02-01 18:07:41 +03:00
### protobuf
2021-12-09 19:15:21 +03:00
#### Examples
2022-02-01 18:07:41 +03:00
2021-12-09 19:15:21 +03:00
Can be used to decode sub messages
2022-02-01 18:07:41 +03:00
```
2021-12-09 19:15:21 +03:00
$ fq -d protobuf '.fields[6].wire_value | protobuf | d'
2022-02-01 18:07:41 +03:00
```
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
#### References and links
- https://developers.google.com/protocol-buffers/docs/encoding
2022-03-29 23:41:11 +03:00
### rtmp
Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
#### References and links
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
- https://rtmp.veriskope.com/docs/spec/
- https://rtmp.veriskope.com/pdf/video_file_format_spec_v10.pdf
[#]: sh-end
2021-09-19 11:27:56 +03:00
2022-02-05 20:15:18 +03:00
## Dependency graph
2021-09-19 11:27:56 +03:00
