## Supported formats
[./formats_table.sh]: sh-start
|Name |Description |Dependencies|
|- |- |-|
|`aac_frame` |Advanced Audio Coding frame ||
|`adts` |Audio Data Transport Stream |`adts_frame`|
|`adts_frame` |Audio Data Transport Stream frame |`aac_frame`|
|`amf0` |Action Message Format 0 ||
|`apev2` |APEv2 metadata tag |`image`|
|`ar` |Unix archive |`probe`|
|[`asn1_ber`](#asn1_ber) |ASN1 Basic Encoding Rules (also CER and DER) ||
|`av1_ccr` |AV1 Codec Configuration Record ||
|`av1_frame` |AV1 frame |`av1_obu`|
|`av1_obu` |AV1 Open Bitstream Unit ||
|`avc_annexb` |H.264/AVC Annex B |`avc_nalu`|
|`avc_au` |H.264/AVC Access Unit |`avc_nalu`|
|`avc_dcr` |H.264/AVC Decoder Configuration Record |`avc_nalu`|
|`avc_nalu` |H.264/AVC Network Access Layer Unit |`avc_sps` `avc_pps` `avc_sei`|
|`avc_pps` |H.264/AVC Picture Parameter Set ||
|`avc_sei` |H.264/AVC Supplemental Enhancement Information ||
|`avc_sps` |H.264/AVC Sequence Parameter Set ||
|[`avro_ocf`](#avro_ocf) |Avro object container file ||
|`bencode` |BitTorrent bencoding ||
|`bsd_loopback_frame` |BSD loopback frame |`inet_packet`|
|[`bson`](#bson) |Binary JSON ||
|`bzip2` |bzip2 compression |`probe`|
|[`cbor`](#cbor) |Concise Binary Object Representation ||
|`dns` |DNS packet ||
|`dns_tcp` |DNS packet (TCP) ||
|`elf` |Executable and Linkable Format ||
|`ether8023_frame` |Ethernet 802.3 frame |`inet_packet`|
|`exif` |Exchangeable Image File Format ||
|`flac` |Free Lossless Audio Codec file |`flac_metadatablocks` `flac_frame`|
|`flac_frame` |FLAC frame ||
|`flac_metadatablock` |FLAC metadatablock |`flac_streaminfo` `flac_picture` `vorbis_comment`|
|`flac_metadatablocks` |FLAC metadatablocks |`flac_metadatablock`|
|`flac_picture` |FLAC metadatablock picture |`image`|
|`flac_streaminfo` |FLAC streaminfo ||
|`gif` |Graphics Interchange Format ||
|`gzip` |gzip compression |`probe`|
|`hevc_annexb` |H.265/HEVC Annex B |`hevc_nalu`|
|`hevc_au` |H.265/HEVC Access Unit |`hevc_nalu`|
|`hevc_dcr` |H.265/HEVC Decoder Configuration Record |`hevc_nalu`|
|`hevc_nalu` |H.265/HEVC Network Access Layer Unit |`hevc_vps` `hevc_pps` `hevc_sps`|
|`hevc_pps` |H.265/HEVC Picture Parameter Set ||
|`hevc_sps` |H.265/HEVC Sequence Parameter Set ||
|`hevc_vps` |H.265/HEVC Video Parameter Set ||
|`icc_profile` |International Color Consortium profile ||
|`icmp` |Internet Control Message Protocol ||
|`icmpv6` |Internet Control Message Protocol v6 ||
|`id3v1` |ID3v1 metadata ||
|`id3v11` |ID3v1.1 metadata ||
|`id3v2` |ID3v2 metadata |`image`|
|`ipv4_packet` |Internet protocol v4 packet |`ip_packet`|
|`ipv6_packet` |Internet protocol v6 packet |`ip_packet`|
|`jpeg` |Joint Photographic Experts Group file |`exif` `icc_profile`|
|`json` |JSON ||
|[`macho`](#macho) |Mach-O macOS executable ||
|[`matroska`](#matroska) |Matroska file |`aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `image` `mp3_frame` `mpeg_asc` `mpeg_pes_packet` `mpeg_spu` `opus_packet` `vorbis_packet` `vp8_frame` `vp9_cfm` `vp9_frame`|
|`mp3` |MP3 file |`id3v2` `id3v1` `id3v11` `apev2` `mp3_frame`|
|`mp3_frame` |MPEG audio layer 3 frame |`xing`|
|[`mp4`](#mp4) |MPEG-4 file and similar |`aac_frame` `av1_ccr` `av1_frame` `flac_frame` `flac_metadatablocks` `id3v2` `image` `jpeg` `mp3_frame` `avc_au` `avc_dcr` `mpeg_es` `hevc_au` `hevc_dcr` `mpeg_pes_packet` `opus_packet` `protobuf_widevine` `pssh_playready` `vorbis_packet` `vp9_frame` `vpx_ccr` `icc_profile`|
|`mpeg_asc` |MPEG-4 Audio Specific Config ||
|`mpeg_es` |MPEG Elementary Stream |`mpeg_asc` `vorbis_packet`|
|`mpeg_pes` |MPEG Packetized elementary stream |`mpeg_pes_packet` `mpeg_spu`|
|`mpeg_pes_packet` |MPEG Packetized elementary stream packet ||
|`mpeg_spu` |Sub Picture Unit (DVD subtitle) ||
|`mpeg_ts` |MPEG Transport Stream ||
|[`msgpack`](#msgpack) |MessagePack ||
|`ogg` |OGG file |`ogg_page` `vorbis_packet` `opus_packet` `flac_metadatablock` `flac_frame`|
|`ogg_page` |OGG page ||
|`opus_packet` |Opus packet |`vorbis_comment`|
|`pcap` |PCAP packet capture |`link_frame` `tcp_stream` `ipv4_packet`|
|`pcapng` |PCAPNG packet capture |`link_frame` `tcp_stream` `ipv4_packet`|
|`png` |Portable Network Graphics file |`icc_profile` `exif`|
|[`protobuf`](#protobuf) |Protobuf ||
|`protobuf_widevine` |Widevine protobuf |`protobuf`|
|`pssh_playready` |PlayReady PSSH ||
|`raw` |Raw bits ||
|[`rtmp`](#rtmp) |Real-Time Messaging Protocol |`amf0` `mpeg_asc`|
|`sll2_packet` |Linux cooked capture encapsulation v2 |`inet_packet`|
|`sll_packet` |Linux cooked capture encapsulation |`inet_packet`|
|`tar` |Tar archive |`probe`|
|`tcp_segment` |Transmission control protocol segment ||
|`tiff` |Tag Image File Format |`icc_profile`|
|`udp_datagram` |User datagram protocol |`udp_payload`|
|`vorbis_comment` |Vorbis comment |`flac_picture`|
|`vorbis_packet` |Vorbis packet |`vorbis_comment`|
|`vp8_frame` |VP8 frame ||
|`vp9_cfm` |VP9 Codec Feature Metadata ||
|`vp9_frame` |VP9 frame ||
|`vpx_ccr` |VPX Codec Configuration Record ||
|`wav` |WAV file |`id3v2` `id3v1` `id3v11`|
|`webp` |WebP image |`vp8_frame`|
|`xing` |Xing header ||
|`zip` |ZIP archive |`probe`|
|`image` |Group |`gif` `jpeg` `mp4` `png` `tiff` `webp`|
|`inet_packet` |Group |`ipv4_packet` `ipv6_packet`|
|`ip_packet` |Group |`icmp` `icmpv6` `tcp_segment` `udp_datagram`|
|`link_frame` |Group |`bsd_loopback_frame` `ether8023_frame` `sll2_packet` `sll_packet`|
|`probe` |Group |`adts` `ar` `avro_ocf` `bzip2` `elf` `flac` `gif` `gzip` `jpeg` `json` `macho` `matroska` `mp3` `mp4` `mpeg_ts` `ogg` `pcap` `pcapng` `png` `tar` `tiff` `wav` `webp` `zip`|
|`tcp_stream` |Group |`dns` `rtmp`|
|`udp_payload` |Group |`dns`|
[#]: sh-end
## Format options
Currently the only option is `force` and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI `-o` option:
```
fq -d mp4 -o force=true file.mp4
fq -d raw 'mp4({force: true})' file.mp4
```
## Format details
[./formats_collect.sh]: sh-start
### asn1_ber
Supports decoding BER, CER and DER ([X.690]([X.690_1297.pdf)).
- Currently no extra validation is done for CER and DER.
- Does not support specifying a schema.
- Supports `torepr` but without schema all sequences and sets will be arrays.
```
fq -d asn1_ber torepr file.ber
```
Functions `frompem` and `topem` can help working with PEM format:
```
fq -d raw 'frompem | asn1_ber | d' cert.pem
```
If the schema is known and not that complicated it can be reproduced:
```
fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der
```
Can be used to decode nested parts:
```
fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber
```
References and tools:
- https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf
- https://en.wikipedia.org/wiki/X.690
- https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
- https://lapo.it/asn1js/
### avro_ocf
Supports reading Avro Object Container Format (OCF) files based on the [1.11.0 specification](https://avro.apache.org/docs/current/spec.html#Object+Container+Files).
Capable of handling null, deflate, and snappy codecs for data compression.
Limitations:
- Schema does not support self-referential types, only built-in types.
- Decimal logical types are not supported for decoding, will just be treated as their primitive type
### becode
Supports `torepr`:
```
fq -d bencode torepr file.torrent
```
### bson
Supports `torepr`:
```
fq -d bson torepr file.bson
```
### cbor
Supports `torepr`:
```
fq -d cbor torepr file.cbor
fq -d cbor 'torepr.field' file.cbor
fq -d cbor 'torepr | .field' file.cbor
fq -d cbor 'torepr | grep("abc")' file.cbor
```
### macho
Supports decoding vanilla and FAT Mach-O binaries.
#### Examples
To decode the macOS build of `fq`:
```
fq . /path/to/fq
```
```
fq '.load_commands[] | select(.cmd=="segment_64")' /path/to/fq
```
Note you can use `-d macho` to decode a broken Mach-O binary.
#### References:
- https://github.com/aidansteele/osx-abi-macho-file-format-reference
### matroska
Supports `matroska_path`:
```
$ fq 'matroska_path(".Segment.Tracks[0]")' file.mkv
│00 01 02 03 04 05 06 07 08 09│0123456789│.elements[1].elements[3]{}:
0x122│ 16 54 ae 6b │ .T.k │ id: "Tracks" (0x1654ae6b) (A Top-Level Element of information with many tracks described.)
│ │ │ type: "master" (7)
0x122│ 4d bf │ M. │ size: 3519
0x122│ bf│ .│ elements[0:3]:
0x12c│84 cf 8b db a0 ae 01 00 00 00│..........│
0x136│00 00 00 78 d7 81 01 73 c5 88│...x...s..│
* │until 0xee9.7 (3519) │ │
```
```
$ fq 'first(grep_by(.id == "Tracks")) | matroska_path' test.mkv
".Segment.Tracks"
```
### mp4
Supports `mp4_path`:
```
$ fq 'mp4_path(".moov.trak[1]")' file.mp4
│00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[1]{}:
0x4f6│ 00 00 02│ ...│ size: 573
0x500│3d │= │
0x500│ 74 72 61 6b │ trak │ type: "trak" (Container for an individual track or stream)
0x500│ 00 00 00 5c 74│ ...\t│ boxes[0:3]:
0x50a│6b 68 64 00 00 00 03 00 00 00│khd.......│
0x514│00 00 00 00 00 00 00 00 01 00│..........│
* │until 0x739.7 (565) │ │
```
```
$ fq 'first(grep_by(.type == "trak")) | mp4_path' file.mp4
".moov.trak"
```
### msgpack
Supports `torepr`:
```
fq -d msgpack torepr file.msgpack
```
### protobuf
`protobuf` decoder can be used to decode sub messages:
```
fq -d protobuf '.fields[6].wire_value | protobuf | d'
```
### rtmp
Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).
[#]: sh-end
## Dependency graph
