## Supported formats
[fq -rn -L . 'include "formats"; formats_table']: sh-start
|Name |Description |Dependencies|
|- |- |-|
|[`aac_frame`](#aac_frame) |Advanced Audio Coding frame ||
|`adts` |Audio Data Transport Stream |`adts_frame`|
|`adts_frame` |Audio Data Transport Stream frame |`aac_frame`|
|`amf0` |Action Message Format 0 ||
|`apev2` |APEv2 metadata tag |`image`|
|`ar` |Unix archive |`probe`|
|[`asn1_ber`](#asn1_ber) |ASN1 BER (basic encoding rules, also CER and DER)||
|`av1_ccr` |AV1 Codec Configuration Record ||
|`av1_frame` |AV1 frame |`av1_obu`|
|`av1_obu` |AV1 Open Bitstream Unit ||
|`avc_annexb` |H.264/AVC Annex B |`avc_nalu`|
|[`avc_au`](#avc_au) |H.264/AVC Access Unit |`avc_nalu`|
|`avc_dcr` |H.264/AVC Decoder Configuration Record |`avc_nalu`|
|`avc_nalu` |H.264/AVC Network Access Layer Unit |`avc_sps` `avc_pps` `avc_sei`|
|`avc_pps` |H.264/AVC Picture Parameter Set ||
|`avc_sei` |H.264/AVC Supplemental Enhancement Information ||
|`avc_sps` |H.264/AVC Sequence Parameter Set ||
|[`avro_ocf`](#avro_ocf) |Avro object container file ||
|[`bencode`](#bencode) |BitTorrent bencoding ||
|`bitcoin_blkdat` |Bitcoin blk.dat |`bitcoin_block`|
|[`bitcoin_block`](#bitcoin_block) |Bitcoin block |`bitcoin_transaction`|
|`bitcoin_script` |Bitcoin script ||
|`bitcoin_transaction` |Bitcoin transaction |`bitcoin_script`|
|[`bplist`](#bplist) |Apple Binary Property List ||
|`bsd_loopback_frame` |BSD loopback frame |`inet_packet`|
|[`bson`](#bson) |Binary JSON ||
|`bzip2` |bzip2 compression |`probe`|
|[`cbor`](#cbor) |Concise Binary Object Representation ||
|[`csv`](#csv) |Comma separated values ||
|`dns` |DNS packet ||
|`dns_tcp` |DNS packet (TCP) ||
|`elf` |Executable and Linkable Format ||
|`ether8023_frame` |Ethernet 802.3 frame |`inet_packet`|
|`exif` |Exchangeable Image File Format ||
|`fairplay_spc` |FairPlay Server Playback Context ||
|`flac` |Free Lossless Audio Codec file |`flac_metadatablocks` `flac_frame`|
|[`flac_frame`](#flac_frame) |FLAC frame ||
|`flac_metadatablock` |FLAC metadatablock |`flac_streaminfo` `flac_picture` `vorbis_comment`|
|`flac_metadatablocks` |FLAC metadatablocks |`flac_metadatablock`|
|`flac_picture` |FLAC metadatablock picture |`image`|
|`flac_streaminfo` |FLAC streaminfo ||
|`gif` |Graphics Interchange Format ||
|`gzip` |gzip compression |`probe`|
|`hevc_annexb` |H.265/HEVC Annex B |`hevc_nalu`|
|[`hevc_au`](#hevc_au) |H.265/HEVC Access Unit |`hevc_nalu`|
|`hevc_dcr` |H.265/HEVC Decoder Configuration Record |`hevc_nalu`|
|`hevc_nalu` |H.265/HEVC Network Access Layer Unit |`hevc_vps` `hevc_pps` `hevc_sps`|
|`hevc_pps` |H.265/HEVC Picture Parameter Set ||
|`hevc_sps` |H.265/HEVC Sequence Parameter Set ||
|`hevc_vps` |H.265/HEVC Video Parameter Set ||
|[`html`](#html) |HyperText Markup Language ||
|`icc_profile` |International Color Consortium profile ||
|`icmp` |Internet Control Message Protocol ||
|`icmpv6` |Internet Control Message Protocol v6 ||
|`id3v1` |ID3v1 metadata ||
|`id3v11` |ID3v1.1 metadata ||
|`id3v2` |ID3v2 metadata |`image`|
|`ipv4_packet` |Internet protocol v4 packet |`ip_packet`|
|`ipv6_packet` |Internet protocol v6 packet |`ip_packet`|
|`jpeg` |Joint Photographic Experts Group file |`exif` `icc_profile`|
|`json` |JavaScript Object Notation ||
|`jsonl` |JavaScript Object Notation Lines ||
|[`macho`](#macho) |Mach-O macOS executable ||
|`macho_fat` |Fat Mach-O macOS executable (multi-architecture) |`macho`|
|`markdown` |Markdown ||
|[`matroska`](#matroska) |Matroska file |`aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `image` `mp3_frame` `mpeg_asc` `mpeg_pes_packet` `mpeg_spu` `opus_packet` `vorbis_packet` `vp8_frame` `vp9_cfm` `vp9_frame`|
|[`mp3`](#mp3) |MP3 file |`id3v2` `id3v1` `id3v11` `apev2` `mp3_frame`|
|`mp3_frame` |MPEG audio layer 3 frame |`xing`|
|[`mp4`](#mp4) |ISOBMFF, QuickTime and similar |`aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `icc_profile` `id3v2` `image` `jpeg` `mp3_frame` `mpeg_es` `mpeg_pes_packet` `opus_packet` `prores_frame` `protobuf_widevine` `pssh_playready` `vorbis_packet` `vp9_frame` `vpx_ccr`|
|`mpeg_asc` |MPEG-4 Audio Specific Config ||
|`mpeg_es` |MPEG Elementary Stream |`mpeg_asc` `vorbis_packet`|
|`mpeg_pes` |MPEG Packetized elementary stream |`mpeg_pes_packet` `mpeg_spu`|
|`mpeg_pes_packet` |MPEG Packetized elementary stream packet ||
|`mpeg_spu` |Sub Picture Unit (DVD subtitle) ||
|`mpeg_ts` |MPEG Transport Stream ||
|[`msgpack`](#msgpack) |MessagePack ||
|`ogg` |OGG file |`ogg_page` `vorbis_packet` `opus_packet` `flac_metadatablock` `flac_frame`|
|`ogg_page` |OGG page ||
|`opus_packet` |Opus packet |`vorbis_comment`|
|`pcap` |PCAP packet capture |`link_frame` `tcp_stream` `ipv4_packet`|
|`pcapng` |PCAPNG packet capture |`link_frame` `tcp_stream` `ipv4_packet`|
|`png` |Portable Network Graphics file |`icc_profile` `exif`|
|`prores_frame` |Apple ProRes frame ||
|[`protobuf`](#protobuf) |Protobuf ||
|`protobuf_widevine` |Widevine protobuf |`protobuf`|
|`pssh_playready` |PlayReady PSSH ||
|`raw` |Raw bits ||
|[`rtmp`](#rtmp) |Real-Time Messaging Protocol |`amf0` `mpeg_asc`|
|`sll2_packet` |Linux cooked capture encapsulation v2 |`inet_packet`|
|`sll_packet` |Linux cooked capture encapsulation |`inet_packet`|
|`tar` |Tar archive |`probe`|
|`tcp_segment` |Transmission control protocol segment ||
|`tiff` |Tag Image File Format |`icc_profile`|
|`toml` |Tom's Obvious, Minimal Language ||
|`udp_datagram` |User datagram protocol |`udp_payload`|
|`vorbis_comment` |Vorbis comment |`flac_picture`|
|`vorbis_packet` |Vorbis packet |`vorbis_comment`|
|`vp8_frame` |VP8 frame ||
|`vp9_cfm` |VP9 Codec Feature Metadata ||
|`vp9_frame` |VP9 frame ||
|`vpx_ccr` |VPX Codec Configuration Record ||
|[`wasm`](#wasm) |WebAssembly Binary Format ||
|`wav` |WAV file |`id3v2` `id3v1` `id3v11`|
|`webp` |WebP image |`vp8_frame`|
|`xing` |Xing header ||
|[`xml`](#xml) |Extensible Markup Language ||
|`yaml` |YAML Ain't Markup Language ||
|[`zip`](#zip) |ZIP archive |`probe`|
|`image` |Group |`gif` `jpeg` `mp4` `png` `tiff` `webp`|
|`inet_packet` |Group |`ipv4_packet` `ipv6_packet`|
|`ip_packet` |Group |`icmp` `icmpv6` `tcp_segment` `udp_datagram`|
|`link_frame` |Group |`bsd_loopback_frame` `ether8023_frame` `sll2_packet` `sll_packet`|
|`probe` |Group |`adts` `ar` `avro_ocf` `bitcoin_blkdat` `bplist` `bzip2` `elf` `flac` `gif` `gzip` `jpeg` `json` `jsonl` `macho` `macho_fat` `matroska` `mp3` `mp4` `mpeg_ts` `ogg` `pcap` `pcapng` `png` `tar` `tiff` `toml` `wasm` `wav` `webp` `xml` `yaml` `zip`|
|`tcp_stream` |Group |`dns_tcp` `rtmp`|
|`udp_payload` |Group |`dns`|
[#]: sh-end
## Global format options
Currently the only global option is `force` and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI `-o` option:
```
fq -d mp4 -o force=true file.mp4
fq -d raw 'mp4({force: true})' file.mp4
```
## Format details
[fq -rn -L . 'include "formats"; formats_sections']: sh-start
## aac_frame
### Options
|Name |Default|Description|
|- |- |-|
|`object_type`|1 |Audio object type|
### Examples
Decode file using aac_frame options
```
$ fq -d aac_frame -o object_type=1 . file
```
Decode value as aac_frame
```
... | aac_frame({object_type:1})
```
## asn1_ber
Supports decoding BER, CER and DER (X.690).
- Currently no extra validation is done for CER and DER.
- Does not support specifying a schema.
- Supports `torepr` but without schema all sequences and sets will be arrays.
### Can be used to decode certificates etc
```sh
$ fq -d raw 'frompem | asn1_ber | d' cert.pem
```
### Can decode nested values
```sh
$ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber
```
### Manual schema
```sh
$ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der
```
### References
- https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf
- https://en.wikipedia.org/wiki/X.690
- https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
- https://lapo.it/asn1js/
## avc_au
### Options
|Name |Default|Description|
|- |- |-|
|`length_size`|4 |Length value size|
### Examples
Decode file using avc_au options
```
$ fq -d avc_au -o length_size=4 . file
```
Decode value as avc_au
```
... | avc_au({length_size:4})
```
## avro_ocf
Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.
Capable of handling null, deflate, and snappy codecs for data compression.
Limitations:
- Schema does not support self-referential types, only built-in types.
- Decimal logical types are not supported for decoding, will just be treated as their primitive type
### References
- https://avro.apache.org/docs/current/spec.html#Object+Container+Files
### Authors
- Xentripetal
xentripetal@fastmail.com
[@xentripetal](https://github.com/xentripetal)
## bencode
### Convert represented value to JSON
```
$ fq -d bencode torepr file.torrent
```
### References
- https://wiki.theory.org/BitTorrentSpecification#Bencoding
## bitcoin_block
### Options
|Name |Default|Description|
|- |- |-|
|`has_header`|false |Has blkdat header|
### Examples
Decode file using bitcoin_block options
```
$ fq -d bitcoin_block -o has_header=false . file
```
Decode value as bitcoin_block
```
... | bitcoin_block({has_header:false})
```
## bplist
### Show full decoding
```sh
$ fq -d bplist dv Info.plist
```
### Timestamps
Timestamps in Apple Binary Property Lists are encoded as Cocoa Core Data
timestamps, where the raw value is the floating point number of seconds since
January 1, 2001. By default, `fq` will render the raw floating point value. In
order to get the raw value or string description, use the `todescription`
function, you can use the `tovalue` and `todescription` functions:
```sh
$ fq 'torepr.SomeTimeStamp | tovalue' Info.plist
685135328
$ fq 'torepr.SomeTimeStamp | todescription' Info.plist
"2022-09-17T19:22:08Z"
```
### Get JSON representation
```sh
$ fq torepr com.apple.UIAutomation.plist
{
"UIAutomationEnabled": true
}
```
### Authors
- David McDonald
[@dgmcdona](https://github.com/dgmcdona)
### References
- http://fileformats.archiveteam.org/wiki/Property_List/Binary
- https://medium.com/@karaiskc/understanding-apples-binary-property-list-format-281e6da00dbd
- https://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c
## bson
### Convert represented value to JSON
```
$ fq -d bson torepr file.bson
```
### Filter represented value
```
$ fq -d bson 'torepr | select(.name=="bob")' file.bson
```
### References
- https://bsonspec.org/spec.html
## cbor
### Convert represented value to JSON
```
$ fq -d cbor torepr file.cbor
```
### References
- https://en.wikipedia.org/wiki/CBOR
- https://www.rfc-editor.org/rfc/rfc8949.html
## csv
### Options
|Name |Default|Description|
|- |- |-|
|`comma` |, |Separator character|
|`comment`|# |Comment line character|
### Examples
Decode file using csv options
```
$ fq -d csv -o comma="," -o comment="#" . file
```
Decode value as csv
```
... | csv({comma:",",comment:"#"})
```
### TSV to CSV
```sh
$ fq -d csv -o comma="\t" tocsv file.tsv
```
### Convert rows to objects based on header row
```sh
$ fq -d csv '.[0] as $t | .[1:] | map(with_entries(.key = $t[.key]))' file.csv
```
## flac_frame
### Options
|Name |Default|Description|
|- |- |-|
|`bits_per_sample`|16 |Bits per sample|
### Examples
Decode file using flac_frame options
```
$ fq -d flac_frame -o bits_per_sample=16 . file
```
Decode value as flac_frame
```
... | flac_frame({bits_per_sample:16})
```
## hevc_au
### Options
|Name |Default|Description|
|- |- |-|
|`length_size`|4 |Length value size|
### Examples
Decode file using hevc_au options
```
$ fq -d hevc_au -o length_size=4 . file
```
Decode value as hevc_au
```
... | hevc_au({length_size:4})
```
## html
### Options
|Name |Default|Description|
|- |- |-|
|`array` |false |Decode as nested arrays|
|`attribute_prefix`|@ |Prefix for attribute keys|
|`seq` |false |Use seq attribute to preserve element order|
### Examples
Decode file using html options
```
$ fq -d html -o array=false -o attribute_prefix="@" -o seq=false . file
```
Decode value as html
```
... | html({array:false,attribute_prefix:"@",seq:false})
```
HTML is decoded in HTML5 mode and will always include ``, `` and `` element.
See xml format for more examples and how to preserve element order and how to encode to xml.
There is no `tohtml` function, see `toxml` instead.
### Element as object
```sh
# decode as object is the default
$ echo 'text' | fq -d html
{
"html": {
"body": {
"a": {
"#text": "text",
"@href": "url"
}
},
"head": ""
}
}
```
### Element as array
```sh
$ 'text' | fq -d html -o array=true
[
"html",
null,
[
[
"head",
null,
[]
],
[
"body",
null,
[
[
"a",
{
"#text": "text",
"href": "url"
},
[]
]
]
]
]
]
# decode html files to a {file: "title", ...} object
$ fq -n -d html '[inputs | {key: input_filename, value: .html.head.title?}] | from_entries' *.html
# href:s in file
$ fq -r -o array=true -d html '.. | select(.[0] == "a" and .[1].href)?.[1].href' file.html
```
## macho
Supports decoding vanilla and FAT Mach-O binaries.
### Select 64bit load segments
```sh
$ fq '.load_commands[] | select(.cmd=="segment_64")' file
```
### References
- https://github.com/aidansteele/osx-abi-macho-file-format-reference
### Authors
- Sıddık AÇIL
acils@itu.edu.tr
[@Akaame](https://github.com/Akaame)
## matroska
### Lookup element using path
```sh
$ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv
```
### Get path to element
```sh
$ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv
```
### References
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus
## mp3
### Options
|Name |Default|Description|
|- |- |-|
|`max_sync_seek` |32768 |Max byte distance to next sync|
|`max_unique_header_configs`|5 |Max number of unique frame header configs allowed|
### Examples
Decode file using mp3 options
```
$ fq -d mp3 -o max_sync_seek=32768 -o max_unique_header_configs=5 . file
```
Decode value as mp3
```
... | mp3({max_sync_seek:32768,max_unique_header_configs:5})
```
## mp4
### Options
|Name |Default|Description|
|- |- |-|
|`allow_truncated`|false |Allow box to be truncated|
|`decode_samples` |true |Decode supported media samples|
### Examples
Decode file using mp4 options
```
$ fq -d mp4 -o allow_truncated=false -o decode_samples=true . file
```
Decode value as mp4
```
... | mp4({allow_truncated:false,decode_samples:true})
```
### Lookup mp4 box using a mp4 box path.
```sh
# | mp4_path($path) ->
$ fq 'mp4_path(".moov.trak[1]")' file.mp4
```
### Get mp4 box path for a decode value box.
```sh
# | mp4_path -> string
$ fq 'grep_by(.type == "trak") | mp4_path' file.mp4
```
### Force decode a single box
```sh
$ fq -n '"AAAAHGVsc3QAAAAAAAAAAQAAADIAAAQAAAEAAA==" | frombase64 | mp4({force:true}) | d'
```
### Speed up decoding by not decoding samples
```sh
# manually decode first sample as a aac_frame
$ fq -o decode_samples=false '.tracks[0].samples[0] | aac_frame | d' file.mp4
```
### Entries for first edit list as values
```sh
$ fq 'first(grep_by(.type=="elst").entries) | tovalue' file.mp4
```
### References
- [ISO/IEC base media file format (MPEG-4 Part 12)](https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format)
- [Quicktime file format](https://developer.apple.com/standards/qtff-2001.pdf)
## msgpack
### Convert represented value to JSON
```
$ fq -d msgpack torepr file.msgpack
```
### References
- https://github.com/msgpack/msgpack/blob/master/spec.md
## protobuf
### Can decode sub messages
```sh
$ fq -d protobuf '.fields[6].wire_value | protobuf | d' file
```
### References
- https://developers.google.com/protocol-buffers/docs/encoding
## rtmp
Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).
### References
- https://rtmp.veriskope.com/docs/spec/
- https://rtmp.veriskope.com/pdf/video_file_format_spec_v10.pdf
## wasm
### Count opcode usage
```sh
$ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' file.wasm
```
### List exports and imports
```sh
$ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' file.wasm
```
### Authors
- Takashi Oguma
[@bitbears-dev](https://github.com/bitbears-dev)
[@0xb17bea125](https://twitter.com/0xb17bea125)
### References
- https://webassembly.github.io/spec/core/
## xml
### Options
|Name |Default|Description|
|- |- |-|
|`array` |false |Decode as nested arrays|
|`attribute_prefix`|@ |Prefix for attribute keys|
|`seq` |false |Use seq attribute to preserve element order|
### Examples
Decode file using xml options
```
$ fq -d xml -o array=false -o attribute_prefix="@" -o seq=false . file
```
Decode value as xml
```
... | xml({array:false,attribute_prefix:"@",seq:false})
```
XML can be decoded and encoded into jq values in two ways, elements as object or array.
Which variant to use depends a bit what you want to do. The object variant might be easier
to query for a specific value but array might be easier to use to generate xml or to query
after all elements of some kind etc.
Encoding is done using the `toxml` function and it will figure what variant that is used based on the input value.
Is has two optional options `indent` and `attribute_prefix`.
### Elements as object
Element can have different shapes depending on body text, attributes and children:
- `text` is `{"a":{"#text":"text","@key":"value"}}`, has text (`#text`) and attributes (`@key`)
- `text` is `{"a":"text"}`
- `text` is `{"a":{"b":"text"}}` one child with only text and no attributes
- `text` is `{"a":{"b":["","text"]}}` two children with same name end up in an array
- `text` is `{"a":{"b":["",{"#text":"text","@key":"value"}]}}`
If there is `#seq` attribute it encodes the child element order. Use `-o seq=true` to include sequence number when decoding,
otherwise order might be lost.
```sh
# decode as object is the default
$ echo 'bbbccc' | fq -d xml -o seq=true
{
"a": {
"b": [
{
"#seq": 0
},
{
"#seq": 1,
"#text": "bbb"
}
],
"c": {
"#seq": 2,
"#text": "ccc",
"@attr": "value"
}
}
}
# access text of the element
$ echo 'bbbccc' | fq '.a.c["#text"]'
"ccc"
# decode to object and encode to xml
$ echo 'bbbccc' | fq -r -d xml -o seq=true 'toxml({indent:2})'
bbb
ccc
```
### Elements as array
Elements are arrays of the shape `["#text": "body text", "attr_name", {key: "attr value"}|null, [, ...]]`.
```sh
# decode as array
$ echo 'bbbccc' | fq -d xml -o array=true
[
"a",
null,
[
[
"b",
null,
[]
],
[
"b",
{
"#text": "bbb"
},
[]
],
[
"c",
{
"#text": "ccc",
"attr": "value"
},
[]
]
]
]
# decode to array and encode to xml
$ echo 'bbbccc' | fq -r -d xml -o array=true -o seq=true 'toxml({indent:2})'
bbb
ccc
# access text of the element, the object variant above is probably easier to use
$ echo 'bbbccc' | fq -o array=true '.[2][2][1]["#text"]'
"ccc"
```
### References
- [xml.com's Converting Between XML and JSON](https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html)
## zip
### Options
|Name |Default|Description|
|- |- |-|
|`uncompress`|true |Uncompress and probe files|
### Examples
Decode file using zip options
```
$ fq -d zip -o uncompress=true . file
```
Decode value as zip
```
... | zip({uncompress:true})
```
Supports ZIP64.
### References
- https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
- https://opensource.apple.com/source/zip/zip-6/unzip/unzip/proginfo/extra.fld
[#]: sh-end
## Dependency graph
