mirror of
https://github.com/wader/fq.git
synced 2024-11-24 03:05:22 +03:00
814 lines
32 KiB
Markdown
814 lines
32 KiB
Markdown
## Supported formats
|
||
|
||
[fq -rn -L . 'include "formats"; formats_table']: sh-start
|
||
|
||
|Name |Description |Dependencies|
|
||
|- |- |-|
|
||
|[`aac_frame`](#aac_frame) |Advanced Audio Coding frame |<sub></sub>|
|
||
|`adts` |Audio Data Transport Stream |<sub>`adts_frame`</sub>|
|
||
|`adts_frame` |Audio Data Transport Stream frame |<sub>`aac_frame`</sub>|
|
||
|`amf0` |Action Message Format 0 |<sub></sub>|
|
||
|`apev2` |APEv2 metadata tag |<sub>`image`</sub>|
|
||
|`ar` |Unix archive |<sub>`probe`</sub>|
|
||
|[`asn1_ber`](#asn1_ber) |ASN1 BER (basic encoding rules, also CER and DER)|<sub></sub>|
|
||
|`av1_ccr` |AV1 Codec Configuration Record |<sub></sub>|
|
||
|`av1_frame` |AV1 frame |<sub>`av1_obu`</sub>|
|
||
|`av1_obu` |AV1 Open Bitstream Unit |<sub></sub>|
|
||
|`avc_annexb` |H.264/AVC Annex B |<sub>`avc_nalu`</sub>|
|
||
|[`avc_au`](#avc_au) |H.264/AVC Access Unit |<sub>`avc_nalu`</sub>|
|
||
|`avc_dcr` |H.264/AVC Decoder Configuration Record |<sub>`avc_nalu`</sub>|
|
||
|`avc_nalu` |H.264/AVC Network Access Layer Unit |<sub>`avc_sps` `avc_pps` `avc_sei`</sub>|
|
||
|`avc_pps` |H.264/AVC Picture Parameter Set |<sub></sub>|
|
||
|`avc_sei` |H.264/AVC Supplemental Enhancement Information |<sub></sub>|
|
||
|`avc_sps` |H.264/AVC Sequence Parameter Set |<sub></sub>|
|
||
|[`avro_ocf`](#avro_ocf) |Avro object container file |<sub></sub>|
|
||
|[`bencode`](#bencode) |BitTorrent bencoding |<sub></sub>|
|
||
|`bitcoin_blkdat` |Bitcoin blk.dat |<sub>`bitcoin_block`</sub>|
|
||
|[`bitcoin_block`](#bitcoin_block) |Bitcoin block |<sub>`bitcoin_transaction`</sub>|
|
||
|`bitcoin_script` |Bitcoin script |<sub></sub>|
|
||
|`bitcoin_transaction` |Bitcoin transaction |<sub>`bitcoin_script`</sub>|
|
||
|[`bplist`](#bplist) |Apple Binary Property List |<sub></sub>|
|
||
|`bsd_loopback_frame` |BSD loopback frame |<sub>`inet_packet`</sub>|
|
||
|[`bson`](#bson) |Binary JSON |<sub></sub>|
|
||
|`bzip2` |bzip2 compression |<sub>`probe`</sub>|
|
||
|[`cbor`](#cbor) |Concise Binary Object Representation |<sub></sub>|
|
||
|[`csv`](#csv) |Comma separated values |<sub></sub>|
|
||
|`dns` |DNS packet |<sub></sub>|
|
||
|`dns_tcp` |DNS packet (TCP) |<sub></sub>|
|
||
|`elf` |Executable and Linkable Format |<sub></sub>|
|
||
|`ether8023_frame` |Ethernet 802.3 frame |<sub>`inet_packet`</sub>|
|
||
|`exif` |Exchangeable Image File Format |<sub></sub>|
|
||
|`fairplay_spc` |FairPlay Server Playback Context |<sub></sub>|
|
||
|`flac` |Free Lossless Audio Codec file |<sub>`flac_metadatablocks` `flac_frame`</sub>|
|
||
|[`flac_frame`](#flac_frame) |FLAC frame |<sub></sub>|
|
||
|`flac_metadatablock` |FLAC metadatablock |<sub>`flac_streaminfo` `flac_picture` `vorbis_comment`</sub>|
|
||
|`flac_metadatablocks` |FLAC metadatablocks |<sub>`flac_metadatablock`</sub>|
|
||
|`flac_picture` |FLAC metadatablock picture |<sub>`image`</sub>|
|
||
|`flac_streaminfo` |FLAC streaminfo |<sub></sub>|
|
||
|`gif` |Graphics Interchange Format |<sub></sub>|
|
||
|`gzip` |gzip compression |<sub>`probe`</sub>|
|
||
|`hevc_annexb` |H.265/HEVC Annex B |<sub>`hevc_nalu`</sub>|
|
||
|[`hevc_au`](#hevc_au) |H.265/HEVC Access Unit |<sub>`hevc_nalu`</sub>|
|
||
|`hevc_dcr` |H.265/HEVC Decoder Configuration Record |<sub>`hevc_nalu`</sub>|
|
||
|`hevc_nalu` |H.265/HEVC Network Access Layer Unit |<sub>`hevc_vps` `hevc_pps` `hevc_sps`</sub>|
|
||
|`hevc_pps` |H.265/HEVC Picture Parameter Set |<sub></sub>|
|
||
|`hevc_sps` |H.265/HEVC Sequence Parameter Set |<sub></sub>|
|
||
|`hevc_vps` |H.265/HEVC Video Parameter Set |<sub></sub>|
|
||
|[`html`](#html) |HyperText Markup Language |<sub></sub>|
|
||
|`icc_profile` |International Color Consortium profile |<sub></sub>|
|
||
|`icmp` |Internet Control Message Protocol |<sub></sub>|
|
||
|`icmpv6` |Internet Control Message Protocol v6 |<sub></sub>|
|
||
|`id3v1` |ID3v1 metadata |<sub></sub>|
|
||
|`id3v11` |ID3v1.1 metadata |<sub></sub>|
|
||
|`id3v2` |ID3v2 metadata |<sub>`image`</sub>|
|
||
|`ipv4_packet` |Internet protocol v4 packet |<sub>`ip_packet`</sub>|
|
||
|`ipv6_packet` |Internet protocol v6 packet |<sub>`ip_packet`</sub>|
|
||
|`jpeg` |Joint Photographic Experts Group file |<sub>`exif` `icc_profile`</sub>|
|
||
|`json` |JavaScript Object Notation |<sub></sub>|
|
||
|`jsonl` |JavaScript Object Notation Lines |<sub></sub>|
|
||
|[`macho`](#macho) |Mach-O macOS executable |<sub></sub>|
|
||
|`macho_fat` |Fat Mach-O macOS executable (multi-architecture) |<sub>`macho`</sub>|
|
||
|`markdown` |Markdown |<sub></sub>|
|
||
|[`matroska`](#matroska) |Matroska file |<sub>`aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `image` `mp3_frame` `mpeg_asc` `mpeg_pes_packet` `mpeg_spu` `opus_packet` `vorbis_packet` `vp8_frame` `vp9_cfm` `vp9_frame`</sub>|
|
||
|[`mp3`](#mp3) |MP3 file |<sub>`id3v2` `id3v1` `id3v11` `apev2` `mp3_frame`</sub>|
|
||
|`mp3_frame` |MPEG audio layer 3 frame |<sub>`xing`</sub>|
|
||
|[`mp4`](#mp4) |ISOBMFF, QuickTime and similar |<sub>`aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `icc_profile` `id3v2` `image` `jpeg` `mp3_frame` `mpeg_es` `mpeg_pes_packet` `opus_packet` `prores_frame` `protobuf_widevine` `pssh_playready` `vorbis_packet` `vp9_frame` `vpx_ccr`</sub>|
|
||
|`mpeg_asc` |MPEG-4 Audio Specific Config |<sub></sub>|
|
||
|`mpeg_es` |MPEG Elementary Stream |<sub>`mpeg_asc` `vorbis_packet`</sub>|
|
||
|`mpeg_pes` |MPEG Packetized elementary stream |<sub>`mpeg_pes_packet` `mpeg_spu`</sub>|
|
||
|`mpeg_pes_packet` |MPEG Packetized elementary stream packet |<sub></sub>|
|
||
|`mpeg_spu` |Sub Picture Unit (DVD subtitle) |<sub></sub>|
|
||
|`mpeg_ts` |MPEG Transport Stream |<sub></sub>|
|
||
|[`msgpack`](#msgpack) |MessagePack |<sub></sub>|
|
||
|`ogg` |OGG file |<sub>`ogg_page` `vorbis_packet` `opus_packet` `flac_metadatablock` `flac_frame`</sub>|
|
||
|`ogg_page` |OGG page |<sub></sub>|
|
||
|`opus_packet` |Opus packet |<sub>`vorbis_comment`</sub>|
|
||
|`pcap` |PCAP packet capture |<sub>`link_frame` `tcp_stream` `ipv4_packet`</sub>|
|
||
|`pcapng` |PCAPNG packet capture |<sub>`link_frame` `tcp_stream` `ipv4_packet`</sub>|
|
||
|`png` |Portable Network Graphics file |<sub>`icc_profile` `exif`</sub>|
|
||
|`prores_frame` |Apple ProRes frame |<sub></sub>|
|
||
|[`protobuf`](#protobuf) |Protobuf |<sub></sub>|
|
||
|`protobuf_widevine` |Widevine protobuf |<sub>`protobuf`</sub>|
|
||
|`pssh_playready` |PlayReady PSSH |<sub></sub>|
|
||
|`raw` |Raw bits |<sub></sub>|
|
||
|[`rtmp`](#rtmp) |Real-Time Messaging Protocol |<sub>`amf0` `mpeg_asc`</sub>|
|
||
|`sll2_packet` |Linux cooked capture encapsulation v2 |<sub>`inet_packet`</sub>|
|
||
|`sll_packet` |Linux cooked capture encapsulation |<sub>`inet_packet`</sub>|
|
||
|`tar` |Tar archive |<sub>`probe`</sub>|
|
||
|`tcp_segment` |Transmission control protocol segment |<sub></sub>|
|
||
|`tiff` |Tag Image File Format |<sub>`icc_profile`</sub>|
|
||
|`toml` |Tom's Obvious, Minimal Language |<sub></sub>|
|
||
|`udp_datagram` |User datagram protocol |<sub>`udp_payload`</sub>|
|
||
|`vorbis_comment` |Vorbis comment |<sub>`flac_picture`</sub>|
|
||
|`vorbis_packet` |Vorbis packet |<sub>`vorbis_comment`</sub>|
|
||
|`vp8_frame` |VP8 frame |<sub></sub>|
|
||
|`vp9_cfm` |VP9 Codec Feature Metadata |<sub></sub>|
|
||
|`vp9_frame` |VP9 frame |<sub></sub>|
|
||
|`vpx_ccr` |VPX Codec Configuration Record |<sub></sub>|
|
||
|[`wasm`](#wasm) |WebAssembly Binary Format |<sub></sub>|
|
||
|`wav` |WAV file |<sub>`id3v2` `id3v1` `id3v11`</sub>|
|
||
|`webp` |WebP image |<sub>`vp8_frame`</sub>|
|
||
|`xing` |Xing header |<sub></sub>|
|
||
|[`xml`](#xml) |Extensible Markup Language |<sub></sub>|
|
||
|`yaml` |YAML Ain't Markup Language |<sub></sub>|
|
||
|[`zip`](#zip) |ZIP archive |<sub>`probe`</sub>|
|
||
|`image` |Group |<sub>`gif` `jpeg` `mp4` `png` `tiff` `webp`</sub>|
|
||
|`inet_packet` |Group |<sub>`ipv4_packet` `ipv6_packet`</sub>|
|
||
|`ip_packet` |Group |<sub>`icmp` `icmpv6` `tcp_segment` `udp_datagram`</sub>|
|
||
|`link_frame` |Group |<sub>`bsd_loopback_frame` `ether8023_frame` `sll2_packet` `sll_packet`</sub>|
|
||
|`probe` |Group |<sub>`adts` `ar` `avro_ocf` `bitcoin_blkdat` `bplist` `bzip2` `elf` `flac` `gif` `gzip` `jpeg` `json` `jsonl` `macho` `macho_fat` `matroska` `mp3` `mp4` `mpeg_ts` `ogg` `pcap` `pcapng` `png` `tar` `tiff` `toml` `wasm` `wav` `webp` `xml` `yaml` `zip`</sub>|
|
||
|`tcp_stream` |Group |<sub>`dns_tcp` `rtmp`</sub>|
|
||
|`udp_payload` |Group |<sub>`dns`</sub>|
|
||
|
||
[#]: sh-end
|
||
|
||
## Global format options
|
||
|
||
Currently the only global option is `force` and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI `-o` option:
|
||
|
||
```
|
||
fq -d mp4 -o force=true file.mp4
|
||
fq -d raw 'mp4({force: true})' file.mp4
|
||
```
|
||
|
||
## Format details
|
||
|
||
[fq -rn -L . 'include "formats"; formats_sections']: sh-start
|
||
|
||
## aac_frame
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`object_type`|1 |Audio object type|
|
||
|
||
### Examples
|
||
|
||
Decode file using aac_frame options
|
||
```
|
||
$ fq -d aac_frame -o object_type=1 . file
|
||
```
|
||
|
||
Decode value as aac_frame
|
||
```
|
||
... | aac_frame({object_type:1})
|
||
```
|
||
|
||
## asn1_ber
|
||
|
||
Supports decoding BER, CER and DER (X.690).
|
||
|
||
- Currently no extra validation is done for CER and DER.
|
||
- Does not support specifying a schema.
|
||
- Supports `torepr` but without schema all sequences and sets will be arrays.
|
||
|
||
### Can be used to decode certificates etc
|
||
|
||
```sh
|
||
$ fq -d raw 'frompem | asn1_ber | d' cert.pem
|
||
```
|
||
|
||
### Can decode nested values
|
||
|
||
```sh
|
||
$ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber
|
||
```
|
||
|
||
### Manual schema
|
||
|
||
```sh
|
||
$ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der
|
||
```
|
||
|
||
### References
|
||
- https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf
|
||
- https://en.wikipedia.org/wiki/X.690
|
||
- https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
|
||
- https://lapo.it/asn1js/
|
||
|
||
## avc_au
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`length_size`|4 |Length value size|
|
||
|
||
### Examples
|
||
|
||
Decode file using avc_au options
|
||
```
|
||
$ fq -d avc_au -o length_size=4 . file
|
||
```
|
||
|
||
Decode value as avc_au
|
||
```
|
||
... | avc_au({length_size:4})
|
||
```
|
||
|
||
## avro_ocf
|
||
|
||
Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.
|
||
|
||
Capable of handling null, deflate, and snappy codecs for data compression.
|
||
|
||
Limitations:
|
||
|
||
- Schema does not support self-referential types, only built-in types.
|
||
- Decimal logical types are not supported for decoding, will just be treated as their primitive type
|
||
|
||
### References
|
||
- https://avro.apache.org/docs/current/spec.html#Object+Container+Files
|
||
|
||
### Authors
|
||
- Xentripetal
|
||
xentripetal@fastmail.com
|
||
[@xentripetal](https://github.com/xentripetal)
|
||
|
||
## bencode
|
||
|
||
### Convert represented value to JSON
|
||
|
||
```
|
||
$ fq -d bencode torepr file.torrent
|
||
```
|
||
|
||
### References
|
||
- https://wiki.theory.org/BitTorrentSpecification#Bencoding
|
||
|
||
## bitcoin_block
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`has_header`|false |Has blkdat header|
|
||
|
||
### Examples
|
||
|
||
Decode file using bitcoin_block options
|
||
```
|
||
$ fq -d bitcoin_block -o has_header=false . file
|
||
```
|
||
|
||
Decode value as bitcoin_block
|
||
```
|
||
... | bitcoin_block({has_header:false})
|
||
```
|
||
|
||
## bplist
|
||
|
||
### Show full decoding
|
||
```sh
|
||
$ fq -d bplist dv Info.plist
|
||
```
|
||
|
||
### Timestamps
|
||
Timestamps in Apple Binary Property Lists are encoded as Cocoa Core Data
|
||
timestamps, where the raw value is the floating point number of seconds since
|
||
January 1, 2001. By default, `fq` will render the raw floating point value. In
|
||
order to get the raw value or string description, use the `todescription`
|
||
function, you can use the `tovalue` and `todescription` functions:
|
||
|
||
```sh
|
||
$ fq 'torepr.SomeTimeStamp | tovalue' Info.plist
|
||
685135328
|
||
|
||
$ fq 'torepr.SomeTimeStamp | todescription' Info.plist
|
||
"2022-09-17T19:22:08Z"
|
||
```
|
||
|
||
|
||
### Get JSON representation
|
||
```sh
|
||
$ fq torepr com.apple.UIAutomation.plist
|
||
{
|
||
"UIAutomationEnabled": true
|
||
}
|
||
```
|
||
|
||
### Authors
|
||
- David McDonald
|
||
[@dgmcdona](https://github.com/dgmcdona)
|
||
|
||
### References
|
||
- http://fileformats.archiveteam.org/wiki/Property_List/Binary
|
||
- https://medium.com/@karaiskc/understanding-apples-binary-property-list-format-281e6da00dbd
|
||
- https://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c
|
||
|
||
## bson
|
||
|
||
### Convert represented value to JSON
|
||
|
||
```
|
||
$ fq -d bson torepr file.bson
|
||
```
|
||
|
||
### Filter represented value
|
||
|
||
```
|
||
$ fq -d bson 'torepr | select(.name=="bob")' file.bson
|
||
```
|
||
|
||
### References
|
||
- https://bsonspec.org/spec.html
|
||
|
||
## cbor
|
||
|
||
### Convert represented value to JSON
|
||
|
||
```
|
||
$ fq -d cbor torepr file.cbor
|
||
```
|
||
|
||
### References
|
||
- https://en.wikipedia.org/wiki/CBOR
|
||
- https://www.rfc-editor.org/rfc/rfc8949.html
|
||
|
||
## csv
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`comma` |, |Separator character|
|
||
|`comment`|# |Comment line character|
|
||
|
||
### Examples
|
||
|
||
Decode file using csv options
|
||
```
|
||
$ fq -d csv -o comma="," -o comment="#" . file
|
||
```
|
||
|
||
Decode value as csv
|
||
```
|
||
... | csv({comma:",",comment:"#"})
|
||
```
|
||
|
||
### TSV to CSV
|
||
|
||
```sh
|
||
$ fq -d csv -o comma="\t" tocsv file.tsv
|
||
```
|
||
|
||
### Convert rows to objects based on header row
|
||
|
||
```sh
|
||
$ fq -d csv '.[0] as $t | .[1:] | map(with_entries(.key = $t[.key]))' file.csv
|
||
```
|
||
|
||
## flac_frame
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`bits_per_sample`|16 |Bits per sample|
|
||
|
||
### Examples
|
||
|
||
Decode file using flac_frame options
|
||
```
|
||
$ fq -d flac_frame -o bits_per_sample=16 . file
|
||
```
|
||
|
||
Decode value as flac_frame
|
||
```
|
||
... | flac_frame({bits_per_sample:16})
|
||
```
|
||
|
||
## hevc_au
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`length_size`|4 |Length value size|
|
||
|
||
### Examples
|
||
|
||
Decode file using hevc_au options
|
||
```
|
||
$ fq -d hevc_au -o length_size=4 . file
|
||
```
|
||
|
||
Decode value as hevc_au
|
||
```
|
||
... | hevc_au({length_size:4})
|
||
```
|
||
|
||
## html
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`array` |false |Decode as nested arrays|
|
||
|`attribute_prefix`|@ |Prefix for attribute keys|
|
||
|`seq` |false |Use seq attribute to preserve element order|
|
||
|
||
### Examples
|
||
|
||
Decode file using html options
|
||
```
|
||
$ fq -d html -o array=false -o attribute_prefix="@" -o seq=false . file
|
||
```
|
||
|
||
Decode value as html
|
||
```
|
||
... | html({array:false,attribute_prefix:"@",seq:false})
|
||
```
|
||
|
||
HTML is decoded in HTML5 mode and will always include `<html>`, `<body>` and `<head>` element.
|
||
|
||
See xml format for more examples and how to preserve element order and how to encode to xml.
|
||
|
||
There is no `tohtml` function, see `toxml` instead.
|
||
|
||
### Element as object
|
||
|
||
```sh
|
||
# decode as object is the default
|
||
$ echo '<a href="url">text</a>' | fq -d html
|
||
{
|
||
"html": {
|
||
"body": {
|
||
"a": {
|
||
"#text": "text",
|
||
"@href": "url"
|
||
}
|
||
},
|
||
"head": ""
|
||
}
|
||
}
|
||
```
|
||
|
||
### Element as array
|
||
|
||
```sh
|
||
$ '<a href="url">text</a>' | fq -d html -o array=true
|
||
[
|
||
"html",
|
||
null,
|
||
[
|
||
[
|
||
"head",
|
||
null,
|
||
[]
|
||
],
|
||
[
|
||
"body",
|
||
null,
|
||
[
|
||
[
|
||
"a",
|
||
{
|
||
"#text": "text",
|
||
"href": "url"
|
||
},
|
||
[]
|
||
]
|
||
]
|
||
]
|
||
]
|
||
]
|
||
|
||
# decode html files to a {file: "title", ...} object
|
||
$ fq -n -d html '[inputs | {key: input_filename, value: .html.head.title?}] | from_entries' *.html
|
||
|
||
# <a> href:s in file
|
||
$ fq -r -o array=true -d html '.. | select(.[0] == "a" and .[1].href)?.[1].href' file.html
|
||
```
|
||
|
||
## macho
|
||
|
||
Supports decoding vanilla and FAT Mach-O binaries.
|
||
|
||
### Select 64bit load segments
|
||
|
||
```sh
|
||
$ fq '.load_commands[] | select(.cmd=="segment_64")' file
|
||
```
|
||
|
||
### References
|
||
- https://github.com/aidansteele/osx-abi-macho-file-format-reference
|
||
|
||
### Authors
|
||
- Sıddık AÇIL
|
||
acils@itu.edu.tr
|
||
[@Akaame](https://github.com/Akaame)
|
||
|
||
## matroska
|
||
|
||
### Lookup element using path
|
||
|
||
```sh
|
||
$ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv
|
||
```
|
||
|
||
### Get path to element
|
||
|
||
```sh
|
||
$ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv
|
||
```
|
||
|
||
### References
|
||
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
|
||
- https://matroska.org/technical/specs/index.html
|
||
- https://www.matroska.org/technical/basics.html
|
||
- https://www.matroska.org/technical/codec_specs.html
|
||
- https://wiki.xiph.org/MatroskaOpus
|
||
|
||
## mp3
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`max_sync_seek` |32768 |Max byte distance to next sync|
|
||
|`max_unique_header_configs`|5 |Max number of unique frame header configs allowed|
|
||
|
||
### Examples
|
||
|
||
Decode file using mp3 options
|
||
```
|
||
$ fq -d mp3 -o max_sync_seek=32768 -o max_unique_header_configs=5 . file
|
||
```
|
||
|
||
Decode value as mp3
|
||
```
|
||
... | mp3({max_sync_seek:32768,max_unique_header_configs:5})
|
||
```
|
||
|
||
## mp4
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`allow_truncated`|false |Allow box to be truncated|
|
||
|`decode_samples` |true |Decode supported media samples|
|
||
|
||
### Examples
|
||
|
||
Decode file using mp4 options
|
||
```
|
||
$ fq -d mp4 -o allow_truncated=false -o decode_samples=true . file
|
||
```
|
||
|
||
Decode value as mp4
|
||
```
|
||
... | mp4({allow_truncated:false,decode_samples:true})
|
||
```
|
||
|
||
### Lookup mp4 box using a mp4 box path.
|
||
|
||
```sh
|
||
# <decode value box> | mp4_path($path) -> <decode value box>
|
||
$ fq 'mp4_path(".moov.trak[1]")' file.mp4
|
||
```
|
||
|
||
### Get mp4 box path for a decode value box.
|
||
|
||
```sh
|
||
# <decode value box> | mp4_path -> string
|
||
$ fq 'grep_by(.type == "trak") | mp4_path' file.mp4
|
||
```
|
||
|
||
### Force decode a single box
|
||
|
||
```sh
|
||
$ fq -n '"AAAAHGVsc3QAAAAAAAAAAQAAADIAAAQAAAEAAA==" | frombase64 | mp4({force:true}) | d'
|
||
```
|
||
|
||
### Speed up decoding by not decoding samples
|
||
|
||
```sh
|
||
# manually decode first sample as a aac_frame
|
||
$ fq -o decode_samples=false '.tracks[0].samples[0] | aac_frame | d' file.mp4
|
||
```
|
||
|
||
### Entries for first edit list as values
|
||
|
||
```sh
|
||
$ fq 'first(grep_by(.type=="elst").entries) | tovalue' file.mp4
|
||
```
|
||
|
||
### References
|
||
|
||
- [ISO/IEC base media file format (MPEG-4 Part 12)](https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format)
|
||
- [Quicktime file format](https://developer.apple.com/standards/qtff-2001.pdf)
|
||
|
||
## msgpack
|
||
|
||
### Convert represented value to JSON
|
||
|
||
```
|
||
$ fq -d msgpack torepr file.msgpack
|
||
```
|
||
|
||
### References
|
||
- https://github.com/msgpack/msgpack/blob/master/spec.md
|
||
|
||
## protobuf
|
||
|
||
### Can decode sub messages
|
||
|
||
```sh
|
||
$ fq -d protobuf '.fields[6].wire_value | protobuf | d' file
|
||
```
|
||
|
||
### References
|
||
- https://developers.google.com/protocol-buffers/docs/encoding
|
||
|
||
## rtmp
|
||
|
||
Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).
|
||
|
||
### References
|
||
- https://rtmp.veriskope.com/docs/spec/
|
||
- https://rtmp.veriskope.com/pdf/video_file_format_spec_v10.pdf
|
||
|
||
## wasm
|
||
|
||
### Count opcode usage
|
||
```sh
|
||
$ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' file.wasm
|
||
```
|
||
|
||
### List exports and imports
|
||
```sh
|
||
$ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' file.wasm
|
||
```
|
||
|
||
### Authors
|
||
- Takashi Oguma
|
||
[@bitbears-dev](https://github.com/bitbears-dev)
|
||
[@0xb17bea125](https://twitter.com/0xb17bea125)
|
||
|
||
### References
|
||
- https://webassembly.github.io/spec/core/
|
||
|
||
## xml
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`array` |false |Decode as nested arrays|
|
||
|`attribute_prefix`|@ |Prefix for attribute keys|
|
||
|`seq` |false |Use seq attribute to preserve element order|
|
||
|
||
### Examples
|
||
|
||
Decode file using xml options
|
||
```
|
||
$ fq -d xml -o array=false -o attribute_prefix="@" -o seq=false . file
|
||
```
|
||
|
||
Decode value as xml
|
||
```
|
||
... | xml({array:false,attribute_prefix:"@",seq:false})
|
||
```
|
||
|
||
XML can be decoded and encoded into jq values in two ways, elements as object or array.
|
||
Which variant to use depends a bit what you want to do. The object variant might be easier
|
||
to query for a specific value but array might be easier to use to generate xml or to query
|
||
after all elements of some kind etc.
|
||
|
||
Encoding is done using the `toxml` function and it will figure what variant that is used based on the input value.
|
||
Is has two optional options `indent` and `attribute_prefix`.
|
||
|
||
### Elements as object
|
||
|
||
Element can have different shapes depending on body text, attributes and children:
|
||
|
||
- `<a key="value">text</a>` is `{"a":{"#text":"text","@key":"value"}}`, has text (`#text`) and attributes (`@key`)
|
||
- `<a>text</a>` is `{"a":"text"}`
|
||
- `<a><b>text</b></a>` is `{"a":{"b":"text"}}` one child with only text and no attributes
|
||
- `<a><b/><b>text</b></a>` is `{"a":{"b":["","text"]}}` two children with same name end up in an array
|
||
- `<a><b/><b key="value">text</b></a>` is `{"a":{"b":["",{"#text":"text","@key":"value"}]}}`
|
||
|
||
If there is `#seq` attribute it encodes the child element order. Use `-o seq=true` to include sequence number when decoding,
|
||
otherwise order might be lost.
|
||
|
||
```sh
|
||
# decode as object is the default
|
||
$ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -d xml -o seq=true
|
||
{
|
||
"a": {
|
||
"b": [
|
||
{
|
||
"#seq": 0
|
||
},
|
||
{
|
||
"#seq": 1,
|
||
"#text": "bbb"
|
||
}
|
||
],
|
||
"c": {
|
||
"#seq": 2,
|
||
"#text": "ccc",
|
||
"@attr": "value"
|
||
}
|
||
}
|
||
}
|
||
|
||
# access text of the <c> element
|
||
$ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq '.a.c["#text"]'
|
||
"ccc"
|
||
|
||
# decode to object and encode to xml
|
||
$ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -r -d xml -o seq=true 'toxml({indent:2})'
|
||
<a>
|
||
<b></b>
|
||
<b>bbb</b>
|
||
<c attr="value">ccc</c>
|
||
</a>
|
||
```
|
||
|
||
### Elements as array
|
||
|
||
Elements are arrays of the shape `["#text": "body text", "attr_name", {key: "attr value"}|null, [<child element>, ...]]`.
|
||
|
||
```sh
|
||
# decode as array
|
||
$ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -d xml -o array=true
|
||
[
|
||
"a",
|
||
null,
|
||
[
|
||
[
|
||
"b",
|
||
null,
|
||
[]
|
||
],
|
||
[
|
||
"b",
|
||
{
|
||
"#text": "bbb"
|
||
},
|
||
[]
|
||
],
|
||
[
|
||
"c",
|
||
{
|
||
"#text": "ccc",
|
||
"attr": "value"
|
||
},
|
||
[]
|
||
]
|
||
]
|
||
]
|
||
|
||
# decode to array and encode to xml
|
||
$ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -r -d xml -o array=true -o seq=true 'toxml({indent:2})'
|
||
<a>
|
||
<b></b>
|
||
<b>bbb</b>
|
||
<c attr="value">ccc</c>
|
||
</a>
|
||
|
||
# access text of the <c> element, the object variant above is probably easier to use
|
||
$ echo '<a><b/><b>bbb</b><c attr="value">ccc</c></a>' | fq -o array=true '.[2][2][1]["#text"]'
|
||
"ccc"
|
||
```
|
||
|
||
### References
|
||
- [xml.com's Converting Between XML and JSON](https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html)
|
||
|
||
## zip
|
||
|
||
### Options
|
||
|
||
|Name |Default|Description|
|
||
|- |- |-|
|
||
|`uncompress`|true |Uncompress and probe files|
|
||
|
||
### Examples
|
||
|
||
Decode file using zip options
|
||
```
|
||
$ fq -d zip -o uncompress=true . file
|
||
```
|
||
|
||
Decode value as zip
|
||
```
|
||
... | zip({uncompress:true})
|
||
```
|
||
|
||
Supports ZIP64.
|
||
|
||
### References
|
||
- https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
|
||
- https://opensource.apple.com/source/zip/zip-6/unzip/unzip/proginfo/extra.fld
|
||
|
||
|
||
[#]: sh-end
|
||
|
||
## Dependency graph
|
||
|
||
|