2021-09-19 11:27:56 +03:00
## Supported formats
2021-12-09 19:15:21 +03:00
[fq -rn -L . 'include "formats"; formats_table']: sh-start
2022-12-06 07:03:26 +03:00
|Name |Description |Dependencies|
|- |- |-|
|[`aac_frame`](#aac_frame) |Advanced Audio Coding frame |< sub > < / sub > |
|`adts` |Audio Data Transport Stream |< sub > `adts_frame`< / sub > |
|`adts_frame` |Audio Data Transport Stream frame |< sub > `aac_frame`< / sub > |
2023-03-09 16:02:59 +03:00
|`aiff` |Audio Interchange File Format |< sub > < / sub > |
2022-12-06 07:03:26 +03:00
|`amf0` |Action Message Format 0 |< sub > < / sub > |
|`apev2` |APEv2 metadata tag |< sub > `image`< / sub > |
|[`apple_bookmark`](#apple_bookmark) |Apple BookmarkData |< sub > < / sub > |
|`ar` |Unix archive |< sub > `probe`< / sub > |
|[`asn1_ber`](#asn1_ber) |ASN1 BER (basic encoding rules, also CER and DER) |< sub > < / sub > |
|`av1_ccr` |AV1 Codec Configuration Record |< sub > < / sub > |
|`av1_frame` |AV1 frame |< sub > `av1_obu`< / sub > |
|`av1_obu` |AV1 Open Bitstream Unit |< sub > < / sub > |
|`avc_annexb` |H.264/AVC Annex B |< sub > `avc_nalu`< / sub > |
|[`avc_au`](#avc_au) |H.264/AVC Access Unit |< sub > `avc_nalu`< / sub > |
|`avc_dcr` |H.264/AVC Decoder Configuration Record |< sub > `avc_nalu`< / sub > |
|`avc_nalu` |H.264/AVC Network Access Layer Unit |< sub > `avc_sps` `avc_pps` `avc_sei` </ sub > |
|`avc_pps` |H.264/AVC Picture Parameter Set |< sub > < / sub > |
|`avc_sei` |H.264/AVC Supplemental Enhancement Information |< sub > < / sub > |
|`avc_sps` |H.264/AVC Sequence Parameter Set |< sub > < / sub > |
|[`avi`](#avi) |Audio Video Interleaved |< sub > `avc_au` `hevc_au` `mp3_frame` `flac_frame` </ sub > |
|[`avro_ocf`](#avro_ocf) |Avro object container file |< sub > < / sub > |
|[`bencode`](#bencode) |BitTorrent bencoding |< sub > < / sub > |
|`bitcoin_blkdat` |Bitcoin blk.dat |< sub > `bitcoin_block`< / sub > |
|[`bitcoin_block`](#bitcoin_block) |Bitcoin block |< sub > `bitcoin_transaction`< / sub > |
|`bitcoin_script` |Bitcoin script |< sub > < / sub > |
|`bitcoin_transaction` |Bitcoin transaction |< sub > `bitcoin_script`< / sub > |
|[`bits`](#bits) |Raw bits |< sub > < / sub > |
|[`bplist`](#bplist) |Apple Binary Property List |< sub > < / sub > |
|`bsd_loopback_frame` |BSD loopback frame |< sub > `inet_packet`< / sub > |
|[`bson`](#bson) |Binary JSON |< sub > < / sub > |
|[`bytes`](#bytes) |Raw bytes |< sub > < / sub > |
|`bzip2` |bzip2 compression |< sub > `probe`< / sub > |
2023-08-18 21:04:57 +03:00
|[`caff`](#caff) |Live2D Cubism archive |< sub > `probe`< / sub > |
2022-12-06 07:03:26 +03:00
|[`cbor`](#cbor) |Concise Binary Object Representation |< sub > < / sub > |
|[`csv`](#csv) |Comma separated values |< sub > < / sub > |
|`dns` |DNS packet |< sub > < / sub > |
|`dns_tcp` |DNS packet (TCP) |< sub > < / sub > |
|`elf` |Executable and Linkable Format |< sub > < / sub > |
|`ether8023_frame` |Ethernet 802.3 frame |< sub > `inet_packet`< / sub > |
|`exif` |Exchangeable Image File Format |< sub > < / sub > |
|`fairplay_spc` |FairPlay Server Playback Context |< sub > < / sub > |
|`flac` |Free Lossless Audio Codec file |< sub > `flac_metadatablocks` `flac_frame` </ sub > |
|[`flac_frame`](#flac_frame) |FLAC frame |< sub > < / sub > |
|`flac_metadatablock` |FLAC metadatablock |< sub > `flac_streaminfo` `flac_picture` `vorbis_comment` </ sub > |
|`flac_metadatablocks` |FLAC metadatablocks |< sub > `flac_metadatablock`< / sub > |
|`flac_picture` |FLAC metadatablock picture |< sub > `image`< / sub > |
|`flac_streaminfo` |FLAC streaminfo |< sub > < / sub > |
|`gif` |Graphics Interchange Format |< sub > < / sub > |
|`gzip` |gzip compression |< sub > `probe`< / sub > |
|`hevc_annexb` |H.265/HEVC Annex B |< sub > `hevc_nalu`< / sub > |
|[`hevc_au`](#hevc_au) |H.265/HEVC Access Unit |< sub > `hevc_nalu`< / sub > |
|`hevc_dcr` |H.265/HEVC Decoder Configuration Record |< sub > `hevc_nalu`< / sub > |
|`hevc_nalu` |H.265/HEVC Network Access Layer Unit |< sub > `hevc_vps` `hevc_pps` `hevc_sps` </ sub > |
|`hevc_pps` |H.265/HEVC Picture Parameter Set |< sub > < / sub > |
|`hevc_sps` |H.265/HEVC Sequence Parameter Set |< sub > < / sub > |
|`hevc_vps` |H.265/HEVC Video Parameter Set |< sub > < / sub > |
|[`html`](#html) |HyperText Markup Language |< sub > < / sub > |
|`icc_profile` |International Color Consortium profile |< sub > < / sub > |
|`icmp` |Internet Control Message Protocol |< sub > < / sub > |
|`icmpv6` |Internet Control Message Protocol v6 |< sub > < / sub > |
|`id3v1` |ID3v1 metadata |< sub > < / sub > |
|`id3v11` |ID3v1.1 metadata |< sub > < / sub > |
|`id3v2` |ID3v2 metadata |< sub > `image`< / sub > |
|`ipv4_packet` |Internet protocol v4 packet |< sub > `ip_packet`< / sub > |
|`ipv6_packet` |Internet protocol v6 packet |< sub > `ip_packet`< / sub > |
|`jpeg` |Joint Photographic Experts Group file |< sub > `exif` `icc_profile` </ sub > |
|`json` |JavaScript Object Notation |< sub > < / sub > |
|`jsonl` |JavaScript Object Notation Lines |< sub > < / sub > |
2023-06-23 00:54:49 +03:00
|[`luajit`](#luajit) |LuaJIT 2.0 bytecode |< sub > < / sub > |
2022-12-06 07:03:26 +03:00
|[`macho`](#macho) |Mach-O macOS executable |< sub > < / sub > |
|`macho_fat` |Fat Mach-O macOS executable (multi-architecture) |< sub > `macho`< / sub > |
|[`markdown`](#markdown) |Markdown |< sub > < / sub > |
|[`matroska`](#matroska) |Matroska file |< sub > `aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `image` `mp3_frame` `mpeg_asc` `mpeg_pes_packet` `mpeg_spu` `opus_packet` `vorbis_packet` `vp8_frame` `vp9_cfm` `vp9_frame` </ sub > |
2023-08-21 10:37:44 +03:00
|[`moc3`](#moc3) |MOC3 file |< sub > < / sub > |
2022-12-06 07:03:26 +03:00
|[`mp3`](#mp3) |MP3 file |< sub > `id3v2` `id3v1` `id3v11` `apev2` `mp3_frame` </ sub > |
|`mp3_frame` |MPEG audio layer 3 frame |< sub > `mp3_frame_tags`< / sub > |
2022-12-15 13:56:50 +03:00
|`mp3_frame_vbri` |MP3 frame Fraunhofer encoder variable bitrate tag |< sub > < / sub > |
|`mp3_frame_xing` |MP3 frame Xing/Info tag |< sub > < / sub > |
2022-12-06 07:03:26 +03:00
|[`mp4`](#mp4) |ISOBMFF, QuickTime and similar |< sub > `aac_frame` `av1_ccr` `av1_frame` `avc_au` `avc_dcr` `flac_frame` `flac_metadatablocks` `hevc_au` `hevc_dcr` `icc_profile` `id3v2` `image` `jpeg` `mp3_frame` `mpeg_es` `mpeg_pes_packet` `opus_packet` `png` `prores_frame` `protobuf_widevine` `pssh_playready` `vorbis_packet` `vp9_frame` `vpx_ccr` </ sub > |
|`mpeg_asc` |MPEG-4 Audio Specific Config |< sub > < / sub > |
|`mpeg_es` |MPEG Elementary Stream |< sub > `mpeg_asc` `vorbis_packet` </ sub > |
|`mpeg_pes` |MPEG Packetized elementary stream |< sub > `mpeg_pes_packet` `mpeg_spu` </ sub > |
|`mpeg_pes_packet` |MPEG Packetized elementary stream packet |< sub > < / sub > |
|`mpeg_spu` |Sub Picture Unit (DVD subtitle) |< sub > < / sub > |
|`mpeg_ts` |MPEG Transport Stream |< sub > < / sub > |
|[`msgpack`](#msgpack) |MessagePack |< sub > < / sub > |
|`ogg` |OGG file |< sub > `ogg_page` `vorbis_packet` `opus_packet` `flac_metadatablock` `flac_frame` </ sub > |
|`ogg_page` |OGG page |< sub > < / sub > |
2023-09-26 21:42:02 +03:00
|[`opentimestamps`](#opentimestamps) |OpenTimestamps file |< sub > < / sub > |
2022-12-06 07:03:26 +03:00
|`opus_packet` |Opus packet |< sub > `vorbis_comment`< / sub > |
|[`pcap`](#pcap) |PCAP packet capture |< sub > `link_frame` `tcp_stream` `ipv4_packet` </ sub > |
|`pcapng` |PCAPNG packet capture |< sub > `link_frame` `tcp_stream` `ipv4_packet` </ sub > |
2023-05-04 08:34:32 +03:00
|[`pg_btree`](#pg_btree) |PostgreSQL btree index file |< sub > < / sub > |
|[`pg_control`](#pg_control) |PostgreSQL control file |< sub > < / sub > |
|[`pg_heap`](#pg_heap) |PostgreSQL heap file |< sub > < / sub > |
2022-12-06 07:03:26 +03:00
|`png` |Portable Network Graphics file |< sub > `icc_profile` `exif` </ sub > |
|`prores_frame` |Apple ProRes frame |< sub > < / sub > |
|[`protobuf`](#protobuf) |Protobuf |< sub > < / sub > |
|`protobuf_widevine` |Widevine protobuf |< sub > `protobuf`< / sub > |
|`pssh_playready` |PlayReady PSSH |< sub > < / sub > |
|[`rtmp`](#rtmp) |Real-Time Messaging Protocol |< sub > `amf0` `mpeg_asc` </ sub > |
|`sll2_packet` |Linux cooked capture encapsulation v2 |< sub > `inet_packet`< / sub > |
|`sll_packet` |Linux cooked capture encapsulation |< sub > `inet_packet`< / sub > |
|`tar` |Tar archive |< sub > `probe`< / sub > |
|`tcp_segment` |Transmission control protocol segment |< sub > < / sub > |
|`tiff` |Tag Image File Format |< sub > `icc_profile`< / sub > |
2023-02-10 22:06:38 +03:00
|[`tls`](#tls) |Transport layer security |< sub > `asn1_ber`< / sub > |
2022-12-06 07:03:26 +03:00
|`toml` |Tom's Obvious, Minimal Language |< sub > < / sub > |
|[`tzif`](#tzif) |Time Zone Information Format |< sub > < / sub > |
|`udp_datagram` |User datagram protocol |< sub > `udp_payload`< / sub > |
|`vorbis_comment` |Vorbis comment |< sub > `flac_picture`< / sub > |
|`vorbis_packet` |Vorbis packet |< sub > `vorbis_comment`< / sub > |
|`vp8_frame` |VP8 frame |< sub > < / sub > |
|`vp9_cfm` |VP9 Codec Feature Metadata |< sub > < / sub > |
|`vp9_frame` |VP9 frame |< sub > < / sub > |
|`vpx_ccr` |VPX Codec Configuration Record |< sub > < / sub > |
|[`wasm`](#wasm) |WebAssembly Binary Format |< sub > < / sub > |
|`wav` |WAV file |< sub > `id3v2` `id3v1` `id3v11` </ sub > |
2023-11-01 19:31:32 +03:00
|`webp` |WebP image |< sub > `exif` `vp8_frame` `icc_profile` `xml` </ sub > |
2022-12-06 07:03:26 +03:00
|[`xml`](#xml) |Extensible Markup Language |< sub > < / sub > |
|`yaml` |YAML Ain't Markup Language |< sub > < / sub > |
|[`zip`](#zip) |ZIP archive |< sub > `probe`< / sub > |
|`image` |Group |< sub > `gif` `jpeg` `mp4` `png` `tiff` `webp` </ sub > |
|`inet_packet` |Group |< sub > `ipv4_packet` `ipv6_packet` </ sub > |
|`ip_packet` |Group |< sub > `icmp` `icmpv6` `tcp_segment` `udp_datagram` </ sub > |
2023-02-26 23:41:46 +03:00
|`link_frame` |Group |< sub > `bsd_loopback_frame` `ether8023_frame` `ipv4_packet` `ipv6_packet` `sll2_packet` `sll_packet` </ sub > |
2022-12-15 13:56:50 +03:00
|`mp3_frame_tags` |Group |< sub > `mp3_frame_vbri` `mp3_frame_xing` </ sub > |
2023-09-26 18:32:40 +03:00
|`probe` |Group |< sub > `adts` `aiff` `apple_bookmark` `ar` `avi` `avro_ocf` `bitcoin_blkdat` `bplist` `bzip2` `caff` `elf` `flac` `gif` `gzip` `html` `jpeg` `json` `jsonl` `luajit` `macho` `macho_fat` `matroska` `moc3` `mp3` `mp4` `mpeg_ts` `ogg` `opentimestamps` `pcap` `pcapng` `png` `tar` `tiff` `toml` `tzif` `wasm` `wav` `webp` `xml` `yaml` `zip` </ sub > |
2023-02-10 22:06:38 +03:00
|`tcp_stream` |Group |< sub > `dns_tcp` `rtmp` `tls` </ sub > |
2022-12-06 07:03:26 +03:00
|`udp_payload` |Group |< sub > `dns`< / sub > |
2021-09-19 11:27:56 +03:00
[#]: sh-end
2021-12-09 19:15:21 +03:00
## Global format options
2022-02-05 20:15:18 +03:00
2021-12-09 19:15:21 +03:00
Currently the only global option is `force` and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI `-o` option:
2022-02-05 20:15:18 +03:00
```
fq -d mp4 -o force=true file.mp4
2022-11-20 21:22:16 +03:00
fq -d bytes 'mp4({force: true})' file.mp4
2022-02-05 20:15:18 +03:00
```
2022-02-19 03:33:45 +03:00
## Format details
2022-01-29 14:01:36 +03:00
2021-12-09 19:15:21 +03:00
[fq -rn -L . 'include "formats"; formats_sections']: sh-start
2022-09-11 10:55:30 +03:00
## aac_frame
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Options
2021-12-09 19:15:21 +03:00
|Name |Default|Description|
|- |- |-|
|`object_type`|1 |Audio object type|
2022-09-11 10:55:30 +03:00
### Examples
2021-12-09 19:15:21 +03:00
2022-05-26 13:51:54 +03:00
Decode file using aac_frame options
2021-12-09 19:15:21 +03:00
```
2022-05-26 13:51:54 +03:00
$ fq -d aac_frame -o object_type=1 . file
2021-12-09 19:15:21 +03:00
```
Decode value as aac_frame
```
2022-06-13 19:49:30 +03:00
... | aac_frame({object_type:1})
2021-12-09 19:15:21 +03:00
```
2022-01-29 14:01:36 +03:00
2022-11-28 01:43:19 +03:00
## apple_bookmark
Apple's `bookmarkData` format is used to encode information that can be resolved
into a `URL` object for a file even if the user moves or renames it. Can also
contain security scoping information for App Sandbox support.
2023-06-23 00:54:49 +03:00
These `bookmarkData` blobs are often found endcoded in data fields of Binary
2022-11-28 01:43:19 +03:00
Property Lists. Notable examples include:
- `com.apple.finder.plist` - contains an `FXRecentFolders` value, which is an
array of ten objects, each of which consists of a `name` and `file-bookmark`
field, which is a `bookmarkData` object for each recently accessed folder
location.
- `com.apple.LSSharedFileList.RecentApplications.sfl2` - `sfl2` files are
actually `plist` files of the `NSKeyedArchiver` format. They can be parsed the
same as `plist` files, but they have a more complicated tree-like structure
than would typically be found, which can make locating and retrieving specific
values difficult, even once it has been converted to a JSON representation.
2022-12-02 08:31:07 +03:00
For more information about these types of files, see Sarah Edwards' excellent
research on the subject (link in references).
2022-11-28 01:43:19 +03:00
2022-12-04 02:36:56 +03:00
`fq` 's `grep_by` function can be used to recursively descend through the decoded
tree, probing for and selecting any `bookmark` blobs, then converting them to
readable JSON with `torepr` :
2022-11-28 01:43:19 +03:00
```
2022-12-04 02:36:56 +03:00
fq 'grep_by(.type=="data" and .value[0:4] == "book") | .value | apple_bookmark |
torepr' < sfl2 file >
2022-11-28 01:43:19 +03:00
```
### Authors
- David McDonald
[@dgmcdona ](https://github.com/dgmcdona )
[@river_rat_504 ](https://twitter.com/river_rat_504 )
### References
- https://developer.apple.com/documentation/foundation/url/2143023-bookmarkdata
- https://mac-alias.readthedocs.io/en/latest/bookmark_fmt.html
- https://www.mac4n6.com/blog/2016/1/1/manual-analysis-of-nskeyedarchiver-formatted-plist-files-a-review-of-the-new-os-x-1011-recent-items
- https://michaellynn.github.io/2015/10/24/apples-bookmarkdata-exposed/
2022-09-11 10:55:30 +03:00
## asn1_ber
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
Supports decoding BER, CER and DER (X.690).
2022-01-13 20:34:59 +03:00
2022-02-16 22:13:28 +03:00
- Currently no extra validation is done for CER and DER.
- Does not support specifying a schema.
- Supports `torepr` but without schema all sequences and sets will be arrays.
2022-01-13 20:34:59 +03:00
2022-09-22 19:32:13 +03:00
### Can be used to decode certificates etc
2021-12-09 19:15:21 +03:00
2022-09-10 19:28:54 +03:00
```sh
2022-12-21 15:59:54 +03:00
$ fq -d bytes 'from_pem | asn1_ber | d' cert.pem
2022-01-13 20:34:59 +03:00
```
2022-09-11 10:55:30 +03:00
### Can decode nested values
2022-02-07 19:41:05 +03:00
2022-09-10 19:28:54 +03:00
```sh
$ fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber
2022-02-07 19:41:05 +03:00
```
2022-02-16 22:13:28 +03:00
2022-09-11 10:55:30 +03:00
### Manual schema
2022-02-16 22:13:28 +03:00
2022-09-10 19:28:54 +03:00
```sh
$ fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der
2022-02-16 22:13:28 +03:00
```
2022-09-11 10:55:30 +03:00
### References
2022-02-16 22:13:28 +03:00
- https://www.itu.int/ITU-T/studygroups/com10/languages/X.690_1297.pdf
- https://en.wikipedia.org/wiki/X.690
- https://letsencrypt.org/docs/a-warm-welcome-to-asn1-and-der/
- https://lapo.it/asn1js/
2022-09-11 10:55:30 +03:00
## avc_au
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Options
2021-12-09 19:15:21 +03:00
|Name |Default|Description|
|- |- |-|
2022-10-29 20:23:50 +03:00
|`length_size`|0 |Length value size|
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2021-12-09 19:15:21 +03:00
2022-05-26 13:51:54 +03:00
Decode file using avc_au options
2021-12-09 19:15:21 +03:00
```
2022-10-29 20:23:50 +03:00
$ fq -d avc_au -o length_size=0 . file
2021-12-09 19:15:21 +03:00
```
Decode value as avc_au
```
2022-10-29 20:23:50 +03:00
... | avc_au({length_size:0})
2021-12-09 19:15:21 +03:00
```
2022-10-29 20:23:50 +03:00
## avi
### Options
2023-10-12 16:48:36 +03:00
|Name |Default|Description|
|- |- |-|
|`decode_extended_chunks`|true |Decode extended chunks|
|`decode_samples` |true |Decode samples|
2022-10-29 20:23:50 +03:00
### Examples
Decode file using avi options
```
2023-10-12 16:48:36 +03:00
$ fq -d avi -o decode_extended_chunks=true -o decode_samples=true . file
2022-10-29 20:23:50 +03:00
```
Decode value as avi
```
2023-10-12 16:48:36 +03:00
... | avi({decode_extended_chunks:true,decode_samples:true})
2022-10-29 20:23:50 +03:00
```
### Samples
AVI has many redundant ways to index samples so currently `.streams[].samples` will only include samples the most "modern" way used in the file. That is in order of stream super index, movi ix index then idx1 index.
### Extract samples for stream 1
```sh
$ fq '.streams[1].samples[] | tobytes' file.avi > stream01.mp3
```
### Show stream summary
```sh
$ fq -o decode_samples=false '[.chunks[0] | grep_by(.id=="LIST" and .type=="strl") | grep_by(.id=="strh") as {$type} | grep_by(.id=="strf") as {$format_tag, $compression} | {$type,$format_tag,$compression}]' *.avi
```
2023-10-12 16:48:36 +03:00
### Speed up decoding by disabling sample and extended chunks decoding
If your not interested in sample details or extended chunks you can speed up decoding by using:
```sh
$ fq -o decode_samples=false -o decode_extended_chunks=false d file.avi
```
2022-10-29 20:23:50 +03:00
### References
- [AVI RIFF File Reference ](https://learn.microsoft.com/en-us/windows/win32/directshow/avi-riff-file-reference )
- [OpenDML AVI File Format Extensions ](http://www.jmcgowan.com/odmlff2.pdf )
2022-09-11 10:55:30 +03:00
## avro_ocf
2022-02-10 05:46:12 +03:00
2021-12-09 19:15:21 +03:00
Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.
2022-02-10 05:46:12 +03:00
Capable of handling null, deflate, and snappy codecs for data compression.
Limitations:
2022-01-29 14:01:36 +03:00
2022-09-10 19:28:54 +03:00
- Schema does not support self-referential types, only built-in types.
- Decimal logical types are not supported for decoding, will just be treated as their primitive type
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### References
2021-12-09 19:15:21 +03:00
- https://avro.apache.org/docs/current/spec.html#Object+Container+Files
2022-01-29 14:01:36 +03:00
2022-09-22 19:32:13 +03:00
### Authors
- Xentripetal
xentripetal@fastmail.com
[@xentripetal ](https://github.com/xentripetal )
2022-09-11 10:55:30 +03:00
## bencode
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Convert represented value to JSON
2021-12-09 19:15:21 +03:00
```
2022-09-10 19:28:54 +03:00
$ fq -d bencode torepr file.torrent
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2022-09-11 10:55:30 +03:00
### References
2022-09-10 19:28:54 +03:00
- https://wiki.theory.org/BitTorrentSpecification#Bencoding
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
## bitcoin_block
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Options
2021-12-09 19:15:21 +03:00
2022-09-10 19:28:54 +03:00
|Name |Default|Description|
|- |- |-|
|`has_header`|false |Has blkdat header|
2022-01-29 14:01:36 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2021-12-09 19:15:21 +03:00
2022-09-10 19:28:54 +03:00
Decode file using bitcoin_block options
2021-12-09 19:15:21 +03:00
```
2022-09-10 19:28:54 +03:00
$ fq -d bitcoin_block -o has_header=false . file
2021-12-09 19:15:21 +03:00
```
2022-01-29 14:01:36 +03:00
2022-09-10 19:28:54 +03:00
Decode value as bitcoin_block
2022-01-29 14:01:36 +03:00
```
2022-09-10 19:28:54 +03:00
... | bitcoin_block({has_header:false})
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2022-11-20 21:22:16 +03:00
## bits
Decode to a slice and indexable binary of bits.
### Slice and decode bit range
```sh
$ echo 'some {"a":1} json' | fq -d bits '.[40:-48] | fromjson'
{
"a": 1
}
```
## Index bits
```sh
✗ echo 'hello' | fq -d bits '.[4]'
1
$ echo 'hello' | fq -c -d bits '[.[range(8)]]'
[0,1,1,0,1,0,0,0]
```
2022-09-24 20:57:11 +03:00
## bplist
### Show full decoding
```sh
2022-10-09 20:05:30 +03:00
$ fq d Info.plist
2022-09-24 20:57:11 +03:00
```
### Timestamps
Timestamps in Apple Binary Property Lists are encoded as Cocoa Core Data
timestamps, where the raw value is the floating point number of seconds since
January 1, 2001. By default, `fq` will render the raw floating point value. In
order to get the raw value or string description, use the `todescription`
function, you can use the `tovalue` and `todescription` functions:
```sh
$ fq 'torepr.SomeTimeStamp | tovalue' Info.plist
685135328
$ fq 'torepr.SomeTimeStamp | todescription' Info.plist
"2022-09-17T19:22:08Z"
```
### Get JSON representation
2022-12-20 00:27:48 +03:00
`bplist` files can be converted to a JSON representation using the `torepr` filter:
2022-09-24 20:57:11 +03:00
```sh
$ fq torepr com.apple.UIAutomation.plist
{
"UIAutomationEnabled": true
}
```
2022-12-20 00:27:48 +03:00
### Decoding NSKeyedArchiver serialized objects
A common way that Swift and Objective-C libraries on macOS serialize objects
is through the NSKeyedArchiver API, which flattens objects into a list of elements
and class descriptions that are reconstructed into an object graph using CFUID
elements in the property list. `fq` includes a function, `from_ns_keyed_archiver` ,
which will rebuild this object graph into a friendly representation.
If no parameters are supplied, it will assume that there is a CFUID located at
`."$top".root` that specifies the root from which decoding should occur. If this
is not present, an error will be produced, asking the user to specify a root
object in the `.$objects` list from which to decode.
The following examples show how this might be used (in this case, within the `fq` REPL):
```
# Assume $top.root is present
bplist> from_ns_keyed_archiver
# Specify optional root
bplist> from_ns_keyed_archiver(1)
```
2022-09-24 20:57:11 +03:00
### Authors
- David McDonald
[@dgmcdona ](https://github.com/dgmcdona )
### References
- http://fileformats.archiveteam.org/wiki/Property_List/Binary
- https://medium.com/@karaiskc/understanding-apples-binary-property-list-format-281e6da00dbd
- https://opensource.apple.com/source/CF/CF-550/CFBinaryPList.c
2022-09-11 10:55:30 +03:00
## bson
2022-09-10 19:28:54 +03:00
2023-05-04 08:34:32 +03:00
### Limitations
2023-04-29 08:30:45 +03:00
- The decimal128 type is not supported for decoding, will just be treated as binary
2022-09-11 10:55:30 +03:00
### Convert represented value to JSON
2022-09-10 19:28:54 +03:00
2021-12-09 19:15:21 +03:00
```
2022-09-10 19:28:54 +03:00
$ fq -d bson torepr file.bson
2021-12-09 19:15:21 +03:00
```
2022-09-10 19:28:54 +03:00
### Filter represented value
2021-12-09 19:15:21 +03:00
2022-09-10 19:28:54 +03:00
```
$ fq -d bson 'torepr | select(.name=="bob")' file.bson
```
2021-12-09 19:15:21 +03:00
2023-05-04 08:34:32 +03:00
### Authors
- Mattias Wadman mattias.wadman@gmail.com, original author
- Matt Dale [@matthewdale ](https://github.com/matthewdale ), additional types and bug fixes
2022-09-11 10:55:30 +03:00
### References
2022-09-10 19:28:54 +03:00
- https://bsonspec.org/spec.html
2022-01-29 14:01:36 +03:00
2022-11-20 21:22:16 +03:00
## bytes
Decode to a slice and indexable binary of bytes.
### Slice out byte ranges
```sh
$ echo -n 'hello' | fq -d bytes '.[-3:]' > last_3_bytes
$ echo -n 'hello' | fq -d bytes '[.[-2:], .[0:2]] | tobytes' > first_last_2_bytes_swapped
```
### Slice and decode byte range
```sh
$ echo 'some {"a":1} json' | fq -d bytes '.[5:-6] | fromjson'
{
"a": 1
}
```
## Index bytes
```sh
$ echo 'hello' | fq -d bytes '.[1]'
101
```
2023-08-18 21:04:57 +03:00
## caff
### Options
|Name |Default|Description|
|- |- |-|
|`uncompress`|true |Uncompress and probe files|
### Examples
Decode file using caff options
```
$ fq -d caff -o uncompress=true . file
```
Decode value as caff
```
... | caff({uncompress:true})
```
2023-08-21 10:37:44 +03:00
### Authors
- [@ronsor ](https://github.com/ronsor )
2022-09-11 10:55:30 +03:00
## cbor
2022-01-29 14:01:36 +03:00
2022-09-11 10:55:30 +03:00
### Convert represented value to JSON
2022-01-13 20:34:59 +03:00
2021-12-09 19:15:21 +03:00
```
2022-09-10 19:28:54 +03:00
$ fq -d cbor torepr file.cbor
2021-12-09 19:15:21 +03:00
```
2022-02-28 12:28:21 +03:00
2022-09-11 10:55:30 +03:00
### References
2021-12-09 19:15:21 +03:00
- https://en.wikipedia.org/wiki/CBOR
- https://www.rfc-editor.org/rfc/rfc8949.html
2022-09-11 10:55:30 +03:00
## csv
2022-06-01 17:55:55 +03:00
2022-09-11 10:55:30 +03:00
### Options
2022-06-01 17:55:55 +03:00
|Name |Default|Description|
|- |- |-|
|`comma` |, |Separator character|
|`comment`|# |Comment line character|
2022-09-11 10:55:30 +03:00
### Examples
2022-06-01 17:55:55 +03:00
Decode file using csv options
```
$ fq -d csv -o comma="," -o comment="#" . file
```
Decode value as csv
```
... | csv({comma:",",comment:"#"})
```
2022-09-15 02:13:49 +03:00
### TSV to CSV
```sh
2022-12-21 15:59:54 +03:00
$ fq -d csv -o comma="\t" to_csv file.tsv
2022-09-15 02:13:49 +03:00
```
### Convert rows to objects based on header row
```sh
$ fq -d csv '.[0] as $t | .[1:] | map(with_entries(.key = $t[.key]))' file.csv
```
2022-09-11 10:55:30 +03:00
## flac_frame
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Options
2021-12-09 19:15:21 +03:00
|Name |Default|Description|
|- |- |-|
|`bits_per_sample`|16 |Bits per sample|
2022-02-28 12:28:21 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2022-02-28 12:28:21 +03:00
2022-05-26 13:51:54 +03:00
Decode file using flac_frame options
2021-12-09 19:15:21 +03:00
```
2022-05-26 13:51:54 +03:00
$ fq -d flac_frame -o bits_per_sample=16 . file
2021-12-09 19:15:21 +03:00
```
2022-02-28 12:28:21 +03:00
2021-12-09 19:15:21 +03:00
Decode value as flac_frame
2022-02-28 12:28:21 +03:00
```
2022-06-13 19:49:30 +03:00
... | flac_frame({bits_per_sample:16})
2022-02-28 12:28:21 +03:00
```
2022-09-11 10:55:30 +03:00
## hevc_au
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Options
2021-12-09 19:15:21 +03:00
|Name |Default|Description|
|- |- |-|
|`length_size`|4 |Length value size|
2022-09-11 10:55:30 +03:00
### Examples
2021-12-09 19:15:21 +03:00
2022-05-26 13:51:54 +03:00
Decode file using hevc_au options
2022-02-28 12:28:21 +03:00
```
2022-05-26 13:51:54 +03:00
$ fq -d hevc_au -o length_size=4 . file
2022-02-28 12:28:21 +03:00
```
2021-12-09 19:15:21 +03:00
Decode value as hevc_au
```
2022-06-13 19:49:30 +03:00
... | hevc_au({length_size:4})
2021-12-09 19:15:21 +03:00
```
2022-09-11 10:55:30 +03:00
## html
2022-06-01 17:55:55 +03:00
2022-09-11 10:55:30 +03:00
### Options
2022-06-01 17:55:55 +03:00
2022-08-25 17:06:25 +03:00
|Name |Default|Description|
|- |- |-|
|`array` |false |Decode as nested arrays|
|`attribute_prefix`|@ |Prefix for attribute keys|
|`seq` |false |Use seq attribute to preserve element order|
2022-06-01 17:55:55 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2022-06-01 17:55:55 +03:00
Decode file using html options
```
2022-08-25 17:06:25 +03:00
$ fq -d html -o array=false -o attribute_prefix="@" -o seq=false . file
2022-06-01 17:55:55 +03:00
```
Decode value as html
```
2022-08-25 17:06:25 +03:00
... | html({array:false,attribute_prefix:"@",seq:false})
2022-06-01 17:55:55 +03:00
```
2022-09-20 12:16:18 +03:00
HTML is decoded in HTML5 mode and will always include `<html>` , `<body>` and `<head>` element.
See xml format for more examples and how to preserve element order and how to encode to xml.
2022-12-21 15:59:54 +03:00
There is no `to_html` function, see `to_xml` instead.
2022-09-20 12:16:18 +03:00
### Element as object
```sh
# decode as object is the default
$ echo '< a href = "url" > text< / a > ' | fq -d html
{
"html": {
"body": {
"a": {
"#text": "text",
"@href": "url"
}
},
"head": ""
}
}
```
### Element as array
```sh
$ '< a href = "url" > text< / a > ' | fq -d html -o array=true
[
"html",
null,
[
[
"head",
null,
[]
],
[
"body",
null,
[
[
"a",
{
"#text": "text",
"href": "url"
},
[]
]
]
]
]
]
2022-09-22 19:32:13 +03:00
# decode html files to a {file: "title", ...} object
2022-09-20 12:16:18 +03:00
$ fq -n -d html '[inputs | {key: input_filename, value: .html.head.title?}] | from_entries' *.html
# <a> href:s in file
$ fq -r -o array=true -d html '.. | select(.[0] == "a" and .[1].href)?.[1].href' file.html
```
2023-06-23 00:54:49 +03:00
## luajit
### Authors
- [@dlatchx ](https://github.com/dlatchx )
### References
- https://github.com/LuaJIT/LuaJIT/blob/v2.1/src/lj_bcdump.h
- http://scm.zoomquiet.top/data/20131216145900/index.html
2022-09-11 10:55:30 +03:00
## macho
2021-12-09 19:15:21 +03:00
Supports decoding vanilla and FAT Mach-O binaries.
2022-09-11 10:55:30 +03:00
### Select 64bit load segments
2021-12-09 19:15:21 +03:00
2022-09-10 19:28:54 +03:00
```sh
2021-12-09 19:15:21 +03:00
$ fq '.load_commands[] | select(.cmd=="segment_64")' file
```
2022-09-11 10:55:30 +03:00
### References
2022-02-28 12:28:21 +03:00
- https://github.com/aidansteele/osx-abi-macho-file-format-reference
2022-09-22 19:32:13 +03:00
### Authors
- Sı ddı k AÇIL
acils@itu.edu.tr
[@Akaame ](https://github.com/Akaame )
2022-10-09 20:05:30 +03:00
## markdown
### Array with all level 1 and 2 headers
```sh
$ fq -d markdown '[.. | select(.type=="heading" and .level< =2)?.children[0]]' file.md
```
2022-09-11 10:55:30 +03:00
## matroska
2022-01-29 14:01:36 +03:00
2023-02-08 13:21:08 +03:00
### Options
|Name |Default|Description|
|- |- |-|
|`decode_samples`|true |Decode samples|
### Examples
Decode file using matroska options
```
$ fq -d matroska -o decode_samples=true . file
```
Decode value as matroska
```
... | matroska({decode_samples:true})
```
2022-09-22 19:32:13 +03:00
### Lookup element using path
2021-12-09 19:15:21 +03:00
2022-09-10 19:28:54 +03:00
```sh
$ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv
2021-12-09 19:15:21 +03:00
```
2022-09-22 19:32:13 +03:00
### Get path to element
2021-12-09 19:15:21 +03:00
2022-09-10 19:28:54 +03:00
```sh
$ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv
```
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### References
2021-12-09 19:15:21 +03:00
- https://tools.ietf.org/html/draft-ietf-cellar-ebml-00
- https://matroska.org/technical/specs/index.html
- https://www.matroska.org/technical/basics.html
- https://www.matroska.org/technical/codec_specs.html
- https://wiki.xiph.org/MatroskaOpus
2023-08-21 10:37:44 +03:00
## moc3
### Authors
- [@ronsor ](https://github.com/ronsor )
2022-09-11 10:55:30 +03:00
## mp3
2022-01-29 14:01:36 +03:00
2022-09-11 10:55:30 +03:00
### Options
2021-12-09 19:15:21 +03:00
|Name |Default|Description|
|- |- |-|
|`max_sync_seek` |32768 |Max byte distance to next sync|
|`max_unique_header_configs`|5 |Max number of unique frame header configs allowed|
2023-01-25 17:27:31 +03:00
|`max_unknown` |50 |Max percent (0-100) unknown bits|
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2021-12-09 19:15:21 +03:00
2022-05-26 13:51:54 +03:00
Decode file using mp3 options
2022-01-29 14:01:36 +03:00
```
2023-01-25 17:27:31 +03:00
$ fq -d mp3 -o max_sync_seek=32768 -o max_unique_header_configs=5 -o max_unknown=50 . file
2022-01-29 14:01:36 +03:00
```
2021-12-09 19:15:21 +03:00
Decode value as mp3
2022-01-29 14:01:36 +03:00
```
2023-01-25 17:27:31 +03:00
... | mp3({max_sync_seek:32768,max_unique_header_configs:5,max_unknown:50})
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2022-09-11 10:55:30 +03:00
## mp4
2022-01-29 14:01:36 +03:00
2022-09-11 10:55:30 +03:00
### Options
2021-12-09 19:15:21 +03:00
|Name |Default|Description|
|- |- |-|
|`allow_truncated`|false |Allow box to be truncated|
2023-02-08 13:21:08 +03:00
|`decode_samples` |true |Decode samples|
2021-12-09 19:15:21 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2022-01-29 14:01:36 +03:00
2022-09-10 19:28:54 +03:00
Decode file using mp4 options
2022-01-29 14:01:36 +03:00
```
2022-09-10 19:28:54 +03:00
$ fq -d mp4 -o allow_truncated=false -o decode_samples=true . file
2022-01-29 14:01:36 +03:00
```
2022-09-10 19:28:54 +03:00
Decode value as mp4
2022-01-29 14:01:36 +03:00
```
2022-09-10 19:28:54 +03:00
... | mp4({allow_truncated:false,decode_samples:true})
2022-01-29 14:01:36 +03:00
```
2022-01-13 20:34:59 +03:00
2022-12-15 16:15:31 +03:00
### Speed up decoding by not decoding samples
2022-09-10 19:28:54 +03:00
```sh
2022-12-15 16:15:31 +03:00
# manually decode first sample as a aac_frame
$ fq -o decode_samples=false '.tracks[0].samples[0] | aac_frame | d' file.mp4
2021-12-09 19:15:21 +03:00
```
2022-09-10 19:28:54 +03:00
2022-12-15 16:15:31 +03:00
### Entries for first edit list as values
2022-09-10 19:28:54 +03:00
```sh
2022-12-15 16:15:31 +03:00
$ fq 'first(grep_by(.type=="elst").entries) | tovalue' file.mp4
```
### Whole box tree as JSON (exclude mdat data and tracks)
```sh
$ fq 'del(.tracks) | grep_by(.type=="mdat").data = "< excluded > " | tovalue' file.mp4
2021-12-09 19:15:21 +03:00
```
2022-09-11 10:55:30 +03:00
### Force decode a single box
2022-09-10 19:28:54 +03:00
```sh
2022-12-21 15:59:54 +03:00
$ fq -n '"AAAAHGVsc3QAAAAAAAAAAQAAADIAAAQAAAEAAA==" | from_base64 | mp4({force:true}) | d'
2021-12-09 19:15:21 +03:00
```
2022-09-10 19:28:54 +03:00
2022-12-15 16:15:31 +03:00
### Lookup mp4 box using a mp4 box path.
2022-09-10 19:28:54 +03:00
```sh
2022-12-15 16:15:31 +03:00
# <decode value box> | mp4_path($path) -> <decode value box>
$ fq 'mp4_path(".moov.trak[1]")' file.mp4
2022-09-10 19:28:54 +03:00
```
2022-12-15 16:15:31 +03:00
### Get mp4 box path for a decode value box.
2022-09-10 19:28:54 +03:00
```sh
2022-12-15 16:15:31 +03:00
# <decode value box> | mp4_path -> string
$ fq 'grep_by(.type == "trak") | mp4_path' file.mp4
2021-12-09 19:15:21 +03:00
```
2022-09-11 10:55:30 +03:00
### References
2021-12-09 19:15:21 +03:00
- [ISO/IEC base media file format (MPEG-4 Part 12) ](https://en.wikipedia.org/wiki/ISO/IEC_base_media_file_format )
- [Quicktime file format ](https://developer.apple.com/standards/qtff-2001.pdf )
2022-09-11 10:55:30 +03:00
## msgpack
2022-01-29 14:01:36 +03:00
2022-09-11 10:55:30 +03:00
### Convert represented value to JSON
2021-12-09 19:15:21 +03:00
2022-01-29 14:01:36 +03:00
```
2022-09-10 19:28:54 +03:00
$ fq -d msgpack torepr file.msgpack
2021-12-09 19:15:21 +03:00
```
2022-09-11 10:55:30 +03:00
### References
2021-12-09 19:15:21 +03:00
- https://github.com/msgpack/msgpack/blob/master/spec.md
2022-01-13 20:34:59 +03:00
2023-09-26 21:42:02 +03:00
## opentimestamps
### View a full OpenTimestamps file
```
$ fq dd file.ots
```
### List the names of the Calendar servers used
```
$ fq '.operations | map(select(.attestation_type == "calendar") | .url)' file.ots
```
### Check if there are Bitcoin attestations present
```
$ fq '.operations | map(select(.attestation_type == "bitcoin")) | length > 0' file.ots
```
### Authors
- fiatjaf, https://fiatjaf.com
### References
- https://opentimestamps.org/
- https://github.com/opentimestamps/python-opentimestamps
2022-10-09 20:05:30 +03:00
## pcap
### Build object with number of (reassembled) TCP bytes sent to/from client IP
```sh
# for a pcapng file you would use .[0].tcp_connections for first section
$ fq '.tcp_connections | group_by(.client.ip) | map({key: .[0].client.ip, value: map(.client.stream, .server.stream | tobytes.size) | add}) | from_entries'
{
"10.1.0.22": 15116,
"10.99.12.136": 234,
"10.99.12.150": 218
}
```
2023-05-04 08:34:32 +03:00
## pg_btree
### Options
|Name |Default|Description|
|- |- |-|
|`page`|0 |First page number in file, default is 0|
### Examples
Decode file using pg_btree options
```
$ fq -d pg_btree -o page=0 . file
```
Decode value as pg_btree
```
... | pg_btree({page:0})
```
### Btree index meta page
```sh
$ fq -d pg_btree -o flavour=postgres14 ".[0] | d" 16404
```
### Btree index page
```sh
$ fq -d pg_btree -o flavour=postgres14 ".[1]" 16404
```
2023-05-06 08:54:56 +03:00
### Authors
- Pavel Safonov
p.n.safonov@gmail.com
[@pnsafonov ](https://github.com/pnsafonov )
2023-05-04 08:34:32 +03:00
### References
- https://www.postgresql.org/docs/current/storage-page-layout.html
## pg_control
### Options
|Name |Default|Description|
|- |- |-|
|`flavour`| |PostgreSQL flavour: postgres14, pgproee14.., postgres10|
### Examples
Decode file using pg_control options
```
$ fq -d pg_control -o flavour="" . file
```
Decode value as pg_control
```
... | pg_control({flavour:""})
```
### Decode content of pg_control file
```sh
$ fq -d pg_control -o flavour=postgres14 d pg_control
```
### Specific fields can be got by request
```sh
$ fq -d pg_control -o flavour=postgres14 ".state, .check_point_copy.redo, .wal_level" pg_control
```
2023-05-06 08:54:56 +03:00
### Authors
- Pavel Safonov
p.n.safonov@gmail.com
[@pnsafonov ](https://github.com/pnsafonov )
2023-05-04 08:34:32 +03:00
### References
- https://github.com/postgres/postgres/blob/REL_14_2/src/include/catalog/pg_control.h
## pg_heap
### Options
|Name |Default |Description|
|- |- |-|
|`flavour`|postgres14|PostgreSQL flavour: postgres14, pgproee14.., postgres10|
|`page` |0 |First page number in file, default is 0|
|`segment`|0 |Segment file number (16790.1 is 1), default is 0|
### Examples
Decode file using pg_heap options
```
$ fq -d pg_heap -o flavour="postgres14" -o page=0 -o segment=0 . file
```
Decode value as pg_heap
```
... | pg_heap({flavour:"postgres14",page:0,segment:0})
```
### To see heap page's content
```sh
$ fq -d pg_heap -o flavour=postgres14 ".[0]" 16994
```
### To see page's header
```sh
$ fq -d pg_heap -o flavour=postgres14 ".[0].page_header" 16994
```
### First and last item pointers on first page
```sh
$ fq -d pg_heap -o flavour=postgres14 ".[0].pd_linp[0, -1]" 16994
```
### First and last tuple on first page
```sh
$ fq -d pg_heap -o flavour=postgres14 ".[0].tuples[0, -1]" 16994
```
2023-05-06 08:54:56 +03:00
### Authors
- Pavel Safonov
p.n.safonov@gmail.com
[@pnsafonov ](https://github.com/pnsafonov )
2023-05-04 08:34:32 +03:00
### References
- https://www.postgresql.org/docs/current/storage-page-layout.html
2022-09-11 10:55:30 +03:00
## protobuf
2022-02-01 18:07:41 +03:00
2022-09-11 10:55:30 +03:00
### Can decode sub messages
2022-02-01 18:07:41 +03:00
2022-09-10 19:28:54 +03:00
```sh
$ fq -d protobuf '.fields[6].wire_value | protobuf | d' file
2022-02-01 18:07:41 +03:00
```
2022-01-29 14:01:36 +03:00
2022-09-11 10:55:30 +03:00
### References
2021-12-09 19:15:21 +03:00
- https://developers.google.com/protocol-buffers/docs/encoding
2022-09-11 10:55:30 +03:00
## rtmp
2022-03-29 23:41:11 +03:00
Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).
2022-01-13 20:34:59 +03:00
2022-10-09 20:05:30 +03:00
### Show rtmp streams in PCAP file
```sh
fq '.tcp_connections[] | select(.server.port=="rtmp") | d' file.cap
```
2022-09-11 10:55:30 +03:00
### References
2021-12-09 19:15:21 +03:00
- https://rtmp.veriskope.com/docs/spec/
- https://rtmp.veriskope.com/pdf/video_file_format_spec_v10.pdf
2023-02-10 22:06:38 +03:00
## tls
### Options
|Name |Default|Description|
|- |- |-|
|`keylog`| |NSS Key Log content|
### Examples
Decode file using tls options
```
$ fq -d tls -o keylog="" . file
```
Decode value as tls
```
... | tls({keylog:""})
```
Supports decoding of most standard records, messages and extensions. Can also decrypt most standard cipher suits in a PCAP with traffic in both directions if a NSS key log is provided.
2023-06-23 00:54:49 +03:00
### Decode and decrypt provding a PCAP and key log
2023-02-10 22:06:38 +03:00
Write traffic to a PCAP file:
```sh
$ tcpdump -i < iface > -w traffic.pcap
```
Make sure your curl TLS backend support `SSLKEYLOGFILE` and do:
```sh
$ SSLKEYLOGFILE=traffic.keylog curl --tls-max 1.2 https://host/path
```
Decode, decrypt and query. Uses `keylog=@<path>` to read option value from keylog file:
```sh
# decode and show whole tree
$ fq -o keylog=@traffic.keylog d traffic.pcap
# write unencrypted server response to a file.
# first .stream is the TCP stream, second .stream is TLS application data stream
#
# first TCP connections:
$ fq -o keylog=@traffic.keylog '.tcp_connections[0].server.stream.stream | tobytes' traffic.pcap > data
# first TLS connection:
$ fq -o keylog=@traffic.keylog 'first(grep_by(.server.stream | format == "tls")).server.stream.stream | tobytes' > data
```
### Supported cipher suites for decryption
`TLS_DH_ANON_EXPORT_WITH_DES40_CBC_SHA` ,
`TLS_DH_ANON_EXPORT_WITH_RC4_40_MD5` ,
`TLS_DHE_DSS_EXPORT_WITH_DES40_CBC_SHA` ,
`TLS_DHE_DSS_WITH_3DES_EDE_CBC_SHA` ,
`TLS_DHE_DSS_WITH_AES_128_CBC_SHA` ,
`TLS_DHE_DSS_WITH_AES_128_CBC_SHA256` ,
`TLS_DHE_DSS_WITH_AES_128_GCM_SHA256` ,
`TLS_DHE_DSS_WITH_AES_256_CBC_SHA` ,
`TLS_DHE_DSS_WITH_AES_256_CBC_SHA256` ,
`TLS_DHE_DSS_WITH_AES_256_GCM_SHA384` ,
`TLS_DHE_DSS_WITH_DES_CBC_SHA` ,
`TLS_DHE_DSS_WITH_RC4_128_SHA` ,
`TLS_DHE_RSA_EXPORT_WITH_DES40_CBC_SHA` ,
`TLS_DHE_RSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_DHE_RSA_WITH_AES_128_CBC_SHA` ,
`TLS_DHE_RSA_WITH_AES_128_CBC_SHA256` ,
`TLS_DHE_RSA_WITH_AES_128_GCM_SHA256` ,
`TLS_DHE_RSA_WITH_AES_256_CBC_SHA` ,
`TLS_DHE_RSA_WITH_AES_256_CBC_SHA256` ,
`TLS_DHE_RSA_WITH_AES_256_GCM_SHA384` ,
`TLS_DHE_RSA_WITH_CHACHA20_POLY1305_SHA256` ,
`TLS_DHE_RSA_WITH_DES_CBC_SHA` ,
`TLS_ECDH_ECDSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA` ,
`TLS_ECDH_ECDSA_WITH_AES_128_CBC_SHA256` ,
`TLS_ECDH_ECDSA_WITH_AES_128_GCM_SHA256` ,
`TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA` ,
`TLS_ECDH_ECDSA_WITH_AES_256_CBC_SHA384` ,
`TLS_ECDH_ECDSA_WITH_AES_256_GCM_SHA384` ,
`TLS_ECDH_ECDSA_WITH_RC4_128_SHA` ,
`TLS_ECDH_RSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_ECDH_RSA_WITH_AES_128_CBC_SHA` ,
`TLS_ECDH_RSA_WITH_AES_128_CBC_SHA256` ,
`TLS_ECDH_RSA_WITH_AES_128_GCM_SHA256` ,
`TLS_ECDH_RSA_WITH_AES_256_CBC_SHA` ,
`TLS_ECDH_RSA_WITH_AES_256_CBC_SHA384` ,
`TLS_ECDH_RSA_WITH_AES_256_GCM_SHA384` ,
`TLS_ECDH_RSA_WITH_RC4_128_SHA` ,
`TLS_ECDHE_ECDSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA` ,
`TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA` ,
`TLS_ECDHE_ECDSA_WITH_AES_128_CBC_SHA256` ,
`TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256` ,
`TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA` ,
`TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA` ,
`TLS_ECDHE_ECDSA_WITH_AES_256_CBC_SHA384` ,
`TLS_ECDHE_ECDSA_WITH_AES_256_GCM_SHA384eadAESGCM` ,
`TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305_SHA256` ,
`TLS_ECDHE_ECDSA_WITH_CHACHA20_POLY1305` ,
`TLS_ECDHE_ECDSA_WITH_RC4_128_SHA` ,
`TLS_ECDHE_ECDSA_WITH_RC4_128_SHA` ,
`TLS_ECDHE_PSK_WITH_AES_128_CBC_SHA` ,
`TLS_ECDHE_PSK_WITH_AES_128_GCM_SHA256` ,
`TLS_ECDHE_PSK_WITH_AES_256_CBC_SHA` ,
`TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_ECDHE_RSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA` ,
`TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA` ,
`TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA256` ,
`TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256` ,
`TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA` ,
`TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA` ,
`TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384` ,
`TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384` ,
`TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305_SHA256` ,
`TLS_ECDHE_RSA_WITH_CHACHA20_POLY1305` ,
`TLS_ECDHE_RSA_WITH_RC4_128_SHA` ,
`TLS_ECDHE_RSA_WITH_RC4_128_SHA` ,
`TLS_PSK_WITH_AES_128_CBC_SHA` ,
`TLS_PSK_WITH_AES_256_CBC_SHA` ,
`TLS_PSK_WITH_RC4_128_SHA` ,
`TLS_RSA_EXPORT_WITH_DES40_CBC_SHA` ,
`TLS_RSA_EXPORT_WITH_RC4_40_MD5` ,
`TLS_RSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_RSA_WITH_3DES_EDE_CBC_SHA` ,
`TLS_RSA_WITH_AES_128_CBC_SHA` ,
`TLS_RSA_WITH_AES_128_CBC_SHA` ,
`TLS_RSA_WITH_AES_128_CBC_SHA256` ,
`TLS_RSA_WITH_AES_128_CBC_SHA256` ,
`TLS_RSA_WITH_AES_128_GCM_SHA256` ,
`TLS_RSA_WITH_AES_128_GCM_SHA256` ,
`TLS_RSA_WITH_AES_256_CBC_SHA` ,
`TLS_RSA_WITH_AES_256_CBC_SHA` ,
`TLS_RSA_WITH_AES_256_CBC_SHA256` ,
`TLS_RSA_WITH_AES_256_GCM_SHA384` ,
`TLS_RSA_WITH_AES_256_GCM_SHA384` ,
`TLS_RSA_WITH_DES_CBC_SHA` ,
`TLS_RSA_WITH_RC4_128_MD5` ,
`TLS_RSA_WITH_RC4_128_SHA` ,
`TLS_RSA_WITH_RC4_128_SHA`
### References
- [RFC 5246: The Transport Layer Security (TLS) Protocol ](https://www.rfc-editor.org/rfc/rfc5246 )
- [RFC 6101: The Secure Sockets Layer (SSL) Protocol Version 3.0 ](https://www.rfc-editor.org/rfc/rfc )
2022-12-01 15:44:57 +03:00
## tzif
2022-12-03 10:06:46 +03:00
### Get last transition time
```sh
fq '.v2plusdatablock.transition_times[-1] | tovalue' tziffile
```
### Count leap second records
```sh
fq '.v2plusdatablock.leap_second_records | length' tziffile
```
2022-12-01 15:44:57 +03:00
### Authors
- Takashi Oguma
[@bitbears-dev ](https://github.com/bitbears-dev )
[@0xb17bea125 ](https://twitter.com/0xb17bea125 )
### References
- https://datatracker.ietf.org/doc/html/rfc8536
2022-09-12 12:47:16 +03:00
## wasm
### Count opcode usage
```sh
$ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' file.wasm
```
### List exports and imports
```sh
$ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' file.wasm
```
### Authors
- Takashi Oguma
[@bitbears-dev ](https://github.com/bitbears-dev )
[@0xb17bea125 ](https://twitter.com/0xb17bea125 )
### References
- https://webassembly.github.io/spec/core/
2022-09-11 10:55:30 +03:00
## xml
2022-06-01 17:55:55 +03:00
2022-09-11 10:55:30 +03:00
### Options
2022-06-01 17:55:55 +03:00
2022-08-25 17:06:25 +03:00
|Name |Default|Description|
|- |- |-|
|`array` |false |Decode as nested arrays|
|`attribute_prefix`|@ |Prefix for attribute keys|
|`seq` |false |Use seq attribute to preserve element order|
2022-06-01 17:55:55 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2022-06-01 17:55:55 +03:00
Decode file using xml options
```
2022-08-25 17:06:25 +03:00
$ fq -d xml -o array=false -o attribute_prefix="@" -o seq=false . file
2022-06-01 17:55:55 +03:00
```
Decode value as xml
```
2022-08-25 17:06:25 +03:00
... | xml({array:false,attribute_prefix:"@",seq:false})
2022-06-01 17:55:55 +03:00
```
2022-09-20 12:16:18 +03:00
XML can be decoded and encoded into jq values in two ways, elements as object or array.
Which variant to use depends a bit what you want to do. The object variant might be easier
to query for a specific value but array might be easier to use to generate xml or to query
after all elements of some kind etc.
2022-12-21 15:59:54 +03:00
Encoding is done using the `to_xml` function and it will figure what variant that is used based on the input value.
2022-09-20 12:16:18 +03:00
Is has two optional options `indent` and `attribute_prefix` .
### Elements as object
Element can have different shapes depending on body text, attributes and children:
- `<a key="value">text</a>` is `{"a":{"#text":"text","@key":"value"}}` , has text (`#text`) and attributes (`@key`)
- `<a>text</a>` is `{"a":"text"}`
- `<a><b>text</b></a>` is `{"a":{"b":"text"}}` one child with only text and no attributes
- `<a><b/><b>text</b></a>` is `{"a":{"b":["","text"]}}` two children with same name end up in an array
- `<a><b/><b key="value">text</b></a>` is `{"a":{"b":["",{"#text":"text","@key":"value"}]}}`
If there is `#seq` attribute it encodes the child element order. Use `-o seq=true` to include sequence number when decoding,
otherwise order might be lost.
```sh
# decode as object is the default
$ echo '< a > < b / > < b > bbb< / b > < c attr = "value" > ccc< / c > < / a > ' | fq -d xml -o seq=true
{
"a": {
"b": [
{
"#seq": 0
},
{
"#seq": 1,
"#text": "bbb"
}
],
"c": {
"#seq": 2,
"#text": "ccc",
"@attr": "value"
}
}
}
# access text of the <c> element
$ echo '< a > < b / > < b > bbb< / b > < c attr = "value" > ccc< / c > < / a > ' | fq '.a.c["#text"]'
"ccc"
# decode to object and encode to xml
2022-12-21 15:59:54 +03:00
$ echo '< a > < b / > < b > bbb< / b > < c attr = "value" > ccc< / c > < / a > ' | fq -r -d xml -o seq=true 'to_xml({indent:2})'
2022-09-20 12:16:18 +03:00
< a >
< b > < / b >
< b > bbb< / b >
< c attr = "value" > ccc< / c >
< / a >
```
### Elements as array
Elements are arrays of the shape `["#text": "body text", "attr_name", {key: "attr value"}|null, [<child element>, ...]]` .
```sh
# decode as array
2022-09-27 14:42:31 +03:00
$ echo '< a > < b / > < b > bbb< / b > < c attr = "value" > ccc< / c > < / a > ' | fq -d xml -o array=true
2022-09-20 12:16:18 +03:00
[
"a",
null,
[
[
"b",
null,
[]
],
[
"b",
{
"#text": "bbb"
},
[]
],
[
"c",
{
"#text": "ccc",
"attr": "value"
},
[]
]
]
]
# decode to array and encode to xml
2022-12-21 15:59:54 +03:00
$ echo '< a > < b / > < b > bbb< / b > < c attr = "value" > ccc< / c > < / a > ' | fq -r -d xml -o array=true -o seq=true 'to_xml({indent:2})'
2022-09-20 12:16:18 +03:00
< a >
< b > < / b >
< b > bbb< / b >
< c attr = "value" > ccc< / c >
< / a >
# access text of the <c> element, the object variant above is probably easier to use
$ echo '< a > < b / > < b > bbb< / b > < c attr = "value" > ccc< / c > < / a > ' | fq -o array=true '.[2][2][1]["#text"]'
"ccc"
```
2022-09-11 10:55:30 +03:00
### References
2022-08-25 17:06:25 +03:00
- [xml.com's Converting Between XML and JSON ](https://www.xml.com/pub/a/2006/05/31/converting-between-xml-and-json.html )
2022-09-11 10:55:30 +03:00
## zip
2022-05-26 13:39:14 +03:00
2022-09-11 10:55:30 +03:00
### Options
2022-05-26 13:39:14 +03:00
|Name |Default|Description|
|- |- |-|
2022-05-26 20:27:17 +03:00
|`uncompress`|true |Uncompress and probe files|
2022-05-26 13:39:14 +03:00
2022-09-11 10:55:30 +03:00
### Examples
2022-05-26 13:39:14 +03:00
2022-05-26 13:51:54 +03:00
Decode file using zip options
2022-05-26 13:39:14 +03:00
```
2022-05-26 13:51:54 +03:00
$ fq -d zip -o uncompress=true . file
2022-05-26 13:39:14 +03:00
```
Decode value as zip
```
2022-06-13 19:49:30 +03:00
... | zip({uncompress:true})
2022-05-26 13:39:14 +03:00
```
2022-09-10 19:28:54 +03:00
Supports ZIP64.
2022-05-26 17:28:45 +03:00
2023-10-21 19:11:56 +03:00
## Timestamp and time zones
The timestamp accessed via `.local_files[].last_modification` is encoded in ZIP files using [MS-DOS representation ](https://learn.microsoft.com/en-us/windows/win32/api/oleauto/nf-oleauto-dosdatetimetovarianttime ) which lacks a known time zone. Probably the local time/date was used at creation. The `unix_guess` field in `last_modification` is a guess assuming the local time zone was UTC at creation.
2022-09-11 10:55:30 +03:00
### References
2022-05-26 17:28:45 +03:00
- https://pkware.cachefly.net/webdocs/casestudies/APPNOTE.TXT
2022-09-10 19:28:54 +03:00
- https://opensource.apple.com/source/zip/zip-6/unzip/unzip/proginfo/extra.fld
2023-10-21 19:11:56 +03:00
- https://formats.kaitai.io/dos_datetime/
- https://learn.microsoft.com/en-us/windows/win32/api/oleauto/nf-oleauto-dosdatetimetovarianttime
2022-05-26 17:28:45 +03:00
2021-12-09 19:15:21 +03:00
[#]: sh-end
2021-09-19 11:27:56 +03:00
2022-02-05 20:15:18 +03:00
## Dependency graph
2021-09-19 11:27:56 +03:00
![alt text ](formats.svg "Format diagram" )