wader/fq
1
1
Fork 0
mirror of https://github.com/wader/fq.git synced 2024-11-24 11:16:09 +03:00
Code Issues Projects Releases Wiki Activity
21ad628add
fq/doc/formats.md
Mattias Wadman 21ad628add interp: dump: Show field name for compound values in arrays 
mp3: Remove one level of struct for granales
dump: Makes it a bit easier to understand array of arrays/structs.
avro_ocf: Rename ocf.go -> avro_ocr.go to have same name of source file as format
pcap,pcapng: Rename tcp flow struct to tcp_connection, makes more sense i think
mp4: Add ISOBMFF to format description
2022-04-21 17:47:15 +02:00

18 KiB
Raw Blame History

Supported formats

Name Description Dependencies
aac_frame Advanced Audio Coding frame
adts Audio Data Transport Stream adts_frame
adts_frame Audio Data Transport Stream frame aac_frame
amf0 Action Message Format 0
apev2 APEv2 metadata tag image
ar Unix archive probe
asn1_ber ASN1 Basic Encoding Rules (also CER and DER)
av1_ccr AV1 Codec Configuration Record
av1_frame AV1 frame av1_obu
av1_obu AV1 Open Bitstream Unit
avc_annexb H.264/AVC Annex B avc_nalu
avc_au H.264/AVC Access Unit avc_nalu
avc_dcr H.264/AVC Decoder Configuration Record avc_nalu
avc_nalu H.264/AVC Network Access Layer Unit avc_sps avc_pps avc_sei
avc_pps H.264/AVC Picture Parameter Set
avc_sei H.264/AVC Supplemental Enhancement Information
avc_sps H.264/AVC Sequence Parameter Set
avro_ocf Avro object container file
bencode BitTorrent bencoding
bsd_loopback_frame BSD loopback frame inet_packet
bson Binary JSON
bzip2 bzip2 compression probe
cbor Concise Binary Object Representation
dns DNS packet
dns_tcp DNS packet (TCP)
elf Executable and Linkable Format
ether8023_frame Ethernet 802.3 frame inet_packet
exif Exchangeable Image File Format
flac Free Lossless Audio Codec file flac_metadatablocks flac_frame
flac_frame FLAC frame
flac_metadatablock FLAC metadatablock flac_streaminfo flac_picture vorbis_comment
flac_metadatablocks FLAC metadatablocks flac_metadatablock
flac_picture FLAC metadatablock picture image
flac_streaminfo FLAC streaminfo
gif Graphics Interchange Format
gzip gzip compression probe
hevc_annexb H.265/HEVC Annex B hevc_nalu
hevc_au H.265/HEVC Access Unit hevc_nalu
hevc_dcr H.265/HEVC Decoder Configuration Record hevc_nalu
hevc_nalu H.265/HEVC Network Access Layer Unit hevc_vps hevc_pps hevc_sps
hevc_pps H.265/HEVC Picture Parameter Set
hevc_sps H.265/HEVC Sequence Parameter Set
hevc_vps H.265/HEVC Video Parameter Set
icc_profile International Color Consortium profile
icmp Internet Control Message Protocol
icmpv6 Internet Control Message Protocol v6
id3v1 ID3v1 metadata
id3v11 ID3v1.1 metadata
id3v2 ID3v2 metadata image
ipv4_packet Internet protocol v4 packet ip_packet
ipv6_packet Internet protocol v6 packet ip_packet
jpeg Joint Photographic Experts Group file exif icc_profile
json JSON
macho Mach-O macOS executable
matroska Matroska file aac_frame av1_ccr av1_frame avc_au avc_dcr flac_frame flac_metadatablocks hevc_au hevc_dcr image mp3_frame mpeg_asc mpeg_pes_packet mpeg_spu opus_packet vorbis_packet vp8_frame vp9_cfm vp9_frame
mp3 MP3 file id3v2 id3v1 id3v11 apev2 mp3_frame
mp3_frame MPEG audio layer 3 frame xing
mp4 ISOBMFF MPEG-4 part 12 and similar aac_frame av1_ccr av1_frame flac_frame flac_metadatablocks id3v2 image jpeg mp3_frame avc_au avc_dcr mpeg_es hevc_au hevc_dcr mpeg_pes_packet opus_packet protobuf_widevine pssh_playready vorbis_packet vp9_frame vpx_ccr icc_profile
mpeg_asc MPEG-4 Audio Specific Config
mpeg_es MPEG Elementary Stream mpeg_asc vorbis_packet
mpeg_pes MPEG Packetized elementary stream mpeg_pes_packet mpeg_spu
mpeg_pes_packet MPEG Packetized elementary stream packet
mpeg_spu Sub Picture Unit (DVD subtitle)
mpeg_ts MPEG Transport Stream
msgpack MessagePack
ogg OGG file ogg_page vorbis_packet opus_packet flac_metadatablock flac_frame
ogg_page OGG page
opus_packet Opus packet vorbis_comment
pcap PCAP packet capture link_frame tcp_stream ipv4_packet
pcapng PCAPNG packet capture link_frame tcp_stream ipv4_packet
png Portable Network Graphics file icc_profile exif
protobuf Protobuf
protobuf_widevine Widevine protobuf protobuf
pssh_playready PlayReady PSSH
raw Raw bits
rtmp Real-Time Messaging Protocol amf0 mpeg_asc
sll2_packet Linux cooked capture encapsulation v2 inet_packet
sll_packet Linux cooked capture encapsulation inet_packet
tar Tar archive probe
tcp_segment Transmission control protocol segment
tiff Tag Image File Format icc_profile
udp_datagram User datagram protocol udp_payload
vorbis_comment Vorbis comment flac_picture
vorbis_packet Vorbis packet vorbis_comment
vp8_frame VP8 frame
vp9_cfm VP9 Codec Feature Metadata
vp9_frame VP9 frame
vpx_ccr VPX Codec Configuration Record
wav WAV file id3v2 id3v1 id3v11
webp WebP image vp8_frame
xing Xing header
zip ZIP archive probe
image Group gif jpeg mp4 png tiff webp
inet_packet Group ipv4_packet ipv6_packet
ip_packet Group icmp icmpv6 tcp_segment udp_datagram
link_frame Group bsd_loopback_frame ether8023_frame sll2_packet sll_packet
probe Group adts ar avro_ocf bzip2 elf flac gif gzip jpeg json macho matroska mp3 mp4 mpeg_ts ogg pcap pcapng png tar tiff wav webp zip
tcp_stream Group dns rtmp
udp_payload Group dns

Format options

Currently the only option is force and is used to ignore some format assertion errors. It can be used as a decode option or as a CLI -o option:

fq -d mp4 -o force=true file.mp4
fq -d raw 'mp4({force: true})' file.mp4

Format details

asn1_ber

Supports decoding BER, CER and DER (X.690).

  • Currently no extra validation is done for CER and DER.
  • Does not support specifying a schema.
  • Supports torepr but without schema all sequences and sets will be arrays.
fq -d asn1_ber torepr file.ber

Functions frompem and topem can help working with PEM format:

fq -d raw 'frompem | asn1_ber | d' cert.pem

If the schema is known and not that complicated it can be reproduced:

fq -d asn1_ber 'torepr as $r | ["version", "modulus", "private_exponent", "private_exponen", "prime1", "prime2", "exponent1", "exponent2", "coefficient"] | with_entries({key: .value, value: $r[.key]})' pkcs1.der

Can be used to decode nested parts:

fq -d asn1_ber '.constructed[1].value | asn1_ber' file.ber

References and tools:

avro_ocf

Supports reading Avro Object Container Format (OCF) files based on the 1.11.0 specification.

Capable of handling null, deflate, and snappy codecs for data compression.

Limitations:

  • Schema does not support self-referential types, only built-in types.
  • Decimal logical types are not supported for decoding, will just be treated as their primitive type

becode

Supports torepr:

fq -d bencode torepr file.torrent

bson

Supports torepr:

fq -d bson torepr file.bson

cbor

Supports torepr:

fq -d cbor torepr file.cbor
fq -d cbor 'torepr.field' file.cbor
fq -d cbor 'torepr | .field' file.cbor
fq -d cbor 'torepr | grep("abc")' file.cbor

macho

Supports decoding vanilla and FAT Mach-O binaries.

Examples

To decode the macOS build of fq:

fq . /path/to/fq

fq '.load_commands[] | select(.cmd=="segment_64")' /path/to/fq

Note you can use -d macho to decode a broken Mach-O binary.

References:

matroska

Supports matroska_path:

$ fq 'matroska_path(".Segment.Tracks[0]")' file.mkv
     │00 01 02 03 04 05 06 07 08 09│0123456789│.elements[1].elements[3]{}:
0x122│         16 54 ae 6b         │   .T.k   │  id: "Tracks" (0x1654ae6b) (A Top-Level Element of information with many tracks described.)
     │                             │          │  type: "master" (7)
0x122│                     4d bf   │       M. │  size: 3519
0x122│                           bf│         .│  elements[0:3]:
0x12c│84 cf 8b db a0 ae 01 00 00 00│..........│
0x136│00 00 00 78 d7 81 01 73 c5 88│...x...s..│
*    │until 0xee9.7 (3519)         │          │

$ fq 'first(grep_by(.id == "Tracks")) | matroska_path' test.mkv
".Segment.Tracks"

mp4

Supports mp4_path:

$ fq 'mp4_path(".moov.trak[1]")' file.mp4
     │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[1]{}:
0x4f6│                     00 00 02│       ...│  size: 573
0x500│3d                           │=         │
0x500│   74 72 61 6b               │ trak     │  type: "trak" (Container for an individual track or stream)
0x500│               00 00 00 5c 74│     ...\t│  boxes[0:3]:
0x50a│6b 68 64 00 00 00 03 00 00 00│khd.......│
0x514│00 00 00 00 00 00 00 00 01 00│..........│
*    │until 0x739.7 (565)          │          │

$ fq 'first(grep_by(.type == "trak")) | mp4_path' file.mp4
".moov.trak"

msgpack

Supports torepr:

fq -d msgpack torepr file.msgpack

protobuf

protobuf decoder can be used to decode sub messages:

fq -d protobuf '.fields[6].wire_value | protobuf | d'

rtmp

Current only supports plain RTMP (not RTMPT or encrypted variants etc) with AMF0 (not AMF3).

Dependency graph

alt text