92 KiB
0.2.0
This ended up being a release to cleanup old sins in the decoder internals and change some defaults how binary values work with JSON and string functions.
It also adds a new Time Zone Information Format decoder tzif
(Thanks Takashi Oguma @bitbears-dev) and a new Apple BookmarkData decoder apple_bookmark
decoder (Thanks David McDonald @dgmcdona). Also a new function from_ns_keyed_archiver
was added to convert NSKeyedArchiver encoded objects into JSON.
A possible breaking change is that now all from
/to
prefix functions now has a from_
/to_
prefix, ex: from_mp3
instead of frommp3
. There are some few exceptions to this. Note that the functions named just be the format name, ex mp3
are still around.
In other fq related news jq-lsp got some fixed and additions and seems to work fine with neovim. It's also possible to use jq-lsp with vscode using vscode-jq.
Changes
- All functions that had a
from
/to
prefix now has the prefixfrom_
/to_
. This is to be easier to read and more consistent, there are still some exceptions liketovalue
,torepr
,tobytes
etc but in general anything that does not deal with primitive types is nowsnake_case
. #535 - Change default
bit_formats
option value (how raw bits values are represented in JSON) fromsnippet
tostring
.snippet
meant truncated bits as base64. Now all bits are included as a UTF-8 string. The string will be binary safe (not lose any data) when used internally in fq but will lose data when represented in JSON as some bytes can't be encoded as UTF-8. #499 - Don't auto convert to binary for string/regexp functions, turned out this is very confusing. Now you have to manually use
tobytes
etc to convert to binary value. #540# This used to not work as test/1 would convert decode values to the source bytes # (0x00 0x00 0x00 0x01) in this case. Now the jq value (symbolic in this case) will # be used instead. You can do ".test | tobytes" to get old behavior. # # find all types with a "mdta." prefix $ fq -o line_bytes=10 'grep_by(.type | test(`^mdta\.`))' file.mp4 │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[0]{}: box 0x528│ 00 00 00 1c │ .... │ size: 28 0x528│ 00 00 00 01│ ....│ type: "mdta.title" ("\x00\x00\x00\x01") 0x532│00 00 00 14 64 61 74 61 00 00│....data..│ boxes[0:1]: 0x53c│00 01 00 00 00 00 74 65 73 74│......test│ │00 01 02 03 04 05 06 07 08 09│0123456789│.boxes[3].boxes[2].boxes[0].boxes[2].boxes[1]{}: box 0x546│00 00 00 25 │...% │ size: 37 0x546│ 00 00 00 02 │ .... │ type: "mdta.encoder" ("\x00\x00\x00\x02") 0x546│ 00 00│ ..│ boxes[0:1]: 0x550│00 1d 64 61 74 61 00 00 00 01│..data....│ 0x55a│00 00 00 00 4c 61 76 66 35 39│....Lavf59│ 0x564│2e 32 37 2e 31 30 30│ │.27.100│ │
- Fix panic when cancel (ctrl-c etc) before interpreter is executing. Thanks @pldin601 for reporting. #495
- Fix error using JQValue:s in assign/update paths, ex
.[<JQValue here>] = 123
#509 - Rename fields added for bit-ranges not used by a decoder from
unknown#
togap#
. "unknown" is probably a useful field name in some formats and "gap" describe better what it is. #500 - Big decode API internals refactor to split scalars types into their own go types so they can store per type specific values. This also opens up for more ways to make fq both faster and more memory efficient. It also makes the decode API more type safe and makes it possible to experiment with decode DLS that uses chained methods etc. #523
Decoder changes
apple_bookmark
New Apple BookmarkData decoder. Thanks David McDonald @dgmcdona. #493bplist
- Fix decoding of UID types
- Adds a
lost_and_found
array with unused values - Fix an endian issue for unicode strings
- Add NSKeyedArchiver to JSON helper function
from_ns_keyed_archiver
, seebplist
docs for details on how to use it. Thanks David McDonald @dgmcdona. #493
# decode bplist, from_ns_keyed_archiver converts to JSON plist and then into object data as JSON, find app bookmarks keys and expand them as bookmark data and convert to represented JSON, and finally build path to applications $ fq -r 'from_ns_keyed_archiver | (.. | .Bookmark? // empty) |= (apple_bookmark | torepr) | .. | .target_path? // empty | join("/")' recentapps.sfl2 System/Applications/Utilities/Terminal.app Applications/Spotify.app System/Applications/Calculator.app System/Applications/Preview.app Applications/Alacritty.app Applications/DB Browser for SQLite.app System/Applications/System Preferences.app System/Library/CoreServices/Applications/Directory Utility.app System/Applications/Utilities/Activity Monitor.app Applications/Safari.app
tzif
new Time Zone Information Format decoder. Thanks Takashi Oguma @bitbears-dev. #498mp4
- Map
mdta
metadata namespace and key names forilst
child boxes. #521
$ fq 'grep_by(.type=="ilst").boxes | map({key: .type, value: .boxes[0].data}) | from_entries' file.mp4 # create object with all ilst key/value pairs { "mdta.encoder": "Lavf59.27.100", "mdta.title": "test" } # query specific value $ fq -r 'grep_by(.type=="mdta.encoder").boxes[0].data | tovalue' file.mp4 Lavf59.27.100
- Support
sidx
version 1. #506 - Add description and symbolic values for traf sample flags, makes it easier to see and query for I-frames etc. #514
# which boxes has depends_on flags $ fq 'grep_by(.sample_depends_on) | parent.type' fragmented.mp4
- Support PNG codec mapping. #492
- Decode
pdin
boxes. #524 - Decode
hnti
boxes. #513
- Map
mp3_tags
Add VBRI support and split into intomp3_frame_xing
andmp3_frame_vbri
decoders. #525
Changelog
7fa8b635
Add related file format projects to README4fdb7362
Update docker-golang to 1.19.4 from 1.19.3519eff6c
Update github-go-version to 1.19.4 from 1.19.3, 1.19.3, 1.19.32a91d293
Update gomod-golang/text to 0.5.0 from 0.4.0cb15b371
added checks to prevent infinite looping and recursionc2445335
added some sfl2 test files to bplist package7d13cf73
adds flag parsing to applebookmark71b17d03
apple bookmarkdata decoder initial commit8f39ef63
bplist: Harmonize ns_keyed_archive jq style a bitcba72dbd
bplist: added overload for from_ns_keyed_archiver jq func129b4b70
bplist: doc: update docs to reflect changes to ns_keyed_archiver9dab3c60
bplist: minor fix to from_ns_keyed_archiver448c3efb
bplist: update docs with from_ns_keyed_archiver reference, add error case to functiona9047c02
bplist: updates from_ns_keyed_archiver to do automatic torepr based on format detection4a28e44f
changes decoder package name from bookmark to apple_bookmarkd0b044c2
converts to snake_case and refactors decode helperd199793a
created stack typee77f7769
decode,interp: Rename unknown gap fields from "unknown#" to "gap#"a85da295
decode: Make FieldFormat usage more consistent9b81d4d3
decode: More type safe API and split scalar into multiple types3ec0ba3f
decode: add ns_keyed_archiver, restructure apple decoder into apple package330d5f7f
decode: apple_bookmark: simplifies flag decoding93f2aa5d
decode: change PosLoopDetector to use generics7e98b538
decode: fix type on defer function call, test: add loop.fqtesta873819e
decode: fixes endian of unicode stringsf747873d
decode: implements lost and found for unreferenced objectsb45f9fa6
decode: improve stack push/popa162e07b
decode: minor change to method receiver name3232f9cc
decode: moves PosLoopDetector into its own package7c9504c7
decode: moves macho decoder to apple package70834678
decode: remove dead code from ns_keyed_archiver7ab44662
decode: remove unused field from decoder, unused parens from toreprbdb81662
decode: removed unnecessary struct98eab8cb
decode: rename parameter for consistency04379df8
decode: revert decode.D back, place posLoopDetector in apple_bookmark7fb674b5
decode: unexport methodsfa368bb7
decode: updates all.go with correct macho path0287ffa4
decoding well but torepr needs work42debe58
dev,doc,make: Cleanup makefile and have proper targets for *.md and *.svg423bab9e
dev,test: Use jqtest code from jqjq for jq tests6fc84a88
doc,dev: Add more usage and dev tips2fc16ae2
doc: Add some padding margin to formats table to make it less likely to cause git conflicts62f377c2
doc: fixes snippet for recursive bookmark searching22064f50
doc: remake4aad2fde
doc: remakeb872b1a3
doc: remake1e1fc551
fixed one more snake_case letterd0b76cae
fixes broken test and removes long link from markdown body5146f28d
fixes broken test for all.fqtest253033cc
fixes broken uid parsing in plist decoderf535ad3d
fixes spacing in jq files64351e8b
fixes tests and adds torepr testc7d00b87
fixes unknown bit ranges8f930aac
forgot to add bookmark.jq in last commit164e527b
gojq: Update rebased fq fork6c869451
gojq: Update rebased fq fork578b84d4
interp,display: Add workaround for go 1.18 when escaping 0x7f42d9f2c2
interp,help: Properly count line length when breaking on whole words8d69f1fb
interp: Change default bits_format=string6c229d73
interp: Don't auto convert to binary for string functions, is just confusing568afff3
interp: Fix panic when trigger before any context has been pushede3ae1440
interp: Rename to/from functions to to_/from_ba88a684
interp: mimic jq: if expr arg is given read stdin even if tty9bd65f93
migrates tests to per-sample filesf7d7a49f
missed a letter on last commit - converting to snake_case2f37cb55
mod: Update modules not tracked with bump55f4f1aa
moved a flag bit fields into correct positions9e5a072e
mp3_frame_tags: Covert to decode group and split to mp3_frame_{xing,vbri} decoders48522e3c
mp3_tags,mp3: Add VBRI header support and rename tags to tag as there is only one83ccedc5
mp4,decode: Properly decode ilst items (both mdta and mdir)1dea40e6
mp4,doc: Add JSON box tree example and reorder a bitb1b3b63d
mp4: Add namespace to mdta ilst boxes7b60b24a
mp4: Add pdin box supportef2d5232
mp4: Add png mapping5fb81a14
mp4: Add sym and description for traf sample flags1d6ce2c0
mp4: Decode hint and hnti child boxes9ac453a1
mp4: Fix typo in sample flags sample_is_depended_on descriptiona23fe618
mp4: sidx version 1 segment_duration is s643942db79
pkg/decode/D: Adds PushAndPop, Push, Pop methods. doc: adds help_applebookmark.fqtestdecode: converts applebookmark to use new d.PushAndPop method0c216dff
refactors some decoder logic in apple_bookmark for better querying34db9d7f
regenerated docs, added tests, fixed torepr0a72635a
remade documentation1352598a
removed commented out line81269430
removed unnecessary conversions5b1455e7
removed unused function63a3ca20
removes underscore from apple_bookmark package namea351c346
removes unused function2ee6360b
support tzif (time zone information format)8d5dcff8
test: applebookmark: adds problematic test case63a4e80c
test: fixed doc test47a568e0
text,test: Unbeak base64 tests44c91d82
tweaks apple_bookmark markdown documentationfd22426b
tzif: add help_tzif.fqtestc4e7fc79
tzif: moved document to tzif.mdabde823a
tzif: use PeekFindByte() to find end of the string4481a77a
tzif: use scalar.Fn() to define a mapper ad hocdbc6fccd
updated doc with apple referencef5e25fca
updated docs6f4d1cb1
updated documentationb2aeac6a
updates bplist fq testsa23ac8f5
updates fqtest for torepr in apple_bookmarkdata
0.1.0
Adds avi
decoder and replace raw
with more convenient bits
and bytes
format. Otherwise mostly small updates and bug fixes.
Increase minor version. fq does not use sementic versioning (yet) but it's probably a good idea to start increase minor version when adding features to be able to do patch releases.
In other fq related news:
- I gave a talk about fq at No Time To Wait 6 a conference about open media, standardization, and audiovisual preservation.
- While prototyping writing decoders directly in jq for fq I ended up implementing jq in jq. Still thinking and working on how to do decoders in jq.
Changes
- Replace
raw
format withbits
andbytes
format that decodes directly to a binary with different unit size.$ echo -n 'hello' | fq -d bytes '.[-3:]' > last_3_bytes $ echo 'hello' | fq -d bytes '.[1]' 101 $ echo 'hello' | fq -c -d bits '[.[range(8)]]' [0,1,1,0,1,0,0,0]
Decoder changes
avc_au
Support annexb format (used in AVI). #476avi
Add AVI (Audio Video Interleaved) decoder. #476# extract samples for stream 1 $ fq '.streams[1].samples[] | tobytes' file.avi > stream01.mp3
bits
Replacesraw
but is a binary using bit units. #485bytes
Replacesraw
but is a binary using byte units. #485bplist
- Fix signed interger decoding. #451 @dgmcdona
- Use correct size for references and check for infinite loops. #454 @dgmcdona
flac_frame
Correctly decode zero escape sample size. #461id3v2
Fix decoding of COMM and TXXX with missing null terminator. #468matroska
Updated to latest specification. #455mp3_frame
Use frame size calculation from spec instead of own as it seems to not work in some cases. #480mp3_frame_tags
Replacesxing
and also decodes "lame extensions" for both Xing and Info. #481raw
Removed. #485wav
More codec symbol names and now shares RIFF code with AVI decoder. #476yaml
Fix type panic for large intergers. #462
Changelog
7b6492ee
Improve README.md a bit, one more demo and move up usage4e069625
Update docker-golang to 1.19.2 from 1.19.1e0334497
Update docker-golang to 1.19.3 from 1.19.2f3f2648b
Update github-go-version to 1.19.2 from 1.19.1, 1.19.1, 1.19.1003197eb
Update github-go-version to 1.19.3 from 1.19.2, 1.19.2, 1.19.2453963dd
Update github-golangci-lint to 1.50.1 from 1.50.056dcb3a0
Update gomod-BurntSushi/toml to 1.2.1 from 1.2.0101b2806
Update gomod-golang/text to 0.3.8 from 0.3.7d80f12c7
Update gomod-golang/text to 0.4.0 from 0.3.8753927ba
Update make-golangci-lint to 1.50.1 from 1.50.04d8dd5c5
adds check for recursion in decodeReference, adds test to verify fixb7c4576c
adds necessary cast46b7ab32
adds test to verify fix4ee7dd8a
changes Errorf to Fatalf on infinite loops41b2d1ad
cli: Better decode error help7254b0f9
decode,elf,fuzz: TryBytesRange error on negative sizebafd1f56
decode,fuzz: Signed integer (S) read require at least one bit2a86d323
doc,rtmp,pcap,markdown: Add more examples0b9b0173
doc: Add gomarkdown to license/dependencies4bfd9d81
doc: Add link to nttw6 presentation video and slidesfb1a91ac
drop indented else block per lint4dd594c1
fixes bad path in test outputf9a1d3f4
fixes calculation of floating point lengths236fbc17
fixes reference calculation to use reference size from trailerac86f931
fixes signed integer parsingfb2a2b94
flac,fuzz: Fatal error on negative partition sample count7859be1e
flac_frame: Properly decode zero escape sample size7cb2a6c9
fuzz: gotip not needed anymorecef4245b
fuzz: make fuzz GROUP=mp4 to fuzz one group413d4250
gofmt349d9497
gojq: Update rebased fq fork450f5844
gojq: Update rebased fq forkd8641ab1
gomod: Update modules that lack bump configf66e2244
id3v2: In the wild COMM and TXXX frame might not have a null terminatorb09d6116
makes dictionary key type checking more sensibled07b2eec
markdown,fuzz: Update gomarkdown646f32d5
matroska: Fix path tests and make _tree_path more robuste748079e
matroska: Update spec and regenerate1c7d3252
mod: Update ones without bump config2de87539
mp3_frame: Fix issue calc frame size for some configsc3a0686c
mp3_frame_tags: Refactor and rename xing format to mp3_frame_tagsd75748d8
mp4: Decode more sample flagsc93301fc
raw,bits,bytes: Replace raw format with bits and bytes format that decode to a binaryb08e25ce
removes unnecessary cast2b3adbe8
renames test data file0cf46e11
wav,avi,avc_au: Add avi decoder and refactor wav decoder26069167
yaml,fuzz: gojq.Normalize value to fix type panic
0.0.10
Changes
- Add
bplist
Apple Binary Property List decoder. Thanks David McDonald @dgmcdona #427 - Add
markdown
decoder. #422 - Fix panic when interrupting (ctrl-c) JSON output (
fq tovalue file
etc), #440 - Fix issue using
debug
(and some other native go iterator functions) insidepath(...)
, which is used by assign (... = ...
) expressions etc. #439 - Fix issue and also make
toactual
andtosym
work more similar totovalue
. #432 - Fix issue with unknown fields (gaps found after decoding) where one continuous gap could end up split into two of more unknown fields. #431
- More format documentation and also nicer help output. Also now all documentation is in markdown format. #430 #422
# or help(matroska) in the REPL $ fq -h matroska matroska: Matroska file decoder Decode examples =============== # Decode file as matroska $ fq -d matroska . file # Decode value as matroska ... | matroska Lookup element using path ========================= $ fq 'matroska_path(".Segment.Tracks[0)")' file.mkv Get path to element =================== $ fq 'grep_by(.id == "Tracks") | matroska_path' file.mkv References ========== - https://tools.ietf.org/html/draft-ietf-cellar-ebml-00 - https://matroska.org/technical/specs/index.html - https://www.matroska.org/technical/basics.html - https://www.matroska.org/technical/codec_specs.html - https://wiki.xiph.org/MatroskaOpus
Decoder changes
ar
Allow empty integer strings. For example owner id can be an empty string. #428bitcoin_blkdat
Assert that there is a header. As the format is part of the probe group this speeds up probing. #402bplist
Add Apple Binary Property List decoder.$ fq '.objects.entries[0] | .key, .value' Info.plist │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].key{}: 0x32│ 5c │ \ │ type: "ascii_string" (5) (ASCII encoded string) 0x32│ 5c │ \ │ size_bits: 12 │ │ │ size: 12 0x32│ 43 46 42 75│ CFBu│ value: "CFBundleName" 0x3c│6e 64 6c 65 4e 61 6d 65 │ndleName │ │00 01 02 03 04 05 06 07 08 09│0123456789│.objects.entries[0].value{}: 0x1ea│ 5f │ _ │ type: "ascii_string" (5) (ASCII encoded string) 0x1ea│ 5f │ _ │ size_bits: 15 0x1ea│ 10 │ . │ large_size_marker: 1 (valid) 0x1ea│ 10 │ . │ exponent: 0 0x1ea│ 18 │ . │ size_bigint: 24 │ │ │ size: 24 0x1ea│ 41 70 70 6c│ Appl│ value: "AppleProResCodecEmbedded" 0x1f4│65 50 72 6f 52 65 73 43 6f 64│eProResCod│ 0x1fe│65 63 45 6d 62 65 64 64 65 64│ecEmbedded│
- Supports
torepr
$ fq torepr.CFBundleName Info.plist "AppleProResCodecEmbedded"
- Supports
elf
- More robust decoding when string tables are missing. #417
# extract entry opcodes and disassemble with ndisasm $ fq -n '"f0VMRgIBAQAAAAAAAAAAAAIAPgABAAAAeABAAAAAAABAAAAAAAAAAAAAAAAAAAAAAAAAAEAAOAABAAAAAAAAAAEAAAAFAAAAAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAAAAAAAAEAAAAAAAAAAQAAAAAAIAAAAAAAsDxmvwYADwU=" | frombase64 | . as $b | elf | $b[.header.entry-.program_headers[0].vaddr:]' \ | ndisasm -b 64 - 00000000 B03C mov al,0x3c 00000002 66BF0600 mov di,0x6 00000006 0F05 syscall
- Now decodes program header notes. #421
- More robust decoding when string tables are missing. #417
markdown
Add decoder. Is used in fq to render CLI help. #422# array with all level 1 and 2 headers $ fq -d markdown '[.. | select(.type=="heading" and .level<=2)?.children[0]]' README.md [ "fq", "Usage", "Presentations", "Install", "TODO and ideas", "Development and adding a new decoder", "Thanks and related projects", "License" ]
matroska
Add support for sample lacing. Used by FLAC samples etc. #404mp4
More codec names and also use official names from mp4ra.org. #424
# show details of first two track in file $ fq -o line_bytes=10 '.tracks[0,1]' big_buck_bunny.mp4 │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[0]{}: track 0x00910a│20 68 10 01 a0 40 0e 20 8c 1b│ h...@. ..│ samples[0:1295]: 0x009114│c2 2b 99 09 84 42 60 a8 c4 60│.+...B`..`│ * │until 0x541697.7 (5473678) │ │ │ │ │ id: 1 │ │ │ data_format: "mp4a" (MPEG-4 Audio) │00 01 02 03 04 05 06 07 08 09│0123456789│.tracks[1]{}: track 0x00a5e6│ 00│ .│ samples[0:1440]: 0x00a5f0│00 00 0c 06 00 07 8b 71 b0 00│.......q..│ 0x00a5fa│00 03 00 40 80 00 00 00 15 06│...@......│ * │until 0x540959.7 (5464939) │ │ │ │ │ id: 2 │ │ │ data_format: "avc1" (Advanced Video Coding)
html
Handle leading doc type and processing directives. #414
Changelog
a77cec92
Added documentation and tests, fixed bad date parsingd784db69
Adds support for Apple Binary Plist, version 005711f290
Code fixes from PR, still need to add tests and testdata6b04f2de
Documentation cleanupbcccde23
Fixes and embeds documentationebae938d
Fixes bug in integer parsing368d183b
Size check on nBits to save memory84ca1010
Update docker-golang from 1.19.0 to 1.19.1c47c3866
Update github-go-version from 1.19.0, 1.19.0, 1.19.0 to 1.19.1816169b6
Update github-golangci-lint to 1.50.0 from 1.49.021f2980e
Update make-golangci-lint to 1.50.0 from 1.49.05f619940
adds function for decoding fixed sized arraysf08f44f1
ar: Integer strings might be empty004406de
bitcoin_blkdat,bitcoin_block: Make sure there is a header if blkdat421b2b30
bplist: Fix unknown field for singletons and add torepr tests16b01211
bplist: Make torepr convert to valuesfe64530e
csv: Add tsv and header examplecb3dc802
decode,tar: Add scalar description and Try* helpersa6429ffe
decode: Remove RangeSorted flag as we can decide on array/struct insteada468684a
deps: Manual update ones not using bumpa7a101ca
doc,help: Nicer format help and move help tests into each format725ab1b1
doc,html,xml: Add more documentation and examplesabd19ed8
doc: Fix format sections a bit0fdc03a4
doc: Fix some incorrect example prompts5382d46a
elf: Basic program header notes decoding12105d8c
elf: Treat missing string tables as empty to be more robust3deceeeb
fixes from PR comments226a9a3e
generics: Use more from x/exp404b1704
gojq: Update fq fork376f0ebb
gojq: Update rebased fq fork87b2c6c1
help,doc: Use markdown for format documentation again8016352b
html: Handle html with leading doctype etc768df301
interp,decode: For struct use map to lookup fieldc4219d69
interp: Fix interrupt panic for cli eval00ee10a1
interp: Make to{actual,sym} behave similar to tovalue00a50662
markdown: Add decoder7749e1b5
matroska: Add proper lacing support20a15372
mp4: Fix data_format typo2655ba09
mp4: More codec names (from mp4ra.org)7cd43b49
perfomance: increase performance by map usage6a6fec54
range,decode: Use own range sort impl to speed up a bit0f35fe48
ranges,decode: Correctly skip empty ranges when adding unknown fieldsea81efec
readline: Update rebased fq fork369f4016
removed unneccessary type conversions3198602d
removed unused return type7d865343
sortex: Package with type safe sort helpers808202fa
test: Skip go test with -race by default12836abe
updates fqtest1e47f4f2
updates tests post integer-bug fix3d8ea1de
updates torepr for data type1385b5d0
wasm: Add some documentationd6316d5c
wav: Decode smpl chunk
0.0.9
Changes
- New
wasm
WebAssembly Binary Format decoder by Takashi Oguma @bitbears-dev
# show part of code_section $ fq '.sections[4].content.code.x[0].code.e | d' add.wasm │00 01 02 03 04 05 06 07 08 09│0123456789│.sections[4].content.code.x[0].code.e[0:4]: │ │ │ [0]{}: in 0x3c│ 20│ │ opcode: "local.get" (0x20) 0x46│01 │. │ x: 1 (valid) │ │ │ [1]{}: in 0x46│ 20 │ │ opcode: "local.get" (0x20) 0x46│ 00 │ . │ x: 0 (valid) │ │ │ [2]{}: in 0x46│ 6a │ j │ opcode: "i32.add" (0x6a) │ │ │ [3]{}: in 0x46│ 0b │ . │ opcode: "end" (0xb)
# count opcode usage $ fq '.sections[] | select(.id == "code_section") | [.. | .opcode? // empty] | count | map({key: .[0], value: .[1]}) | from_entries' add.wasm { "end": 1, "i32.add": 1, "local.get": 2 }
# list exports and imports $ fq '.sections | {import: map(select(.id == "import_section").content.im.x[].nm.b), export: map(select(.id == "export_section").content.ex.x[].nm.b)}' add.wasm { "export": [ "memory", "add" ], "import": [] }
- Decode value display now shows address bar on new format or buffer. Should make it easier to spot changes and read hex and ASCII view. #365
Examples of PCAP with different formats and TCP stream buffers:... │00 01 02 03 04 05 06 07 08 09│0123456789│ packet{}: (ether8023_frame) 0x00668│ 00 0a 95 67 49 3c │ ...gI< │ destination: "00:0a:95:67:49:3c" (0xa9567493c) 0x00668│ 00 c0 f0│ ...│ source: "00:c0:f0:2d:4a:a3" (0xc0f02d4aa3) 0x00672│2d 4a a3 │-J. │ 0x00672│ 08 00 │ .. │ ether_type: "ipv4" (0x800) (Internet Prot... │00 01 02 03 04 05 06 07 08 09│0123456789│ payload{}: (ipv4_packet) 0x00672│ 45 │ E │ version: 4 ... │00 01 02 03 04 05 06 07 08 09│0123456789│ payload{}: (tcp_segment) 0x00686│ 00 50 │ .P │ source_port: "http" (80) (World Wide ... ... │ │ │ ipv4_reassembled[0:0]: │ │ │ tcp_connections[0:1]: │ │ │ [0]{}: tcp_connection │ │ │ client{}: │ │ │ ip: "192.168.69.2" │ │ │ port: 34059 │ │ │ has_start: true │ │ │ has_end: true │ │ │ skipped_bytes: 0 │00 01 02 03 04 05 06 07 08 09│0123456789│ 0x000│47 45 54 20 2f 74 65 73 74 2f│GET /test/│ stream: raw bits 0x000│65 74 68 65 72 65 61 6c 2e 68│ethereal.h│ * │until 0x1bc.7 (end) (445) │ │ ...
- Add
--unicode-output
/-U
argument to force use of Unicode characters to improve output readability. #377- For example useful when piping to less and you want fancy unicode and colors:
fq -cU d file | less -r
- For example useful when piping to less and you want fancy unicode and colors:
to_entries
now preserves struct field order. #340- Experimental
`raw string`
literal support. Work the same as golang raw string literals. Useful in REPL when pasting things etc but should probably be avoided in jq scripts. #371 - Properly fail lexing of invalid binary, octal and hex number literals. #371
- REPL completion now include all functions. Before some functions with multiple argument counts were skipped. #375
- Switch to new gopacket fork with speedup and bug fixes. Remove SLL2 workarounds in fq. #386
Decoder changes
csv
Correctly handle decode values whentocsv
normalize to strings. Before array and object ended up being JSON serialized to strings. #341- Normalize to strings is done so that non-string scalars can be used:
$ fq -n '[[1,true,null,"a"]] | tocsv' "1,true,,a\n"
- Normalize to strings is done so that non-string scalars can be used:
dns
DNS over UDP format was accidentally used to probe TCP streams #366elf
Remove redundantprogram_header
structflac
- Add 32 bit samples support. #378 Thanks @ktmf01
- Properly decode/checksum samples in partitions with raw samples. #379 Thanks @ktmf01
Now successfully decodes all test cases from https://github.com/ietf-wg-cellar/flac-test-files
jsonl
Add decoder. Decodes JSON lines. There is alsofromjsonl
andtojsonl
. #374macho
- Split FAT Macho decoding into
macho_fat
format which also fixed handling of file offsets in sections. #362 - Decode symbol and string sections. #352
- Split FAT Macho decoding into
matroska
Remove new lines in descriptions. Messes up tree. #369mp3_frame
- Support LSF (low sampling frequency) frames. #376
- Skip trying to figure out what main data is for current frame and not. Was incorrect and doing it properly probably requires hoffman decoding. #376
pcap
Support files with nanosecond precision. Has a different magic. #382prores_frame
Add basic decoder. Decodes container and fram header. #396 Thanks @Doctor-love for test filestar
Fix regression when decoding number fields. Now ok again to be empty string. #372wasm
Add WebAssembly Binary Format decoder. #383 Thanks to Takashi Oguma @bitbears-dev- Decodes to a tree following the WASM binary grammar specification
yaml
Fail on trailing data. Before it succeeded with the last value. #373zip
- Don't require PK header as there seems to be zip files with prepended data. #359
- Correctly limit amount of backwards search for EOCD (end of content directory). #363
xml
Correctly handle decode values whentoxml
normalize to strings. Before array and object ended up being JSON serialized to strings. #341xml
- Change attribute prefix to
@
instead of-
and make it an optionattribute_prefix
. #401 - Skip default namespace in element names. #389
- Always include attributes and children even when empty in array mode. Makes it a lot easer to work with as you can assume
.[1]
will be attributes and so on. #357 - Normalize to strings is done so that non-string scalars can be used:
$ fq -nr '{a: {"-boolean": true, "-number": 123, "-null": null}} | toxml' <a boolean="true" null="" number="123"></a>
- Allow and ignore trailing
<?procinstr?>
and improve trailing data error message. #368 - Correctly sort if any
#seq
is found and also properly sort negative#seq
. #384
- Change attribute prefix to
Changelog
0cd846a1
*extra: Rename extra to just ex and refactor to use genericsfb583e2c
Add 32 bps FLAC to testc1d5b2b1
Add sample size entry to list for 32bps flac streams3f209c46
Fix decoding of FLAC raw entropy partition25061aca
Update docker-golang from 1.18.4 to 1.18.50de2c906
Update docker-golang from 1.18.5 to 1.19.07b8d95bf
Update github-go-version from 1.18.4, 1.18.4, 1.18.4 to 1.18.5103991f7
Update github-go-version from 1.18.5, 1.18.5, 1.18.5 to 1.19.04255b87a
Update github-golangci-lint from 1.47.2 to 1.47.3198305ec
Update github-golangci-lint from 1.47.3 to 1.48.0fa9fec30
Update github-golangci-lint from 1.48.0 to 1.49.0f579e9c3
Update make-golangci-lint from 1.47.2 to 1.47.3c8069d22
Update make-golangci-lint from 1.47.3 to 1.48.0004eb564
Update make-golangci-lint from 1.48.0 to 1.49.0abcc7366
add ULEB and SLEB to known words for spell check9238251b
ci: Skip -race for windows and macos913f5780
columnwriter,dump: Add Column interface and refactor into BarColumn and MultiLineColumn5d9ffead
decode,scalar: Map empty string also else sym might ends up nil326dada7
decode: Add LEB128 readers502f451c
decode: Refactor to use scalar type assert helper840292ba
decode: Simplify compound range sort behaviour15f7c67a
dev,fuzz: Add some useful retrigger snippets46dca8cd
dns: Don't use dns (udp) format for tcp alsoc233215a
dns: Rename isTCP to hasLengthHeadered424783
doc,interp: Update and add more examplesf247edb5
doc: Update README demo a bit with new features3613b6d4
elf: Remove redundant program_header struct8a19978b
flac: Make gen script generate correct fqtest files2bfbe9a9
flac_frame: Cleanup some dev lefterovers and todos64b23659
fqtest: Run tests in parallellaf35b284
gojq: Preserve keys order for to_entries when used with JQValue804ad1e2
gojq: Update fq forkadd3dcfd
gojq: Update fq fork, fix scope argcount issued898732c
gojq: Update fq fork, new scope function, rawstring, stricter integers394717ca
gopacket: Switch/update to new fork, remove SLL2 hack4eae7ffd
interp,doc: Add -R raw string slurp hint to -s helpd8792fd1
interp,dump: Correctly flush columns if data will be shown29005c70
interp,dump: Show address bar for root, nested roots and on format changec7559b59
interp: Add --unicode-output/-U to force use of unicode9e447c9a
interp: Use RegisterFS instead of format files701c67c1
jsonl: Add decoder, also tojsonl encoderbc6cffde
lint,decode,fuzz:: Fix nilerr warnings, one real one should be ignored for now3c21b058
lint: Fix ioutil deprecation, reformat for new doc standardb2d4e6d9
macho: Decode cmd symtab symbols725c8e83
macho: Split into macho/macho_fat, fix offset issue and add string decoding2e407386
matroska: Strip newlines in descriptioncf15661e
mp3_frame: Add LSF support and fix incorrect main data handling74c7dc4e
pcap: Add ns support and add header field8fc43533
prores_frame: Add basic container and frame header decoderdc32ac08
script: Use strings.Builder to collect output0d44b937
tar: Some number fields can be empty545dac8c
test: Update tests, go 1.19 uses \xff instead of \u00ffce438872
wasm:make doc
074c22c9
wasm: add-timeout 20m
for go test to workaround ci test failcd037c51
wasm: add comment to clarify lazy initializationf73965d2
wasm: add wasm to probe list00869b37
wasm: avoid race conditiondb8021c9
wasm: define and use constants for some insturctionsbcc0dfd9
wasm: fix comment format289ddf59
wasm: fix lint issues3fca7cc0
wasm: fix lint issuescbb5a8ed
wasm: further simplification934ed9a8
wasm: initial versione5cf1731
wasm: make the godoc formatter happyb0f3fec8
wasm: remove nolint:unparam which is no longer needede1691dec
wasm: remove unused functionae4529c4
wasm: rungolangci-lint run --fix
fead68de
wasm: tidy up3298d181
wasm: to be able to probe2eb17505
wasm: update testsd5d9e738
wasm: use FieldULEB128() / FieldSLEB128() defined in the upstream7401d141
wasm: use WRITE_ACTUAL=1 to generate .fqtest files2037b86a
wasm: use map, not switchae08bf70
wasm: use s.ActualU() instead of s.Actual.(uint64)63f4a726
wasm: use scalar.UToSymStr for simplicity0ad5a8ec
wasm: use underscores for symbol valuesfa20c74c
xml,csv,interp: Handle JQValue when string normalizingf4e01372
xml,html: Always include attrs and children in array mode9a5fcc89
xml: Allow trailing71900c2a
xml: Correctly sort if one #seq is found and allow negative seq numbers716323ce
xml: Even more namespace fixesf24d685a
xml: Keep track of default namespace and skip it element names095e1161
xml: Switch from "-" to "@" as attribute prefix and make it an option3623eac3
yaml: Error on trailing yaml/jsond607bee1
zip: Correctly limit max EOCD find19b70899
zip: Skip header assert as there are zip files with other things appended
0.0.8
Changes
- Add support for some common structured serialization formats: #284 #335
- XML,
toxml
,fromxml
options for indent, jq mapping variants (object or array) and order preservation - HTML,
fromhtml
options for indent, jq mapping variants (object or array) and order preservation - TOML,
totoml
,fromtoml
- YAML,
toyaml
,fromyaml
- jq-flavored JSON (optional key quotes and trailing comma)
tojq
,fromjq
options for indent #284# query a YAML file $ fq '...' file.yml # convert YAML to JSON # note -r for raw string output, without a JSON string with JSON would outputted $ fq -r 'tojson({indent:2})' file.yml $ fq -nr '{hello: {world: "test"}} | toyaml, totoml, toxml, tojq({indent: 2})' hello: world: test [hello] world = "test" <hello> <world>test</world> </hello> { hello: { world: "test" } } $ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a"' b $ echo '<doc><element a="b"></doc>' | fq -r '.doc.element."-a" = "<test>" | toxml({indent: 2})' <doc> <element a="<test>"></element> </doc>
- CSV,
tocsv
,fromcsv
options for separator and comment character$ echo -e '1,2\n3,4' | fq -rRs 'fromcsv | . + [["a","b"]] | tocsv' 1,2 3,4 a,b
- XML,
- Add support for binary encodings
- Base64.
tobase64
,frombase64
options for encoding variants.$ echo -n hello | base64 | fq -rRs 'frombase64 | tostring' hello
- Hex string.
tohex
,fromhex
- Base64.
- Add support for text formats
- XML entities
toxmlentities
,fromxmlentities
- URL
tourl
,fromurl
$ echo -n 'https://host/path/?key=value#fragment' | fq -Rs 'fromurl | ., (.host = "changed" | tourl)' { "fragment": "fragment", "host": "host", "path": "/path/", "query": { "key": "value" }, "rawquery": "key=value", "scheme": "https" } "https://changed/path/?key=value#fragment"
- URL path encoding
tourlpath
,fromurlpath
- URL encoding
tourlencode
,fromurlencode
- URL query
tourlquery
,fromurlquery
- XML entities
- Add support for common hash functions:
- MD4
tomd4
- MD5
tomd5
$ echo -n hello | fq -rRs 'tomd5 | tohex' 5d41402abc4b2a76b9719d911017c592
- SHA1
tosha1
- SHA256
tosha256
- SHA512
tosha512
- SHA3 224
tosha3_224
- SHA3 256
tosha3_256
- SHA3 384
tosha3_384
- SHA3 512
tosha3_512
- MD4
- Add support for common text encodings:
- ISO8859-1
toiso8859_1
,fromiso8859_1
- UTF8
tutf8
,fromutf8
- UTF16
toutf16
,fromutf16
- UTF16LE
toutf16le
,fromutf16le
- UTF16BE
toutf16be
,fromutf16be
$ echo -n 00680065006c006c006f | fq -rRs 'fromhex | fromutf16be' hello
- ISO8859-1
- Add
group
function, same asgroup_by(.)
#299 - Update/rebase readline dependency (based on @tpodowd https://github.com/chzyer/readline/pull/207) #305 #308
- Less blinking/redraw in REPL
- Lots of small bug fixes
- Update/rebase gojq dependency #247
- Fixes JQValue destructing issue (ex:
<some object JQValue> as {$key}
)
- Fixes JQValue destructing issue (ex:
- Major rewrite/refactor how native function are implemented. Less verbose and less error-prone as now shared code takes care of type casting and some argument errors. #316
- Add
tojson($opts)
that support indent option.tojson
still works as before (no indent).$ echo '{a: 1}' | fq -r 'tojson({indent: 2})' { "a": 1 }
- Rename
--decode-file
(will still work) to--argdecode
be be more consistent with existing--arg*
arguments. #309 - On some decode error cases fq can now keep more of partial tree making it easier to know where it stopped #245
- Build with go 1.18 #272
Decoder changes
bitcoin
Add Bitcoin blkdat, block, transcation and script decoders #239elf
Use correct offset to dynamic linking string table #304tcp
Restructure into separate client/server objects and addskipped_bytes
(number of bytes with known missing ACK),has_start
(has first byte in stream) andhas_end
(has last byte in stream) per direction #251- Old:
│00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection │ │ │ source_ip: "192.168.69.2" │ │ │ source_port: 34059 │ │ │ destination_ip: "192.168.69.1" │ │ │ destination_port: "http" (80) (World Wide Web HTTP) │ │ │ has_start: true │ │ │ has_end: true 0x000│47 45 54 20 2f 74 65 73│GET /tes│ client_stream: raw bits 0x008│74 2f 65 74 68 65 72 65│t/ethere│ * │until 0x1bc.7 (end) (44│ │ 0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│ server_stream: raw bits 0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│ * │until 0x191.7 (end) (40│ │
- New:
│00 01 02 03 04 05 06 07│01234567│.tcp_connections[0]{}: tcp_connection │ │ │ client{}: │ │ │ ip: "192.168.69.2" │ │ │ port: 34059 │ │ │ has_start: true │ │ │ has_end: true │ │ │ skipped_bytes: 0 0x000│47 45 54 20 2f 74 65 73│GET /tes│ stream: raw bits 0x008│74 2f 65 74 68 65 72 65│t/ethere│ * │until 0x1bc.7 (end) (44│ │ │ │ │ server{}: │ │ │ ip: "192.168.69.1" │ │ │ port: "http" (80) (World Wide Web HTTP) │ │ │ has_start: true │ │ │ has_end: true │ │ │ skipped_bytes: 0 0x000│48 54 54 50 2f 31 2e 31│HTTP/1.1│ stream: raw bits 0x008│20 32 30 30 20 4f 4b 0d│ 200 OK.│ * │until 0x191.7 (end) (40│ │
zip
Add 64-bit support and adduncompress
option #278matroska
Update and regenerate based on latest spec and also handle unknown ids better #291mp4
Changes:- Fix PSSH decode issue #283
- Add track for track_id references without tfhd box
- Makes it possible to see samples in fragments without having an init segment.
Note it is possible to decode samples in a fragment file by concatenating the init and fragment file ex:
cat init frag | fq ...
. - Add
senc
box support #290 - Don't decode encrypted samples #311
- Add
track_id
to tracks #254 - Add fairplay PSSH system ID #310
- Properly handle
trun
data offset #294 - Skip decoding of individual PCM samples for now #268
- Add
mvhd
,tkhd
,mdhd
andmehd
version 1 support #258 - Make sure to preserve sample table order #330
fairplay_spc
Add basic FairPlay Server Playback Context decoder #310avc_pps
Correctly check for more rbsp data
Changelog
210940a4
Update docker-golang from 1.18.1 to 1.18.2fbeabdc3
Update docker-golang from 1.18.2 to 1.18.351a414db
Update docker-golang from 1.18.3 to 1.18.43017e8b4
Update github-go-version from 1.18.1, 1.18.1, 1.18.1 to 1.18.2c597f7f7
Update github-go-version from 1.18.2, 1.18.2, 1.18.2 to 1.18.3dd283923
Update github-go-version from 1.18.3, 1.18.3, 1.18.3 to 1.18.4d10a3616
Update github-golangci-lint from 1.45.2 to 1.46.075b5946c
Update github-golangci-lint from 1.46.0 to 1.46.13ffa9efb
Update github-golangci-lint from 1.46.1 to 1.46.24be8cb91
Update github-golangci-lint from 1.46.2 to 1.47.01b8f4be8
Update github-golangci-lint from 1.47.0 to 1.47.1fc596a7a
Update github-golangci-lint from 1.47.1 to 1.47.262be9223
Update gomod-BurntSushi/toml from 1.1.0 to 1.2.05db7397a
Update make-golangci-lint from 1.45.2 to 1.46.0456742ea
Update make-golangci-lint from 1.46.0 to 1.46.106757119
Update make-golangci-lint from 1.46.1 to 1.46.23d69e9d0
Update make-golangci-lint from 1.46.2 to 1.47.02170925d
Update make-golangci-lint from 1.47.0 to 1.47.1c4199c0f
Update make-golangci-lint from 1.47.1 to 1.47.202f00be9
Update usage.md75169a65
asn1: Add regression test for range decode fix ##330b0096bc1
avc_pps: Correct check if there is more rbsp data5d67df47
avro_ocf: Fix panic on missing meta schema417255b7
bitcoin: Add blkdat, block, transcation and script decodera6a97136
decode: Cleanup Try/ pairs3ce660a2
decode: Keep decode tree on RangeFn errorc4dd518e
decode: Make compound range sort optional8bb4a6d2
decode: Range decode with new decoder to preserve bit reader342612eb
dev: Cleanup linters and fix some unused args78aa96b0
dev: Cleanup some code to fix a bunch of new linter warnings3570f1f0
doc: Add more related tools7aff654a
doc: Clarify decode, slurp and spew args0863374f
doc: Correct bencode spec URL10cc5518
doc: Improve and cleanup text formatsb1006119
doc: Typos and add note about Try* functionsc27646a6
doc: Update and shorten README.md a bitb0388722
doc: Use singular jq value to refer to jq valuea980656c
doc: go 1.18 and improve intro text a bita64c28d9
dump: Skip JQValueEx if there are not options40481f66
elf,fuzz: Error on too large string tablef66a359c
elf: Use correct offset to dynamic linking string table64f3e5c7
fairplay: Add basic SPC decoder and PSSH system idcae288e6
format,intepr: Refactor json, yaml, etc into formats also move out related functionse9d9f8ae
fq: Use go 1.18377af133
fqtest: Cleanup path usage2464ebc2
fuzz: Replace built tag with FUZZTEST env and use new interp api0f78687b
gojq: Fix JQValue index and destructuring issue and rebase fq fork59c7d0df
gojq: Rebase fq forkc57dc17d
gojq: Rebase fq fork9a7ce148
gojq: Update rebased fq forkc1a0cda5
gojq: Update rebased fq fork32361dee
gojqextra: Cleanup gojq type cast code9b2e474e
gojqextra: Simplify function type helpersfd302093
hevc_vps,fuzz: Error on too many vps layersefa5e23a
icc_profile: Correctly clamp align padding on EOF1ddea1ad
interp,format: Refactor registry usage and use function helpersa3c33fc1
interp: Add group/095e61965
interp: Add internal is helpers3b717c3b
interp: Add to/from for some common serialzations, encodings and hashes6b088000
interp: Cast jq value to go value properly for encoding functionsf5be5180
interp: Cleanup and clarify some format namingc7701851
interp: Extract to/from map/struct to own package8dde3ef5
interp: Fix crash when including relatve path when no search paths are set735c443b
interp: Improve type normalization and use it for toyaml and totoml81a014ce
interp: Make empty _finally fin error on error2dc509ab
interp: Refactor dump and revert #259 for nowab8c728a
interp: Rename --decode-file to --argdecode to be more consistentdff3cc11
interp: dump: Fix column truncate issue with unicode bars5109df4a
interp: dump: Show address bar for nested roots80214921
interp: help: Fix incorrect options example76714349
mapstruct: Handle nested values when converting to camel casec92f4f13
matroska: Update ebml_matroska.xml and allow unknown idsc2a359bd
mod: Update golang.org/x/{crypto,net}3780375d
mp3: Use d.FieldValueU and some cleanup7b27e506
mp4,bitio: Fix broken pssh decoding and add proper reader cloning to bitio6b00297e
mp4,senc: Refactor current track/moof tracking and add senc box support8228ecae
mp4: Add track id field and add track for tfhd with unseen track_idea2cc3c2
mp4: Don't decode encrypted samplesc6d0d89c
mp4: Don't range sort samples, keep sample table order7d25fbfd
mp4: Properly use trun data offsetba844eb0
mp4: Skip fields for pcm samples for now0e02bb66
mp4: iinf: Only assume sub boxes for version 02e328180
mp4: mvhd,tkhd,mdhd,mehd: Add version 1 support44bab274
readline: Rebase on top of tpodowd's redraw/completion fixes PRa5122690
readline: Rebase on top of tpodowd's update PR54dcdce9
readline: Update fq fork6e7267d2
readme: add MacPorts install details76161a1b
scalar,mp4,gzip,tar: Add timestamp to description9133f0e5
scalar: Add *Fn type to map value and clearer naming34cf5442
tcp: Split into client/server structs and add skipped_bytes and has_start/end per direction1aaaefb0
wav,bencode,mpeg_ps_packet,id3v1: Random fixes47350e46
zip: Add uncompress=false test and some docse6412744
zip: Add zip64 support and uncompress optionaa694e3f
zip: s/Decompress/Uncompress/
0.0.7
Changes
- Format specific options
- Formats can now have own options
- Example to skip decoding of samples in a mp4 file use:
fq -d decode_samples=false d file.mp4
or... | mp4({decode_samples: false}})
- To see supported options for a formats see formats documentation, use
fq -h mp4
orhelp(mp4)
in the REPL.
- gojq fork rebase:
- Many performance improvements from upstream
- Assign to a JQValue will now shallowly turn it into a jq value and then be assigned.
- Refactor and rewrote large parts to make it easier to rebase and maintain in the future.
Decoder changes
amf0
Add Action Message Format 0 decoder #214hevc_pps
Add H.265/HEVC Picture Parameter Set decoder #210hevc_sps
Add H.265/HEVC Sequence Parameter Set decoder #210hevc_vpc
Add H.265/HEVC Video Parameter Set decoder #210mp3
Addmax_unique_header_config
andmax_sync_seek
options #242mp4
Simplify granule structure a bit #242mp4
Adddecode_samples
andallow_truncate
options #242flac_frame
Addbits_per_sample
option #242icmpv6
Add Internet Control Message Protocol v6 decoder #216id3v2
Add v2.0 PIC supportipv6_packet
Add Internet protocol v6 packet decoder #216macho
Remove redundant arch struct level and cleanup some sym values #226macho
Add raw fields for section and encryption info #238mp4
Add more HEIF boxes support #221mpeg_pes
Support MPEG1 #219rtmp
Add Real-Time Messaging Protocol decoder. Only plain RTMP for now. #214matroska
Symbol name cleanup #220tcp
Better port matching and make it possible to know if byte stream has start/end. #223udp
Better port matching #223
Changelog
010f6430
Update docker-golang from 1.17.8 to 1.18.005096f50
Update docker-golang from 1.18.0 to 1.18.1e5f61e22
Update github-go-version from 1.17.7, 1.17.7, 1.17.7 to 1.18.0fdfc5c5b
Update github-go-version from 1.18.0, 1.18.0, 1.18.0 to 1.18.14ea362e3
Update github-golangci-lint from 1.44.2 to 1.45.02a90485b
Update github-golangci-lint from 1.45.0 to 1.45.2d9195ac4
Update gomod-mapstructure from 1.4.3 to 1.5.0cf88bc11
Update make-golangci-lint from 1.44.2 to 1.45.03a0799cb
Update make-golangci-lint from 1.45.0 to 1.45.234cbe487
amf0: Decode strings in more detailb2a865ea
avc_sps: Add chroma format name mappingb35b1804
decode,format: Add d.FieldFormatOrRaw(Len)f4480c6f
decode,interp: Support for format specific options5ff67e4c
formats: Sym and field name cleanup to be more jq friendly3c029925
github: Update action versions02a97fa3
gojq: Rebase fq fork2e240447
gojq: Rebase fq fork518f6af4
gojq: Rebase fq fork88f791e0
gojq: Rebase fq fork8c918702
gojq: Rebase fq forkadde8c70
gojq: Rebase fq forkd79afeb3
gojq: Rebase fq forkdd0d97ea
gojq: Rebase fq fork (speedup and fix range with JQValue)afd724bf
gojq: Rebase fq fork. Fixes JQValue path tracking when iterating74978c9d
hevc: Add hevc_vps, hevc_sps and hevc_pps decodersc0202483
hevc_vpc,hevc_sps: Use same nameing for profile as in spec09385c61
id3v2: Add 2.0 PIC support9cb4b57a
interp,cli: Handle ctrl-c properly607202bb
interp: Don't truncate last display column6f03471d
interp: Paths with a array as root was missing start dotdabad850
interp: Proper display column truncatee8678ca8
interp: Remove opts refactor leftoverd376520f
interp: Remove to*range pad argument and fix stdout padding issue087d1241
interp: Simpler and more efficient hexdump21ad628a
interp: dump: Show field name for compound values in arrayse8dc7112
ipv6,icmpv6: Add decoderd6c31dac
macho: Add section and encryption_info raw data fields5424eed7
macho: Cleanup syms and remove redundant fat_arch structf8d79a57
matroska: More sym cleanupf34ebd83
mp4: Add more HEIF boxesf8fd6b7f
mp4: Add more HEIF boxes39ba5c4d
mpeg_pes: Support mpeg1 and some cleanupd8aaf303
rtmp,amf0: Add decoders788b0ac1
rtmp,amf0: Improve decoders, aac asc, chunk stream interrupt, fix amf0 ecma arrays5d25bbc2
tcp,udp: Refactor and make port matching better
0.0.6
Added macho
decoder (thanks @Akaame), nicer REPL interrupt, error and prompt, add slurp
/spew
functions and explode
for binary.
Added fq talk slides from Binary Tools Summit 2022 to README.md
.
Changes
- Major query rewrite refactor to share code for slurp-ish functions
repl
,slurp
and futurehelp
system. #178 - REPL improvements:
- Much improved eval and output interrupt. Should fix more or less all issue with un-interruptable long outputs. It is still possible to get "hangs" if some decode value ends up being expanded into a huge string etc. #191 #192
- Prompt paths now has colors support. #181
- Shows an arrow on parse error.
- Faster on multi inputs. #165
- Speedup interpeter by skipping redundant includes. #172
- gojq fork rebase: #179
- Fixes
try ... | ... catch
precedence issue. tonumber
now supports non-base-10 numbers.
- Fixes
- Add
slurp
/spew
to collect outputs and outputs them later.1,2,3 | slurp("a")
collects, later dospew("a")
to output them. Also a global array$a
will be available. #178
- Add
explode
for binary. #188"a" | tobits | explode
return bits[0,1,1,0,0,0,0,1]
."åäö" | tobytes | explode
return utf8 bytes instead of codepoints[195,165,195,164,195,182]
.
- Add optional sub topic to
--help
: #177- Replace
--formats
with--help formats
. #181 - Add
--help options
to see all default option values. #181
- Replace
- Remove
var
, useslupr
instead.
Decoder changes
macho
Add decoder. Thanks @Akaame #43mp4
Supportcolr
box. #176
Changelog
ee5e4718
Update docker-golang from 1.17.7 to 1.17.8ca04cc20
Update github-golangci-lint from 1.44.0 to 1.44.15c6e1d32
Update github-golangci-lint from 1.44.1 to 1.44.21b8e6936
Update make-golangci-lint from 1.44.0 to 1.44.19d5ba826
Update make-golangci-lint from 1.44.1 to 1.44.2cd2cbef6
decode: Some cleanup9e4f2641
dev: Add .jq-lsp.jq to add additional builtins for jq-lspc6a90cfc
doc,asn1_ber: Add more documentationc53bd777
doc: Add bts2022 videob97776c9
doc: Add fq bts2022 presentationd334c2d4
doc: Add href in supported format listc95b0d6d
doc: Forgot make doca202df9a
doc: Improve and fix some typos9ec1d357
doc: Improve project description758b2d0e
doc: Regenerate after macho merge920629f5
doc: Regenerate and fix macho section sized3397cf9
doc: Tweak format diagramd47e04c4
fixup! macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff27e76157
format: Simplify torepr, no need for _f function206dcd02
fuzz: Include more testdata seed filesbe6f0093
gojq: Rebase fq fork and add support for non-10 base for tonumber33efb02a
interp,repl: Add path and value colors to prompt41551de3
interp,repl: Improved eval and output interruptdff7e7da
interp: Cleanup binary regexp overloading and add explodefe8183b5
interp: Color parse in jq6f10745a
interp: Fix interrupt regression after query rewrite refactorf66f115e
interp: Make _finally handle null, call fin once and lasteeb59152
interp: Make help output less wide9dc59e5d
interp: Move _is_decode_value to jq0bc11719
interp: Move opts eval to options.jq3f50bb90
interp: Rework formats and options help03f450f8
interp: Skip redundant includesc5918d23
macho: CPU_SUBTYPE_MULTIPLE and TYPE_ALL are 0xff_ff_ff_ff5c974209
macho: TS string to UTC04eae939
macho: add basic docs5e95d1c3
macho: add cpuSubTypes2638f419
macho: add darwin_amd64 test5c5bd879
macho: add fqtest actualizationbf214d5e
macho: add nolint suppression to const defs333a3243
macho: add scalar.Hex mapper to addr fieldsa86e7043
macho: add section type parsing90b94631
macho: adopt plural-singular scheme for FieldStructArrayLoopb78ed02f
macho: barebones decoder imple199d219
macho: basic impl for ar and fat file parsing66feebc5
macho: change parseFlags impl for ordered resultsb5fe9ce6
macho: change registry description20e5be3f
macho: delete ar decoder codeefdd0bf5
macho: discard lc_ and lowercase command namesb0911af2
macho: docs review changesa29bfca5
macho: expand filetypes and header flagsfb0654ec
macho: fix FieldUTF8NullFixedLen for segnamed1f093ce
macho: fix fat header decode bug0d648928
macho: fix null in segname sectname9eb71dc6
macho: generate doc via make doc3991c51a
macho: handle unknown lc_commands betteref2919b3
macho: introduce arm and fat tests98c9840d
macho: linting changes for ar parse1feb81c9
macho: little-endian to little_endian141a8e84
macho: mach_header_X to header9206d9d8
macho: magicToHex to scalar.Hex2021b054
macho: make actual70b84cde
macho: ntools fix LC_MAIN fix78699f3a
macho: parse flags individually4016ad0b
macho: parse segment section flags5a48cb30
macho: refactor prebound_dylib2e7767cd
macho: remake docs33347503
macho: reuse ar decoder228757b9
macho: review fixes5ee9a23c
macho: review fixese3daee7d
macho: simplify thread state decoder70c9d519
macho: thread state visualizationa4789dc1
macho: timestamp mapper2ccb8087
macho: update test cases v to dv74abe990
macho: update tests12eb7cc5
macho: use FieldUTF8NullFixedLen5f4ad410
macho: use FieldUTF8NullFixedLen for segnamef8690e6c
mp4: Add colr box supportb157751a
mp4: Reformat and use dv in test0a043f90
repl,interp: Refactor repl and slurpca8cdadb
repl: Add comments and query from/to helper9cb4205b
repl: Correct error arrow position in color modee238f292
repl: Speedup multi input to sub-repl56ae4a0c
test: Make expect cli test more robust
0.0.5
Improved binary slicing and bit reading, avro_ocf
decoder (thanks @xentripetal), asn1_ber
decoder, renamed display
aliases, new grep_by
and paste
function.
Changes
- Big internal bit reader refactor. Now much more consistent code and fixes some issues reading and decoding of binary arrays and binary slices. #123
- Bit reading and IO have been moved to a
bitio
package. - Non-simple bit reading have been move out of
bitio
todecode
package. [0,1,1,0,0,1,1,0,0,1,1,1,0,0,0,1 | tobits] | tobytes | tostring
returns"fq"
.[.frames[0], .frames[-1]] | mp3
decode mp3 based on first and last frame of other mp3.
- Bit reading and IO have been moved to a
- Add
grep_by
that recursively selects using a filter condition and ignores errors. #102grep_by(.type == "trak")
finds all objects where.type
is "trak" (all mp4 track boxes).grep_by(tonumber | . >= 40 and . <= 100)
find all numbers between 40 and 100.grep_by(format == "jpeg")
find all jpegs.
- Add
paste
function to read string from stdin util ^D (EOF). Can be used to paste in REPl etc. #143paste | frompem | asn1_ber | repl
wait for PEM encoded text on stdin (paste and press ^D), decode it withasn1_ber
and start a sub-REPL with the result.paste | fromjson
decode pasted JSON.eval(paste)
eval pasted jq expression.
- Cleanup display aliases. Remove
v
andf
, addda
,dd
,dv
andddv
. #112d
/d($opts)
display value and truncate long arrays and buffersda
/da($opts)
display value and don't truncate arraysdd
/dd($opts)
display value and don't truncate arrays or buffersdv
/dv($opts)
verbosely display value and don't truncate arrays but truncate buffersddv
/ddv($opts)
verbosely display value and don't truncate arrays or buffers
- Refactor
radix
intotoradix($base)
/fromradix($base)
. #139 - Remove
number_to_bytes
. Can be done withtobytes
. #139 - Change
tobytes
to zero pad most significant bits to byte alignment if needed. #133 - Add
tobytes
/tobits
variant that takes an argument to add extra padding. #1330xf | tobytes
8 bit binary with last 4 bits set0xf | tobytes(4)
32 bit binary with last 4 bits set0xf | tobits(12)
12 bit binary with last 4 bits set
- Rename fq type buffer to binary as it makes more sense. #133
- Add
topem
/frompem
to work with PEM encoding. #92 - Add Windows scoop install. #137 Thanks @thushan
- Add
display
, decode value, binary and binary array documentation. #118 #136 #133 - Add decode API documentation. #149
- Improved REPL completion for keys. #144
- Add
-o force=<bool>
option that sets force decode option. Same asmp4({force: true})
. #126
Decoder changes
avro_ocf
Add decoder. #38 Thanks @xentripetal- Full avro OCF support. Handles all primitive, complex, and logical types besides decimals.
- Able to handle deflate, snappy, and null codecs for blocks.
asn1_ber
Add decoder. #92- Also decodes CER and DER (X.690) but with no additional validation at the moment.
- Support all types but real type is currently limited to range for 64 bit interger/float.
- Has
torepr
support. - No schema support.
aac_frame
Only decode object types we know about. #130mp3
Shorter sync find heuristics. #104mp4
Addstz2
supportmp4
Addpnot
(preview container) andjP
(JPEG 2000) signature. #125
Also thanks to @Doctor-love for keeping things tidy.
Changelog
6fc1efd9
Add test case with all data typesae4a6243
Adds Windows Scoop instructions for fq.4b809a73
Change avro codec to funcs66ca1f10
Change tests to use new verbose syntax7345b8c7
Cleanup07ddf36f
Cleanup for linting4508241b
Cleanup snappy0909fb6d
Comment on snappy decompression21cfc70c
Dates need to specify UTC too75b84961
Fix lint7a8e3ca2
Hook into registry, add codecs251053ef
Initial pass on logical types2605bce4
Lint and add basic docee184075
Parse header using avro decoders. Still not certain this is the best idea. Will get opinions before finalizing.ab50088d
Polish of problem template and clarifying questions27789f2d
Regenerate docs5a1d35e7
Remove redudant question and fix typo31c4c0d3
Support snappy and deflate codecs0300c955
Take heading off doc to match make doc format6f57cdbf
Timestamps should be UTC06085a26
Undo change to doc/file.mp4. I have no idea how this got changed in the first place? Maybe some macos shenanigans.d137a72a
Update docker-golang from 1.17.6 to 1.17.7267e30ec
Update github-go-version from 1.17.6, 1.17.6 to 1.17.71e859cda
Update github-golangci-lint from 1.43.0 to 1.44.016849c8f
Update lintingd02d8968
Update make-golangci-lint from 1.43.0 to 1.44.068e85a2d
Use existing scalar description helper3bab3d65
aac_frame: Only try decode object types we know about0829c167
asn1_ber: Add decoder0312c92c
asn1_ber: Add more doc and multiple outputs for frompem06245d12
binary,decode,doc: Rename buffer to binary and add some documentation7c521534
bitio,decode: Refactor bitio usage and make buffer slicing more correct0d74e879
bitio,doc: Even more cleanupd854ed57
bitio: Cleanup documentation a bit82aeb355
bitio: More doc cleanup01ecde64
bump: Add snappy configde64a99e
cleanup some docs, change enum to mapper, error zigzag on more than 8 bytes6cd1c38f
decode,scalar: Add scalar.Str{Uint/Int/F}ToSym to parse numbers4ab6381d
decode: Add scalars args to FieldRootBitBufbe71eb01
decode: Rename LenFn() to LimitedFn, add FramedFn and document7bc25219
doc,interp: Add some example usages to cli help8e47fb1a
doc,matroska: Fix filesname in examplec15f5283
doc: Add format links to format tableb86da7ae
doc: Add inital decoder API documentation49c90f89
doc: Add macOS security notes and move supported format up a bit06b67e4b
doc: Add more license detailsa8664ed5
doc: Add per format documentation09552628
doc: Add snappy license36307857
doc: Cleanup and add more decode value and binary documentation710c29b2
doc: Color edges in diagram based on destf0ce7179
doc: Document display and some more jq hintsb3504680
doc: More decode API details and polish6b51b067
doc: More display alias leftover fixesdd3e40fb
doc: Unbreak formats_digaram.jq since radix changec52a1a23
doc: Use f($a; $b) instead of jq f/2 notation233d86a3
fq: Add arch and os to --versionb8efd8e5
fuzz: Fuzz all formatse1bdfdf8
fuzz: List seed numbers and make it build again6090b65e
fuzz: Make it compile again and run one format per fuzzaea48847
github: Add basic issue templateb55ca2cd
gojq: Rebase fq branch47c978e4
goreleaser: Use zip for macos85371173
id3v2: Should assert not validate magicd6ca4818
initial work for avro OCF filesca68e6a1
interp: Add Platform() method to OS interfacee792598c
interp: Add grep_by/1 to recursively match using a filter0a1a5610
interp: Add missing default opts for tovalue48a19cb8
interp: Add paste function to allow pasting text into REPL etcfc0aacb6
interp: Cleanup display aliases, now: d, da, dd, dv, ddvbf7a483f
interp: Fix handling of group decode error from stdin26d9650b
interp: Refactor radix* into toradix($base)/fromradix($base)366f6b18
interp: Support force decode as -o force=true77ab667b
interp: Use absolute path in errorsc31ec2a3
interp: Use correct sym color898dfec1
lint: Fix typeassert and case exhaustive warningsd5401166
make docbf170be8
make: Cleanup some not very used targets8d2d88f4
mp3: Decrease max sync seek length between frames to 4kd555c324
mp4,fuzz: Fatal error on infinite sgpd box entries45b00aab
mp4: Add stz2 support092609be
mp4: Add video preview (pnot) and JPEG 2000 (jP) signaturesfebce5a5
mpeg_spu: Fatal error on infinite loopc58ba28d
mpeg_spu: Fatal error on unknown cmdd1943dad
pcapng,fuzz: Fix infinite loop by fatal error on block length <= 02ab395a0
protobuf: Add note about sub message decodingaf053811
repl,interp: Make stdio work during completionbd9be2c5
repl: Fix completion of non-underscore extkeys69c745d3
simplify scalar usage778a1a41
zip: Assert signature not validate
(Some commits have been removed from list for clarity)
0.0.4
Changes
- Add a
torepr/0
function that converts decoded value into what it represents. Initially works for:bencode
,cbor
,bson
andmsgpack
#74 Example usage:fq torepr file.cbor
fq -i torepr file.cbor
fq torepr.field file.cbor
fq 'torepr | .field | ...' file.cbor
- Add
stderr/0
function for jq compatibility #58 - Bitwise operators
band
,bor
etc are now normal functions instead of operators. Was done to be syntax compatible with jq #61 Uses the same convention as jq math functions: Unary uses input123 | bnot
, more than one argument all as argumentsband(123; 456)
- Decode API now supports null values #81
- Decode API now supports arbitrary large integers #79
- TCP reassembly now supports streams with missing SYN/ACK #57
- Update readline package to version with less dependencies #83
- Make REPL prompt more jqish #70
Decoder changes
bencode
Add decoder #64cbor
Add decoder #75msgpack
Add decoder #69mp4
Much improved sample decode #82 #78png
Decode PLTE and tRNS chunks #59tar
Don't assume there is a end marker and support more than 2 blocks #86 #87
Also thanks to @Doctor-love for keeping things tidy.
Changelog
af8e7ef
bencode: Add decoder0b0f28e
cbor: Add decoder1383b41
decode,interp: Add arbitrary large integer support (BigInt)548a065
decode,interp: Finish up nil value supportff5c0b8
decode: Error on negative number of bits when reading numberscf8a50c
decode: Use stable sort for values to not change order or values with same range startb4694b6
doc,dev: Add some more decoder implementation help0c1716b
doc: Add alpine and go run809210b
doc: Add more dev tips59b8803
doc: Document dev dependencies and related PRs/issues etc6ca4767
doc: Improve formats graph a bit8e9700d
doc: Improve readme a bit and add torepr example0cf486d
elf: fix all-platforms naming typo263f1ae
flac: Don't allow zero subframe sample size729a6ca
formats: Sort and make lists less likely to cause collision78c0775
fq: Embed version in sourceaa7adb6
fq: Update version to 0.0.47461264
fuzz: Skip other tests when fuzzingbe0ef80
interp,fq: Make bit operators normal functionsa3cfcd0
interp: Add stderr again for jq compat149cb3f
interp: Add torepr/0 that converts decode value into what it reptresentsb3a0980
interp: Document bit opts funcs and add some error tests8d10423
make: Fix quote issue in release script4a1e859
mp4: Improved stsz handling61bf2ce
mp4: Refactor sample decode into something more sanea6bf62c
msgpack: Add decoderedad481
num,mathextra: Rename num package to mathextrabfc977b
png: Decode PLTE and tRNS chunks and cleanup syms a bit36d2891
readline: Update to verison with less deps9770b00
repl: Make prompt for array and iter more jqishba1edef
tar: Allow more than 2 zero end blocks at end5921d76
tar: Don't assume there is a end markeredd0ae1
tcp,flow: By default allow missing syn/ack for now
(Some commits have been removed from list for clarity)
0.0.3
Changes
- Now works on Windows #52
- Now builds and runs on 32-bit CPUs #30 @danfe
print/0
,println/0
function now properly convert input to string if needed. #54match
functions now don't try to be smart with buffers, use.tobytes
etc instead. Less confusing and also unbreakgrep
:ing decode values. #53 Now this works:fq 'grep("^strtab$") | parent | {name, string}' /bin/ls
- Add Arch package #19 @orhun @ulrichSchreiner @dundee
- Add Nix package #22 @siraben @jtojnar @portothree
- Add FreeBSD port @danfe
Decoder changes
bson
Add Binary JSON deccoderar
Add Unix archive decoderbsd_loopback_frame
Add BSD lookback frame decoder (used in pcap files)elf
Now does a two-pass decode to correctly handle string table referenceself
Decode more sections: symbol tables and hashesmatroska
Assert sane tag size only for stringspcap
Don't fail if incl_len > spanlen
Also thanks to @Doctor-love @mathieu-aubin for keeping things tidy.
Changelog
628f0f4
bson: Add decoder46b59d0
crc: Unbreak build on 32-bit arch681dbc2
elf,ar: Add ar decoder, improved elf decodere5c620d
github,ci: Add windows, macos and 32-bit linux52dddbb
goreleaser: Use draft release to allow release note changese365f22
interp: Cleanup stdio usage and functions55b1d5c
interp: Move _registry to decodeb6515c8
interp: Remove buffer smartness for regexp match functionsb867113
matroska: Assert sane tag size only for stringsb9aef39
pcap,pcapng,bsd_loopback_frame: Add decoder, refactor link frame into a groupaf23eb8
pcap: Don't fail if incl_len > spanlena41f0d4
windows: Correct @builtin include path joinbf9e13c
windows: Unbreak tests
(Some commits have been removed from list for clarity)
0.0.2
Changelog
00f34c2
Update docker-golang from 1.17.3 to 1.17.405b179c
Update docker-golang from 1.17.4 to 1.17.5c721ac8
Update github-go-version from 1.17.3, 1.17.3 to 1.17.4befe783
Update github-go-version from 1.17.4, 1.17.4 to 1.17.575aa475
decode: Generate Try?Scalar* methods for readers too9b683cd
deocde: Cleanup some io panic(err)e1e8a23
doc: Add color/unicode section and move config sectionee023d7
doc: Add some more related and similar projectsd02c7c4
doc: Add some more usage examplesbe46d5f
doc: Cleanup todo and add some dev notes47deb4d
doc: Fix interpretation typofe68b51
doc: Improve readme text a bit916cb30
doc: Improve usage examples as bit5cb3496
docker: Fix broken build, copy fq.goc2131bb
flac: Cleanup scalar usage and fix incorrect sample rates1500fd9
gojq: Update to rebased fq fork3601fe3
gzip: fuzz: Don't uncompress on unknown compress method (nil create reader fn)f4f6383
interp: Add ._index for values in arrays4558192
interp: Cleanup buffer code and implement ExtType()e823475
interp: Fix help a bitfa350c6
interp: Move display to jqf7c7801
interp: Rework buffer regex support6ed2e2e
interp: dump: Indicate arrays using jq-syntax9aec91a
interp: match: Fix issue with regexp meta characters when matching using a buffere5e81e7
make: Fix prof build issuee91b22b
matroska,ebml: Use scalar and require sane tag sizeeb9698f
mp4,ctts: Seem more usable to treat sample count/offset as signedc149732
mp4,trun,fuzz: Limit number of constant sample entries406263b
mp4: Add comment about hdlr.component_name prefix byte28a3b71
mp4: Cleanup sample decode code6278529
mp4: fuzz: Make sure stsz has sane number of entries on constant sample size9f08af3
mpeg,aac: Factor out escape value decodingdc1aea3
opus: Cleanup endian usage and fix incorrect preskip decode2b2320d
pcap,flows: fuzz: Handle broken packets more nicely1d7ace3
pcap,pcapng,tcp: Use capture length not original lengthb525d0b
pcap: fuzz: Skip ssl2 packet if too short713ffe4
scalar: Add Require* and Require/Validate/AssertRange*f348002
sll2: fuzz: Limit address length to max 8 byteseb4718f
tar: Cleanup api usage45026eb
tar: Cleanup constant usage a bit more91cc6d8
tar: Fix size decode regression after cleanup07a2ebe
tiff,fuzz: Fatal error on infinite ifd loops91217e8
tiff: Fix endian typo and cleanup todos3850968
udp: Use proper udp payload format var name6a8d77b
vorbis_comment: Cleanup endian usage and naming a bit57e9f41
vorbis_packet: Cleanup endian usagef5a4d26
vscode: Use tabsize 2 for jq filesdd883b3
wav: Cleanup endian usagee260830
webp: Cleanup endian usage
0.0.1
Changelog
4242bf6
*_annexb: Refactor into avc/hevc_annexbe86b45b
Add *grep/1/2 and find/1/236fd74a
Add comment how raw byte regexp matching worksd1be167
Add decode struct each order test7f36f70
Add to/bytes/bits[range]571bf29
Change project title95ec5e1
Cleanup and rename s/BufferView/BufferRange9797cdc
Cleanup dev/snippets.jq565f18d
Correct avc_au format variable name6a1fa04
Decode hvc1 as hevc samplesa also74bad2d
Fix broken value.fqtest514739a
Give proper error on missing short flag57f0ec1
Improve cli help a bit9704659
Initf33b310
Refactor decode.Value gojq bindings3d90b6d
Remove fixed commente4e269b
Rename and move cli test to pkg/cli1bd34bf
Same args error behavior as jq3693667
Start of configurable json bit buffer formats590ee52
Update ci-golang from 1.17.0 to 1.17.172a3f69
Update ci-golangci-lint from 1.42.0 to 1.42.1078cf29
Update docker-golang from 1.17.1 to 1.17.2f5a8484
Update docker-golang from 1.17.2 to 1.17.31371bc7
Update docker-golangci-lint from 1.42.1 to 1.43.0f1ad700
Update github-go-version from 1.17.1, 1.17.1 to 1.17.2314bd17
Update github-go-version from 1.17.2, 1.17.2 to 1.17.353a8d91
Update github-golangci-lint from 1.42.1 to 1.43.0898cd26
Update golang from 1.17.0 to 1.17.131cd26e
Update golangci-lint from 1.42.0 to 1.42.1d4b2d58
Update gomod-mapstructure from 1.4.2 to 1.4.31decf85
Update make-golangci-lint from 1.42.0 to 1.42.1233aaa1
Update make-golangci-lint from 1.42.1 to 1.43.015e9f6f
ansi: Correct background reset code1ccab2d
apev2: Add test27e4770
apev2: Fatal if > 1000 tags3bf1a57
avc: Cleanup and add color names etc777191f
avc: Correct sign expgolomb decode20021f4
bitio: Handle < 0 nbits7c4b0b3
bitio: Simplify by embedding readerf600f2e
build: Require go 1.1777f97aa
builtin: Add chunk_by, count_by and debug27ba359
bump: Add action and cleanup names71e87e6
bump: Cleanup config, add config for release.yml8f2f524
bump: Move bump config to where it's used184df0a
cli: A bit clearner array and iter promptd350971
cli: Add --decode-file VAR PATH support17104f0
cli: Add --options to make help a bit nicer6356a84
cli: Add --raw-string93fd097
cli: Add -M -C support and default to color if tty8dc0f06
cli: Add completion tests2010cac
cli: Add error test706b2f2
cli: Add exit 2 (like jq) for no args569b631
cli: Add output join tests49f541c
cli: Add proper repl iterator support3304f29
cli: Add string_input options test0dd848d
cli: Better filenames in errors22eb53d
cli: Cleanup MaybeLogFilec14c29a
cli: Cleanup and more commens10d7ed7
cli: Fix error filename on script error2b8d11d
cli: Fix indent2d4eb9c
cli: Fix non-string variables and var(; f) variant to delete etcf5ffd32
cli: Implement --arg, --argjson and --rawfileae5566a
cli: Include paths and some refactorec98fd3
cli: Make --argjson and --decode-file error similarac8cfca
cli: Make --raw-string work with input/0 and inputs/0b33f2cd
cli: Make profile build optional and move it to clie2ff2a2
cli: Move help/0 to inter.jq, better help for -nd0bb9a5
cli: Nicer grammar for --null-input help78eb737
cli: Nicer usage and indent input iteration394e2b3
cli: Only compelete at end or whitespace8d1fafe
cli: Only show fq info for --help2684ed2
cli: Prepare completion for better variables supporte666380
cli: REPL and multiple files75cf46f
cli: Refactor options codef8ab00e
cli: Remove unused eval debug arg5c8fb5f
cli: Revert accidental history path change55cd45f
cli: Simplify code2874bc7
cli: Unbreak colors in windows48517c7
cli: Unbreak part of completion518b725
cli: Use format/0 to check if value is a format21bef18
cli: User defined global vars538f4ff
cli: add -nul-output compat85d1719
cli: jq compat, multiple -Lc8f0264
cli: more jq compatf893295
cli: rename fq.jq to interp.jq1436fdc
completion: Better and _internal handlingc7416e6
decode, interp: More buffer reuse6ee7977
decode,format: Allow root array58ba84f
decode,interp: Add RecoverableErrorer interface instead of enumerateb66ed32
decode,interp: Make fuzzing work again and cleanup fatal/error code5052bae
decode,interp: Refactor to allow decode/fillgap a ranged4142b8
decode,png: Add FieldFormatReaderLen, refactor out zlib to format826c509
decode: Add Generated header for scalar3db11d3
decode: Add UTF8Fn functions and trim some null terminated stringsc083a9e
decode: Fix MapRawToScalar regressionc17483d
decode: Fix accidental rename6fba1a8
decode: Fix bitbuf root handling a bitd1e1cd9
decode: Fix walk root depth issue causing dump to indent incorrectly1b32b42
decode: Major decode API refactor7af191d
decode: Move io helper into *D26d615b
decode: Move name/description into DecodeOptions986d5ec
decode: Move registry package to decode/registry and add a format group typeede2e77
decode: Nicer scalar template and add doc6207fcc
decode: Pass context to be able to cancel properly5d98a69
decode: Refactor Error/Fatal into printf functions9f55b6e
decode: Refactor and add symbol struct and mapping2fc0a71
decode: Refactor scalar usage8eaba88
decode: Refactor walk code a bit, add WalkRoot* to stay inside one rootf40320b
decode: Remove D.Scalar* and add d.(Try)FieldScala*Fn insteadc155c89
decode: Rename format *Decode to *Formatf801cc0
decode: Rename s/FieldTryFormat/TryFieldFormat for consistency776a6b3
decode: Reuse read buffer per decode to speed things up9d116df
decode: Rework use of TryFieldReaderRangeFormatd48ebc1
decode: Simplify Compound.Children473b224
decode: Simplify and move format arg into DecodeOptions0480a2f
decode: Some format decode and sub buffer worka49e924
decode: Use golang.org/x/text for text decodingaf3e6b1
dev: Add format_summary.jq4c6de82
dev: Add summary how start to dump tree worksa926c8f
dev: Some debug notesb17a715
difftest: Remove accidental log1e1ad14
difftest: Sync code3cea849
dns: Cleanup a bitd469edf
doc,make: Correctly strip out graphviz version from svg15d85e1
doc: Add find/1 find/226ea6d8
doc: Cleanupa131210
doc: Cleanup and add note about . argument0a97f86
doc: Cleanup and note about repl limit8440e8a
doc: Cleanup todo and dev a bit15b6d64
doc: Cleanup up a bit1047d90
doc: Document io packages a bit1a0089e
doc: Fix typo and some improvments97c7403
doc: Fix typo in README0814206
doc: Fix typos and old examples0e8c82a
doc: Improve readme goals a bit2f9d93d
doc: Improved readme and cleanup todo242525f
doc: Move formats to own file583bc38
doc: Regenerate demo.svga050adc
doc: Regenerate demo.svg18e3e20
doc: Regenerate demo.svg with newer ansisvg1f61704
doc: Regenerate svg after ansisvg monospace updatea7459b3
doc: Some basic usage and cleanup6fa5ae8
doc: Some fixes00b7c18
doc: Some more doc work07c7daa
doc: Some rewording5a82224
doc: Update21a74fa
doc: Updated6d3265
doc: Update README a bitdf5bd19
doc: Update TODOfb13fe5
doc: Update todoeee3c4e
doc: Use unicode pipe in demob9b0326
doc: fq - jq for files790198c
docker: Make image build again and install expect2387ec8
docker: Use golang 1.17f8e5944
dump,json: Properly figure if compound or notee972f4
dump: Add ascii header69c6d15
editlit.jq: Update after mp4 field renames27095b5
es: Decode MPEG ASC if audio stream761c411
exif: Add note about JPEGInterchangeFormat344f628
flac: Calculate correct md5 when total samples count is zeroce044ba
flac: Cleanup2af08da
flac: Fix block_size regression4f0bf92
flac: Make md5_calculated be a buffer797bd4d
flac: Refactor flac_metadatablock into flac_metadatablocks1f26d4f
flac_frame: Correctly read escaped samples and also a bit less allocationsbc9951c
flac_frame: Fail if trying to decode outside block size723542a
flac_frame: Make utf8Uint reader more correct and robustb643e22
flac_frame: Support non-8 bit align sample size2daa738
flac_frame: Use d.Invalid for posssible errorsc1d9b4d
flac_metadatablocks: typ > 127 can't happen, add app id984ba1a
flac_metadatablocks: type >= 127 is invalid2b35d28
flac_picture,mp4: avif images and data fallback is image format fails509b8f8
flac_picture: Add picture_type names4f8d037
format,decode: Some crc and endian refactoraa38ccf
format,interp: Use MustGroup and add probe order test577c0f5
format: Add panic if register after resolve606c0b6
format: Add vorbis-comment-picture test, add .gitignore and cleanup798141a
format: Cleanup commentsc0eebcc
format: Remove unused ProtoBufType4b48828
format: Rename source file to match format named1b514e
format: Some claeanupec97eca
format: Split default.go into format.go and shared.go25f5ad7
fq,cli: Rename chunk to streaks, cleanup905c0ab
fq: Add chunk/146d37ef
fq: Add cli sanity testb849895
fq: Add truncate array support to dump/display8cb380e
fq: Generate decode alises codedfcefc1
fq: Make format/0 native for performanceba273be
fq: Make relative include work with @builtin etcd23edaa
fq: Rename bits/0 bytes/0 to tobits tobytes, remove string/0a7a58c8
fq: Rename main.go to fq.go834f4a5
fq: use jq functions for all display alises31d7611
fqtest: Add env support and isterminal/width/height support285356d
fqtest: Cleanup and dont assert when WRITE_ACTUAL86b34a3
fqtest: Fix section regexpdee10db
fqtest: No need to escape empty stdout6916880
fqtest: Refactor our script part to own package01d8a90
funcs: Add delta/0 delta_by/096f7a75
funcs: Fix typo add count/0962d84d
funcs: Make intdiv truncate to int08ec4f0
funcs: Remove unsued string functiond5c084c
funcs: chunk_by comment80eaa46
funcs: format helperc770a75
funcs: make in_bytes/bits_range more generic6ff5aca
gif: Support GIF87a0d87018
github: Install expect to cli test8e3d9d8
github: bump: Checkout with bump token so it's used when push0c7fa09
gojq: Initial update support1888bb2
gojq: Remove div operatord7dbe7c
gojq: Update fq branch to fix mod (-1 % 256) differencebf5c222
gojq: Update fq fork093ee71
gojq: Update fq fork1d15c1d
gojq: Update fq fork3044fef
gojq: Update fq fork5a5f01e
gojq: Update fq forkbfec366
gojq: Update fq fork4104a18
gojq: Update fq fork31a5047
gojq: Update fq fork0b90558
gojq: Update fq fork8277b79
gojq: Update fq fork, support JQValue alt //a34784d
gojq: Update fq gojq fork7ad3d25
gojq: Update fq gojq forkf5164ee
gojq: Update fq gojq forkac7568d
gojq: Update gojq fq forkf828ae1
gojq: Update gojq fq fork845bc6b
gojq: Update gojq fq forkd365ab7
gojq: Update gojq fq fork6420928
gojq: Update gojq fq forkc8776ab
gojq: Update gojq fq forkf4cd7bf
gojq: Update jq forkb75da30
gojq: Update jq fork03af2b5
gojq: Update rebased fq branchc3b7b5c
gojq: Update to rebased fq branch2d6573d
gojq: Update to rebased fq branchfdb811e
gojqextra,decode: Add generic lazy JQValueaab32cf
gojqextra,interp: Add lazy string to speed usage of decode value buffer where string is not used9035278
gojqextra: Move errors to own file86092e6
golangci: unused: Assume go 1.17ac106d9
goreleaser: Cleanupee611a4
gzip,bzip2: Calculate CRC5344c7e
icc: Add mluc support50d00d8
icc_profile: Add element alignment bytes field59e7fd4
icc_profile: Clamp alignment to end instead of check last tag (might not be last in buffer)a5b802b
icc_profile: Correct alignment byte count57a1207
id3v2: Support GEOB tagsf55b1af
inet: Add tcp and ipv4 reassembly26c594f
input: make -R mimic jq on io errord9b45ba
internal: _global_var returns new value instead of _global_statef9f8660
interp,decode: Add force option to ignore asserts6a15625
interp,decode: Refactor out Scalar from Value and merge Array/Struct into Compoundcc5f405
interp,format: Update tests after decode refactor/tosym/toactual9f2dddc
interp,gojqextra: Make buffers values even more lazy and error early on non-scalar callsbf19588
interp: Add ansi helper7298a4c
interp: Add buffer match support to find and grep0d693aa
interp: Add line between usage and args helpc997536
interp: Add root, buffer_root, format_root, parent and parents69e4eea
interp: Better error if format/group is not found7423f45
interp: Cleanup output types8d442b8
interp: Cleanup stdin reading and add more option testsb641c77
interp: Cleanup unfinished/broken preview67898cb
interp: Cleanup, use BufferRange for _open, progress for all decode0660ff0
interp: Clear up confusing --rawfile (add a jq alias)3fafee8
interp: Clearer help for -d879bb56
interp: Correctly check if _decode_value, add more has/1 tests2e964fa
interp: Disable progress after decode is done80b5d66
interp: Document inputs and repl/cli details13fae09
interp: Don't print context cancel96cc128
interp: Eval options in jq instead of calling jq from go178032e
interp: Fix $opts shadowing in decode4eccb1e
interp: Fix broken aes_ctr, should return buffer instead of []byte110c86b
interp: Fix broken dynamic width/height80a6997
interp: Fix file leak and always cache for nowcfdd922
interp: Fix prompt issue with format93322bc
interp: Implement format/0 in jq8e5442f
interp: Limit how often decode progress fn is calledf1fcbe5
interp: Make has/1 work for _ext keys826c8bd
interp: Make include abs path work again6034ad7
interp: More sure stdOS stops the signal forward gorutine6cacc9b
interp: Move *CtxWriter to ioextra, some comments cleanupafb1050
interp: Move _decode_value to value.jq3e7e133
interp: Move formats func def to jq5cd5633
interp: Move jq function impls closer to where they belong996be0f
interp: Move more options code into options.jq976e992
interp: Move progress logic to jq528e6b9
interp: Refactor and use mapstructure472c1ef
interp: Refactor out string_input to own function54e121c
interp: Refactor repl inputs a bit618c1ea
interp: Refactor/Rename BufferViewd6d9484
interp: Remove --options, probably just confusingb024316
interp: Remove accidental extra space after path in promptff143d0
interp: Remove redundant decode argabcecb8
interp: Remove unused []byte type01a407a
interp: Rename s/bufferrange/buffer and cleanup567bc4b
interp: Rename to* funcs to match36e5562
interp: Rename value.* to decode.* as it makes more sense0cce5ec
interp: Reorganize, move out repl and options, more functions to funcs.jqeedfd16
interp: Replace find with overloaded match that support buffer58bf069
interp: Return []byte value as a buffer for now1c3c65b
interp: Rework buffer, still confusing97f7317
interp: Rework repl prompt code and fix some whitespace issues again4af5739
interp: Rework string/buffer for decode valuesff2077b
interp: Simplify Function, aliases done in jq now1fe5d95
interp: Some better naming and typos1b7b2f9
interp: Use gojqextra.NonUpdatableTypeError6ce4ba9
interp: Use snake_case for all options9cba69e
interp: Use todescription in testscf26b1f
interp: _readline: Use _repeat_break, add test07b4210
interp: add topath/0 and make todescription return null if there is none16d1f45
interp: find buffer should always use ByteRuneReader3ff0c9b
lint: Enable errcheck adnd revive13e98d4
lint: Fix unused bufferRange and toBufferView13d5cbd
lint: Remove usused nolint, should somehow tell about decode.Copyad850d6
lint: want explicit types in gojqextra.go90c19c6
make,test: Move testjq to own script and reuse fq binary052b9c0
make: Build with -s -w same as goreleaser1557e14
make: CGO_ENABLED=0 for static build8019212
make: Enable -race for tests263a77f
make: Mark actual and cover as phonyb0694f5
make: Move build flags to vars75b59db
make: Move doc generate to helper scriptfcbfc29
make: Move testjq.sh into pkg/interp5bf4bc7
make: Rename testwrite to actual40e26e8
make: Rename to testjqab8080f
make: Reorganize test targets98a8bae
make: Silence git if no repo version found95b9c32
make: doc/formats.svg: Ignore graphviz verison to get less diff30ad643
make: go build args should not be quotedf1507f7
mod: Use proper path and dont use replacec4a3120
mp3,README: some cleanupda386ea
mp3: Be more relaxed with zero padding, just warn1325e5c
mp3: Cleanup commentse104748
mp3: Continue try find frames on error046f2fd
mp3: Don't allow more than 64k between frames527f917
mp3: Error if > 5 unique header configs4344b62
mp3: Only look for supported mp3 sync headers8a4f66f
mp3: Probe order after formwats with raw samples and similar sync headers60df04b
mp3_frame: Only supports layer 3, fixes some probe issuesed21f36
mp3_frame: Rename samples_per_frame to sample_countdb586eb
mp4,matroska: Add *_path/0 variant that uses format_root9ac17bd
mp4: Add comment about future truncate to size option2e71fa1
mp4: Add smhd box278e909
mp4: Add tapt, prof, enof and clap boxes161dcaf
mp4: Better fragmented mp4 supporte47888e
mp4: Fix 64bit size regression0801882
mp4: Fix field name typo for sample_composition_time_offsets_presentf322e78
mp4: More _time decoding6b8d26c
mp4: Properly decode tfraf2c1327
mp4: Use ISOBMFF naming and some more tests589207d
mp4: Use descriptor field for all descriptor boxesc7d45ff
mp4: add pssh_playready formate6cb708
mpeg: Nicer sym and descriptionf4b11b4
mpeg_annexb: Add format6a8ba31
number_to_bytes: Force int to make it work with bigint88eade9
ogg: Add flac support7f76986
ogg: Cleanup bitio in format out, maybe later7b7faaf
pcap: Add pcap, pcapng, ether8023, ipv4, udp, udpfc76907
png: Add proper color type5c733ad
readline: Update fq fork4cee498
readme: Nicer demoffb5adf
registry: Move to pkg/registry, feels betterdcceaa4
repl: Fix help and error message a bit44d8b66
repl: Give error if repl is used in non-repl mode1d0ebb5
repl: Handle directives, add testsc9777aa
repl: completion in jqccf6cab
repl: use map in _query_slurp_wrap74b5750
shquote: Remove unnecessary sb.Reset()44251ca
snippet: Add mp4_matrix_structure_rotation79a1aea
snippets: Add urldecode684193a
snippets: add changes/1fda1dda
snippets: urlencode: only 0-9a-f5ad048d
tar: Fix 0 trim regression49d2e61
tar: Unbreak num parsing and add testc8fad57
tiff: Fix reading of mluc tags with multiple recordsb55f24a
todo: Add ignore range check ideabc1b3bf
todo: Add note about symbols and iprint improvments1d83554
todo: Add note about test and capture with bufferf839317
todo: Add some known issues4d94c9a
todo: Clenaup a bit17a708f
todo: Remove fixed repl item0af4c2b
todo: Update about readlined03a1c9
todo: add echo '{} {} {}' | jq differencef9622c2
vorbis_comment: Fix field name typobe0fdbe
vp9: Add profile and fix reserved_zero field collision646f902
vpx_ccr: Add color namesb0ad3f2
w45afbe6
wipd838d2f
zip: Add format decoder9029143
zip: Fix nested decode for none compress