fq/doc/TODO.md

Known bugs to fix

• fq -n '"aabbccdd" | hex | tobytes[1:] | decode("bytes") | tobytes' create binary aabbcc should be bbccdd. I think decode (raw in this case) is confused by root value buffer.
• Buffers/string duality is confusing, most string functions should be wrapped to understand binary.
• REPL cancel seems to sometimes exit a sub-REPl without properly cleanup options.
• Value errors, can only be accessed with ._error.
• Framed (add unknown gaps) decode should be on struct level not format?
• tovalue({bits_format: "base64"}) only affect root value.
• Auto complete of non-global variables is broken. scope is broken for variables.
• echo '{} {} {}' | jq vs echo '{} {} {}' | fq works differently. fq currently decodes one root format and might add unknown gap fields etc. Maybe should work differently for json format?
• format/0 overlap with jq builtin format/1. What to rename it to? decode_format?
• repl expression returning a value that produced lots of output can't be interrupted. This is because ctrl-c currently only interrupts the eval interpreter, outputted value is printed (display) by parent interpreter.
• Rework cli/repl user interrupt (context cancel via ctrl-c), see comment in Interp.Main
• Optimize Interp.Options calls, now called per display. Cache per eval? needs to handle nested evals.
• <array decode value>[{start: ...: end: ...}] syntax a bit broken.
• REPL completion might have side effcts. Make interp.Function type know and wrap somehow? input, inputs, open, ...
• Rework group arguments so that {is_probe:true} is not needed. Look up group name and see if it has an argument somehow?

TODO and ideas

CLI

• --args support
• Reset color at prompt? context cancel

CLI and REPL

• ctxstack index cancel wrong order, should just skip?
• Pager for long output. Configurable? $PAGER? only explicit with some kind of syntax? .. | less but how?
• dump cancel output of large root value, ex: .frames. Problem is dump is done by parent repl.
• Error position "^" pointer?
• Configurable history file/name?
• Auto complete $variables
• Auto complete keys that need escaping, now just filtered out
• Auto complete add "." just one and is object
• Use JQ_COLORS but extended to allow name= also?

Language

• Cleanup/Make binary buffers make sense.
• gojq uses golang int for slice indexes, might be issue for non-64bit cpus

Functions

• buffer truncate, left/right pad?
• toimage? can be done in CLI with "\x1b]1337" but maybe something for a UI?
• toplot?
• dump should handle binary, make column code more generic? share with hexdump? (bindump also?)
• dump colorize/notify row range discontinuity
• hexdump etc should handle binary non byte aligned data
• Cleanup rework cipher functions, ctr(aes("key"), "iv") or cipher(ctr("iv"), aes("key))?
• open when to close file?
• Safe mode interpreter?
• Allow/deny open in autocomplete
• open leak, file and ctxreadseeker
• Summary tree with format specific summaries for each format, sample count etc etc?
• List all unique paths in some compact form?

Tests

• update tests does not preserve comment order for readlines
• empty file test
• CLI tests, raw write, colors?
• Interactive tests

Documentation

• help("topic")?
• Generate from source
• -n, inputs/0 and input/0 behavior. Same as jq.
• Mention empty.something?
• Use https://github.com/fadado/JBOL/blob/master/doc/JQ-Distilled.md notation

Decode

• Use interfaces to save memory, "Value V" interface so can have U, Str, etc implementations?
• Array of "decorations" sym, display format?
• Store original filename somewhere? description for now
• Nicer "synthetic" values? now zero length
• Cleanup and rethink nested buffers (zip, muxed like ogg)
• Endian bitfield helper (elf etc)
• Cleanup checksums, should just be fields and add warning if mismatch?
• Decoder in jq
  • Use jq array/object syntax and pass around decode context, collect fields and build tree
• Somehow control/limit nested decoding, depth/exclude/include? probe({depth:1}) etc? per format skip options?
• Can't use range while decoding, not calculated yet
• Keep track of encoding for values, u16le, utf8, varint etc
• Option to ignore range checks, decode until read error instead. Ex: mp4 with truncated mdat.

Formats

• asn1_ber asn1_der, asn1_cer decoder
• flatbuffer decoder
• capnproto decoder
• Pass argument to format
• Value decoder in jq u(32), u32?
• Warnings and errors
  • mp4 sample counts
  • flac truncated picture, mix sample rate, bitdepth etc?
• protobuf schema?
• matroska crc
• mp4 styp segment test
• Document maturity/completeness
• Add dsf format
• Make json format more normal? is a bit a of a special case now
• exif in mp4 (heif/heic):
  . as $r | grep("iloc") | parent.items[] | select(.id == (first($r | grep("exif";"i")) | parent.id)).extends[0] as $e | $r | tobytes[$e.offset+10:$e.offset+$e.length] | exif

Scripts

• Probe tool with common field names
• MIME codec encode/decoder "avc1.PPCCLL" etc https://tools.ietf.org/html/rfc6381#section-3.3
• Validate scripts for mp4, matroska

gojq

• Common errors with gojq? re-implemented now
• 0b -> 1.7976931348623157e+308 something fishy with bin/hex/... literals change
• Do something similar to builtin.go in gojq to speedup a bit
• remove scopedump?

Big things

• fq play website?
• UI, web interface? tree interface, multiple repl windows? nicer way of showing overlapping fiends in hex etc?
• jupyter notebook integration
• FUSE interface
• Lazy decode, should work on known sizes? could also save memory be re-decode?