mirror of
https://github.com/wader/fq.git
synced 2024-12-24 13:52:02 +03:00
6.2 KiB
6.2 KiB
Known issues
format/0
overlap with jq builtinformat/1
. What to rename it to?decode_format
?- repl expression returning a value that produced lots of output can't be interrupted. This is becaus ctrl-C currently only interrupts the evaluation of the expression, outputted value is printed (
display
) by parent. - Auto complete of non-global variables is broken.
scope
is broken for variables. tovalue({bits_format: "base64"})
only affect root value.- Unknown field filling is currently only done on root, should be done per "framed" decode.
- Sub buffers range is broken, could have a different size in parent (ex: compressed).
- Symbolic values should be more intuitive, should not need to use ._symbol.
echo '{} {} {}' | jq
vsecho '{} {} {}' | fq
works differently. fq currently decodes one root format and might add unknown fields etc. Maybe should work differently forjson
format?
TODO and ideas
CLI
--args
support- Reset color at prompt? context cancel
CLI and REPL
- ctxstack index cancel wrong order, should just skip?
- Pager for long output. Configurable?
$PAGER
? only explicit with some kind of syntax?.. | less
but how? - Nicer context cancel message
dump
cancel output of large root value, ex:.frames
. Problem is dump is done by parent repl.- Error position "^" pointer?
- Configurable history file/name?
- Auto complete $variables
- Auto complete keys that need escaping, now just filtered out
- Auto complete add "." just one and is object
- Use JQ_COLORS but extended to allow name= also?
Language
- Nicer variables somehow?
... | var($VAR)
? make slurp and rewrite$var
to$var[]
? - Cleanup/Make binary buffers make sense.
- gojq uses golang
int
for slice indexes, might be issue for non-64bit cpus
Functions
- buffer truncate, left/right pad?
toimage
? can be done in CLI with "\x1b]1337" but maybe something for a UI?toplot
?dump
should handle binary, make column code more generic? share withhexdump
? (bindump also?)dump
colorize/notify row range discontinuitydump
truncate long arrays in output unless verbose?dd
?hexdump
etc should handle binary non byte aligned datatojvalue
handle binary somehow, base64 string, truncate? md5 digest etc? configurable?{binarystring: "md5"}
- Function to search in binary data, regexp?
grep
like function somehow?- Cleanup rework cipher functions,
ctr(aes("key"), "iv")
orcipher(ctr("iv"), aes("key))
? open
when to close file?- Safe mode interpreter?
- Allow/deny
open
in autocomplete open
leak, file and ctxreadseeker
Tests
- WRITE_ACTUAL does not preserve comment order
- empty file test
- CLI tests, raw write, colors?
- Interactive tests
Documentation
- Nicer README, repl example (generate somehow)
help("topic")
?- Generate from source
-n
,inputs/0
andinput/0
behavior. Same as jq.- Mention
empty.something
? - Known issus and confusing things
- Symbolic number has to use
._symbol
for now. For example matroska ids are number ID:s that have symbolic string names.
- Symbolic number has to use
- Use https://github.com/fadado/JBOL/blob/master/doc/JQ-Distilled.md notation
- Decoder write guide
- Endian inherited per buffer, reset on new buffer
- Invalid on zero length input, assert one valid frame etc
- Try validate input to make it not ambiguous with other decoders
- Try to not seek and read at end while validating or early, will break progress indicator if not
- Split bit flags etc into a field with subfields for each bit
- Try keep code as declarative as possible
- Split into multiple sub formats if possible
- See the decoded tree as user interface but still has to represent the actual bit structure
Decode
- Store original filename somewhere? description for now
- Nicer DSL
- More optional things? optional args or return value to modify?
- Nicer "synthetic" values? now zero length
- Value should have raw, tranlated, symbolic and description? struct of map functions?
- If symbolic should resolve to string? would make
.field == "abc"
work instead of now.field._symbolic
- If symbolic should resolve to string? would make
- Array as root value, adts, avc_au etc
- Decode framed/limited? framed adds unknown fields?
- Add unknown for arrays?
- Cleanup and rethink nested buffers (zip, muxed like ogg)
- Root of nested buffer, what range?
dump
has some bug not showing buffer "nesting" level in the address column- Endian bitfield helper (elf etc)
- Cleanup checksums, should just be fields and add warning if mismatch?
- Decoder in jq
- Use jq array/object syntax and pass around decode context, collect fields and build tree
- Somehow control/limit nested decoding, depth/exclude/include?
probe({depth:1})
etc? per format skip options? - Can't use range while decoding, not calculated yet
- Keep track of encoding for values, u16le, utf8, varint etc
- Option to ignore range checks, decode until read error instead. Ex: mp4 with truncated mdat.
Formats
- Pass argument to format
- Value decoder in jq
u(32)
,u32
? - Warnings and errors
mp4
sample countsflac
truncated picture, mix sample rate, bitdepth etc?
protobuf
schema?matroska
crcmp4
styp segment test- Document maturity/completeness
- Refactor *[]decode.Format into something more abstract, group?
- Add
dsf
format - Make
json
format more normal? is a bit a of a special case now
Scripts
- Summary script, tree with format specific summaries like codec, sample count etc etc?
- Probe tool with common field names
- MIME codec encode/decoder "avc1.PPCCLL" etc https://tools.ietf.org/html/rfc6381#section-3.3
- Validate scripts for mp4, matroska
gojq
- Common errors with gojq? re-implemented now
join
can be exponential, try add with strings faster, use add["a","b","c"] | add
?0b
->1.7976931348623157e+308
something fishy with bin/hex/... literals change- Do something similar to
builtin.go
in gojq to speedup a bit - remove
scopedump
?
Readline
- Use something else than
github.com/chzyer/readline
? - Fix backspace as start clears whole line bug
- Undo ctrl+_ would be nice
Big things
- UI, web interface? multiple repl windows? nicer way of showing overlapping fiends in hex etc?
- jupyter notebook integration
- FUSE interface
- Lazy decode, should work on known sizes? could also save memory be re-decode?