mirror of
https://github.com/wader/fq.git
synced 2024-09-11 12:05:39 +03:00
doc,dev: Add more usage and dev tips
This commit is contained in:
parent
1dd9c8759a
commit
6fc84a885c
22
doc/dev.md
22
doc/dev.md
@ -17,6 +17,7 @@ But maybe in lowercase to be jq/JSON-ish.
|
||||
- Decode only ranges you know what they are. If possible let "parent" decide what to do with unknown gaps
|
||||
bits by using `*Decode*Len/Range/Limit` functions. fq will also automatically add "gap" fields if
|
||||
it finds gaps.
|
||||
- If you have decode helpers functions that decode a bunch of fields etc it is usually nice to make it only decode fields, not seek or add it's own "containing" struct. That way the function will be easier to reuse and only do one thing. Ex the helper `func decodeHeader(d *decode.D)` can then be use as `d.FieldStruct("header", decodeHeader)`, `d.SeekRel(1234, decodeHeader)` or `d.SeekRel(1234, func(d *decode.D) { d.FieldStruct("header, decodeHeader") }`
|
||||
- Try to not decode too much as one value.
|
||||
A length encoded int could be two fields, but maybe a length prefixed string should be one.
|
||||
Flags can be struct with bit-fields.
|
||||
@ -32,6 +33,25 @@ Flags can be struct with bit-fields.
|
||||
- Can new formats be added to other formats
|
||||
- Does the new format include existing formats
|
||||
|
||||
### Checklist
|
||||
|
||||
- Commits:
|
||||
- Use commit messages with a context prefix to make it easier to find and understand, ex: `mp3: Validate sync correctly`
|
||||
- Tests:
|
||||
- If possible use a pair of `testdata/file` and `testdata/file.fqtest` where `file.fqtest` is `$ fq dv file` and optionally `$ fq torepr file` if there is `torepr` support.
|
||||
- If `dv` produces a lof of output maybe use `dv({array_truncate: 50})` etc
|
||||
- Run `go test ./format -run TestFormats/<name>` to test expected output.
|
||||
- Run `WRITE_ACTUAL=1 go test ./format -run TestFormats/<name>` to write current output as expected output.
|
||||
- If you have format specific documentation:
|
||||
- Put it in `format/*/<name>.md` and use `//go:embed <name>.md`/`interp.RegisterFS(..)` to embed/register it.
|
||||
- Use simple markdown, just sections (depth starts at 3, `### Section`), paragraphs, lists and links.
|
||||
- No heading section is needs with format name, will be added by `make doc` and fq cli help system.
|
||||
- Add a `testdata/<name>_help.fqtest` with just `$ fq -h <name>` to test CLI help.
|
||||
- Run `make doc` to add documentation to `doc/formats.md`.
|
||||
- If in doubt look at `mp4.md`/`mp4.go` etc.
|
||||
- Run linter `make lint`
|
||||
- Run fuzzer `make fuzz GROUP=<name>`, see usage in Makefile
|
||||
|
||||
### Decoder API
|
||||
|
||||
`*decode.D` reader methods use this name convention:
|
||||
@ -215,7 +235,7 @@ make test
|
||||
# run all go tests
|
||||
go test ./...
|
||||
# run all tests for one format
|
||||
go test -run TestFQTests/mp4 ./format/
|
||||
go test -run TestFormats/mp4 ./format/
|
||||
# write all actual outputs
|
||||
WRITE_ACTUAL=1 go test ./...
|
||||
# write actual output for specific tests
|
||||
|
43
doc/usage.md
43
doc/usage.md
@ -9,6 +9,7 @@ cat file | fq
|
||||
fq . < file
|
||||
fq . *.png *.mp3
|
||||
fq '.frames[0]' *.mp3
|
||||
fq '.frames[-1] | tobytes' file.mp3 > last_frame
|
||||
```
|
||||
|
||||
### Common usages
|
||||
@ -20,18 +21,40 @@ fq d file
|
||||
fq display file
|
||||
|
||||
# display all bytes for each value
|
||||
fq dd file
|
||||
# same as
|
||||
fq 'd({display_bytes: 0})' file
|
||||
|
||||
# display 200 bytes for each value
|
||||
fq 'd({display_bytes: 200})' file
|
||||
|
||||
# recursively display decode tree without truncating
|
||||
fq da file
|
||||
# same as
|
||||
fq 'd({array_truncate: 0})' file
|
||||
|
||||
# recursively and verbosely display decode tree
|
||||
fq dv file
|
||||
# same as
|
||||
fq 'd({verbose: true})' file
|
||||
|
||||
# JSON repersenation for whole file
|
||||
fq tovalue file
|
||||
# JSON but raw bit fields truncated
|
||||
fq -o bits_format=truncate tovalue file
|
||||
# JSON but raw bit fields as md5 hex string
|
||||
fq -o bits_format=md5 tovalue file
|
||||
|
||||
# grep whole tree by value
|
||||
fq 'grep("^prefix")' file
|
||||
fq 'grep(123)' file
|
||||
# grep whole tree by condition
|
||||
fq 'grep_by(. >= 100 and . =< 100)' file
|
||||
|
||||
# recursively look for values fullfilling some condition
|
||||
fq '.. | select(.type=="trak")?' file
|
||||
fq 'grep_by(.type=="trak")' file
|
||||
# grep_by(f) is alias for .. | select(f)?, that is: recuse, select and ignore errors
|
||||
|
||||
# recursively look for decode value roots for a format
|
||||
fq '.. | select(format=="jpeg")' file
|
||||
@ -42,15 +65,6 @@ fq 'grep_by(format=="jpeg")' file
|
||||
fq 'first(.. | select(format=="jpeg"))' file
|
||||
fq 'first(grep_by(format=="jpeg"))' file
|
||||
|
||||
# recursively look for objects fullfilling condition
|
||||
fq '.. | select(.type=="trak")?' file
|
||||
fq 'grep_by(.type=="trak")' file
|
||||
|
||||
# grep whole tree
|
||||
fq 'grep("^prefix")' file
|
||||
fq 'grep(123)' file
|
||||
fq 'grep_by(. >= 100 and . =< 100)' file
|
||||
|
||||
# decode file as mp4 and return a result even if there are some errors
|
||||
fq -d mp4 file.mp4
|
||||
# decode file as mp4 and also ignore validity assertions
|
||||
@ -179,9 +193,9 @@ fq '.frames[0:10] | map(tobytesrange.start)' file.mp3
|
||||
#### Decode at range
|
||||
|
||||
```sh
|
||||
# decode byte range 100 to end
|
||||
# decode byte range 100 to end as mp3_frame
|
||||
fq -d bytes '.[100:] | mp3_frame | d' file.mp3
|
||||
# decode byte range 10 bytes into .somefield and preseve relative position in file
|
||||
# decode byte range 10 bytes from .somefield and preseve relative position in file
|
||||
fq '.somefield | tobytesrange[10:] | mp3_frame | d' file.mp3
|
||||
```
|
||||
|
||||
@ -261,6 +275,12 @@ To get the most out of fq it's recommended to learn more about jq, here are some
|
||||
- [jq wiki: Pitfalls](https://github.com/stedolan/jq/wiki/How-to:-Avoid-Pitfalls)
|
||||
- [FAQ](https://github.com/stedolan/jq/wiki/FAQ)
|
||||
|
||||
For a more convenient jq experience these might be interesting:
|
||||
|
||||
- [jq Dash docset](https://github.com/wader/jq-dash-docset)
|
||||
- [vscode-jq](https://github.com/wader/vscode-jq)
|
||||
- [jq-lsp](https://github.com/wader/jq-lsp)
|
||||
|
||||
Common beginner gotcha are:
|
||||
- jq's use of `;` and `,`. jq uses `;` as argument separator
|
||||
and `,` as output separator. To call a function `f` with two arguments use `f(1; 2)`. If you do `f(1, 2)` you pass a
|
||||
@ -269,7 +289,6 @@ single argument `1, 2` (a lambda expression that output `1` and then output `2`)
|
||||
achieved.
|
||||
- Expressions have one implicit input and output value. This how pipelines like `1 | . * 2` work.
|
||||
|
||||
|
||||
## Types specific to fq
|
||||
|
||||
fq has two additional types compared to jq, decode value and binary. In standard jq expressions they will in most case behave as some standard jq type.
|
||||
|
@ -8,6 +8,6 @@ import (
|
||||
"github.com/wader/fq/pkg/interp"
|
||||
)
|
||||
|
||||
func TestFQTests(t *testing.T) {
|
||||
func TestFormats(t *testing.T) {
|
||||
fqtest.TestPath(t, interp.DefaultRegistry)
|
||||
}
|
||||
|
@ -8,6 +8,6 @@ import (
|
||||
"github.com/wader/fq/pkg/interp"
|
||||
)
|
||||
|
||||
func TestFQTests(t *testing.T) {
|
||||
func TestInterp(t *testing.T) {
|
||||
fqtest.TestPath(t, interp.DefaultRegistry)
|
||||
}
|
||||
|
Loading…
Reference in New Issue
Block a user