doc,dev: Add more usage and dev tips

doc/dev.md

@@ -17,6 +17,7 @@ But maybe in lowercase to be jq/JSON-ish.
 - Decode only ranges you know what they are. If possible let "parent" decide what to do with unknown gaps
 bits by using `*Decode*Len/Range/Limit` functions. fq will also automatically add "gap" fields if
 it finds gaps.
+- If you have decode helpers functions that decode a bunch of fields etc it is usually nice to make it only decode fields, not seek or add it's own "containing" struct. That way the function will be easier to reuse and only do one thing. Ex the helper `func decodeHeader(d *decode.D)` can then be use as `d.FieldStruct("header", decodeHeader)`, `d.SeekRel(1234, decodeHeader)` or `d.SeekRel(1234, func(d *decode.D) { d.FieldStruct("header, decodeHeader") }`
 - Try to not decode too much as one value.
 A length encoded int could be two fields, but maybe a length prefixed string should be one.
 Flags can be struct with bit-fields.
@@ -32,6 +33,25 @@ Flags can be struct with bit-fields.
 - Can new formats be added to other formats
 - Does the new format include existing formats
 
+### Checklist
+
+- Commits:
+  - Use commit messages with a context prefix to make it easier to find and understand, ex: `mp3: Validate sync correctly`
+- Tests:
+  - If possible use a pair of `testdata/file` and `testdata/file.fqtest` where `file.fqtest` is `$ fq dv file` and optionally `$ fq torepr file` if there is `torepr` support.
+  - If `dv` produces a lof of output maybe use `dv({array_truncate: 50})` etc
+  - Run `go test ./format -run TestFormats/<name>` to test expected output.
+  - Run `WRITE_ACTUAL=1 go test ./format -run TestFormats/<name>` to write current output as expected output.
+- If you have format specific documentation:
+  - Put it in `format/*/<name>.md` and use `//go:embed <name>.md`/`interp.RegisterFS(..)` to embed/register it.
+  - Use simple markdown, just sections (depth starts at 3, `### Section`),  paragraphs, lists and links.
+  - No heading section is needs with format name, will be added by `make doc` and fq cli help system.
+  - Add a `testdata/<name>_help.fqtest` with just `$ fq -h <name>` to test CLI help.
+  - Run `make doc` to add documentation to `doc/formats.md`.
+  - If in doubt look at `mp4.md`/`mp4.go` etc.
+- Run linter `make lint`
+- Run fuzzer `make fuzz GROUP=<name>`, see usage in Makefile
+
 ### Decoder API
 
 `*decode.D` reader methods use this name convention:
@@ -215,7 +235,7 @@ make test
 # run all go tests
 go test ./...
 # run all tests for one format
-go test -run TestFQTests/mp4 ./format/
+go test -run TestFormats/mp4 ./format/
 # write all actual outputs
 WRITE_ACTUAL=1 go test ./...
 # write actual output for specific tests

doc/usage.md

@@ -9,6 +9,7 @@ cat file | fq
 fq . < file
 fq . *.png *.mp3
 fq '.frames[0]' *.mp3
+fq '.frames[-1] | tobytes' file.mp3 > last_frame
 ```
 
 ### Common usages
@@ -20,18 +21,40 @@ fq d file
 fq display file
 
 # display all bytes for each value
+fq dd file
+# same as
 fq 'd({display_bytes: 0})' file
+
 # display 200 bytes for each value
 fq 'd({display_bytes: 200})' file
 
 # recursively display decode tree without truncating
 fq da file
+# same as
+fq 'd({array_truncate: 0})' file
 
 # recursively and verbosely display decode tree
 fq dv file
+# same as
+fq 'd({verbose: true})' file
 
 # JSON repersenation for whole file
 fq tovalue file
+# JSON but raw bit fields truncated
+fq -o bits_format=truncate tovalue file
+# JSON but raw bit fields as md5 hex string
+fq -o bits_format=md5 tovalue file
+
+# grep whole tree by value
+fq 'grep("^prefix")' file
+fq 'grep(123)' file
+# grep whole tree by condition
+fq 'grep_by(. >= 100 and . =< 100)' file
+
+# recursively look for values fullfilling some condition
+fq '.. | select(.type=="trak")?' file
+fq 'grep_by(.type=="trak")' file
+# grep_by(f) is alias for .. | select(f)?, that is: recuse, select and ignore errors
 
 # recursively look for decode value roots for a format
 fq '.. | select(format=="jpeg")' file
@@ -42,15 +65,6 @@ fq 'grep_by(format=="jpeg")' file
 fq 'first(.. | select(format=="jpeg"))' file
 fq 'first(grep_by(format=="jpeg"))' file
 
-# recursively look for objects fullfilling condition
-fq '.. | select(.type=="trak")?' file
-fq 'grep_by(.type=="trak")' file
-
-# grep whole tree
-fq 'grep("^prefix")' file
-fq 'grep(123)' file
-fq 'grep_by(. >= 100 and . =< 100)' file
-
 # decode file as mp4 and return a result even if there are some errors
 fq -d mp4 file.mp4
 # decode file as mp4 and also ignore validity assertions
@@ -179,9 +193,9 @@ fq '.frames[0:10] | map(tobytesrange.start)' file.mp3
 #### Decode at range
 
 ```sh
-# decode byte range 100 to end
+# decode byte range 100 to end as mp3_frame
 fq -d bytes '.[100:] | mp3_frame | d' file.mp3
-# decode byte range 10 bytes into .somefield and preseve relative position in file
+# decode byte range 10 bytes from .somefield and preseve relative position in file
 fq '.somefield | tobytesrange[10:] | mp3_frame | d' file.mp3
 ```
 
@@ -261,6 +275,12 @@ To get the most out of fq it's recommended to learn more about jq, here are some
 - [jq wiki: Pitfalls](https://github.com/stedolan/jq/wiki/How-to:-Avoid-Pitfalls)
 - [FAQ](https://github.com/stedolan/jq/wiki/FAQ)
 
+For a more convenient jq experience these might be interesting:
+
+- [jq Dash docset](https://github.com/wader/jq-dash-docset)
+- [vscode-jq](https://github.com/wader/vscode-jq)
+- [jq-lsp](https://github.com/wader/jq-lsp)
+
 Common beginner gotcha are:
 - jq's use of `;` and `,`. jq uses `;` as argument separator
 and `,` as output separator. To call a function `f` with two arguments use `f(1; 2)`. If you do `f(1, 2)` you pass a
@@ -269,7 +289,6 @@ single argument `1, 2` (a lambda expression that output `1` and then output `2`)
 achieved.
 - Expressions have one implicit input and output value. This how pipelines like `1 | . * 2` work.
 
-
 ## Types specific to fq
 
 fq has two additional types compared to jq, decode value and binary. In standard jq expressions they will in most case behave as some standard jq type.

format/fqtest_test.go

@@ -8,6 +8,6 @@ import (
 	"github.com/wader/fq/pkg/interp"
 )
 
-func TestFQTests(t *testing.T) {
+func TestFormats(t *testing.T) {
 	fqtest.TestPath(t, interp.DefaultRegistry)
 }

pkg/interp/fqtest_test.go

@@ -8,6 +8,6 @@ import (
 	"github.com/wader/fq/pkg/interp"
 )
 
-func TestFQTests(t *testing.T) {
+func TestInterp(t *testing.T) {
 	fqtest.TestPath(t, interp.DefaultRegistry)
 }