Merge pull request #72 from wader/doc-dev-tips

doc: Add more dev tips

doc/dev.md

@@ -1,32 +1,29 @@
-# Implementation details
-
-- fq uses a gojq fork that can be found at https://github.com/wader/gojq/tree/fq (the "fq" branch)
-- fq uses a readline fork that can be found at https://github.com/wader/readline/tree/fq (the "fq" branch)
-- cli readline uses raw mode so blocks ctrl-c to become a SIGINT
-
 ## Implement a decoder
 
 ### Steps to add new decoder
 
 - Create a directory  `format/<name>`
 - Copy some similar decoder, `format/format/bson.go` is quite small, to `format/<name>/<name>.go`
-- Cleanup and fill in the register struct, rename `format.BSON` and don't forget to change the string constant.
+- Cleanup and fill in the register struct, rename `format.BSON` and add it
+to `format/fromat.go` and don't forget to change the string constant.
 - Add an import to `format/all/all.go`
 
-## Things to think about
+### Some general tips
 
-- Main goal in the end is to produce a tree that is user-friendly and easy to work with.
-Prefer a nice and easy to use tree over nice decoder implementation.
-- Use same names, symbols, constant number base etc as in specification. But prefer lowercase to jq/JSON-ish.
+- Main goal is to produce a tree structure that is user-friendly and easy to work with.
+Prefer a nice and easy query tree structure over nice decoder implementation.
+- Use same names, symbols, constant number bases etc as in specification.
+But maybe in lowercase to be jq/JSON-ish.
 - Decode only ranges you know what they are. If possible let "parent" decide what to do with unknown
-bits by using `*Decode*Len/Range/Limit` functions. fq will automatically add "unknown" fields.
-- Try to no decode too much as one value.
+bits by using `*Decode*Len/Range/Limit` functions. fq will also automatically add "unknown" fields if
+it finds gaps.
+- Try to not decode too much as one value.
 A length encoded int could be two fields, but maybe a length prefixed string should be one.
 Flags can be struct with bit-fields.
-- Map as many value as possible to more usage symbolic values.
+- Map as many value as possible to more symbolic values.
 - Endian is inherited inside one format decoder, defaults to big endian for new format decoder
-- Make sure zero length or no frames etc found fails decoding
-- If in the probe group make sure to validate input to make it non-ambiguous with other decoders
+- Make sure zero length or no frames found etc fails decoding
+- If format is in the probe group make sure to validate input to make it non-ambiguous with other decoders
 - Try keep decoder code as declarative as possible
 - Split into multiple sub formats if possible. Makes it possible to use them separately.
 - Validate/Assert
@@ -35,23 +32,63 @@ Flags can be struct with bit-fields.
 - Can new formats be added to other formats
 - Does the new format include existing formats
 
-Run `make doc` generate some of the documentation (requires [FFmpeg](https://github.com/FFmpeg/FFmpeg) and [Graphviz](https://gitlab.com/graphviz/graphviz)).
+### Development tips
 
-Run `make lint` to lint source code.
-
-TODO: `make fuzz`
-
-## Tests
+I ususally use `-d <format>` and `v` while developing, that way you will get a decode tree
+even if it fails. `v` gives verbose output and also includes stacktrace.
 
 ```sh
+go run fq.go -d <format> v file
+```
+
+If the format is inside some other format it can be handy to first extract the bits and run
+the decode directly. For example if working a `aac_frame` decoder issue:
+
+```sh
+fq '.tracks[0].samples[1234] | tobytes' file.mp4 > aac_frame_1234
+fq -d aac_frame v aac_frame_1234
+```
+
+Sometimes nested decoding fails then maybe a good way is to change the parent decoder to
+use `d.RawLen()` etc instead of `d.FormatLen()` etc temporary to extract the bits. Hopefully
+there will be some option to do this in the future.
+
+When researching or investinging something I can recommend to use `watchexec`, `modd` etc to
+make things more comfortable. Also using vscode/delve for debugging should work fine once
+launch `args` are setup etc.
+
+```
+watchexec "go run fq.go -d aac_frame v aac_frame"
+```
+
+Some different ways to run tests:
+```sh
+# run all tests
+make test
+# run all go tests
+go test ./...
 # run all tests for one format
 go test -run TestFQTests/mp4 ./format/
 # write all actual outputs
 make actual
-# write for specific tests
+# write actual output for specific tests
 WRITE_ACTUAL=1 go run -run ...
+# color diff
+DIFF_COLOR=1 go test ...
 ```
 
+To lint source use:
+```
+make lint
+```
+
+Generate documentation. Requires [FFmpeg](https://github.com/FFmpeg/FFmpeg) and [Graphviz](https://gitlab.com/graphviz/graphviz):
+```sh
+make doc
+```
+
+TODO: `make fuzz`
+
 ## Debug
 
 Split debug and normal output even when using repl:
@@ -138,6 +175,12 @@ cd ../fq
 docker --context 2016-box run --rm -ti -v "C:${PWD//\//\\}:C:${PWD//\//\\}" -w "$PWD" golang:1.17.5-windowsservercore-ltsc2016
 ```
 
+# Implementation details
+
+- fq uses a gojq fork that can be found at https://github.com/wader/gojq/tree/fq (the "fq" branch)
+- fq uses a readline fork that can be found at https://github.com/wader/readline/tree/fq (the "fq" branch)
+- cli readline uses raw mode so blocks ctrl-c to become a SIGINT
+
 ## Release process
 
 Run and follow instructions: