git.iko.soy/rowtype-yoga/purescript-protobuf
on v4.5.0 help sign in
rowtype-yoga/purescript-protobuf/README.md
378 lines 14 kB history raw
1# purescript-protobuf 💝
3[![Pursuit](http://pursuit.purescript.org/packages/purescript-protobuf/badge)](http://pursuit.purescript.org/packages/purescript-protobuf/)
4[![Maintainer: jamesdbrock](https://img.shields.io/badge/maintainer-jamesdbrock-teal.svg)](https://github.com/jamesdbrock)
6PureScript library and code generator for
7[Google Protocol Buffers version 3](https://protobuf.dev/programming-guides/proto3/).
9This library operates on
10[`ArrayBuffer`](https://pursuit.purescript.org/packages/purescript-arraybuffer-types/docs/Data.ArrayBuffer.Types#t:ArrayBuffer), so it will run both
11[in *Node.js*](https://pursuit.purescript.org/packages/purescript-node-buffer/docs/Node.Buffer.Class)
12and in browser environments.
14## Features
16We aim to support binary-encoded protobuf for
17`syntax = "proto3";` descriptor files.
19Many `syntax = "proto2";` descriptor files will
20also work, as long as they don't use `"proto2"` features, especially
21[groups](https://protobuf.dev/programming-guides/proto2/#groups),
22which we do not support. We also do not support `"proto2"`
23[extensions](https://protobuf.dev/programming-guides/proto2/#extensions).
25We do not support
26[services](https://protobuf.dev/programming-guides/proto3/#services).
28We do not support
29[JSON Mapping](https://protobuf.dev/programming-guides/proto3/#json).
31### Conformance and Testing
33In this version, we pass all 684 of the
34[Google conformance tests](https://github.com/protocolbuffers/protobuf/tree/main/conformance)
35of binary-wire-format *proto3* for [Protocol Buffers v28.2](https://github.com/protocolbuffers/protobuf/releases/tag/v28.2).
36See the `conformance/README.md` in this repository for details.
38We also have our own unit tests, see `test/README.md` in this repository.
40## Code Generation
42The `nix develop` environment provides
44* The PureScript toolchain: `purs`, `spago`, and `node`.
45* The [`protoc`](https://protobuf.dev/programming-guides/proto3/#generating) compiler.
46* The `protoc-gen-purescript` executable plugin for `protoc` on the `PATH` so that
47 [`protoc` can find it](https://protobuf.dev/reference/cpp/api-docs/google.protobuf.compiler.plugin/).
49```
50$ nix develop
52PureScript Protobuf development environment
53libprotoc 28.2
54node v20.15.1
55purs 0.15.15
56spago 0.93.40
58To build the protoc compiler plugin, run:
60 cd plugin
61 spago build
63To compile PureScript .purs files from .proto files, run for example:
65 protoc --purescript_out=. google/protobuf/timestamp.proto
66```
68We can test out code generation immediately by
69generating `.purs` files for any of Google’s built-in “well-known types” in the
70[`google.protobuf`](https://protobuf.dev/reference/protobuf/google.protobuf/) package namespace. Try the command
71```console
72protoc --purescript_out=. google/protobuf/any.proto
73```
74or
75```console
76protoc --purescript_out=. google/protobuf/timestamp.proto
77```
79To see
80[all of the `.proto` definitions](https://github.com/protocolbuffers/protobuf/tree/main/src/google/protobuf)
81included with the Nix PureScript Protobuf installation including
82the “well-known types,”
83```console
84ls $(nix path-info .#protobuf)/src/google/protobuf/*.proto
85```
87If you don't want to use Nix,
881. install the PureScript toolchain and `protoc`.
892. Build the [PureScript plugin for `protoc`](plugin/).
903. Run `protoc` with the path to the PureScript plugin executable, like for example
91 ```console
92 protoc --plugin=bin/protoc-gen-purescript --purescript_out=. google/protobuf/timestamp.proto
93 ```
95## Writing programs with the generated code
97The code generator will use the `package` import statement in the `.proto` file
98and the base `.proto` file name as the PureScript module name for that file.
100A message in a `shapes.proto` descriptor file declared as
102```
103syntax = "proto3";
104package interproc;
106message Rectangle {
107 double width = 1;
108 double height = 2;
110```
112will export these four names from module `Interproc.Shapes` in a
113generated `shapes.Interproc.purs` file.
1151. A message data type.
117 ```purescript
118 newtype Rectangle = Rectangle { width :: Maybe Number, height :: Maybe Number }
119 ```
121 The message data type will also include an `__unknown_fields` array field for
122 holding received fields which were not in the descriptor `.proto` file. We can
123 ignore `__unknown_fields` if we want to.
1252. A message maker which constructs a message from a `Record`
126 with some message fields.
128 ```purescript
129 mkRectangle :: forall r. Record r -> Rectangle
130 ```
132 All message fields are optional, and can be omitted when making a message.
133 There are some extra type constraints, not shown here, which will cause a
134 compiler error if we try to add a field which is not in the message data type.
136 If we want the compiler to check that we’ve explicitly supplied all the fields,
137 then we can use the ordinary message data type constructor `Rectangle`.
1393. A message serializer which works with
140 [__arraybuffer-builder__](http://pursuit.purescript.org/packages/purescript-arraybuffer-builder/).
142 ```purescript
143 putRectangle :: forall m. MonadEffect m => Rectangle -> PutM m Unit
144 ```
1464. A message deserializer which works with
147 [__parsing-dataview__](http://pursuit.purescript.org/packages/purescript-parsing-dataview/).
149 ```purescript
150 parseRectangle :: forall m. MonadEffect m => ByteLength -> ParserT DataView m Rectangle
151 ```
153 The message parser needs an argument which tells it the
154 length of the message which it’s about to parse, because
155 [“the Protocol Buffer wire format is not self-delimiting.”](https://protobuf.dev/programming-guides/techniques/#streaming)
157In our program, our imports will look something like this.
158The only module from this package which we will import into our program
159will be the `Protobuf.Library` module.
160We'll import the message modules from the generated `.purs` files.
161We'll also import modules for reading and writing `ArrayBuffer`s.
164```purescript
165import Protobuf.Library (Bytes(..))
166import Interproc.Shapes (Rectangle, mkRectangle, putRectangle, parseRectangle)
167import Parsing (runParserT, ParseError, liftMaybe)
168import Data.ArrayBuffer.Builder (execPutM)
169import Data.ArrayBuffer.DataView (whole)
170import Data.ArrayBuffer.ArrayBuffer (byteLength)
171import Data.Tuple (Tuple)
172import Data.Newtype (unwrap)
173```
175This is how we serialize a `Rectangle` to an `ArrayBuffer`.
176We must be in a `MonadEffect`.
178```purescript
179do
180 arraybuffer <- execPutM $ putRectangle $ mkRectangle
181 { width: Just 3.0
182 , height: Just 4.0
183 }
184```
186Next we’ll deserialize `Rectangle` from the `ArrayBuffer` that we just made.
188```purescript
189 result :: Either ParseError {width :: Number, height :: Number}
190 <- runParserT (whole arraybuffer) do
191 rectangle :: Rectangle <- parseRectangle (byteLength arraybuffer)
192```
194At this point we’ve consumed all of the parser input and constructed our
195`Rectangle` message, but we’re not finished parsing.
196We want to “validate” the `Rectangle` message to make sure it has all of the
197fields that we require, because in
198[*proto3*, all fields are optional](https://github.com/protocolbuffers/protobuf/issues/2497).
200Fortunately we are already in the `ParserT` monad,
201so we can do better than to “validate”:
202[Parse, don't validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/).
204We will construct a record `{width::Number, height::Number}`
205with the width and height of the `Rectangle`. If the width or height
206are missing from the `Rectangle` message, then we will fail in the `ParserT`
207monad.
209For this validation step,
210[pattern matching](https://github.com/purescript/documentation/blob/master/language/Pattern-Matching.md)
211on the `Rectangle` message type works well, so we could validate this way:
213```purescript
214 case rectangle of
215 Rectangle { width: Just width, height: Just height } ->
216 pure {width, height}
217 _ -> fail "Missing required width or height"
218```
220Or we might want to use `liftMaybe` for more fine-grained validation:
222```purescript
223 width <- liftMaybe "Missing required width" (unwrap rectangle).width
224 height <- liftMaybe "Missing required height" (unwrap rectangle).height
225 pure {width, height}
226```
228And now the `result` is either a parsing error or a fully validated rectangle.
230### Dependencies
232The generated code modules will import modules from this
233package.
235The generated code depends on packages which are all in the
236[PureScript Registry](https://github.com/purescript/registry).
238If the runtime environment is *Node.js*, then it must be at least *v11*,
239because that is the version in which
240[`TextDecoder`](https://nodejs.org/docs/latest-v11.x/api/globals.html#globals_textdecoder)
241 and
242[`TextEncoder`](https://nodejs.org/docs/latest-v11.x/api/globals.html#globals_textencoder)
243were added to `Globals`.
246### Generated message instances
248All of the generated message types have instances of
249[`Eq`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Eq#t:Eq),
250[`Show`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Show#t:Show),
251[`Generic`](https://pursuit.purescript.org/packages/purescript-generics-rep/docs/Data.Generic.Rep#t:Generic),
252[`NewType`](https://pursuit.purescript.org/packages/purescript-newtype/docs/Data.Newtype#t:Newtype).
254### Usage Examples
256The __protobuf__ repository contains three executable *Node.js*
257programs which use code generated by __protobuf__. Refer to these
258for further examples of how to use the generated code.
2601. The `protoc`
261 [compiler plugin](https://github.com/xc-jp/purescript-protobuf/blob/master/plugin/src/ProtocPlugin/Main.purs).
262 The code generator imports generated code. This program writes itself.
2632. The
264 [unit test suite](https://github.com/xc-jp/purescript-protobuf/blob/master/library/test/Main.purs)
2653. The Google
266 [conformance test program](https://github.com/xc-jp/purescript-protobuf/blob/master/conformance/src/Main.purs)
268The [Protobuf Decoder Explainer](http://jamesdbrock.github.io/protobuf-decoder-explainer/) shows an
269example of how to use this library to parse binary Protobuf when we don’t
270have access to the `.proto` descriptor schema file and can’t generate
271message-reading code.
273### Presence Discipline
275This is how [*field presence*](https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md) works
276in our implementation.
278#### When deserializing
280A message field will always be `Just` when the field is present on the wire.
282A message field will always be `Nothing` when the field is not present on the wire, even if
283it’s a *no presence* field.
284If we want to interpret a missing *no presence* field as a
285[default value](https://protobuf.dev/programming-guides/proto3/#default) then
286we have the
287[`Protobuf.Library.toDefault`](https://pursuit.purescript.org/packages/purescript-protobuf/docs/Protobuf.Library#v:toDefault)
288 function for that.
290#### When serializing
292A *no presence* field will not be serialized on the wire when it is `Nothing`, or `Just` the
293default value.
295An *explicit presence* (`optional`) field will not be serialized on the wire when it is `Nothing`.
296It will be serialized when it is `Just` the default value.
298### Interpreting invalid encoding parse failures
300When the parser encounters an invalid encoding in the Protobuf input
301stream then it will fail to parse.
303When
304[`runParserT`](https://pursuit.purescript.org/packages/purescript-parsing/docs/Parsing#v:runParserT)
305fails it will return a `ParseError String (Position {index::Int,line::Int,column::Int})`.
307The byte offset at which the parse failure occurred is given by the
308`index`.
310The path to the Protobuf definition which failed to parse will be included
311in the `ParseError String` and delimited by `'/'`, something
312like `"Message1 / string_field_1 / Invalid UTF8 encoding."`.
314### Protobuf Imports
316The Protobuf
317[`import`](https://protobuf.dev/programming-guides/proto3/#importing-definitions)
318statement allows Protobuf messages to have fields
319consisting of Protobuf messages imported from another file, and qualified
320by the package name in that file. In order to generate
321the correct PureScript module name qualifier on the types of imported message
322fields, the code generator must be able to lookup the package name
323statement in the imported file.
325For that reason, we can only use top-level
326(not [nested](https://protobuf.dev/programming-guides/proto3/#nested))
327`message` and `enum` types from a Protobuf `import`.
329### PureScript Imports
331The generated PureScript code will usually have module imports which cause
332the `purs` compiler to emit `ImplicitQualifiedImport` warnings. Sorry. If this causes
333trouble then the imports can be fixed automatically in a precompiling pass
334with the command-line tool
335[__suggest__](https://github.com/nwolverson/purescript-suggest).
336Or you can [censor the warnings](https://discourse.purescript.org/t/suppressing-warnings-in-code/2485).
338## Nix derivation
340The `flake.nix` provides a package `protoc-gen-purescript` so that we
341can run the `.proto` → `.purs` generation step as part of a Nix
342derivation. Include `protoc-gen-purescript` and `protobuf` as `nativeBuildInputs`.
343Then `protoc --purescript_out=path_to_output file.proto` will be runnable
344in our derivation phases.
346Building `protoc-gen-purescript` requires an impure Nix build environment.
348You might need in your `flake.nix`:
350```
351nixConfig.sandbox = "relaxed";
352```
354You might need in your `~/.config/nix.conf`:
356```
357experimental-features = nix-command flakes configurable-impure-env auto-allocate-uids
358```
360The `flake.nix` provides the Google Protocol Buffers conformance tests
361as an `app`. To run the conformance tests right now
362[without installing or cloning](https://determinate.systems/posts/nix-run)
363anything,
365```shell
366nix run github:xc-jp/purescript-protobuf#conformance
367```
369## Contributing
371Pull requests welcome.
373## Other References
375* [__justifill__](https://pursuit.purescript.org/packages/purescript-justifill) package may be useful for message construction.
376* [__morello__](https://pursuit.purescript.org/packages/purescript-morello) package may be useful for message validation.
377* [Third-Party Add-ons for Protocol Buffers](https://github.com/protocolbuffers/protobuf/blob/main/docs/third_party.md) Google’s list of Protocol Buffers language implementations.
378* [A vision for data interchange in Elm](https://gist.github.com/evancz/1c5f2cf34939336ecb79b97bb89d9da6) Comparison of JSON, ProtoBuf, GraphQL by Evan Czaplicki.