git.iko.soy/rowtype-yoga/purescript-protobuf
on master help sign in
rowtype-yoga/purescript-protobuf/library/README.md
383 lines 15 kB history raw
1# purescript-protobuf
3[![Pursuit](http://pursuit.purescript.org/packages/purescript-protobuf/badge)](http://pursuit.purescript.org/packages/purescript-protobuf/)
4[![Maintainer: jamesdbrock](https://img.shields.io/badge/maintainer-jamesdbrock-teal.svg)](https://github.com/jamesdbrock)
6PureScript library and code generator for
7[Google Protocol Buffers version 3](https://protobuf.dev/programming-guides/proto3/).
9This library operates on
10[`ArrayBuffer`](https://pursuit.purescript.org/packages/purescript-arraybuffer-types/docs/Data.ArrayBuffer.Types#t:ArrayBuffer), so it will run both
11[in *Node.js*](https://pursuit.purescript.org/packages/purescript-node-buffer/docs/Node.Buffer.Class)
12and in browser environments.
14## Features
16We aim to support binary-encoded protobuf for
17`syntax = "proto3";` descriptor files.
19Many `syntax = "proto2";` descriptor files will
20also work, as long as they don't use `"proto2"` features, especially
21[groups](https://protobuf.dev/programming-guides/proto2/#groups),
22which we do not support. We also do not support `"proto2"`
23[extensions](https://protobuf.dev/programming-guides/proto2/#extensions).
25We do not support
26[services](https://protobuf.dev/programming-guides/proto3/#services).
28We do not support
29[JSON Mapping](https://protobuf.dev/programming-guides/proto3/#json).
31### Conformance and Testing
33In this version, we pass all 684 of the
34[Google conformance tests](https://github.com/protocolbuffers/protobuf/tree/main/conformance)
35of binary-wire-format *proto3* for [Protocol Buffers v28.2](https://github.com/protocolbuffers/protobuf/releases/tag/v28.2).
36See the `conformance/README.md` in this repository for details.
38We also have our own unit tests, see `test/README.md` in this repository.
40## Code Generation
42The `nix develop` environment provides
44* The PureScript toolchain: `purs`, `spago`, and `node`.
45* The [`protoc`](https://protobuf.dev/programming-guides/proto3/#generating) compiler.
46* The `protoc-gen-purescript` executable plugin for `protoc` on the `PATH` so that
47 [`protoc` can find it](https://protobuf.dev/reference/cpp/api-docs/google.protobuf.compiler.plugin/).
49```
50$ nix develop
52PureScript Protobuf development environment
53libprotoc 28.2
54node v20.15.1
55purs 0.15.15
56spago 0.93.40
58To build the protoc compiler plugin, run:
60 cd plugin
61 spago build
63To compile PureScript .purs files from .proto files, run for example:
65 protoc --purescript_out=. google/protobuf/timestamp.proto
66```
68We can test out code generation immediately by
69generating `.purs` files for any of Google’s built-in “well-known types” in the
70[`google.protobuf`](https://protobuf.dev/reference/protobuf/google.protobuf/) package namespace. Try the command
71```shell
72protoc --purescript_out=. google/protobuf/any.proto
73```
74or
75```shell
76protoc --purescript_out=. google/protobuf/timestamp.proto
77```
79To see
80[all of the `.proto` definitions](https://github.com/protocolbuffers/protobuf/tree/main/src/google/protobuf)
81included with the Nix PureScript Protobuf installation including
82the “well-known types,”
83```shell
84ls $(nix path-info .#protobuf_local)/src/google/protobuf/*.proto
85```
87If you don't want to use Nix,
881. install the PureScript toolchain and `protoc`.
892. Build the [PureScript plugin for `protoc`](plugin/).
903. Run `protoc` with the path to the PureScript plugin executable, like for example
91 ```shell
92 protoc --plugin=bin/protoc-gen-purescript --purescript_out=. google/protobuf/timestamp.proto
93 ```
95## Writing programs with the generated code
97Suppose we have a `Rectangle` message declared in a `shapes.proto` descriptor file:
99```
100syntax = "proto3";
101package interproc;
103message Rectangle {
104 double width = 1;
105 double height = 2;
107```
109We run
111```shell
112protoc --purescript_out=. shapes.proto
113```
115which will generate a file `shapes.Interproc.purs`.
117The code generator will use the `package` import statement in the `.proto` file
118and the base `.proto` file name as the PureScript module name for that file.
120The generated `shapes.Interproc.purs` file
121will export these four names from module `Interproc.Shapes`.
1231. A message data type.
125 ```purescript
126 newtype Rectangle = Rectangle { width :: Maybe Number, height :: Maybe Number }
127 ```
129 The message data type will also include an `__unknown_fields` array field for
130 holding received fields which were not in the descriptor `.proto` file. We can
131 ignore `__unknown_fields` if we want to.
1332. A message maker which constructs a message from a `Record`
134 with some message fields.
136 ```purescript
137 mkRectangle :: forall r. Record r -> Rectangle
138 ```
140 All message fields are optional, and can be omitted when making a message.
141 There are some extra type constraints, not shown here, which will cause a
142 compiler error if we try to add a field which is not in the message data type.
144 If we want the compiler to check that we’ve explicitly supplied all the fields,
145 then we can use the ordinary message data type constructor `Rectangle`.
1473. A message serializer which works with
148 [__arraybuffer-builder__](http://pursuit.purescript.org/packages/purescript-arraybuffer-builder/).
150 ```purescript
151 putRectangle :: forall m. MonadEffect m => Rectangle -> PutM m Unit
152 ```
1544. A message deserializer which works with
155 [__parsing-dataview__](http://pursuit.purescript.org/packages/purescript-parsing-dataview/).
157 ```purescript
158 parseRectangle :: forall m. MonadEffect m => ByteLength -> ParserT DataView m Rectangle
159 ```
161 The message parser needs an argument which tells it the
162 length of the message which it’s about to parse, because
163 [“the Protocol Buffer wire format is not self-delimiting.”](https://protobuf.dev/programming-guides/techniques/#streaming)
165In our program, our imports will look something like this.
166The only module from this package which we will import into our program
167will be the `Protobuf.Library` module.
168We'll import the message modules from the generated `.purs` files.
169We'll also import modules for reading and writing `ArrayBuffer`s.
172```purescript
173import Protobuf.Library (Bytes(..))
174import Interproc.Shapes (Rectangle, mkRectangle, putRectangle, parseRectangle)
175import Parsing (runParserT, ParseError, liftMaybe)
176import Data.ArrayBuffer.Builder (execPutM)
177import Data.ArrayBuffer.DataView (whole)
178import Data.ArrayBuffer.ArrayBuffer (byteLength)
179import Data.Tuple (Tuple)
180import Data.Newtype (unwrap)
181```
183This is how we serialize a `Rectangle` to an `ArrayBuffer`.
184We must be in a `MonadEffect`.
186```purescript
187do
188 arraybuffer <- execPutM $ putRectangle $ mkRectangle
189 { width: Just 3.0
190 , height: Just 4.0
191 }
192```
194Next we’ll deserialize `Rectangle` from the `ArrayBuffer` that we just made.
196```purescript
197 result :: Either ParseError {width :: Number, height :: Number}
198 <- runParserT (whole arraybuffer) do
199 rectangle :: Rectangle <- parseRectangle (byteLength arraybuffer)
200```
202At this point we’ve consumed all of the parser input and constructed our
203`Rectangle` message, but we’re not finished parsing.
204We want to “validate” the `Rectangle` message to make sure it has all of the
205fields that we require, because in
206[*proto3*, all fields are optional](https://github.com/protocolbuffers/protobuf/issues/2497).
208Fortunately we are already in the `ParserT` monad,
209so we can do better than to “validate”:
210[Parse, don't validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/).
212We will construct a record `{width::Number, height::Number}`
213with the width and height of the `Rectangle`. If the width or height
214are missing from the `Rectangle` message, then we will fail in the `ParserT`
215monad.
217For this validation step,
218[pattern matching](https://github.com/purescript/documentation/blob/master/language/Pattern-Matching.md)
219on the `Rectangle` message type works well, so we could validate this way:
221```purescript
222 case rectangle of
223 Rectangle { width: Just width, height: Just height } ->
224 pure {width, height}
225 _ -> fail "Missing required width or height"
226```
228Or we might want to use `liftMaybe` for more fine-grained validation:
230```purescript
231 width <- liftMaybe "Missing required width" (unwrap rectangle).width
232 height <- liftMaybe "Missing required height" (unwrap rectangle).height
233 pure {width, height}
234```
236And now the `result` is either a parsing error or a fully validated rectangle.
238### Dependencies
240The generated code modules will import modules from this
241package.
243The generated code depends on packages which are all in the
244[PureScript Registry](https://github.com/purescript/registry).
246If the runtime environment is *Node.js*, then it must be at least *v11*,
247because that is the version in which
248[`TextDecoder`](https://nodejs.org/docs/latest-v11.x/api/globals.html#globals_textdecoder)
249 and
250[`TextEncoder`](https://nodejs.org/docs/latest-v11.x/api/globals.html#globals_textencoder)
251were added to `Globals`.
254### Generated message instances
256All of the generated message types have instances of
257[`Eq`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Eq#t:Eq),
258[`Show`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Show#t:Show),
259[`Generic`](https://pursuit.purescript.org/packages/purescript-generics-rep/docs/Data.Generic.Rep#t:Generic),
260[`NewType`](https://pursuit.purescript.org/packages/purescript-newtype/docs/Data.Newtype#t:Newtype).
262### Usage Examples
264The __protobuf__ repository contains three executable *Node.js*
265programs which use code generated by __protobuf__. Refer to these
266for further examples of how to use the generated code.
2681. The `protoc`
269 [compiler plugin](https://github.com/rowtype-yoga/purescript-protobuf/blob/master/plugin/src/Main.purs).
270 The code generator imports generated code. This program writes itself.
2712. The
272 [unit test suite](https://github.com/rowtype-yoga/purescript-protobuf/blob/master/plugin/test/Test/Main.purs).
2733. The Google
274 [conformance test program](https://github.com/xc-jp/purescript-protobuf/blob/master/conformance/src/Main.purs).
276The [Protobuf Decoder Explainer](http://jamesdbrock.github.io/protobuf-decoder-explainer/) shows an
277example of how to use this library to parse binary Protobuf when we don’t
278have access to the `.proto` descriptor schema file and can’t generate
279message-reading code.
281### Presence Discipline
283This is how [*field presence*](https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md) works
284in our implementation.
286#### When deserializing
288A message field will always be `Just` when the field is present on the wire.
290A message field will always be `Nothing` when the field is not present on the wire, even if
291it’s a *no presence* field.
292If we want to interpret a missing *no presence* field as a
293[default value](https://protobuf.dev/programming-guides/proto3/#default) then
294we have the
295[`Protobuf.Library.toDefault`](https://pursuit.purescript.org/packages/purescript-protobuf/docs/Protobuf.Library#v:toDefault)
296 function for that.
298#### When serializing
300A *no presence* field will not be serialized on the wire when it is `Nothing`, or `Just` the
301default value.
303An *explicit presence* (`optional`) field will not be serialized on the wire when it is `Nothing`.
304It will be serialized when it is `Just` the default value.
306### Interpreting invalid encoding parse failures
308When the parser encounters an invalid encoding in the Protobuf input
309stream then it will fail to parse.
311When
312[`runParserT`](https://pursuit.purescript.org/packages/purescript-parsing/docs/Parsing#v:runParserT)
313fails it will return a `ParseError String (Position {index::Int,line::Int,column::Int})`.
315The byte offset at which the parse failure occurred is given by the
316`index`.
318The path to the Protobuf definition which failed to parse will be included
319in the `ParseError String` and delimited by `'/'`, something
320like `"Message1 / string_field_1 / Invalid UTF8 encoding."`.
322### Protobuf Imports
324The Protobuf
325[`import`](https://protobuf.dev/programming-guides/proto3/#importing-definitions)
326statement allows Protobuf messages to have fields
327consisting of Protobuf messages imported from another file, and qualified
328by the package name in that file. In order to generate
329the correct PureScript module name qualifier on the types of imported message
330fields, the code generator must be able to lookup the package name
331statement in the imported file.
333For that reason, we can only use top-level
334(not [nested](https://protobuf.dev/programming-guides/proto3/#nested))
335`message` and `enum` types from a Protobuf `import`.
337### PureScript Imports
339The generated PureScript code will usually have module imports which cause
340the `purs` compiler to emit `ImplicitQualifiedImport` warnings. Sorry. If this causes
341trouble then the imports can be fixed automatically in a precompiling pass
342with the command-line tool
343[__suggest__](https://github.com/nwolverson/purescript-suggest).
344Or you can [censor the warnings](https://discourse.purescript.org/t/suppressing-warnings-in-code/2485).
346## Nix derivation
348The `flake.nix` provides a package `protoc-gen-purescript` so that we
349can run the `.proto` → `.purs` generation step as part of a Nix
350derivation. Include `protoc-gen-purescript` and `protobuf` as `nativeBuildInputs`.
351Then `protoc --purescript_out=path_to_output file.proto` will be runnable
352in our derivation phases.
354The `flake.nix` provides the Google Protocol Buffers conformance tests
355as an `app`. To run the conformance tests right now
356[without installing or cloning](https://determinate.systems/posts/nix-run)
357anything,
359```shell
360nix run github:xc-jp/purescript-protobuf#conformance
361```
363## Contributing
365Pull requests welcome.
367This repo is organized as a
368[Spago polyrepo](https://github.com/purescript/spago#polyrepo-support).
370- `purescript-protobuf`
371 - [`library`](library) — `protobuf` package, published to Pursuit
372 - [`spago.yaml`](library/spago.yaml)
373 - [`plugin`](plugin) — protoc plugin app for code generation
374 - [`spago.yaml`](plugin/spago.yaml)
375 - [`conformance`](conformance) — Google Protobuf Conformance test runner app
376 - [`spago.yaml`](conformance/spago.yaml)
378## Other References
380* [__justifill__](https://pursuit.purescript.org/packages/purescript-justifill) package may be useful for message construction.
381* [__morello__](https://pursuit.purescript.org/packages/purescript-morello) package may be useful for message validation.
382* [Third-Party Add-ons for Protocol Buffers](https://github.com/protocolbuffers/protobuf/blob/main/docs/third_party.md) Google’s list of Protocol Buffers language implementations.
383* [A vision for data interchange in Elm](https://gist.github.com/evancz/1c5f2cf34939336ecb79b97bb89d9da6) Comparison of JSON, ProtoBuf, GraphQL by Evan Czaplicki.