git.iko.soy/rowtype-yoga/purescript-protobuf
on v4.1.0 help sign in
rowtype-yoga/purescript-protobuf/README.md
355 lines 14 kB history raw
1# purescript-protobuf 💝
3[![Pursuit](http://pursuit.purescript.org/packages/purescript-protobuf/badge)](http://pursuit.purescript.org/packages/purescript-protobuf/)
5PureScript library and code generator for
6[Google Protocol Buffers version 3](https://developers.google.com/protocol-buffers/docs/proto3).
8This library operates on
9[`ArrayBuffer`](https://pursuit.purescript.org/packages/purescript-arraybuffer-types/docs/Data.ArrayBuffer.Types#t:ArrayBuffer), so it will run both
10[in *Node.js*](https://pursuit.purescript.org/packages/purescript-node-buffer/docs/Node.Buffer.Class)
11and in browser environments.
13## Features
15We aim to support binary-encoded protobuf for
16`syntax = "proto3";` descriptor files.
18Many `syntax = "proto2";` descriptor files will
19also work, as long as they don't use `"proto2"` features, especially
20[groups](https://developers.google.com/protocol-buffers/docs/proto#groups),
21which we do not support. We also do not support `"proto2"`
22[extensions](https://developers.google.com/protocol-buffers/docs/proto?hl=en#extensions).
24We do not support
25[services](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#services).
27We do not support
28[JSON encoding](https://developers.google.com/protocol-buffers/docs/proto3#json).
30### Conformance and Testing
32In this version, we pass all 651 of the
33[Google conformance tests](https://github.com/protocolbuffers/protobuf/tree/master/conformance)
34of binary-wire-format *proto3* for [Protocol Buffers v21.10](https://github.com/protocolbuffers/protobuf/blob/master/CHANGES.txt).
35See the `conformance/README.md` in this repository for details.
37We also have our own unit tests, see `test/README.md` in this repository.
39## Code Generation
41The `nix develop` environment provides
43* The PureScript toolchain: `purs`, `spago`, and `node`.
44* The [`protoc`](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#generating) compiler.
45* The `protoc-gen-purescript` executable plugin for `protoc` on the `PATH` so that
46 [`protoc` can find it](https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.compiler.plugin).
48```
49$ nix develop
51PureScript Protobuf development environment.
52libprotoc 3.21.10
53purs 0.15.6
54node v17.1.0
56To build the protoc compiler plugin, run:
58 spago -x spago-plugin.dhall build
60To compile PureScript .purs files from .proto files, run for example:
62 protoc --purescript_out=. google/protobuf/timestamp.proto
63```
65We can test out code generation immediately by
66generating `.purs` files for any of Google’s built-in “well-known types” in the
67[`google.protobuf`](https://developers.google.com/protocol-buffers/docs/reference/google.protobuf) package namespace. Try the command `protoc --purescript_out=. google/protobuf/any.proto`
68or `protoc --purescript_out=. google/protobuf/timestamp.proto`.
70To see
71[all of the `.proto` definitions](https://github.com/protocolbuffers/protobuf/tree/main/src/google/protobuf)
72included with the Nix PureScript Protobuf installation including
73the “well-known types,”
74`ls -l $(nix path-info .#protobuf)/include/google/protobuf/*.proto`
76If you don't want to use Nix, then install the PureScript toolchain and `protoc`,
77and add the executable script
78[`bin/protoc-gen-purescript`](bin/protoc-gen-purescript)
79to your `PATH`.
81## Writing programs with the generated code
83The code generator will use the `package` import statement in the `.proto` file
84and the base `.proto` file name as the PureScript module name for that file.
86A message in a `shapes.proto` descriptor file declared as
88```
89syntax = "proto3";
90package interproc;
92message Rectangle {
93 double width = 1;
94 double height = 2;
96```
98will export these four names from module `Interproc.Shapes` in a
99generated `shapes.Interproc.purs` file.
1011. A message data type.
103 ```purescript
104 newtype Rectangle = Rectangle { width :: Maybe Number, height :: Maybe Number }
105 ```
107 The message data type will also include an `__unknown_fields` array field for
108 holding received fields which were not in the descriptor `.proto` file. We can
109 ignore `__unknown_fields` if we want to.
1112. A message maker which constructs a message from a `Record`
112 with some message fields.
114 ```purescript
115 mkRectangle :: forall r. Record r -> Rectangle
116 ```
118 All message fields are optional, and can be omitted when making a message.
119 There are some extra type constraints, not shown here, which will cause a
120 compiler error if we try to add a field which is not in the message data type.
122 If we want the compiler to check that we've explicitly supplied all the fields,
123 then we can use the ordinary message data type constructor `Rectangle`.
1253. A message serializer which works with
126 [__arraybuffer-builder__](http://pursuit.purescript.org/packages/purescript-arraybuffer-builder/).
128 ```purescript
129 putRectangle :: forall m. MonadEffect m => Rectangle -> PutM m Unit
130 ```
1324. A message deserializer which works with
133 [__parsing-dataview__](http://pursuit.purescript.org/packages/purescript-parsing-dataview/).
135 ```purescript
136 parseRectangle :: forall m. MonadEffect m => ByteLength -> ParserT DataView m Rectangle
137 ```
139 The message parser needs an argument which tells it the
140 length of the message which it’s about to parse, because
141 [“the Protocol Buffer wire format is not self-delimiting.”](https://developers.google.com/protocol-buffers/docs/techniques#streaming)
143In our program, our imports will look something like this.
144The only module from this package which we will import into our program
145will be the `Protobuf.Library` module.
146We'll import the message modules from the generated `.purs` files.
147We'll also import modules for reading and writing `ArrayBuffer`s.
150```purescript
151import Protobuf.Library (Bytes(..))
152import Interproc.Shapes (Rectangle, mkRectangle, putRectangle, parseRectangle)
153import Text.Parsing.Parser (runParserT, ParseError, liftMaybe)
154import Data.ArrayBuffer.Builder (execPutM)
155import Data.ArrayBuffer.DataView (whole)
156import Data.ArrayBuffer.ArrayBuffer (byteLength)
157import Data.Tuple (Tuple)
158import Data.Newtype (unwrap)
159```
161This is how we serialize a `Rectangle` to an `ArrayBuffer`.
162We must be in a `MonadEffect`.
164```purescript
165do
166 arraybuffer <- execPutM $ putRectangle $ mkRectangle
167 { width: Just 3.0
168 , height: Just 4.0
169 }
170```
172Next we'll deserialize `Rectangle` from the `ArrayBuffer` that we just made.
174```purescript
175 result :: Either ParseError (Tuple Number Number)
176 <- runParserT (whole arraybuffer) $ do
178 rectangle :: Rectangle <- parseRectangle (byteLength arraybuffer)
179```
181At this point we've consumed all of the parser input and constructed our
182`Rectangle` message, but we're not finished parsing.
183We want to “validate” the `Rectangle` message to make sure it has all of the
184fields that we require, because in
185[*proto3*, all fields are optional](https://github.com/protocolbuffers/protobuf/issues/2497).
187Fortunately we are already in the `ParserT` monad,
188so we can do better than to “validate”:
189[Parse, don't validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/).
191We will construct a `Tuple Number Number`
192with the width and height of the `Rectangle`. If the width or height
193are missing from the `Rectangle` message, then we will fail in the `ParserT`
194monad.
196For this validation step,
197[pattern matching](https://github.com/purescript/documentation/blob/master/language/Pattern-Matching.md)
198on the `Rectangle` message type works well, so we could validate this way:
200```purescript
201 case rectangle of
202 Rectangle { width: Just width, height: Just height } ->
203 pure $ Tuple width height
204 _ -> fail "Missing required width or height"
205```
207Or we might want to use `liftMaybe` for more fine-grained validation:
209```purescript
210 width <- liftMaybe "Missing required width" (unwrap rectangle).width
211 height <- liftMaybe "Missing required height" (unwrap rectangle).height
212 pure $ Tuple width height
213```
215And now the `result` is either a parsing error or a fully validated rectangle.
217### Dependencies
219The generated code modules will import modules from this
220package.
222The generated code depends on packages which are all in
223[__package-sets__](https://github.com/purescript/package-sets).
225If the runtime environment is *Node.js*, then it must be at least *v11*,
226because that is the version in which
227[`TextDecoder`](https://nodejs.org/docs/latest-v11.x/api/globals.html#globals_textdecoder)
228 and
229[`TextEncoder`](https://nodejs.org/docs/latest-v11.x/api/globals.html#globals_textencoder)
230were added to `Globals`.
233### Generated message instances
235All of the generated message types have instances of
236[`Eq`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Eq#t:Eq),
237[`Show`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Show#t:Show),
238[`Generic`](https://pursuit.purescript.org/packages/purescript-generics-rep/docs/Data.Generic.Rep#t:Generic),
239[`NewType`](https://pursuit.purescript.org/packages/purescript-newtype/docs/Data.Newtype#t:Newtype).
241### Usage Examples
243The __protobuf__ repository contains three executable *Node.js*
244programs which use code generated by __protobuf__. Refer to these
245for further examples of how to use the generated code.
2471. The `protoc`
248 [compiler plugin](https://github.com/xc-jp/purescript-protobuf/blob/master/plugin/ProtocPlugin/Main.purs).
249 The code generator imports generated code. Trippy, right? This program
250 literally writes itself.
2512. The
252 [unit test suite](https://github.com/xc-jp/purescript-protobuf/blob/master/test/Main.purs)
2533. The Google
254 [conformance test program](https://github.com/xc-jp/purescript-protobuf/blob/master/conformance/Main.purs)
256The [Protobuf Decoder Explainer](http://jamesdbrock.github.io/protobuf-decoder-explainer/) shows an
257example of how to use this library to parse binary Protobuf when we don’t
258have access to the `.proto` descriptor schema file and can’t generate
259message-reading code.
261### Presence Discipline
263This is how [*field presence*](https://github.com/protocolbuffers/protobuf/blob/main/docs/field_presence.md) works
264in our implementation.
266#### When deserializing
268A message field will always be `Just` when the field is present on the wire.
270A message field will always be `Nothing` when the field is not present on the wire, even if
271it’s a *no presence* field.
272If we want to interpret a missing *no presence* field as a
273[default value](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#default) then
274we have the
275[`Protobuf.Library.toDefault`](https://pursuit.purescript.org/packages/purescript-protobuf/docs/Protobuf.Library#v:toDefault)
276 function for that.
278#### When serializing
280A *no presence* field will not be serialized on the wire when it is `Nothing`, or `Just` the
281default value.
283An *explicit presence* (`optional`) field will not be serialized on the wire when it is `Nothing`.
284It will be serialized when it is `Just` the default value.
286### Interpreting invalid encoding parse failures
288When the parser encounters an invalid encoding in the Protobuf input
289stream then it will fail to parse.
291When
292[`ParserT`](https://pursuit.purescript.org/packages/purescript-parsing/docs/Parsing#t:ParserT)
293fails it will return a `ParseError String (Position {index::Int,line::Int,column::Int})`.
295The byte offset at which the parse failure occurred is given by the
296`index`.
298The path to the Protobuf definition which failed to parse will be included
299in the `ParseError String` and delimited by `'/'`, something
300like `"Message1 / string_field_1 / Invalid UTF8 encoding."`.
302### Protobuf Imports
304The Protobuf
305[`import`](https://developers.google.com/protocol-buffers/docs/proto3#importing_definitions)
306statement allows Protobuf messages to have fields
307consisting of Protobuf messages imported from another file, and qualified
308by the package name in that file. In order to generate
309the correct PureScript module name qualifier on the types of imported message
310fields, the code generator must be able to lookup the package name
311statement in the imported file.
313For that reason, we can only use top-level
314(not [nested](https://developers.google.com/protocol-buffers/docs/proto3#nested))
315`message` and `enum` types from a Protobuf `import`.
317### PureScript Imports
319The generated PureScript code will usually have module imports which cause
320the `purs` compiler to emit redundant import warnings. Sorry. If this causes
321trouble then the imports can be fixed automatically in a precompiling pass
322with the command-line tool
323[__suggest__](https://github.com/nwolverson/purescript-suggest).
325## Nix derivation
327The `flake.nix` provides a package `protoc-gen-purescript` so that we
328can run the `.proto` → `.purs` generation step as part of a Nix
329derivation. Include `protoc-gen-purescript` and `protobuf` as `nativeBuildInputs`.
330Then `protoc --purescript_out=path_to_output file.proto` will be runnable
331in our derivation phases.
333(`protoc-gen-purescript` requires an impure build for the `spago2nix` step,
334so you’ll have to grant
335trust at the prompt or configure a relaxed Nix sandbox.)
337The `flake.nix` provides the Google Protocol Buffers conformance tests
338as an `app`. To run the conformance tests right now
339[without installing or cloning](https://determinate.systems/posts/nix-run)
340anything,
342```shell
343nix run github:xc-jp/purescript-protobuf#conformance
344```
346## Contributing
348Pull requests welcome.
350## Other References
352* [__justifill__](https://pursuit.purescript.org/packages/purescript-justifill) package may be useful for message construction.
353* [__morello__](https://pursuit.purescript.org/packages/purescript-morello) package may be useful for message validation.
354* [Third-Party Add-ons for Protocol Buffers](https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md) Google’s list of Protocol Buffers language implementations.
355* [A vision for data interchange in Elm](https://gist.github.com/evancz/1c5f2cf34939336ecb79b97bb89d9da6) Comparison of JSON, ProtoBuf, GraphQL by Evan Czaplicki.