git.iko.soy/rowtype-yoga/purescript-protobuf
on minibench help sign in
rowtype-yoga/purescript-protobuf/README.md
337 lines 13 kB history raw
1# purescript-protobuf 💝
3[![Test](https://github.com/xc-jp/purescript-protobuf/workflows/Test/badge.svg?branch=master)](https://github.com/xc-jp/purescript-protobuf/actions)
4[![Pursuit](http://pursuit.purescript.org/packages/purescript-protobuf/badge)](http://pursuit.purescript.org/packages/purescript-protobuf/)
6Purescript library and code generator for
7[Google Protocol Buffers version 3](https://developers.google.com/protocol-buffers/docs/proto3).
9This library operates on
10[`ArrayBuffer`](https://pursuit.purescript.org/packages/purescript-arraybuffer-types/docs/Data.ArrayBuffer.Types#t:ArrayBuffer), so it will run both
11[in *Node.js*](https://pursuit.purescript.org/packages/purescript-node-buffer/docs/Node.Buffer.Class)
12and in browser environments.
14## Features
16We aim to support binary-encoded (not JSON-encoded)
17`syntax = "proto3";` descriptor files.
19Many `syntax = "proto2";` descriptor files will
20also work, as long as they don't use `"proto2"` features, especially
21[groups](https://developers.google.com/protocol-buffers/docs/proto#groups),
22which we do not support.
24We do not support
25[extensions](https://developers.google.com/protocol-buffers/docs/proto?hl=en#extensions).
27We do not support
28[services](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#services).
30### Conformance and Testing
32In this version, we pass all 651 of the
33[Google conformance tests](https://github.com/protocolbuffers/protobuf/tree/master/conformance)
34of binary-wire-format *proto3* for [Protocol Buffers v3.15.8](https://github.com/protocolbuffers/protobuf/blob/master/CHANGES.txt).
35See the `conformance/README.md` in this repository for details.
37We also have our own unit tests, see `test/README.md` in this repository.
39## Code Generation
41The `shell.nix` environment provides
43* The PureScript toolchain: `purs`, `spago`, and `npm`.
44* The [`protoc`](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#generating) compiler
45* The `protoc-gen-purescript` executable plugin for `protoc` on the `PATH` so that
46 [`protoc` can find it](https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.compiler.plugin).
48```
49$ nix-shell
51Purescript Protobuf development environment.
52To build purescript-protobuf, run:
54 npm install
55 spago build
57To generate Purescript .purs files from .proto files, run:
59 protoc --purescript_out=path_to_output file.proto
61[nix-shell]$
62```
64If you don't want to use Nix, then install the PureScript toolchain and `protoc`,
65and add the executable script
66[`bin/protoc-gen-purescript`](bin/protoc-gen-purescript)
67to your `PATH`.
69## Writing programs with the generated code
71The code generator will use the `package` import statement in the `.proto` file
72and the base `.proto` file name as the PureScript module name for that file.
74A message in a `shapes.proto` descriptor file declared as
76```
77package interproc;
79message Rectangle {
80 double width = 1;
81 double height = 2;
83```
85will export these four names from module `Interproc.Shapes` in a
86generated `shapes.Interproc.purs` file.
881. A message data type
90 ```purescript
91 newtype Rectangle = Rectangle { width :: Maybe Number, height :: Maybe Number }
92 ```
94 The message data type will also include an `__unknown_fields` array field for
95 holding received fields which were not in the descriptor `.proto` file. We can
96 completely ignore `__unknown_fields` if we want to.
982. A message maker which constructs a message from a `Record`
99 with some message fields
101 ```purescript
102 mkRectangle :: forall r. Record r -> Rectangle
103 ```
105 All message fields are optional, and can be elided when making a message.
106 There are some extra type constraints, not shown here, which will cause a
107 compiler error if we try to add a field which is not in the message data type.
109 If we want the compiler to check that we've explicitly supplied all the fields,
110 then we can use the ordinary message data type constructor `Rectangle`.
1123. A message encoder which works with
113 [__purescript-arraybuffer-builder__](http://pursuit.purescript.org/packages/purescript-arraybuffer-builder/)
115 ```purescript
116 putRectangle :: forall m. MonadEffect m => Rectangle -> PutM m Unit
117 ```
1194. A message decoder which works with
120 [__purescript-parsing-dataview__](http://pursuit.purescript.org/packages/purescript-parsing-dataview/)
122 ```purescript
123 parseRectangle :: forall m. MonadEffect m => Int -> ParserT DataView m Rectangle
124 ```
126 The message decoder needs an argument which tells it the
127 length of the message which it’s about to decode, because
128 [“the Protocol Buffer wire format is not self-delimiting.”](https://developers.google.com/protocol-buffers/docs/techniques#streaming)
130In our program, our imports will look something like this.
131The only module from this package which we will import into our program
132will be the `Protobuf.Library` module.
133We'll import the message modules from the generated `.purs` files.
134We'll also import modules for reading and writing `ArrayBuffer`s.
137```purescript
138import Protobuf.Library (Bytes(..), parseMaybe)
139import Interproc.Shapes (Rectangle, mkRectangle, putRectangle, parseRectangle)
140import Text.Parsing.Parser (runParserT, ParseError)
141import Data.ArrayBuffer.Builder (execPutM)
142import Data.ArrayBuffer.DataView (whole)
143import Data.ArrayBuffer.ArrayBuffer (byteLength)
144import Data.Tuple (Tuple)
145import Data.Newtype (unwrap)
146```
148This is how we serialize a `Rectangle` to an `ArrayBuffer`.
149We must be in a `MonadEffect`.
151```purescript
152do
153 arraybuffer <- execPutM $ putRectangle $ mkRectangle
154 { width: Just 3.0
155 , height: Just 4.0
156 }
157```
159Next we'll deserialize `Rectangle` from the `ArrayBuffer` that we just made.
161```purescript
162 result :: Either ParseError (Tuple Number Number)
163 <- runParserT (whole arraybuffer) $ do
165 rectangle :: Rectangle <- parseRectangle (byteLength arraybuffer)
166```
168At this point we've consumed all of the parser input and constructed our
169`Rectangle` message, but we're not finished parsing.
170We want to “validate” the `Rectangle` message to make sure it has all of the
171fields that we require, because in
172[*proto3*, all fields are optional](https://github.com/protocolbuffers/protobuf/issues/2497).
174Fortunately we are already in the `ParserT` monad,
175so we can do better than to “validate”:
176[Parse, don't validate](https://lexi-lambda.github.io/blog/2019/11/05/parse-don-t-validate/).
178We will construct a `Tuple Number Number`
179with the width and height of the `Rectangle`. If the width or height
180are missing from the `Rectangle` message, then we will fail in the `ParserT`
181monad.
183For this validation step,
184[pattern matching](https://github.com/purescript/documentation/blob/master/language/Pattern-Matching.md)
185on the `Rectangle` message type works well, so we could validate this way:
187```purescript
188 case rectangle of
189 Rectangle { width: Just width, height: Just height } ->
190 pure $ Tuple width height
191 _ -> fail "Missing required width or height"
192```
194Or we might want to use `parseMaybe`, one of the
195convenience parsing functions exported by `Protobuf.Library`,
196for more fine-grained validation:
198```purescript
199 width <- parseMaybe "Missing required width" (unwrap rectangle).width
200 height <- parseMaybe "Missing required height" (unwrap rectangle).height
201 pure $ Tuple width height
202```
204And now the `result` is either a parsing error or a fully validated rectangle.
206### Dependencies
208The generated code modules will import modules from this
209package.
211The generated code depends on packages
213```
214 , "protobuf"
215 , "arraybuffer"
216 , "arraybuffer-types"
217 , "arraybuffer-builder"
218 , "parsing"
219 , "parsing-dataview"
220 , "uint"
221 , "longs"
222 , "text-encoding"
223```
225which are all in
226[__package-sets__](https://github.com/purescript/package-sets).
228It also depends on the Javascript package
229[__long__](https://www.npmjs.com/package/long).
231For depending on generated code without pulling in any of the code-generation
232dependencies, we can use `spago-library.dhall`.
233### Generated message instances
235All of the generated message types have instances of
236[`Eq`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Eq#t:Eq),
237[`Show`](https://pursuit.purescript.org/packages/purescript-prelude/docs/Data.Show#t:Show),
238[`Generic`](https://pursuit.purescript.org/packages/purescript-generics-rep/docs/Data.Generic.Rep#t:Generic),
239[`NewType`](https://pursuit.purescript.org/packages/purescript-newtype/docs/Data.Newtype#t:Newtype).
241### Usage Examples
243The __purescript-protobuf__ repository contains three executable *Node.js*
244programs which use code generated by __purescript-protobuf__. Refer to these
245for further examples of how to use the generated code.
2471. The `protoc`
248 [compiler plugin](https://github.com/xc-jp/purescript-protobuf/blob/master/src/ProtocPlugin/Main.purs).
249 The code generator imports generated code. Trippy, right? This program
250 literally writes itself.
2512. The
252 [unit test suite](https://github.com/xc-jp/purescript-protobuf/blob/master/test/Main.purs)
2533. The Google
254 [conformance test program](https://github.com/xc-jp/purescript-protobuf/blob/master/conformance/Main.purs)
256The [Protobuf Decoder Explainer](http://jamesdbrock.github.io/protobuf-decoder-explainer/) shows an
257example of how to use this library to parse binary protobuf when we don’t
258have access to the `.proto` descriptor schema file and can’t generate
259message-reading code.
261### Presence Discipline
263This is how [*field presence*](https://github.com/protocolbuffers/protobuf/blob/master/docs/field_presence.md) works
264in our implementation.
266#### When deserializing
268The generated record fields will always be `Nothing` when a field is not present on the wire.
269If we want to use
270[default values](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#default) then
271we have the `Protobuf.Library.Default` typeclass for that.
273#### When serializing
275The *no presence* fields will not be serialized when they are `Nothing` or `Just` their
276default value.
278The *explicit presence* (`optional`) fields will not be serialized when they are `Nothing`.
279They will be serialized when they are `Just` their default value.
281### Interpreting invalid encoding parse failures
283When the decode parser encounters an invalid encoding in the protobuf input
284stream then it will fail to parse.
286When
287[`Text.Parsing.Parser.ParserT`](https://pursuit.purescript.org/packages/purescript-parsing/docs/Text.Parsing.Parser#t:ParserT)
288fails it will return a `ParseError String (Position {line::Int,column::Int})`.
290The byte offset at which the parse failure occured is given by the
291formula `column - 1`.
293The path to the protobuf definition which failed to parse will be included
294in the `ParseError String` and delimited by `'/'`, something
295like `"Message1 / string_field_1 / Invalid UTF8 encoding."`.
297### Protobuf Imports
299The Protobuf
300[`import`](https://developers.google.com/protocol-buffers/docs/proto3#importing_definitions)
301statement allows Protobuf messages to have fields
302consisting of Protobuf messages imported from another file, and qualified
303by the package name in that file. In order to generate
304the correct Purescript module name qualifier on the types of imported message
305fields, the code generator must be able to lookup the package name
306statement in the imported file.
308For that reason, we can only use top-level
309(not [nested](https://developers.google.com/protocol-buffers/docs/proto3#nested))
310`message` and `enum` types from a Protobuf `import`.
312### PureScript Imports
314The generated Purescript code will usually have module imports which cause
315the `purs` compiler to emit redundant import warnings. Sorry. If this causes
316trouble then the imports can be fixed automatically in a precompiling pass
317with the command-line tool
318[__purescript-suggest__](https://github.com/nwolverson/purescript-suggest).
320## Nix derivation
322If we want to run the `.proto` → `.purs` generation step as part of a pure Nix
323derivation, then `import` the top-level `default.nix` from this repository
324as a `nativeBuildInput`.
326Then `protoc --purescript_out=path_to_output file.proto` will be runnable
327in our derivation phases.
329See the `nix/demo.nix` file for an example.
331## Contributing
333Pull requests welcome.
335## Other References
337* [Third-Party Add-ons for Protocol Buffers](https://github.com/protocolbuffers/protobuf/blob/master/docs/third_party.md)