git.iko.soy/rowtype-yoga/purescript-protobuf
on v1.1.1 help sign in
rowtype-yoga/purescript-protobuf/README.md
234 lines 8.9 kB history raw
1# purescript-protobuf
3[![Test](https://github.com/xc-jp/purescript-protobuf/workflows/Test/badge.svg?branch=master)](https://github.com/xc-jp/purescript-protobuf/actions)
4[![Pursuit](http://pursuit.purescript.org/packages/purescript-protobuf/badge)](http://pursuit.purescript.org/packages/purescript-protobuf/)
6Purescript library and code generator for
7[Google Protocol Buffers version 3](https://developers.google.com/protocol-buffers/docs/proto3).
9This library operates on
10[`ArrayBuffer`](https://pursuit.purescript.org/packages/purescript-arraybuffer-types/docs/Data.ArrayBuffer.Types#t:ArrayBuffer), so it will run both
11[in *Node.js*](https://pursuit.purescript.org/packages/purescript-node-buffer/docs/Node.Buffer.Class)
12and in browser environments.
14## Code Generation
16The `shell.nix` environment provides
18* The Purescript toolchain
19* The [`protoc`](https://github.com/protocolbuffers/protobuf/blob/master/src/README.md) compiler
20* The `protoc-gen-purescript` executable plugin for `protoc` on the `PATH` so that
21 [`protoc` can find it](https://developers.google.com/protocol-buffers/docs/reference/cpp/google.protobuf.compiler.plugin).
23```
24$ nix-shell
26Purescript Protobuf development environment.
27To build purescript-protobuf, run:
29 npm install
30 spago build
32To test purescript-protobuf, run:
34 protoc --purescript_out=./test/generated test/*.proto
35 spago -x test.dhall build
36 spago -x test.dhall test
38To generate Purescript .purs files from .proto files, run:
40 protoc --purescript_out=path_to_output file.proto
42[nix-shell]$
43```
45## Writing programs with the generated code
47*None of the modules in this package should be imported directly in our program.*
49Rather, we'll import the message modules from the generated `.purs` files,
50as well as modules for reading and writing `ArrayBuffer`s.
52For example, a message in a `.proto` file declared as
54```
55message MyMessage {
56 sint32 my_field = 1;
58```
60will export these four names in the generated `.purs` modules.
621. A message record type
63 * ```purescript
64 type MyMessageR = { my_field :: Maybe Int }
65 ```
662. A message data type
67 * ```purescript
68 newtype MyMessage = MyMessage MyMessageR
69 ```
703. A message encoder which works with
71 [__purescript-arraybuffer-builder__](http://pursuit.purescript.org/packages/purescript-arraybuffer-builder/)
72 * ```purescript
73 putMyMessage :: forall m. MonadEffect m => MyMessage -> PutM m Unit
74 ```
754. A message decoder which works with
76 [__purescript-parsing-dataview__](http://pursuit.purescript.org/packages/purescript-parsing-dataview/)
77 * ```purescript
78 parseMyMessage :: forall m. MonadEffect m => Int -> ParserT DataView m MyMessage
79 ```
80 The message decoder needs an argument which tells it the
81 length of the message which it’s about to decode, because
82 [“the Protocol Buffer wire format is not self-delimiting.”](https://developers.google.com/protocol-buffers/docs/techniques#streaming)
84Then, in our program, our imports will look something like this.
87```purescript
88import Generated.Module (MyMessage(..), putMyMessage, parseMyMessage)
89import Text.Parsing.Parser (runParserT)
90import Data.ArrayBuffer.Builder (execPutM)
91```
93The generated code modules will import modules from this
94package.
96The generated code depends on packages
98```
99 , "protobuf"
100 , "arraybuffer"
101 , "arraybuffer-types"
102 , "arraybuffer-builder"
103 , "parsing"
104 , "parsing-dataview"
105 , "uint"
106 , "long"
107 , "text-encoding"
108```
110which are in
111[__package-sets__](https://github.com/purescript/package-sets),
112except for
113[__purescript-longs__](https://pursuit.purescript.org/packages/purescript-longs)
114(see `spago.dhall` in this package for the particulars).
116It also depends on the Javascript package
117[__long__](https://www.npmjs.com/package/long).
119### Generated message instances
121We cannot easily derive common instances like `Eq` for the
122generated message types because
1231. The types [might be recursive](https://github.com/purescript/documentation/blob/master/errors/CycleInDeclaration.md).
1242. The types might contain fields of type
125 [`ArrayBuffer`](https://pursuit.purescript.org/packages/purescript-arraybuffer-types/docs/Data.ArrayBuffer.Types#t:ArrayBuffer),
126 which doesn't have those instances.
128All of the generated message types have an instance of
129[`Generic`](https://pursuit.purescript.org/packages/purescript-generics-rep/docs/Data.Generic.Rep#t:Generic).
130This allows us to sometimes use
131[`genericEq`](https://pursuit.purescript.org/packages/purescript-generics-rep/docs/Data.Generic.Rep.Eq#v:genericEq)
132and
133[`genericShow`](https://pursuit.purescript.org/packages/purescript-generics-rep/docs/Data.Generic.Rep.Show#v:genericShow)
134on a generated message, if the generated message has those instances for
135all of its fields.
137All of the generated message types have an instance of
138[`NewType`](https://pursuit.purescript.org/packages/purescript-newtype/docs/Data.Newtype#t:Newtype).
140### Examples
142The __purescript-protobuf__ repository contains three executable *Node.js*
143programs which use code generated by __purescript-protobuf__. Refer to these
144for further examples of how to use the generated code.
1461. The `protoc`
147 [compiler plugin](https://github.com/xc-jp/purescript-protobuf/blob/master/src/ProtocPlugin/Main.purs).
148 The code generator imports generated code. Trippy, right? This program
149 literally writes itself.
1502. The
151 [unit test suite](https://github.com/xc-jp/purescript-protobuf/blob/master/test/Main.purs)
1523. The Google
153 [conformance test program](https://github.com/xc-jp/purescript-protobuf/blob/master/conformance/Main.purs)
155### Interpreting invalid encoding parse failures
157When the decode parser encounters an invalid encoding in the protobuf input
158stream then it will fail to parse.
160When
161[`Text.Parsing.Parser.ParserT`](https://pursuit.purescript.org/packages/purescript-parsing/docs/Text.Parsing.Parser#t:ParserT)
162fails it will return a `ParseError String (Position {line::Int,column::Int})`.
164The byte offset at which the parse failure occured is given by the
165formula `column - 1`.
167The path to the protobuf definition which failed to parse will be included
168in the `ParseError String` and delimited by `'/'`, something
169like `"Message1 / string_field_1 / Invalid UTF8 encoding."`.
171## Features
173We aim to support binary-encoded (not JSON-encoded)
174[__proto3__](https://developers.google.com/protocol-buffers/docs/proto3).
175Many __proto2__-syntax descriptor files will
176also work, as long as they don't use __proto2__ features.
178We don't support
179[extensions](https://developers.google.com/protocol-buffers/docs/proto?hl=en#extensions).
181The generated optional record fields will use `Nothing` instead of the
182[default values](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#default).
184We do not preserve
185[unknown fields](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#unknowns).
187We do not support
188[services](https://developers.google.com/protocol-buffers/docs/proto3?hl=en#services).
190### Conformance
192At the time of this writing, we pass 193 out of 194 of the
193[Google conformance tests](https://github.com/protocolbuffers/protobuf/tree/master/conformance)
194for binary-encoded proto3.
195The one test we fail is
196the *Required.Proto3.ProtobufInput.UnknownVarint.ProtobufOutput* test, which
197is the test for preserving unknown fields, which we do not support, see above.
199See the `conformance/README.md` in this repository for details.
201### Imports
203The code generator will use the `package` statement in the `.proto` file
204and the base file name as the Purescript module name for that file.
206The Protobuf
207[`import`](https://developers.google.com/protocol-buffers/docs/proto3#importing_definitions)
208statement allows Protobuf messages to have fields
209consisting of Protobuf messages imported from another file, and qualified
210by the package name in that file. In order to generate
211the correct Purescript module name qualifier on the types of imported message
212fields, the code generator must be able to lookup the package name
213statement in the imported file.
215For that reason, we can only use top-level
216(not [nested](https://developers.google.com/protocol-buffers/docs/proto3#nested))
217`message` and `enum` types from an `import`.
219The generated Purescript code will usually have module imports which cause
220the `purs` compiler to emit warnings. Sorry.
222## Performance
224The implementation is simple and straightforward. We haven't done
225any special optimizations. For example, when encoding a protobuf varint, we
226allocate a list of new one-byte `ArrayBuffers`s and then copy them all into
227position in the final `ArrayBuffer`. For another example, when decoding a
228packed field of numbers, we build a list of the numbers, and then copy them
229all into the final `Array`. Also, this whole library is very stack-unsafe.
230This may all be improved in later versions.
232## Contributing
234Pull requests welcome.