git.iko.soy/grenade
on iko/nfdata help sign in
grenade/README.md
192 lines 7.5 kB history raw
1Grenade
2=======
4[![Build Status](https://api.travis-ci.org/HuwCampbell/grenade.svg?branch=master)](https://travis-ci.org/HuwCampbell/grenade)
5[![Hackage page (downloads and API reference)][hackage-png]][hackage]
6[![Hackage-Deps][hackage-deps-png]][hackage-deps]
9```
10First shalt thou take out the Holy Pin, then shalt thou count to three, no more, no less.
11Three shall be the number thou shalt count, and the number of the counting shall be three.
12Four shalt thou not count, neither count thou two, excepting that thou then proceed to three.
13Five is right out.
14```
16💣 Machine learning which might blow up in your face 💣
18Grenade is a composable, dependently typed, practical, and fast recurrent neural network library
19for concise and precise specifications of complex networks in Haskell.
21As an example, a network which can achieve ~1.5% error on MNIST can be
22specified and initialised with random weights in a few lines of code with
23```haskell
24type MNIST
25 = Network
26 '[ Convolution 1 10 5 5 1 1, Pooling 2 2 2 2, Relu
27 , Convolution 10 16 5 5 1 1, Pooling 2 2 2 2, Reshape, Relu
28 , FullyConnected 256 80, Logit, FullyConnected 80 10, Logit]
29 '[ 'D2 28 28, 'D3 24 24 10, 'D3 12 12 10, 'D3 12 12 10
30 , 'D3 8 8 16, 'D3 4 4 16, 'D1 256, 'D1 256
31 , 'D1 80, 'D1 80, 'D1 10, 'D1 10]
33randomMnist :: MonadRandom m => m MNIST
34randomMnist = randomNetwork
35```
37And that's it. Because the types are so rich, there's no specific term level code
38required to construct this network; although it is of course possible and
39easy to construct and deconstruct the networks and layers explicitly oneself.
41If recurrent neural networks are more your style, you can try defining something
42["unreasonably effective"](http://karpathy.github.io/2015/05/21/rnn-effectiveness/)
43with
44```haskell
45type Shakespeare
46 = RecurrentNetwork
47 '[ R (LSTM 40 80), R (LSTM 80 40), F (FullyConnected 40 40), F Logit]
48 '[ 'D1 40, 'D1 80, 'D1 40, 'D1 40, 'D1 40 ]
49```
51Design
52------
54Networks in Grenade can be thought of as a heterogeneous lists of layers, where
55their type includes not only the layers of the network, but also the shapes of
56data that are passed between the layers.
58The definition of a network is surprisingly simple:
59```haskell
60data Network :: [*] -> [Shape] -> * where
61 NNil :: SingI i
62 => Network '[] '[i]
64 (:~>) :: (SingI i, SingI h, Layer x i h)
65 => !x
66 -> !(Network xs (h ': hs))
67 -> Network (x ': xs) (i ': h ': hs)
68```
70The `Layer x i o` constraint ensures that the layer `x` can sensibly perform a
71transformation between the input and output shapes `i` and `o`.
73The lifted data kind `Shape` defines our 1, 2, and 3 dimension types, used to
74declare what shape of data is passed between the layers.
76In the MNIST example above, the input layer can be seen to be a two dimensional
77(`D2`), image with 28 by 28 pixels. When the first *Convolution* layer runs, it
78outputs a three dimensional (`D3`) 24x24x10 image. The last item in the list is
79one dimensional (`D1`) with 10 values, representing the categories of the MNIST
80data.
82Usage
83-----
85To perform back propagation, one can call the eponymous function
86```haskell
87backPropagate :: forall shapes layers.
88 Network layers shapes -> S (Head shapes) -> S (Last shapes) -> Gradients layers
89```
90which takes a network, appropriate input and target data, and returns the
91back propagated gradients for the network. The shapes of the gradients are
92appropriate for each layer, and may be trivial for layers like `Relu` which
93have no learnable parameters.
95The gradients however can always be applied, yielding a new (hopefully better)
96layer with
97```haskell
98applyUpdate :: LearningParameters -> Network ls ss -> Gradients ls -> Network ls ss
99```
101Layers in Grenade are represented as Haskell classes, so creating one's own is
102easy in downstream code. If the shapes of a network are not specified correctly
103and a layer can not sensibly perform the operation between two shapes, then
104it will result in a compile time error.
106Composition
107-----------
109Networks and Layers in Grenade are easily composed at the type level. As a `Network`
110is an instance of `Layer`, one can use a trained Network as a small component in a
111larger network easily. Furthermore, we provide 2 layers which are designed to run
112layers in parallel and merge their output (either by concatenating them across one
113dimension or summing by pointwise adding their activations). This allows one to
114write any Network which can be expressed as a
115[series parallel graph](https://en.wikipedia.org/wiki/Series-parallel_graph).
117A residual network layer specification for instance could be written as
118```haskell
119type Residual net = Merge Trivial net
120```
121If the type `net` is an instance of `Layer`, then `Residual net` will be too. It will
122run the network, while retaining its input by passing it through the `Trivial` layer,
123and merge the original image with the output.
125See the [MNIST](https://github.com/HuwCampbell/grenade/blob/master/examples/main/mnist.hs)
126example, which has been overengineered to contain both residual style learning as well
127as inception style convolutions.
129Generative Adversarial Networks
130-------------------------------
132As Grenade is purely functional, one can compose its training functions in flexible
133ways. [GAN-MNIST](https://github.com/HuwCampbell/grenade/blob/master/examples/main/gan-mnist.hs)
134example displays an interesting, type safe way of writing a generative adversarial
135training function in 10 lines of code.
137Layer Zoo
138---------
140Grenade layers are normal haskell data types which are an instance of `Layer`, so
141it's easy to build one's own downstream code. We do however provide a decent set
142of layers, including convolution, deconvolution, pooling, pad, crop, logit, relu,
143elu, tanh, and fully connected.
145Build Instructions
146------------------
147Grenade is most easily built with the [mafia](https://github.com/ambiata/mafia)
148script that is located in the repository. You will also need the `lapack` and
149`blas` libraries and development tools. Once you have all that, Grenade can be
150build using:
152```
153./mafia build
154```
156and the tests run using:
158```
159./mafia test
160```
162Grenade builds with ghc 7.10, 8.0, 8.2 and 8.4.
164Thanks
165------
166Writing a library like this has been on my mind for a while now, but a big shout
167out must go to [Justin Le](https://github.com/mstksg), whose
168[dependently typed fully connected network](https://blog.jle.im/entry/practical-dependent-types-in-haskell-1.html)
169inspired me to get cracking, gave many ideas for the type level tools I
170needed, and was a great starting point for writing this library.
172Performance
173-----------
174Grenade is backed by hmatrix, BLAS, and LAPACK, with critical functions optimised
175in C. Using the im2col trick popularised by Caffe, it should be sufficient for
176many problems.
178Being purely functional, it should also be easy to run batches in parallel, which
179would be appropriate for larger networks, my current examples however are single
180threaded.
182Training 15 generations over Kaggle's 41000 sample MNIST training set on a single
183core took around 12 minutes, achieving 1.5% error rate on a 1000 sample holdout set.
185Contributing
186------------
187Contributions are welcome.
189 [hackage]: http://hackage.haskell.org/package/grenade
190 [hackage-png]: http://img.shields.io/hackage/v/grenade.svg
191 [hackage-deps]: http://packdeps.haskellers.com/reverse/grenade
192 [hackage-deps-png]: https://img.shields.io/hackage-deps/v/grenade.svg