@@ -10,11 +10,11 @@ Five is right out.
11💣 Machine learning which might blow up in your face 💣
11💣 Machine learning which might blow up in your face 💣
13−Grenade is a dependently typed, practical, and pretty quick neural network library for concise and precise
13+Grenade is a dependently typed, practical, and pretty quick neural network
14−specifications of complex networks in Haskell.
14+library for concise and precise specifications of complex networks in Haskell.
16−As an example, a network which can achieve less than 1.5% error on MNIST can be specified and
16+As an example, a network which can achieve less than 1% error on MNIST can be
17−initialised with random weights in a few lines of code with
17+specified and initialised with random weights in a few lines of code with
19randomMnist :: MonadRandom m
19randomMnist :: MonadRandom m
20 => m (Network '[ Convolution 1 10 5 5 1 1, Pooling 2 2 2 2, Relu, Convolution 10 16 5 5 1 1, Pooling 2 2 2 2, FlattenLayer, Relu, FullyConnected 256 80, Logit, FullyConnected 80 10, Logit]
20 => m (Network '[ Convolution 1 10 5 5 1 1, Pooling 2 2 2 2, Relu, Convolution 10 16 5 5 1 1, Pooling 2 2 2 2, FlattenLayer, Relu, FullyConnected 256 80, Logit, FullyConnected 80 10, Logit]
@@ -22,22 +22,56 @@ randomMnist :: MonadRandom m
22randomMnist = randomNetwork
22randomMnist = randomNetwork
25−The network can be thought of as a heterogeneous list of layers, and its type signature includes a type
25+And that's it. Because the types contain all the information we need, there's
26−level list of the shapes of the data passed between the layers of the network.
26+no specific term level code required; although it is of course possible and
27+easy to build one oneself.
28−In the above example, the input layer can be seen to be a two dimensional (`D2`) image with 28 by 28 pixels.
29+The network can be thought of as a heterogeneous list of layers, where its type
29−The last item in the list is one dimensional (`D1`) with 10 values, representing the categories of the mnist data.
30+includes not only the layers of the network, but also the shapes of data that
31+are passed between the layers.
31−Layers in Grenade are represented as Haskell classes, so creating one's own is easy in downstream code. If the shapes
33+```haskell
32−of a network are not specified correctly and a layer can not sensibly perform the operation between two shapes, then
34+data Network :: [*] -> [Shape] -> * where
33−it will result in a compile time error.
35+ O :: Layer x i o => !x -> Network '[x] '[i, o]
36+ (:~>) :: Layer x i h => !x -> !(Network xs (h ': hs)) -> Network (x ': xs) (i ': h ': hs)
39+In the above example, the input layer can be seen to be a two dimensional (`D2`),
40+image with 28 by 28 pixels. When the first *Convolution* layer runs, it outputs
41+a three dimensional (`D3`) 24x24x10 image. The last item in the list is one
42+dimensional (`D1`) with 10 values, representing the categories of the MNIST
48+To perform back propagation, one can call the eponymous function
50+backPropagate :: forall input target shapes layers. (Head shapes ~ input, Last shapes ~ target)
51+ => Network layers shapes -> S' input -> S' target -> Gradients layers
53+which takes a network, appropriate input and target data, and returns the
54+back propagated gradients for the network. The shapes of the gradients are
55+appropriate for each layer, and may be trivial for layers like `Rulu` which
56+have no learnable parameters.
58+The gradients however can always be applied, yielding a new (hopefully better)
61+applyUpdate :: LearningParameters -> Network ls ss -> Gradients ls -> Network ls ss
64+Layers in Grenade are represented as Haskell classes, so creating one's own is
65+easy in downstream code. If the shapes of a network are not specified correctly
66+and a layer can not sensibly perform the operation between two shapes, then
67+it will result in a compile time error.
36Build Instructions
69Build Instructions
37------------------
70------------------
38−Grenade currently only builds with the [mafia](https://github.com/ambiata/mafia) scipt that is located in the
71+Grenade currently only builds with the [mafia](https://github.com/ambiata/mafia)
72+script that is located in the repository. You will also need the `lapack` and
39−repository. You will also need the `lapack` and `blas` libraries and development tools. Once you have all
73+`blas` libraries and development tools. Once you have all that, Grenade can be
40−that, Grenade can be build using:
74+build using:
43./mafia build
77./mafia build
@@ -54,19 +88,23 @@ Grenade is currently known to build with ghc 7.10 and 8.0.
57−Writing a library like this has been on my mind for a while now, but a big shout out must go to [Justin Le](https://github.com/mstksg), whose
91+Writing a library like this has been on my mind for a while now, but a big shout
92+out must go to [Justin Le](https://github.com/mstksg), whose
58−[dependently typed fully connected network](https://blog.jle.im/entry/practical-dependent-types-in-haskell-1.html) inspired me to get cracking, gave many ideas for the type level tools I
93+[dependently typed fully connected network](https://blog.jle.im/entry/practical-dependent-types-in-haskell-1.html)
94+inspired me to get cracking, gave many ideas for the type level tools I
59needed, and was a great starting point for writing this library.
95needed, and was a great starting point for writing this library.
61Performance
97Performance
62-----------
98-----------
63−Grenade is backed by hmatrix and blas, and uses a pretty clever convolution trick popularised by Caffe, which
99+Grenade is backed by hmatrix and blas, and uses a pretty clever convolution
100+trick popularised by Caffe, which is surprisingly effective and fast. So for many
64−is surprisingly effective and fast. So for many small scale problems it should be sufficient.
101+small scale problems it should be sufficient.
66−That said, it's currently stuck on a single core and doesn't hit up the GPU, so there's a fair bit of performance
103+Being purely functional, it's probably pretty easy to parallelise and batch up.
67−sitting there begging.
104+My current examples however are single threaded.
69−Training 15 generations over Kaggle's mnist training data took a few hours.
106+Training 15 generations over Kaggle's 42000 sample MNIST training set took under
107+an hour, achieving 0.5% error rate on a 1000 sample holdout set.
71Contributing
109Contributing
72------------
110------------