!"#$%&'()*+,-./012 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c defghijklmnopqrstuvwxyz{|}~                     &None None None None      MDependently typed shapes of data which are passed between layers of a network(c) Huw Campbell, 2016-2017BSD2 experimentalNone %&,/:DOQRT %Concrete data structures for a Shape.yAll shapes are held in contiguous memory. 3D is held in a matrix (usually row oriented) which has height depth * rows.qThe current shapes we accept. at the moment this is just one, two, and three dimensional Vectors/Matricies.,These are only used with DataKinds, as Kind , with Types 'D1, 'D2, 'D3.One dimensional vector$Two dimensional matrix. Row, Column.0Three dimensional matrix. Row, Column, Channels.)Generate random data of the desired shape(Generate a shape from a Storable Vector.3Returns Nothing if the vector is of the wrong size.  !"#$%&'() !"*     #$%& !"None %&,:DOQRTFrom an int which is hot, create a 1D Shape with one index hot (1) with the rest 0. Rerurns Nothing if the hot number is larger than the length of the vector.oCreate a one hot map from any enumerable. Returns a map, and the ordered list for the reverse transformationFrom a map and value, create a 1D Shape with one index hot (1) with the rest 0. Rerurns Nothing if the hot number is larger than the length of the vector or the map doesn't contain the value. ! ! ! !/Stochastic gradient descent learning parameters(c) Huw Campbell, 2016-2017BSD2 experimentalSafe"4Learning parameters for stochastic gradient descent."#$%&"#$%&"#$%&"#$%&Defines the Layer Classes(c) Huw Campbell, 2016-2017BSD2 experimentalNone%&,9:;<=DOQRT )Class for a layer. All layers implement this, however, they don't need to implement it for all shapes, only ones which are appropriate.*The Wengert tape for this layer. Includes all that is required to generate the back propagated gradients efficiently. As a default, `S i` is fine.+mUsed in training and scoring. Take the input from the previous layer, and give the output from this layer.,Back propagate a step. Takes the current layer, the input that the layer gave from the input and the back propagated derivatives from the layer above.GReturns the gradient layer and the derivatives to push back further.-mClass for updating a layer. All layers implement this, as it describes how to create and update the layer..YThe type for the gradient for this layer. Unit if there isn't a gradient to pass back./8Update a layer with its gradient and learning parameters00Create a random layer, many layers will use pure1"Update a layer with many Gradients )*+,-./01 )*+,-./01 )*+,-./01)*+,-./01 #Core definition of a Neural Network(c) Huw Campbell, 2016-2017BSD2 experimentalNone%&,9:;<=DQRT +A network can easily be created by hand with (:~>), but an easy way to initialise a random network is with the randomNetwork.23Create a network with randomly initialised weights.ZCalls to this function will not compile if the type of the neural network is not sound.3Wegnert Tape of a network.6Parameterised on the layers and shapes of the network.6Gradient of a network.+Parameterised on the layers of the network.9Type of a network.The [*]( type specifies the types of the layers.The [Shape]= type specifies the shapes of data passed between the layers.uCan be considered to be a heterogeneous list of layers which are able to transform the data shapes of the network.<0Running a network forwards with some input data.MThis gives the output, and the Wengert tape required for back propagation.=1Running a loss gradient back through the network.SThis requires a Wengert tape, generated with the appropriate input for the loss.dGives the gradients for the layer, and the gradient across the input (which may not be required).>@Apply one step of stochastic gradient decent across the network.?Ultimate composition.LThis allows a complete network to be treated as a layer in a larger network.@Ultimate composition.LThis allows a complete network to be treated as a layer in a larger network.B,Add very simple serialisation to the network+23456789:;<=>?@ABCDEF 23456789:;<=> 9:;678345<=>2+23456789:;<=>?@ABCDEF;5 1Functions to perform training and backpropagation(c) Huw Campbell, 2016-2017BSD2 experimentalNone,:GfPerform reverse automatic differentiation on the network for the current input and expected output.Note: The loss function pushed backwards is appropriate for both regression and classification as a squared loss or log-loss respectively.kFor other loss functions, use runNetwork and runGradient with the back propagated gradient of your loss.HBUpdate a network with new weights after training with an instance.I7Run the network with input and return the given output.GHIGHIHGIGHI!None0'() !"* "#$%&)*+,-./0123456789:;<=>GHI Concatenation layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone%&,/9:;<=DOQRTJA Concatentating Layer.XThis layer shares it's input state between two sublayers, and concatenates their output.aWith Networks able to be Layers, this allows for very expressive composition of complex Networks.^The Concat layer has a few instances, which allow one to flexibly "bash" together the outputs.Two 1D vectors, can go to a 2D shape with 2 rows if their lengths are identical. Any 2 1D vectors can also become a longer 1D Vector.3D images become 3D images with more channels. The sizes must be the same, one can use Pad and Crop layers to ensure this is the case.M4Concat 3D shapes, increasing the number of channels.P4Run two layers in parallel, combining their outputs.JKLMNOPQJKJKJKLMNOPQ Convolution layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone!"%&,9:;<=DQRTTA convolution layer for a neural network. This uses the im2col convolution trick popularised by Caffe, which essentially turns the many, many, many, many loop convolution into a single matrix multiplication.The convolution layer takes all of the kernels for the convolution, which are flattened and then put into columns in the matrix.The kernel size dictates which input and output sizes will "fit". Fitting the equation: `out = (in - kernel) / stride + 1` for both dimensions.\One probably shouldn't build their own layer, but rather use the randomConvolution function.WQA three dimensional image can produce a 2D image from a convolution with 1 filterXA two dimensional image may have a convolution filter applied to it producing a two dimensional image if both channels and filters is 1.YCA two dimentional image may have a convolution filter applied to itZyA three dimensional image (or 2d with many channels) can have an appropriately sized convolution filter run across it. RSTUVWXYZ[\]RSTUVTURSV RSTUVWXYZ[\] Cropping layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone %&,:<=DQRT^&A cropping layer for a neural network.`'A two dimentional image can be cropped.a'A two dimentional image can be cropped.^_`abc^_^_^_`abcDeconvolution layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone!"%&,9:;<=DQRTfkA Deconvolution layer for a neural network. This uses the im2col Convolution trick popularised by Caffe.The Deconvolution layer is a way of spreading out a single response into a larger image, and is useful in generating images.i{A three dimensional image (or 2d with many channels) can have an appropriately sized Deconvolution filter run across it.jEA two dimentional image may have a Deconvolution filter applied to itkEA two dimentional image may have a Deconvolution filter applied to itlEA two dimentional image may have a Deconvolution filter applied to it defghijklmnodefghfgdeh defghijklmnoNone,<=DQRpqrstuvpqrstpqrstpqrstuvExponential linear unit layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone,<=DQRxAn exponential linear unit. A layer which can act between any shape of the same dimension, acting as a diode on every neuron individually.xyz{|}~xyxyxyz{|}~None !",<=DQRT@A basic fully connected (or inner product) neural network layer. Sigmoid nonlinear layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone ,9;<=DQRA Logit layer.A layer which can act between any shape of the same dimension, perfoming an sigmoid function. This layer should be used as the output layer of a network for logistic regression (classification) problems.,-,-.Merging layer for parallel network composition(c) Huw Campbell, 2016-2017BSD2 experimentalNone%&,/9:;<=DOQRTA Merging layer.YSimilar to Concat layer, except sums the activations instead of creating a larger shape.<Combine the outputs and the inputs, summing the output shapejRun two layers in parallel, combining their outputs. This just kind of "smooshes" the weights together."Padding layer for 2D and 3D images(c) Huw Campbell, 2016-2017BSD2 experimentalNone %&,:<=DQRT%A padding layer for a neural network.*Pads on the X and Y dimension of an image.&A two dimentional image can be padped.&A two dimentional image can be padped.&Max Pooling layer for 2D and 3D images(c) Huw Campbell, 2016-2017BSD2 experimentalNone%&,/:<=DQRT%A pooling layer for a neural network.Does a max pooling, looking over a kernel similarly to the convolution network, but returning maxarg only. This layer is often used to provide minor amounts of translational invariance.The kernel size dictates which input and output sizes will "fit". Fitting the equation: `out = (in - kernel) / stride + 1` for both dimensions.6A three dimensional image can be pooled on each layer.&A two dimentional image can be pooled.Rectifying linear unit layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone,<=DQRA rectifying linear unit. A layer which can act between any shape of the same dimension, acting as a diode on every neuron individually.Multipurpose reshaping layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone ,:<=DQR Reshape LayerThe Reshape layer can flatten any 2D or 3D image to 1D vector with the same number of activations, as well as cast up from 1D to a 2D or 3D shape.YCan also be used to turn a 3D image with only one channel into a 2D image or vice versa. . .Softmax loss layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone ,9;<=DQRA Softmax layer[This layer is like a logit layer, but normalises a set of matricies to be probabilities._One can use this layer as the last layer in a network if they need normalised probabilities."Hyperbolic tangent nonlinear layer(c) Huw Campbell, 2016-2017BSD2 experimentalNone ,9;<=DQRjA Tanh layer. A layer which can act between any shape of the same dimension, perfoming a tanh function.//5Trivial layer which perfoms no operations on the data(c) Huw Campbell, 2016-2017BSD2 experimentalNone ,9;<=DQRA Trivial layer.jThis can be used to pass an unchanged value up one side of a graph, for a Residual network for example.;Inception style parallel convolutional network composition.(c) Huw Campbell, 2016-2017BSD2 experimentalNone%&,9:;<=DOQRTType of an inception layer.EIt looks like a bit of a handful, but is actually pretty easy to use.The first three type parameters are the size of the (3D) data the inception layer will take. It will emit 3D data with the number of channels being the sum of chx, chy, chzn, which are the number of convolution filters in the 3x3, 5x5, and 7x7 convolutions Layers respectively.The network get padded effectively before each convolution filters such that the output dimension is the same x and y as the input.012012"None,JKRSTUV^_defghxyNone ,9:;<=DR Wengert TapemUsed in training and scoring. Take the input from the previous layer, and give the output from this layer.Back propagate a step. Takes the current layer, the input that the layer gave from the input and the back propagated derivatives from the layer above. Returns the gradient layer and the derivatives to push back further.Class for a recurrent layer. It's quite similar to a normal layer but for the input and output of an extra recurrent data shape.EShape of data that is passed between each subsequent run of the layerNone%&,9:;<=CDOQRT 3A network can easily be created by hand with (:~~>) and (:~@>), but an easy way to initialise a random recurrent network and a set of random inputs for it is with the randomRecurrent.'Create a network of the types requestedEAll the information required to backpropogate through time safely._We index on the time step length as well, to ensure that that all Tape lengths are the same.nRecurrent inputs (sideways shapes on an imaginary unrolled graph) Parameterised on the layers of a Network.Gradient of a network.+Parameterised on the layers of the network.#Type of a recurrent neural network./The [*] type specifies the types of the layers.HThe [Shape] type specifies the shapes of data passed between the layers.The definition is similar to a Network, but every layer in the type is tagged by whether it's a FeedForward Layer of a Recurrent layer.eOften, to make the definitions more concise, one will use a type alias for these empty data types.FWitness type to say indicate we're building up with a recurrent layer.SWitness type to say indicate we're building up with a normal feed forward layer.eApply a batch of gradients to the network Uses runUpdates which can be specialised for a layer.6Add very simple serialisation to the recurrent network%3355None!"%&,:DOQRT<Drive and network and collect its back propogated gradients.+Just forwards propagation with no training.44#NoneNone!"%&,:<=DQR 5656 None!"$%&,:<=DOQRT%Long Short Term Memory Recurrent unitThis is a Peephole formulation, so the recurrent shape is just the cell state, the previous output is not held or used at all.bGenerate an LSTM layer with random Weights one can also just call createRandom from UpdateLayer<Has forget gate biases set to 1 to encourage early learning. Thttps://github.com/karpathy/char-rnn/commit/0dfeaa454e687dd0278f036552ea1e48a0a408c97MathsTODO: move to not here     789                 789$None     %None1     &None\'() !"* "#$%&)*+,-./0123456789:;<=>GHIJKRSTUV^_defghxy:'()*+,-./0123456789:;<=>?@ABCDEFGHIJKKLMNOPQRSTUVWXY Z [ \ ] ^ _ ` a b c d e f g h i j k l m n o p q r r s t u v w x y y z z { | } ~                       ! ! " # $ % & '()*+,-./012345678'9:'9;'(<'(= >?@ABCDEFGHH I J BK$grenade-0.1.0-LtSAx2Qc7WbILGAKal2dtUGrenade.Core.ShapeGrenade.Layers.Internal.UpdateGrenade.Layers.Internal.PoolingGrenade.Layers.Internal.Pad#Grenade.Layers.Internal.ConvolutionGrenade.Utils.OneHotGrenade.Core.LearningParametersGrenade.Core.LayerGrenade.Core.NetworkGrenade.Core.RunnerGrenade.Layers.ConcatGrenade.Layers.ConvolutionGrenade.Layers.CropGrenade.Layers.DeconvolutionGrenade.Layers.DropoutGrenade.Layers.EluGrenade.Layers.FullyConnectedGrenade.Layers.LogitGrenade.Layers.MergeGrenade.Layers.PadGrenade.Layers.PoolingGrenade.Layers.ReluGrenade.Layers.ReshapeGrenade.Layers.SoftmaxGrenade.Layers.TanhGrenade.Layers.TrivialGrenade.Layers.InceptionGrenade.Recurrent.Core.LayerGrenade.Recurrent.Core.NetworkGrenade.Recurrent.Core.Runner'Grenade.Recurrent.Layers.BasicRecurrentGrenade.Recurrent.Layers.LSTM Grenade.CoreGrenade.LayersGrenade.Recurrent.CoreGrenade.Recurrent.LayersGrenade.RecurrentGrenade%singletons-2.2-5sPKGH65fdL9AOYZWYygxxData.SingletonsSing decendMatrix decendVector poolForward poolBackwardpadcropcol2vidcol2imvid2colim2colSS1DS2DS3DShapeD1D2D3 randomOfShape fromStorable $fNFDataS $fFloatingS $fFractionalS$fNumS$fSingIShapeD3$fSingIShapeD2$fSingIShapeD1$fShowSoneHothotMapmakeHotunHotsampleLearningParameters learningRatelearningMomentumlearningRegulariser$fEqLearningParameters$fShowLearningParametersLayerTape runForwards runBackwards UpdateLayerGradient runUpdate createRandom runUpdates randomNetworkTapesTNil:\> GradientsGNil:/>NetworkNNil:~> runNetwork runGradient applyUpdate$fLayerNetworkio$fUpdateLayerNetwork$fSerializeNetwork$fSerializeNetwork0$fCreatableNetwork::$fCreatableNetwork[]: $fShowNetwork$fShowNetwork0 backPropagatetrainrunNetConcat$fSerializeConcat$fLayerConcatiD3$fLayerConcatiD1$fLayerConcatiD2$fUpdateLayerConcat $fShowConcat Convolution' ConvolutionrandomConvolution$fLayerConvolutionD3D2$fLayerConvolutionD2D2$fLayerConvolutionD2D3$fLayerConvolutionD3D3$fSerializeConvolution$fUpdateLayerConvolution$fShowConvolutionCrop$fLayerCropD3D3$fLayerCropD2D2$fUpdateLayerCrop $fShowCropDeconvolution' DeconvolutionrandomDeconvolution$fLayerDeconvolutionD3D3$fLayerDeconvolutionD3D2$fLayerDeconvolutionD2D2$fLayerDeconvolutionD2D3$fSerializeDeconvolution$fUpdateLayerDeconvolution$fShowDeconvolutionDropout dropoutRate dropoutSeed randomDropout$fLayerDropoutD1D1$fUpdateLayerDropout $fShowDropoutElu$fLayerEluD3D3$fLayerEluD2D2$fLayerEluD1D1$fSerializeElu$fUpdateLayerElu $fShowEluFullyConnected'FullyConnectedrandomFullyConnected$fSerializeFullyConnected$fLayerFullyConnectedD1D1$fUpdateLayerFullyConnected$fShowFullyConnectedLogit$fSerializeLogit$fLayerLogitab$fUpdateLayerLogit $fShowLogitMerge$fSerializeMerge$fLayerMergeio$fUpdateLayerMerge $fShowMergePad$fLayerPadD3D3$fLayerPadD2D2$fSerializePad$fUpdateLayerPad $fShowPadPooling$fLayerPoolingD3D3$fLayerPoolingD2D2$fSerializePooling$fUpdateLayerPooling $fShowPoolingRelu$fLayerReluD3D3$fLayerReluD2D2$fLayerReluD1D1$fSerializeRelu$fUpdateLayerRelu $fShowReluReshape$fSerializeReshape$fLayerReshapeD1D3$fLayerReshapeD1D2$fLayerReshapeD2D3$fLayerReshapeD3D2$fLayerReshapeD3D1$fLayerReshapeD2D1$fUpdateLayerReshape $fShowReshapeSoftmaxsoftmaxsoftmax'$fSerializeSoftmax$fLayerSoftmaxD1D1$fUpdateLayerSoftmax $fShowSoftmaxTanh $fLayerTanhab$fSerializeTanh$fUpdateLayerTanh $fShowTanhTrivial$fLayerTrivialab$fUpdateLayerTrivial$fSerializeTrivial $fShowTrivialResnet InceptionMini InceptionRecurrentLayerRecTaperunRecurrentForwardsrunRecurrentBackwardsRecurrentUpdateLayerRecurrentShaperandomRecurrentRecurrentTapesTRNil:\~>:\@>RecurrentInputsRINil:~~+>:~@+>RecurrentGradientsRGNil://>RecurrentNetworkRNil:~~>:~@> Recurrent FeedForwardrunRecurrentNetworkrunRecurrentGradientapplyRecurrentUpdate$fNumRecurrentInputs$fNumRecurrentInputs0$fNumRecurrentInputs1$fSerializeRecurrentInputs$fSerializeRecurrentInputs0$fSerializeRecurrentInputs1$fSerializeRecurrentNetwork$fSerializeRecurrentNetwork0$fSerializeRecurrentNetwork1$fCreatableRecurrent::$fCreatableRecurrent::0$fCreatableRecurrent[]:$fShowRecurrentNetwork$fShowRecurrentNetwork0$fShowRecurrentNetwork1backPropagateRecurrenttrainRecurrent runRecurrentBasicRecurrentrandomBasicRecurrent"$fRecurrentLayerBasicRecurrentD1D1$$fRecurrentUpdateLayerBasicRecurrent$fUpdateLayerBasicRecurrent$fShowBasicRecurrent LSTMWeightslstmWflstmUflstmBflstmWilstmUilstmBilstmWolstmUolstmBolstmWclstmBcLSTM randomLSTM$fSerializeLSTM$fRecurrentLayerLSTMD1D1$fRecurrentUpdateLayerLSTM$fUpdateLayerLSTM $fShowLSTM decend_cpu decendUnsafepool_backwards_cpupool_forwards_cpucrop_cpupad_cpu im2col_cpu col2im_cpucol2im_cim2col_cD1SingD2SingD3Singn1n2nkD:R:SingShapen0!Data.Singletons.TypeLits.InternalSNatSSymSLambda applySingCreatableNetworklogistic logistic' fromJust'tanh' Inception7x7 Inception5x5 Inception3x3CreatableRecurrentupdateRecInputsBasicRecurrent'sigmoidsigmoid'