Copyright | (c) Justin Le 2017 |
---|---|

License | BSD3 |

Maintainer | justin@jle.im |

Stability | experimental |

Portability | non-portable |

Safe Haskell | None |

Language | Haskell2010 |

Provides monomorphic versions of the types and combinators in Numeric.Backprop.Op, for usage with Numeric.Backprop.Mono and Numeric.Backprop.Mono.Implicit.

They are monomorphic in the sense that all of the *inputs* have to be of
the same type. So, something like

`Op`

'[Double, Double, Double] Int

From Numeric.Backprop would, in this module, be:

`Op`

`N3`

Double Int

See the module header for Numeric.Backprop.Op for more explicitly
details on how to encode an `Op`

and how they are implemented. For the
most part, the same principles will apply.

Note that `Op`

is a *subset* or *subtype* of `OpM`

, and so, any function
that expects an

(or an `OpM`

m as a

)
can be given an `OpB`

s as a

and it'll work just fine.`Op`

as a

- type Op n a b = Op (Replicate n a) b
- pattern Op :: forall n a b. Known N Nat n => (Vec n a -> (b, Maybe b -> Vec n a)) -> Op n a b
- type OpM m n a = OpM m (Replicate n a)
- pattern OpM :: forall n m a b. (Known N Nat n, Functor m) => (Vec n a -> m (b, Maybe b -> m (Vec n a))) -> OpM m n a b
- data VecT k n f a :: forall k. N -> (k -> *) -> k -> * where
- type Vec n = VecT * n I
- newtype I a :: * -> * = I {
- getI :: a

- runOp :: Op n a b -> Vec n a -> b
- gradOp :: Op n a b -> Vec n a -> Vec n a
- gradOp' :: Op n a b -> Vec n a -> (b, Vec n a)
- gradOpWith :: Op n a b -> Vec n a -> b -> Vec n a
- gradOpWith' :: Op n a b -> Vec n a -> Maybe b -> Vec n a
- runOp' :: Op n a b -> Vec n a -> (b, Maybe b -> Vec n a)
- runOpM :: Functor m => OpM m n a b -> Vec n a -> m b
- gradOpM :: Monad m => OpM m n a b -> Vec n a -> m (Vec n a)
- gradOpM' :: Monad m => OpM m n a b -> Vec n a -> m (b, Vec n a)
- gradOpWithM :: Monad m => OpM m n a b -> Vec n a -> b -> m (Vec n a)
- gradOpWithM' :: Monad m => OpM m n a b -> Vec n a -> Maybe b -> m (Vec n a)
- runOpM' :: Functor m => OpM m n a b -> Vec n a -> m (b, Maybe b -> m (Vec n a))
- op0 :: a -> Op N0 b a
- opConst :: forall n a b. (Known Nat n, Num b) => a -> Op n b a
- composeOp :: forall m n o a b c. (Monad m, Num a, Known Nat n) => VecT o (OpM m n a) b -> OpM m o b c -> OpM m n a c
- composeOp1 :: forall m n a b c. (Monad m, Num a, Known Nat n) => OpM m n a b -> OpM m N1 b c -> OpM m n a c
- (~.) :: forall m n a b c. (Monad m, Num a, Known Nat n) => OpM m N1 b c -> OpM m n a b -> OpM m n a c
- op1 :: Num a => (forall s. AD s (Forward a) -> AD s (Forward a)) -> Op N1 a a
- op2 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a) -> Op N2 a a
- op3 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a -> Reverse s a) -> Op N3 a a
- opN :: (Num a, Known Nat n) => (forall s. Reifies s Tape => Vec n (Reverse s a) -> Reverse s a) -> Op n a a
- type family Replicate (n :: N) (a :: k) = (as :: [k]) | as -> n where ...
- op1' :: (a -> (b, Maybe b -> a)) -> Op N1 a b
- op2' :: (a -> a -> (b, Maybe b -> (a, a))) -> Op N2 a b
- op3' :: (a -> a -> a -> (b, Maybe b -> (a, a, a))) -> Op N3 a b
- pattern (:+) :: forall a n. a -> Vec n a -> Vec (S n) a
- (*:) :: f a -> f a -> VecT k (S (S Z)) f a
- (+:) :: a -> a -> Vec (S (S Z)) a
- head' :: VecT k (S n) f a -> f a
- type N0 = Z
- type N1 = S N0
- type N2 = S N1
- type N3 = S N2
- type N4 = S N3
- type N5 = S N4
- type N6 = S N5
- type N7 = S N6
- type N8 = S N7
- type N9 = S N8
- type N10 = S N9

# Types

## Op and synonyms

type Op n a b = Op (Replicate n a) b Source #

An

describes a differentiable function from `Op`

n a b`n`

values of
type `a`

to a value of type `b`

.

For example, a value of type

`Op`

`N2`

Int Double

is a function that takes two `Int`

s and returns a `Double`

.
It can be differentiated to give a *gradient* of two `Int`

s, if given
a total derivative for the `Double`

. Mathematically, it is akin to a:

\[ f : \mathbb{Z}^2 \rightarrow \mathbb{R} \]

See `runOp`

, `gradOp`

, and `gradOpWith`

for examples on how to run it,
and `Op`

for instructions on creating it.

This type is abstracted over using the pattern synonym with constructor
`Op`

, so you can create one from scratch with it. However, it's
simplest to create it using `op2'`

, `op1'`

, `op2'`

, and `op3'`

helper
smart constructors And, if your function is a numeric function, they
can even be created automatically using `op1`

, `op2`

, `op3`

, and `opN`

with a little help from Numeric.AD from the *ad* library.

Note that this type is a *subset* or *subtype* of `OpM`

(and also of
`OpB`

). So, if a function ever expects an

(or a `OpM`

m as a`OpB`

), you can always provide an

instead.`Op`

as a

Many functions in this library will expect an

(or
an `OpM`

m as a

), and in all of these cases, you can
provide an `OpB`

s as a

.`Op`

as a

pattern Op :: forall n a b. Known N Nat n => (Vec n a -> (b, Maybe b -> Vec n a)) -> Op n a b Source #

Construct an `Op`

by giving a function creating the result, and also
a continuation on how to create the gradient, given the total derivative
of `a`

.

See the module documentation for Numeric.Backprop.Op for more details
on the function that this constructor and `OpM`

expect.

type OpM m n a = OpM m (Replicate n a) Source #

An

represents a differentiable (monadic) function from
`OpM`

m n a b`n`

values of type `a`

to a value of type `b`

.

For example, an

`OpM`

IO`N2`

Int Double

would be a function that takes two `Int`

s and returns a `Double`

(in
`IO`

). It can be differentiated to give a *gradient* of the two input
`Int`

s (also in `IO`

) if given the total derivative for `a`

.

Note that an `OpM`

is a *superclass* of `Op`

, so any function that
expects an

can also accept an `OpM`

m as a

.`Op`

as a

See `runOpM`

, `gradOpM`

, and `gradOpWithM`

for examples on how to run
it.

pattern OpM :: forall n m a b. (Known N Nat n, Functor m) => (Vec n a -> m (b, Maybe b -> m (Vec n a))) -> OpM m n a b Source #

Construct an `OpM`

by giving a (monadic) function creating the result,
and also a continuation on how to create the gradient, given the total
derivative of `a`

.

See the module documentation for Numeric.Backprop.Op for more details
on the function that this constructor and `Op`

expect.

## Vector types

See Numeric.Backprop.Mono for a mini-tutorial on `VecT`

and
`Vec`

data VecT k n f a :: forall k. N -> (k -> *) -> k -> * where #

Functor1 l l (VecT l n) | |

Foldable1 l l (VecT l n) | |

Traversable1 l l (VecT l n) | |

Witness ØC (Known N Nat n) (VecT k n f a) | |

(Monad f, Known N Nat n) => Monad (VecT * n f) | |

Functor f => Functor (VecT * n f) | |

(Applicative f, Known N Nat n) => Applicative (VecT * n f) | |

Foldable f => Foldable (VecT * n f) | |

Traversable f => Traversable (VecT * n f) | |

Eq (f a) => Eq (VecT k n f a) | |

(Num (f a), Known N Nat n) => Num (VecT k n f a) | |

Ord (f a) => Ord (VecT k n f a) | |

Show (f a) => Show (VecT k n f a) | |

type WitnessC ØC (Known N Nat n) (VecT k n f a) | |

# Running

## Pure

runOp :: Op n a b -> Vec n a -> b Source #

Run the function that an `Op`

encodes, to get the result.

`>>>`

15`runOp (op2 (*)) (3 :+ 5 :+ Ø)`

gradOp :: Op n a b -> Vec n a -> Vec n a Source #

Run the function that an `Op`

encodes, and get the gradient of the
output with respect to the inputs.

`>>>`

5 :+ 3 :+ ØV -- the gradient of x*y is (y, x)`gradOp (op2 (*)) (3 :+ 5 :+ ØV)`

gradOp' :: Op n a b -> Vec n a -> (b, Vec n a) Source #

Run the function that an `Op`

encodes, to get the resulting output and
also its gradient with respect to the inputs.

`>>>`

(15, 5 :+ 3 :+ ØV)`gradOpM' (op2 (*)) (3 :+ 5 :+ ØV) :: IO (Int, Vec N2 Int)`

gradOpWith :: Op n a b -> Vec n a -> b -> Vec n a Source #

Run the function that an `Op`

encodes, and get the gradient of
a "final result" with respect to the inputs, given the total derivative
of the output with the final result.

See `gradOp`

and the module documentaiton for Numeric.Backprop.Op for
more information.

gradOpWith' :: Op n a b -> Vec n a -> Maybe b -> Vec n a Source #

A combination of `gradOp`

and `gradOpWith`

. The third argument is
(optionally) the total derivative the result. Give `Nothing`

and it is
assumed that the result is the final result (and the total derivative is
1), and this behaves the same as `gradOp`

. Give

and it uses
the `Just`

d`d`

as the total derivative of the result, and this behaves like
`gradOpWith`

.

See `gradOp`

and the module documentaiton for Numeric.Backprop.Op for
more information.

runOp' :: Op n a b -> Vec n a -> (b, Maybe b -> Vec n a) Source #

A combination of `runOp`

and `gradOpWith'`

. Given an `Op`

and inputs,
returns the result of the `Op`

and a continuation that gives its
gradient.

The continuation takes the total derivative of the result as input. See
documenation for `gradOpWith'`

and module documentation for
Numeric.Backprop.Op for more information.

## Monadic

gradOpWithM :: Monad m => OpM m n a b -> Vec n a -> b -> m (Vec n a) Source #

The monadic version of `gradOpWith`

, for `OpM`

s.

gradOpWithM' :: Monad m => OpM m n a b -> Vec n a -> Maybe b -> m (Vec n a) Source #

The monadic version of `gradOpWith'`

, for `OpM`

s.

# Creation

op0 :: a -> Op N0 b a Source #

Create an `Op`

that takes no inputs and always returns the given
value.

There is no gradient, of course (using `gradOp`

will give you an empty
vector), because there is no input to have a gradient of.

`>>>`

(10, ØV)`gradOp' (op0 10) ØV`

For a constant `Op`

that takes input and ignores it, see `opConst`

.

Note that because this returns an `Op`

, it can be used with any function
that expects an `OpM`

or `OpB`

, as well.

opConst :: forall n a b. (Known Nat n, Num b) => a -> Op n b a Source #

An `Op`

that ignores all of its inputs and returns a given constant
value.

`>>>`

(10, 0 :+ 0 :+ 0 :+ ØV)`gradOp' (opConst 10) (1 :+ 2 :+ 3 :+ ØV)`

composeOp :: forall m n o a b c. (Monad m, Num a, Known Nat n) => VecT o (OpM m n a) b -> OpM m o b c -> OpM m n a c Source #

composeOp1 :: forall m n a b c. (Monad m, Num a, Known Nat n) => OpM m n a b -> OpM m N1 b c -> OpM m n a c Source #

(~.) :: forall m n a b c. (Monad m, Num a, Known Nat n) => OpM m N1 b c -> OpM m n a b -> OpM m n a c infixr 9 Source #

## Automatic creation using the *ad* library

op2 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a) -> Op N2 a a Source #

op3 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a -> Reverse s a) -> Op N3 a a Source #

opN :: (Num a, Known Nat n) => (forall s. Reifies s Tape => Vec n (Reverse s a) -> Reverse s a) -> Op n a a Source #

type family Replicate (n :: N) (a :: k) = (as :: [k]) | as -> n where ... Source #

is a list of `Replicate`

n a`a`

s repeated `n`

times.

`>>>`

'[Int, Int, Int]`:kind! Replicate N3 Int`

`>>>`

'[Double, Double, Double, Double, Double]`:kind! Replicate N5 Double`

## Giving gradients directly

op1' :: (a -> (b, Maybe b -> a)) -> Op N1 a b Source #

Create an `Op`

of a function taking one input, by giving its explicit
derivative. The function should return a tuple containing the result of
the function, and also a function taking the derivative of the result
and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R} \rightarrow \mathbb{R}\cr y &= f(x)\cr z &= g(y) } \]

Then the derivative \( \frac{dz}{dx} \), it would be:

\[ \frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx} \]

If our `Op`

represents \(f\), then the second item in the resulting
tuple should be a function that takes \(\frac{dz}{dy}\) and returns
\(\frac{dz}{dx}\).

If the input is `Nothing`

, then \(\frac{dz}{dy}\) should be taken to be
\(1\).

As an example, here is an `Op`

that squares its input:

square :: Num a =>`Op`

`N1`

a a square =`op1'`

$ \x -> (x*x, \case Nothing -> 2 * x Just d -> 2 * d * x )

Remember that, generally, end users shouldn't directly construct `Op`

s;
they should be provided by libraries or generated automatically.

For numeric functions, single-input `Op`

s can be generated automatically
using `op1`

.

op2' :: (a -> a -> (b, Maybe b -> (a, a))) -> Op N2 a b Source #

Create an `Op`

of a function taking two inputs, by giving its explicit
gradient. The function should return a tuple containing the result of
the function, and also a function taking the derivative of the result
and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R}^2 \rightarrow \mathbb{R}\cr z &= f(x, y)\cr k &= g(z) } \]

Then the gradient \( \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> \) would be:

\[ \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> = \left< \frac{dk}{dz} \frac{\partial z}{dx}, \frac{dk}{dz} \frac{\partial z}{dy} \right> \]

If our `Op`

represents \(f\), then the second item in the resulting
tuple should be a function that takes \(\frac{dk}{dz}\) and returns
\( \left< \frac{\partial k}{dx}, \frac{\partial k}{dx} \right> \).

If the input is `Nothing`

, then \(\frac{dk}{dz}\) should be taken to be
\(1\).

As an example, here is an `Op`

that multiplies its inputs:

mul :: Num a =>`Op`

`N2`

a a mul =`op2'`

$ \x y -> (x*y, \case Nothing -> (y , x ) Just d -> (d*y, x*d) )

Remember that, generally, end users shouldn't directly construct `Op`

s;
they should be provided by libraries or generated automatically.

For numeric functions, two-input `Op`

s can be generated automatically
using `op2`

.