Numeric.Backprop.Op.Mono

Provides monomorphic versions of the types and combinators in Numeric.Backprop.Op, for usage with Numeric.Backprop.Mono and Numeric.Backprop.Mono.Implicit.

They are monomorphic in the sense that all of the inputs have to be of the same type. So, something like

Op '[Double, Double, Double] Int

From Numeric.Backprop would, in this module, be:

Op N3 Double Int

See the module header for Numeric.Backprop.Op for more explicitly details on how to encode an Op and how they are implemented. For the most part, the same principles will apply.

Note that Op is a subset or subtype of OpM, and so, any function that expects an OpM m as a (or an OpB s as a) can be given an Op as a and it'll work just fine.

Types

Op and synonyms

type Op n a b = Op (Replicate n a) b

An Op n a b describes a differentiable function from n values of type a to a value of type b.

For example, a value of type

Op N2 Int Double

is a function that takes two Ints and returns a Double. It can be differentiated to give a gradient of two Ints, if given a total derivative for the Double. Mathematically, it is akin to a:

\[ f : \mathbb{Z}^2 \rightarrow \mathbb{R} \]

See runOp, gradOp, and gradOpWith for examples on how to run it, and Op for instructions on creating it.

This type is abstracted over using the pattern synonym with constructor Op, so you can create one from scratch with it. However, it's simplest to create it using op2', op1', op2', and op3' helper smart constructors And, if your function is a numeric function, they can even be created automatically using op1, op2, op3, and opN with a little help from Numeric.AD from the ad library.

Note that this type is a subset or subtype of OpM (and also of OpB). So, if a function ever expects an OpM m as a (or a OpB), you can always provide an Op as a instead.

Many functions in this library will expect an OpM m as a (or an OpB s as a), and in all of these cases, you can provide an Op as a.

pattern Op :: forall n a b. Known N Nat n => (Vec n a -> (b, Maybe b -> Vec n a)) -> Op n a b

Construct an Op by giving a function creating the result, and also a continuation on how to create the gradient, given the total derivative of a.

See the module documentation for Numeric.Backprop.Op for more details on the function that this constructor and OpM expect.

type OpM m n a = OpM m (Replicate n a)

An OpM m n a b represents a differentiable (monadic) function from n values of type a to a value of type b.

For example, an

OpM IO N2 Int Double

would be a function that takes two Ints and returns a Double (in IO). It can be differentiated to give a gradient of the two input Ints (also in IO) if given the total derivative for a.

Note that an OpM is a superclass of Op, so any function that expects an OpM m as a can also accept an Op as a.

See runOpM, gradOpM, and gradOpWithM for examples on how to run it.

pattern OpM :: forall n m a b. (Known N Nat n, Functor m) => (Vec n a -> m (b, Maybe b -> m (Vec n a))) -> OpM m n a b

Construct an OpM by giving a (monadic) function creating the result, and also a continuation on how to create the gradient, given the total derivative of a.

See the module documentation for Numeric.Backprop.Op for more details on the function that this constructor and Op expect.

Vector types

See Numeric.Backprop.Mono for a mini-tutorial on VecT and Vec

data VecT k n f a :: forall k. N -> (k -> *) -> k -> * where

Constructors

ØV :: VecT k Z f a

(:*) :: VecT k (S n1) f a infixr 4

Instances

Functor1 l l (VecT l n)

Methods

map1 :: (forall a. f a -> g a) -> t f b -> t g b

Foldable1 l l (VecT l n)

Methods

foldMap1 :: Monoid m => (forall a. f a -> m) -> t f b -> m

Traversable1 l l (VecT l n)

Methods

traverse1 :: Applicative h => (forall a. f a -> h (g a)) -> t f b -> h (t g b)

Witness ØC (Known N Nat n) (VecT k n f a)

Associated Types

type WitnessC (ØC :: Constraint) (Known N Nat n :: Constraint) (VecT k n f a) :: Constraint

Methods

(\\) :: ØC => (Known N Nat n -> r) -> VecT k n f a -> r

(Monad f, Known N Nat n) => Monad (VecT * n f)

Methods

(>>=) :: VecT * n f a -> (a -> VecT * n f b) -> VecT * n f b

(>>) :: VecT * n f a -> VecT * n f b -> VecT * n f b

return :: a -> VecT * n f a

fail :: String -> VecT * n f a

Functor f => Functor (VecT * n f)

Methods

fmap :: (a -> b) -> VecT * n f a -> VecT * n f b

(<$) :: a -> VecT * n f b -> VecT * n f a

(Applicative f, Known N Nat n) => Applicative (VecT * n f)

Methods

pure :: a -> VecT * n f a

(<*>) :: VecT * n f (a -> b) -> VecT * n f a -> VecT * n f b

(*>) :: VecT * n f a -> VecT * n f b -> VecT * n f b

(<*) :: VecT * n f a -> VecT * n f b -> VecT * n f a

Foldable f => Foldable (VecT * n f)

Methods

fold :: Monoid m => VecT * n f m -> m

foldMap :: Monoid m => (a -> m) -> VecT * n f a -> m

foldr :: (a -> b -> b) -> b -> VecT * n f a -> b

foldr' :: (a -> b -> b) -> b -> VecT * n f a -> b

foldl :: (b -> a -> b) -> b -> VecT * n f a -> b

foldl' :: (b -> a -> b) -> b -> VecT * n f a -> b

foldr1 :: (a -> a -> a) -> VecT * n f a -> a

foldl1 :: (a -> a -> a) -> VecT * n f a -> a

toList :: VecT * n f a -> [a]

null :: VecT * n f a -> Bool

length :: VecT * n f a -> Int

elem :: Eq a => a -> VecT * n f a -> Bool

maximum :: Ord a => VecT * n f a -> a

minimum :: Ord a => VecT * n f a -> a

sum :: Num a => VecT * n f a -> a

product :: Num a => VecT * n f a -> a

Traversable f => Traversable (VecT * n f)

Methods

traverse :: Applicative f => (a -> f b) -> VecT * n f a -> f (VecT * n f b)

sequenceA :: Applicative f => VecT * n f (f a) -> f (VecT * n f a)

mapM :: Monad m => (a -> m b) -> VecT * n f a -> m (VecT * n f b)

sequence :: Monad m => VecT * n f (m a) -> m (VecT * n f a)

Eq (f a) => Eq (VecT k n f a)

Methods

(==) :: VecT k n f a -> VecT k n f a -> Bool

(/=) :: VecT k n f a -> VecT k n f a -> Bool

(Num (f a), Known N Nat n) => Num (VecT k n f a)

Methods

(+) :: VecT k n f a -> VecT k n f a -> VecT k n f a

(-) :: VecT k n f a -> VecT k n f a -> VecT k n f a

(*) :: VecT k n f a -> VecT k n f a -> VecT k n f a

negate :: VecT k n f a -> VecT k n f a

abs :: VecT k n f a -> VecT k n f a

signum :: VecT k n f a -> VecT k n f a

fromInteger :: Integer -> VecT k n f a

Ord (f a) => Ord (VecT k n f a)

Methods

compare :: VecT k n f a -> VecT k n f a -> Ordering

(<) :: VecT k n f a -> VecT k n f a -> Bool

(<=) :: VecT k n f a -> VecT k n f a -> Bool

(>) :: VecT k n f a -> VecT k n f a -> Bool

(>=) :: VecT k n f a -> VecT k n f a -> Bool

max :: VecT k n f a -> VecT k n f a -> VecT k n f a

min :: VecT k n f a -> VecT k n f a -> VecT k n f a

Show (f a) => Show (VecT k n f a)

Methods

showsPrec :: Int -> VecT k n f a -> ShowS

show :: VecT k n f a -> String

showList :: [VecT k n f a] -> ShowS

type WitnessC ØC (Known N Nat n) (VecT k n f a)

type WitnessC ØC (Known N Nat n) (VecT k n f a) = ØC

type Vec n = VecT * n I

newtype I a :: * -> *

Constructors

I

Fields

• getI :: a

Instances

Monad I

Methods

(>>=) :: I a -> (a -> I b) -> I b

(>>) :: I a -> I b -> I b

return :: a -> I a

fail :: String -> I a

Functor I

Methods

fmap :: (a -> b) -> I a -> I b

(<$) :: a -> I b -> I a

Applicative I

Methods

pure :: a -> I a

(<*>) :: I (a -> b) -> I a -> I b

(*>) :: I a -> I b -> I b

(<*) :: I a -> I b -> I a

Foldable I

Methods

fold :: Monoid m => I m -> m

foldMap :: Monoid m => (a -> m) -> I a -> m

foldr :: (a -> b -> b) -> b -> I a -> b

foldr' :: (a -> b -> b) -> b -> I a -> b

foldl :: (b -> a -> b) -> b -> I a -> b

foldl' :: (b -> a -> b) -> b -> I a -> b

foldr1 :: (a -> a -> a) -> I a -> a

foldl1 :: (a -> a -> a) -> I a -> a

toList :: I a -> [a]

null :: I a -> Bool

length :: I a -> Int

elem :: Eq a => a -> I a -> Bool

maximum :: Ord a => I a -> a

minimum :: Ord a => I a -> a

sum :: Num a => I a -> a

product :: Num a => I a -> a

Traversable I

Methods

traverse :: Applicative f => (a -> f b) -> I a -> f (I b)

sequenceA :: Applicative f => I (f a) -> f (I a)

mapM :: Monad m => (a -> m b) -> I a -> m (I b)

sequence :: Monad m => I (m a) -> m (I a)

Witness p q a => Witness p q (I a)

Associated Types

type WitnessC (p :: Constraint) (q :: Constraint) (I a) :: Constraint

Methods

(\\) :: p => (q -> r) -> I a -> r

Eq a => Eq (I a)

Methods

(==) :: I a -> I a -> Bool

(/=) :: I a -> I a -> Bool

Num a => Num (I a)

Methods

(+) :: I a -> I a -> I a

(-) :: I a -> I a -> I a

(*) :: I a -> I a -> I a

negate :: I a -> I a

abs :: I a -> I a

signum :: I a -> I a

fromInteger :: Integer -> I a

Ord a => Ord (I a)

Methods

compare :: I a -> I a -> Ordering

(<) :: I a -> I a -> Bool

(<=) :: I a -> I a -> Bool

(>) :: I a -> I a -> Bool

(>=) :: I a -> I a -> Bool

max :: I a -> I a -> I a

min :: I a -> I a -> I a

Show a => Show (I a)

Methods

showsPrec :: Int -> I a -> ShowS

show :: I a -> String

showList :: [I a] -> ShowS

type WitnessC p q (I a)

type WitnessC p q (I a) = Witness p q a

Running

Pure

runOp :: Op n a b -> Vec n a -> b

Run the function that an Op encodes, to get the result.

>>> runOp (op2 (*)) (3 :+ 5 :+ Ø)
15

gradOp :: Op n a b -> Vec n a -> Vec n a

Run the function that an Op encodes, and get the gradient of the output with respect to the inputs.

>>> gradOp (op2 (*)) (3 :+ 5 :+ ØV)
5 :+ 3 :+ ØV
-- the gradient of x*y is (y, x)

gradOp' :: Op n a b -> Vec n a -> (b, Vec n a)

Run the function that an Op encodes, to get the resulting output and also its gradient with respect to the inputs.

>>> gradOpM' (op2 (*)) (3 :+ 5 :+ ØV) :: IO (Int, Vec N2 Int)
(15, 5 :+ 3 :+ ØV)

gradOpWith :: Op n a b -> Vec n a -> b -> Vec n a

Run the function that an Op encodes, and get the gradient of a "final result" with respect to the inputs, given the total derivative of the output with the final result.

See gradOp and the module documentaiton for Numeric.Backprop.Op for more information.

gradOpWith' :: Op n a b -> Vec n a -> Maybe b -> Vec n a

A combination of gradOp and gradOpWith. The third argument is (optionally) the total derivative the result. Give Nothing and it is assumed that the result is the final result (and the total derivative is 1), and this behaves the same as gradOp. Give Just d and it uses the d as the total derivative of the result, and this behaves like gradOpWith.

See gradOp and the module documentaiton for Numeric.Backprop.Op for more information.

runOp' :: Op n a b -> Vec n a -> (b, Maybe b -> Vec n a)

A combination of runOp and gradOpWith'. Given an Op and inputs, returns the result of the Op and a continuation that gives its gradient.

The continuation takes the total derivative of the result as input. See documenation for gradOpWith' and module documentation for Numeric.Backprop.Op for more information.

Monadic

runOpM :: Functor m => OpM m n a b -> Vec n a -> m b

The monadic version of runOp, for OpMs.

>>> runOpM (op2 (*)) (3 :+ 5 :+ ØV) :: IO Int
15

gradOpM :: Monad m => OpM m n a b -> Vec n a -> m (Vec n a)

The monadic version of gradOp, for OpMs.

gradOpM' :: Monad m => OpM m n a b -> Vec n a -> m (b, Vec n a)

The monadic version of gradOp', for OpMs.

gradOpWithM :: Monad m => OpM m n a b -> Vec n a -> b -> m (Vec n a)

The monadic version of gradOpWith, for OpMs.

gradOpWithM' :: Monad m => OpM m n a b -> Vec n a -> Maybe b -> m (Vec n a)

The monadic version of gradOpWith', for OpMs.

runOpM' :: Functor m => OpM m n a b -> Vec n a -> m (b, Maybe b -> m (Vec n a))

The monadic version of runOp, for OpMs.

>>> runOpM (op2 (*)) (3 :+ 5 :+ ØV) :: IO Int
15

Creation

op0 :: a -> Op N0 b a

Create an Op that takes no inputs and always returns the given value.

There is no gradient, of course (using gradOp will give you an empty vector), because there is no input to have a gradient of.

>>> gradOp' (op0 10) ØV
(10, ØV)

For a constant Op that takes input and ignores it, see opConst.

Note that because this returns an Op, it can be used with any function that expects an OpM or OpB, as well.

opConst :: forall n a b. (Known Nat n, Num b) => a -> Op n b a

An Op that ignores all of its inputs and returns a given constant value.

>>> gradOp' (opConst 10) (1 :+ 2 :+ 3 :+ ØV)
(10, 0 :+ 0 :+ 0 :+ ØV)

composeOp :: forall m n o a b c. (Monad m, Num a, Known Nat n) => VecT o (OpM m n a) b -> OpM m o b c -> OpM m n a c

Compose OpMs together, similar to .. But, because all OpMs are \(\mathbb{R}^N \rightarrow \mathbb{R}\), this is more like sequence for functions, or liftAN.

That is, given an o of OpM m n a bs, it can compose them with an OpM m o b c to create an OpM m o a c.

composeOp1 :: forall m n a b c. (Monad m, Num a, Known Nat n) => OpM m n a b -> OpM m N1 b c -> OpM m n a c

Convenient wrappver over composeOp for the case where the second function only takes one input, so the two OpMs can be directly piped together, like for ..

(~.) :: forall m n a b c. (Monad m, Num a, Known Nat n) => OpM m N1 b c -> OpM m n a b -> OpM m n a c infixr 9

Convenient infix synonym for (flipped) composeOp1. Meant to be used just like .:

op1 negate            :: Op '[a]   a
op2 (+)               :: Op '[a,a] a

op1 negate ~. op2 (+) :: Op '[a, a] a

Automatic creation using the ad library

op1 :: Num a => (forall s. AD s (Forward a) -> AD s (Forward a)) -> Op N1 a a

Automatically create an Op of a numerical function taking one argument. Uses diff, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (op1 (recip . negate)) (5 :+ ØV)
(-0.2, 0.04 :+ ØV)

op2 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a) -> Op N2 a a

Automatically create an Op of a numerical function taking two arguments. Uses grad, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (op2 (\x y -> x * sqrt y)) (3 :+ 4 :+ ØV)
(6.0, 2.0 :+ 0.75 :+ ØV)

op3 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a -> Reverse s a) -> Op N3 a a

Automatically create an Op of a numerical function taking three arguments. Uses grad, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (op3 (\x y z -> (x * sqrt y)**z)) (3 :+ 4 :+ 2 :+ ØV)
(36.0, 24.0 :+ 9.0 :+ 64.503 :+ ØV)

opN :: (Num a, Known Nat n) => (forall s. Reifies s Tape => Vec n (Reverse s a) -> Reverse s a) -> Op n a a

Automatically create an Op of a numerical function taking multiple arguments. Uses grad, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (opN (\(x :+ y :+ Ø) -> x * sqrt y)) (3 :+ 4 :+ ØV)
(6.0, 2.0 :+ 0.75 :+ ØV)

type family Replicate (n :: N) (a :: k) = (as :: [k]) | as -> n where ...

Replicate n a is a list of as repeated n times.

>>> :kind! Replicate N3 Int
'[Int, Int, Int]
>>> :kind! Replicate N5 Double
'[Double, Double, Double, Double, Double]

Equations

Replicate Z a = '[]

Replicate (S n) a = a ': Replicate n a

Giving gradients directly

op1' :: (a -> (b, Maybe b -> a)) -> Op N1 a b

Create an Op of a function taking one input, by giving its explicit derivative. The function should return a tuple containing the result of the function, and also a function taking the derivative of the result and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R} \rightarrow \mathbb{R}\cr y &= f(x)\cr z &= g(y) } \]

Then the derivative \( \frac{dz}{dx} \), it would be:

\[ \frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx} \]

If our Op represents \(f\), then the second item in the resulting tuple should be a function that takes \(\frac{dz}{dy}\) and returns \(\frac{dz}{dx}\).

If the input is Nothing, then \(\frac{dz}{dy}\) should be taken to be \(1\).

As an example, here is an Op that squares its input:

square :: Num a => Op N1 a a
square = op1' $ \x -> (x*x, \case Nothing -> 2 * x
                                  Just d  -> 2 * d * x
                      )

Remember that, generally, end users shouldn't directly construct Ops; they should be provided by libraries or generated automatically.

For numeric functions, single-input Ops can be generated automatically using op1.

op2' :: (a -> a -> (b, Maybe b -> (a, a))) -> Op N2 a b

Create an Op of a function taking two inputs, by giving its explicit gradient. The function should return a tuple containing the result of the function, and also a function taking the derivative of the result and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R}^2 \rightarrow \mathbb{R}\cr z &= f(x, y)\cr k &= g(z) } \]

Then the gradient \( \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> \) would be:

\[ \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> = \left< \frac{dk}{dz} \frac{\partial z}{dx}, \frac{dk}{dz} \frac{\partial z}{dy} \right> \]

If our Op represents \(f\), then the second item in the resulting tuple should be a function that takes \(\frac{dk}{dz}\) and returns \( \left< \frac{\partial k}{dx}, \frac{\partial k}{dx} \right> \).

If the input is Nothing, then \(\frac{dk}{dz}\) should be taken to be \(1\).

As an example, here is an Op that multiplies its inputs:

mul :: Num a => Op N2 a a
mul = op2' $ \x y -> (x*y, \case Nothing -> (y  , x  )
                                 Just d  -> (d*y, x*d)
                     )

Remember that, generally, end users shouldn't directly construct Ops; they should be provided by libraries or generated automatically.

For numeric functions, two-input Ops can be generated automatically using op2.

op3' :: (a -> a -> a -> (b, Maybe b -> (a, a, a))) -> Op N3 a b

Create an Op of a function taking three inputs, by giving its explicit gradient. See documentation for op2' for more details.

Utility

pattern (:+) :: forall a n. a -> Vec n a -> Vec (S n) a infixr 4

(*:) :: f a -> f a -> VecT k (S (S Z)) f a infix 5

(+:) :: a -> a -> Vec (S (S Z)) a infix 5

head' :: VecT k (S n) f a -> f a

Nat type synonyms

type N0 = Z

Convenient aliases for low-value Peano numbers.

type N1 = S N0

type N2 = S N1

type N3 = S N2

type N4 = S N3

type N5 = S N4

type N6 = S N5

type N7 = S N6

type N8 = S N7

type N9 = S N8

type N10 = S N9