Numeric.Backprop.Op

Provides the Op (and OpM) type and combinators, which represent differentiable functions/operations on values, and are used by the library to perform back-propagation.

Note that Op is a subset or subtype of OpM, and so, any function that expects an OpM m as a (or an OpB s as a) can be given an Op as a and it'll work just fine.

Implementation

Ops contain information on a function as well as its gradient, but provides that information in a way that allows them to be "chained".

For example, for a function

\[ f : \mathbb{R}^n \rightarrow \mathbb{R} \]

We might want to apply a function \(g\) to the result we get, to get our "final" result:

\[ \eqalign{ y &= f(\mathbf{x})\cr z &= g(y) } \]

Now, we might want the gradient \(\nabla z\) with respect to \(\mathbf{x}\), or \(\nabla_\mathbf{x} z\). Explicitly, this is:

\[ \nabla_\mathbf{x} z = \left< \frac{\partial z}{\partial x_1}, \frac{\partial z}{\partial x_2}, \ldots \right> \]

We can compute that by multiplying the total derivative of \(z\) with respect to \(y\) (that is, \(\frac{dz}{dy}\)) with the gradient of \(f\)) itself:

\[ \eqalign{ \nabla_\mathbf{x} z &= \frac{dz}{dy} \left< \frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots \right>\cr \nabla_\mathbf{x} z &= \frac{dz}{dy} \nabla_\mathbf{x} y } \]

So, to create an Op as a with the Op constructor (or an OpM with the OpM constructor), you give a function that returns a tuple, containing:

1. An a: The result of the function
2. An Maybe a -> Tuple as: A function that, when given \(\frac{dz}{dy}\) (in a Just), returns the total gradient \(\nabla_z \mathbf{x}\). If the function is given is given Nothing, then \(\frac{dz}{dy}\) should be taken to be 1. In other words, you would simply need to return \(\nabla_y \mathbf{x}\), unchanged. That is, an input of Nothing indicates that the "final result" is just simply \(f(\mathbf{x})\), and not some \(g(f(\mathbf{x}))\).

This is done so that Ops can easily be "chained" together, one after the other. If you have an Op for \(f\) and an Op for \(g\), you can compute the gradient of \(f\) knowing that the result target is \(g \circ f\).

Note that end users should probably never be required to construct an Op or OpM explicitly this way. Instead, libraries should provide carefuly pre-constructed ones, or provide ways to generate them automatically (like op1, op2, and op3 here).

For examples of Ops implemented from scratch, see the implementations of +., -., recipOp, sinOp, etc.

type Op as a = forall m. Monad m => OpM m as a

An Op as a describes a differentiable function from as to a.

For example, a value of type

Op '[Int, Bool] Double

is a function from an Int and a Bool, returning a Double. It can be differentiated to give a gradient of an Int and a Bool if given a total derivative for the Double. If we call Bool \(2\), then, mathematically, it is akin to a:

\[ f : \mathbb{Z} \times 2 \rightarrow \mathbb{R} \]

See runOp, gradOp, and gradOpWith for examples on how to run it, and Op for instructions on creating it.

This type is abstracted over using the pattern synonym with constructor Op, so you can create one from scratch with it. However, it's simplest to create it using op2', op1', op2', and op3' helper smart constructors And, if your function is a numeric function, they can even be created automatically using op1, op2, op3, and opN with a little help from Numeric.AD from the ad library.

Note that this type is a subset or subtype of OpM (and also of OpB). So, if a function ever expects an OpM m as a (or a OpB), you can always provide an Op as a instead.

Many functions in this library will expect an OpM m as a (or an OpB s as a), and in all of these cases, you can provide an Op as a.

pattern Op :: forall as a. (Tuple as -> (a, Maybe a -> Tuple as)) -> Op as a

Construct an Op by giving a function creating the result, and also a continuation on how to create the gradient, given the total derivative of a.

See the module documentation for Numeric.Backprop.Op for more details on the function that this constructor and OpM expect.

newtype OpM m as a

An OpM m as a represents a differentiable (monadic) function from as to a, in the context of a Monad m.

For example, an

OpM IO '[Int, Bool] Double

would be a function that takes an Int and a Bool and returns a Double (in IO). It can be differentiated to give a gradient of an Int and a Bool (also in IO) if given the total derivative for the Double.

Note that an OpM is a superclass of Op, so any function that expects an OpM m as a can also accept an Op as a.

See runOpM, gradOpM, and gradOpWithM for examples on how to run it.

Constructors

OpM (Tuple as -> m (a, Maybe a -> m (Tuple as)))

Construct an OpM by giving a (monadic) function creating the result, and also a continuation on how to create the gradient, given the total derivative of a.

See the module documentation for Numeric.Backprop.Op for more details on the function that this constructor and Op expect.

Instances

(Monad m, Known [*] (Length *) as, Every * Floating as, Every * Fractional as, Every * Num as, Floating a) => Floating (OpM m as a)

Methods

pi :: OpM m as a

exp :: OpM m as a -> OpM m as a

log :: OpM m as a -> OpM m as a

sqrt :: OpM m as a -> OpM m as a

(**) :: OpM m as a -> OpM m as a -> OpM m as a

logBase :: OpM m as a -> OpM m as a -> OpM m as a

sin :: OpM m as a -> OpM m as a

cos :: OpM m as a -> OpM m as a

tan :: OpM m as a -> OpM m as a

asin :: OpM m as a -> OpM m as a

acos :: OpM m as a -> OpM m as a

atan :: OpM m as a -> OpM m as a

sinh :: OpM m as a -> OpM m as a

cosh :: OpM m as a -> OpM m as a

tanh :: OpM m as a -> OpM m as a

asinh :: OpM m as a -> OpM m as a

acosh :: OpM m as a -> OpM m as a

atanh :: OpM m as a -> OpM m as a

log1p :: OpM m as a -> OpM m as a

expm1 :: OpM m as a -> OpM m as a

log1pexp :: OpM m as a -> OpM m as a

log1mexp :: OpM m as a -> OpM m as a

(Monad m, Known [*] (Length *) as, Every * Fractional as, Every * Num as, Fractional a) => Fractional (OpM m as a)

Methods

(/) :: OpM m as a -> OpM m as a -> OpM m as a

recip :: OpM m as a -> OpM m as a

fromRational :: Rational -> OpM m as a

(Monad m, Known [*] (Length *) as, Every * Num as, Num a) => Num (OpM m as a)

Methods

(+) :: OpM m as a -> OpM m as a -> OpM m as a

(-) :: OpM m as a -> OpM m as a -> OpM m as a

(*) :: OpM m as a -> OpM m as a -> OpM m as a

negate :: OpM m as a -> OpM m as a

abs :: OpM m as a -> OpM m as a

signum :: OpM m as a -> OpM m as a

fromInteger :: Integer -> OpM m as a

Tuple Types

See Numeric.Backprop for a mini-tutorial on Prod and Tuple

data Prod k f a :: forall k. (k -> *) -> [k] -> * where

Constructors

Ø :: Prod k f ([] k)

(:<) :: Prod k f ((:) k a1 as) infixr 5

Instances

Witness ØC ØC (Prod k f (Ø k))

Associated Types

type WitnessC (ØC :: Constraint) (ØC :: Constraint) (Prod k f (Ø k)) :: Constraint

Methods

(\\) :: ØC => (ØC -> r) -> Prod k f (Ø k) -> r

IxFunctor1 k [k] (Index k) (Prod k)

Methods

imap1 :: (forall a. i b a -> f a -> g a) -> t f b -> t g b

IxFoldable1 k [k] (Index k) (Prod k)

Methods

ifoldMap1 :: Monoid m => (forall a. i b a -> f a -> m) -> t f b -> m

IxTraversable1 k [k] (Index k) (Prod k)

Methods

itraverse1 :: Applicative h => (forall a. i b a -> f a -> h (g a)) -> t f b -> h (t g b)

Functor1 [k] k (Prod k)

Methods

map1 :: (forall a. f a -> g a) -> t f b -> t g b

Foldable1 [k] k (Prod k)

Methods

foldMap1 :: Monoid m => (forall a. f a -> m) -> t f b -> m

Traversable1 [k] k (Prod k)

Methods

traverse1 :: Applicative h => (forall a. f a -> h (g a)) -> t f b -> h (t g b)

TestEquality k f => TestEquality [k] (Prod k f)

Methods

testEquality :: f a -> f b -> Maybe ((Prod k f :~: a) b)

BoolEquality k f => BoolEquality [k] (Prod k f)

Methods

boolEquality :: f a -> f b -> Boolean ((Prod k f == a) b)

Eq1 k f => Eq1 [k] (Prod k f)

Methods

eq1 :: f a -> f a -> Bool

neq1 :: f a -> f a -> Bool

Ord1 k f => Ord1 [k] (Prod k f)

Methods

compare1 :: f a -> f a -> Ordering

(<#) :: f a -> f a -> Bool

(>#) :: f a -> f a -> Bool

(<=#) :: f a -> f a -> Bool

(>=#) :: f a -> f a -> Bool

Show1 k f => Show1 [k] (Prod k f)

Methods

showsPrec1 :: Int -> f a -> ShowS

show1 :: f a -> String

Read1 k f => Read1 [k] (Prod k f)

Methods

readsPrec1 :: Int -> ReadS (Some (Prod k f) f)

(Known [k] (Length k) as, Every k (Known k f) as) => Known [k] (Prod k f) as

Associated Types

type KnownC (Prod k f) (as :: Prod k f -> *) (a :: Prod k f) :: Constraint

Methods

known :: as a

(Witness p q (f a1), Witness s t (Prod a f as)) => Witness (p, s) (q, t) (Prod a f ((:<) a a1 as))

Associated Types

type WitnessC ((p, s) :: Constraint) ((q, t) :: Constraint) (Prod a f ((:<) a a1 as)) :: Constraint

Methods

(\\) :: (p, s) => ((q, t) -> r) -> Prod a f ((a :< a1) as) -> r

ListC ((<$>) Constraint * Eq ((<$>) * k f as)) => Eq (Prod k f as)

Methods

(==) :: Prod k f as -> Prod k f as -> Bool

(/=) :: Prod k f as -> Prod k f as -> Bool

(ListC ((<$>) Constraint * Eq ((<$>) * k f as)), ListC ((<$>) Constraint * Ord ((<$>) * k f as))) => Ord (Prod k f as)

Methods

compare :: Prod k f as -> Prod k f as -> Ordering

(<) :: Prod k f as -> Prod k f as -> Bool

(<=) :: Prod k f as -> Prod k f as -> Bool

(>) :: Prod k f as -> Prod k f as -> Bool

(>=) :: Prod k f as -> Prod k f as -> Bool

max :: Prod k f as -> Prod k f as -> Prod k f as

min :: Prod k f as -> Prod k f as -> Prod k f as

ListC ((<$>) Constraint * Show ((<$>) * k f as)) => Show (Prod k f as)

Methods

showsPrec :: Int -> Prod k f as -> ShowS

show :: Prod k f as -> String

showList :: [Prod k f as] -> ShowS

type WitnessC ØC ØC (Prod k f (Ø k))

type WitnessC ØC ØC (Prod k f (Ø k)) = ØC

type KnownC [k] (Prod k f) as

type KnownC [k] (Prod k f) as = (Known [k] (Length k) as, Every k (Known k f) as)

type WitnessC (p, s) (q, t) (Prod a f ((:<) a a1 as))

type WitnessC (p, s) (q, t) (Prod a f ((:<) a a1 as)) = (Witness p q (f a1), Witness s t (Prod a f as))

type Tuple = Prod * I

A Prod of simple Haskell types.

newtype I a :: * -> *

Constructors

I

Fields

• getI :: a

Instances

Monad I

Methods

(>>=) :: I a -> (a -> I b) -> I b

(>>) :: I a -> I b -> I b

return :: a -> I a

fail :: String -> I a

Functor I

Methods

fmap :: (a -> b) -> I a -> I b

(<$) :: a -> I b -> I a

Applicative I

Methods

pure :: a -> I a

(<*>) :: I (a -> b) -> I a -> I b

(*>) :: I a -> I b -> I b

(<*) :: I a -> I b -> I a

Foldable I

Methods

fold :: Monoid m => I m -> m

foldMap :: Monoid m => (a -> m) -> I a -> m

foldr :: (a -> b -> b) -> b -> I a -> b

foldr' :: (a -> b -> b) -> b -> I a -> b

foldl :: (b -> a -> b) -> b -> I a -> b

foldl' :: (b -> a -> b) -> b -> I a -> b

foldr1 :: (a -> a -> a) -> I a -> a

foldl1 :: (a -> a -> a) -> I a -> a

toList :: I a -> [a]

null :: I a -> Bool

length :: I a -> Int

elem :: Eq a => a -> I a -> Bool

maximum :: Ord a => I a -> a

minimum :: Ord a => I a -> a

sum :: Num a => I a -> a

product :: Num a => I a -> a

Traversable I

Methods

traverse :: Applicative f => (a -> f b) -> I a -> f (I b)

sequenceA :: Applicative f => I (f a) -> f (I a)

mapM :: Monad m => (a -> m b) -> I a -> m (I b)

sequence :: Monad m => I (m a) -> m (I a)

Witness p q a => Witness p q (I a)

Associated Types

type WitnessC (p :: Constraint) (q :: Constraint) (I a) :: Constraint

Methods

(\\) :: p => (q -> r) -> I a -> r

Eq a => Eq (I a)

Methods

(==) :: I a -> I a -> Bool

(/=) :: I a -> I a -> Bool

Num a => Num (I a)

Methods

(+) :: I a -> I a -> I a

(-) :: I a -> I a -> I a

(*) :: I a -> I a -> I a

negate :: I a -> I a

abs :: I a -> I a

signum :: I a -> I a

fromInteger :: Integer -> I a

Ord a => Ord (I a)

Methods

compare :: I a -> I a -> Ordering

(<) :: I a -> I a -> Bool

(<=) :: I a -> I a -> Bool

(>) :: I a -> I a -> Bool

(>=) :: I a -> I a -> Bool

max :: I a -> I a -> I a

min :: I a -> I a -> I a

Show a => Show (I a)

Methods

showsPrec :: Int -> I a -> ShowS

show :: I a -> String

showList :: [I a] -> ShowS

type WitnessC p q (I a)

type WitnessC p q (I a) = Witness p q a

Running

Pure

runOp :: Op as a -> Tuple as -> a

Run the function that an Op encodes, to get the result.

>>> runOp (op2 (*)) (3 ::< 5 ::< Ø)
15

gradOp :: Op as a -> Tuple as -> Tuple as

Run the function that an Op encodes, and get the gradient of the output with respect to the inputs.

>>> gradOp (op2 (*)) (3 ::< 5 ::< Ø)
5 ::< 3 ::< Ø
-- the gradient of x*y is (y, x)

gradOp' :: Op as a -> Tuple as -> (a, Tuple as)

Run the function that an Op encodes, to get the resulting output and also its gradient with respect to the inputs.

>>> gradOpM' (op2 (*)) (3 ::< 5 ::< Ø) :: IO (Int, Tuple '[Int, Int])
(15, 5 ::< 3 ::< Ø)

gradOpWith

Arguments

:: Op as a

Op to run

-> Tuple as

Inputs to run it with

-> a

The total derivative of the result

-> Tuple as

The gradient

Run the function that an Op encodes, and get the gradient of a "final result" with respect to the inputs, given the total derivative of the output with the final result.

See gradOp and the module documentaiton for Numeric.Backprop.Op for more information.

gradOpWith'

Arguments

:: Op as a

Op to run

-> Tuple as

Inputs to run it with

-> Maybe a

If Just, taken as the total derivative of the result. If Nothing, assumes that the result is the final result.

-> Tuple as

The gradient

A combination of gradOp and gradOpWith. The third argument is (optionally) the total derivative the result. Give Nothing and it is assumed that the result is the final result (and the total derivative is 1), and this behaves the same as gradOp. Give Just d and it uses the d as the total derivative of the result, and this behaves like gradOpWith.

See gradOp and the module documentaiton for Numeric.Backprop.Op for more information.

runOp'

Arguments

:: Op as a

Op to run

-> Tuple as

Inputs

-> (a, Maybe a -> Tuple as)

Result, and continuation to get the gradient

A combination of runOp and gradOpWith'. Given an Op and inputs, returns the result of the Op and a continuation that gives its gradient.

The continuation takes the total derivative of the result as input. See documenation for gradOpWith' and module documentation for Numeric.Backprop.Op for more information.

Monadic

runOpM :: Functor m => OpM m as a -> Tuple as -> m a

The monadic version of runOp, for OpMs.

>>> runOpM (op2 (*)) (3 ::< 5 ::< Ø) :: IO Int
15

gradOpM :: Monad m => OpM m as a -> Tuple as -> m (Tuple as)

The monadic version of gradOp, for OpMs.

gradOpM' :: Monad m => OpM m as a -> Tuple as -> m (a, Tuple as)

The monadic version of gradOp', for OpMs.

gradOpWithM

Arguments

:: Monad m

=> OpM m as a

OpM to run

-> Tuple as

Inputs to run it with

-> a

The total derivative of the result

-> m (Tuple as)

the gradient

The monadic version of gradOpWith, for OpMs.

gradOpWithM'

Arguments

:: Monad m

=> OpM m as a

OpM to run

-> Tuple as

Inputs to run it with

-> Maybe a

If Just, taken as the total derivative of the result. If Nothing, assumes that the result is the final result.

-> m (Tuple as)

The gradient

The monadic version of gradOpWith', for OpMs.

runOpM'

Arguments

:: OpM m as a

OpM to run

-> Tuple as

Inputs

-> m (a, Maybe a -> m (Tuple as))

Result, and continuation to get the gradient

A combination of runOpM and gradOpWithM'. Given an OpM and inputs, returns the result of the OpM and a continuation that gives its gradient.

The continuation takes the total derivative of the result as input. See documenation for gradOpWithM' and module documentation for Numeric.Backprop.Op for more information.

Manipulation

composeOp

Arguments

:: (Monad m, Every Num as, Known Length as)

=> Prod (OpM m as) bs

Prod of OpMs taking as and returning different b in bs

-> OpM m bs c

OpM taking eac of the bs from the input Prod.

-> OpM m as c

Composed OpM

Compose OpMs together, similar to .. But, because all OpMs are \(\mathbb{R}^N \rightarrow \mathbb{R}\), this is more like sequence for functions, or liftAN.

That is, given an OpM m as b1, an OpM m as b2, and an OpM m as b3, it can compose them with an OpM m '[b1,b2,b3] c to create an OpM m as c.

composeOp1 :: (Monad m, Every Num as, Known Length as) => OpM m as b -> OpM m '[b] c -> OpM m as c

Convenient wrapper over composeOp for the case where the second function only takes one input, so the two OpMs can be directly piped together, like for ..

(~.) :: (Monad m, Known Length as, Every Num as) => OpM m '[b] c -> OpM m as b -> OpM m as c infixr 9

Convenient infix synonym for (flipped) composeOp1. Meant to be used just like .:

op1 negate            :: Op '[a]   a
op2 (+)               :: Op '[a,a] a

op1 negate ~. op2 (+) :: Op '[a, a] a

composeOp'

Arguments

:: (Monad m, Every Num as)

=> Length as

-> Prod (OpM m as) bs

Prod of OpMs taking as and returning different b in bs

-> OpM m bs c

OpM taking eac of the bs from the input Prod.

-> OpM m as c

Composed OpM

A version of composeOp taking explicit Length, indicating the number of inputs expected and their types.

Requiring an explicit Length is mostly useful for rare "extremely polymorphic" situations, where GHC can't infer the type and length of the the expected input tuple. If you ever actually explicitly write down as as a list of types, you should be able to just use composeOp.

composeOp1' :: (Monad m, Every Num as) => Length as -> OpM m as b -> OpM m '[b] c -> OpM m as c

A version of composeOp1 taking explicit Length, indicating the number of inputs expected and their types.

Requiring an explicit Length is mostly useful for rare "extremely polymorphic" situations, where GHC can't infer the type and length of the the expected input tuple. If you ever actually explicitly write down as as a list of types, you should be able to just use composeOp1.

Creation

op0 :: a -> Op '[] a

Create an Op that takes no inputs and always returns the given value.

There is no gradient, of course (using gradOp will give you an empty tuple), because there is no input to have a gradient of.

>>> gradOp' (op0 10) Ø
(10, Ø)

For a constant Op that takes input and ignores it, see opConst and opConst'.

Note that because this returns an Op, it can be used with any function that expects an OpM or OpB, as well.

opConst :: forall as a. (Every Num as, Known Length as) => a -> Op as a

An Op that ignores all of its inputs and returns a given constant value.

>>> gradOp' (opConst 10) (1 ::< 2 ::< 3 ::< Ø)
(10, 0 ::< 0 ::< 0 ::< Ø)

opConst' :: forall as a. Every Num as => Length as -> a -> Op as a

A version of opConst taking explicit Length, indicating the number of inputs and their types.

Requiring an explicit Length is mostly useful for rare "extremely polymorphic" situations, where GHC can't infer the type and length of the the expected input tuple. If you ever actually explicitly write down as as a list of types, you should be able to just use opConst.

Automatic creation using the ad library

op1 :: Num a => (forall s. AD s (Forward a) -> AD s (Forward a)) -> Op '[a] a

Automatically create an Op of a numerical function taking one argument. Uses diff, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (op1 (recip . negate)) (5 ::< Ø)
(-0.2, 0.04 ::< Ø)

op2 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a) -> Op '[a, a] a

Automatically create an Op of a numerical function taking two arguments. Uses grad, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (op2 (\x y -> x * sqrt y)) (3 ::< 4 ::< Ø)
(6.0, 2.0 ::< 0.75 ::< Ø)

op3 :: Num a => (forall s. Reifies s Tape => Reverse s a -> Reverse s a -> Reverse s a -> Reverse s a) -> Op '[a, a, a] a

Automatically create an Op of a numerical function taking three arguments. Uses grad, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (op3 (\x y z -> (x * sqrt y)**z)) (3 ::< 4 ::< 2 ::< Ø)
(36.0, 24.0 ::< 9.0 ::< 64.503 ::< Ø)

opN :: (Num a, Known Nat n) => (forall s. Reifies s Tape => Vec n (Reverse s a) -> Reverse s a) -> Op (Replicate n a) a

Automatically create an Op of a numerical function taking multiple arguments. Uses grad, and so can take any numerical function polymorphic over the standard numeric types.

>>> gradOp' (opN (\(x :+ y :+ Ø) -> x * sqrt y)) (3 ::< 4 ::< Ø)
(6.0, 2.0 ::< 0.75 ::< Ø)

type family Replicate (n :: N) (a :: k) = (as :: [k]) | as -> n where ...

Replicate n a is a list of as repeated n times.

>>> :kind! Replicate N3 Int
'[Int, Int, Int]
>>> :kind! Replicate N5 Double
'[Double, Double, Double, Double, Double]

Equations

Replicate Z a = '[]

Replicate (S n) a = a ': Replicate n a

Giving gradients directly

op1' :: (a -> (b, Maybe b -> a)) -> Op '[a] b

Create an Op of a function taking one input, by giving its explicit derivative. The function should return a tuple containing the result of the function, and also a function taking the derivative of the result and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R} \rightarrow \mathbb{R}\cr y &= f(x)\cr z &= g(y) } \]

Then the derivative \( \frac{dz}{dx} \), it would be:

\[ \frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx} \]

If our Op represents \(f\), then the second item in the resulting tuple should be a function that takes \(\frac{dz}{dy}\) and returns \(\frac{dz}{dx}\).

If the input is Nothing, then \(\frac{dz}{dy}\) should be taken to be \(1\).

As an example, here is an Op that squares its input:

square :: Num a => Op '[a] a
square = op1' $ \x -> (x*x, \case Nothing -> 2 * x
                                  Just d  -> 2 * d * x
                      )

Remember that, generally, end users shouldn't directly construct Ops; they should be provided by libraries or generated automatically.

For numeric functions, single-input Ops can be generated automatically using op1.

op2' :: (a -> b -> (c, Maybe c -> (a, b))) -> Op '[a, b] c

Create an Op of a function taking two inputs, by giving its explicit gradient. The function should return a tuple containing the result of the function, and also a function taking the derivative of the result and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R}^2 \rightarrow \mathbb{R}\cr z &= f(x, y)\cr k &= g(z) } \]

Then the gradient \( \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> \) would be:

\[ \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> = \left< \frac{dk}{dz} \frac{\partial z}{dx}, \frac{dk}{dz} \frac{\partial z}{dy} \right> \]

If our Op represents \(f\), then the second item in the resulting tuple should be a function that takes \(\frac{dk}{dz}\) and returns \( \left< \frac{\partial k}{dx}, \frac{\partial k}{dx} \right> \).

If the input is Nothing, then \(\frac{dk}{dz}\) should be taken to be \(1\).

As an example, here is an Op that multiplies its inputs:

mul :: Num a => Op '[a, a] a
mul = op2' $ \x y -> (x*y, \case Nothing -> (y  , x  )
                                 Just d  -> (d*y, x*d)
                     )

Remember that, generally, end users shouldn't directly construct Ops; they should be provided by libraries or generated automatically.

For numeric functions, two-input Ops can be generated automatically using op2.

op3' :: (a -> b -> c -> (d, Maybe d -> (a, b, c))) -> Op '[a, b, c] d

Create an Op of a function taking three inputs, by giving its explicit gradient. See documentation for op2' for more details.

From Isomorphisms

opCoerce :: Num a => Coercible a b => Op '[a] b

An Op that coerces an item into another item whose type has the same runtime representation. Requires the input to be an instance of Num.

>>> gradOp' opCoerce (Identity 5) :: (Int, Identity Int)
(5, Identity 1)

opCoerce = opIso coerced

opTup :: (Every Num as, Known Length as) => Length as -> Op as (Tuple as)

An Op that takes as and returns exactly the input tuple.

>>> gradOp' opTup (1 ::< 2 ::< 3 ::< Ø)
(1 ::< 2 ::< 3 ::< Ø, 1 ::< 1 ::< 1 ::< Ø)

opIso :: Num a => Iso' a b -> Op '[a] b

An Op that runs the input value through the isomorphism encoded in the Iso. Requires the input to be an instance of Num.

Warning: This is unsafe! It assumes that the isomorphisms themselves have derivative 1, so will break for things like exponentiating. Basically, don't use this for any "numeric" isomorphisms.

opTup' :: Every Num as => Length as -> Op as (Tuple as)

A version of opTup taking explicit Length, indicating the number of inputs expected and their types.

Requiring an explicit Length is mostly useful for rare "extremely polymorphic" situations, where GHC can't infer the type and length of the the expected input tuple. If you ever actually explicitly write down as as a list of types, you should be able to just use opTup.

Utility

pattern (:>) :: forall k f a b. f a -> f b -> Prod k f ((:) k a ((:) k b ([] k))) infix 6

Construct a two element Prod. Since the precedence of (:>) is higher than (:<), we can conveniently write lists like:

>>> a :< b :> c

Which is identical to:

>>> a :< b :< c :< Ø

only :: f a -> Prod k f ((:) k a ([] k))

Build a singleton Prod.

head' :: Prod k f ((:<) k a as) -> f a

pattern (::<) :: forall a as. a -> Tuple as -> Tuple ((:<) * a as) infixr 5

Cons onto a Tuple.

only_ :: a -> Tuple ((:) * a ([] *))

Singleton Tuple.

Numeric Ops

Built-in ops for common numeric operations, implemented directly so that they are more efficient than using op1 / op2 etc.

The naming scheme is:

(+.) = op2 (+)
negateOp = op1 'negate

Note that the operators (like +.) are meant to be used in prefix form, like:

liftB2 (.+) v1 v2

(+.) :: Num a => Op '[a, a] a

Optimized version of op1 (+).

(-.) :: Num a => Op '[a, a] a

Optimized version of op1 (-).

(*.) :: Num a => Op '[a, a] a

Optimized version of op1 (*).

negateOp :: Num a => Op '[a] a

Optimized version of op1 negate.

absOp :: Num a => Op '[a] a

Optimized version of op1 abs.

signumOp :: Num a => Op '[a] a

Optimized version of op1 signum.

(/.) :: Fractional a => Op '[a, a] a

Optimized version of op1 (/).

recipOp :: Fractional a => Op '[a] a

Optimized version of op1 recip.

expOp :: Floating a => Op '[a] a

Optimized version of op1 exp.

logOp :: Floating a => Op '[a] a

Optimized version of op1 log.

sqrtOp :: Floating a => Op '[a] a

Optimized version of op1 sqrt.

(**.) :: Floating a => Op '[a, a] a

Optimized version of op1 (**).

logBaseOp :: Floating a => Op '[a, a] a

Optimized version of op2 logBase.

sinOp :: Floating a => Op '[a] a

Optimized version of op1 sin.

cosOp :: Floating a => Op '[a] a

Optimized version of op1 cos.

tanOp :: Floating a => Op '[a] a

Optimized version of op1 tan.

asinOp :: Floating a => Op '[a] a

Optimized version of op1 asin.

acosOp :: Floating a => Op '[a] a

Optimized version of op1 acos.

atanOp :: Floating a => Op '[a] a

Optimized version of op1 atan.

sinhOp :: Floating a => Op '[a] a

Optimized version of op1 sinh.

coshOp :: Floating a => Op '[a] a

Optimized version of op1 cosh.

tanhOp :: Floating a => Op '[a] a

Optimized version of op1 tanh.

asinhOp :: Floating a => Op '[a] a

Optimized version of op1 asinh.

acoshOp :: Floating a => Op '[a] a

Optimized version of op1 acosh.

atanhOp :: Floating a => Op '[a] a

Optimized version of op1 atanh.