Numeric.Backprop.Op

Provides the Op type and combinators, which represent differentiable functions/operations on values, and are used internally by the library to perform back-propagation.

Users of the library can ignore this module for the most part. Library authors defining backpropagatable primitives for their functions are recommend to simply use op0, op1, op2, op3, which are re-exported in Numeric.Backprop. However, authors who want more options in defining their primtive functions might find some of these functions useful.

Note that if your entire function is a single non-branching composition of functions, Op and its utility functions alone are sufficient to differentiate/backprop. However, this happens rarely in practice.

To use these Ops with the backprop library, they can be made to work with BVars using liftOp, liftOp1, liftOp2, and liftOp3.

If you are writing a library, see https://backprop.jle.im/06-equipping-your-library.html for a guide for equipping your library with backpropatable operations using Ops.

See also this guide for writing Ops manually on your own numerical functions.

Implementation

Ops contain information on a function as well as its gradient, but provides that information in a way that allows them to be "chained".

For example, for a function

\[ f : \mathbb{R}^n \rightarrow \mathbb{R} \]

We might want to apply a function \(g\) to the result we get, to get our "final" result:

\[ \eqalign{ y &= f(\mathbf{x})\cr z &= g(y) } \]

Now, we might want the gradient \(\nabla z\) with respect to \(\mathbf{x}\), or \(\nabla_\mathbf{x} z\). Explicitly, this is:

\[ \nabla_\mathbf{x} z = \left< \frac{\partial z}{\partial x_1}, \frac{\partial z}{\partial x_2}, \ldots \right> \]

We can compute that by multiplying the total derivative of \(z\) with respect to \(y\) (that is, \(\frac{dz}{dy}\)) with the gradient of \(f\)) itself:

\[ \eqalign{ \nabla_\mathbf{x} z &= \frac{dz}{dy} \left< \frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots \right>\cr \nabla_\mathbf{x} z &= \frac{dz}{dy} \nabla_\mathbf{x} y } \]

So, to create an Op as a with the Op constructor, you give a function that returns a tuple, containing:

1. An a: The result of the function
2. An a -> Rec Identity as: A function that, when given \(\frac{dz}{dy}\), returns the total gradient \(\nabla_z \mathbf{x}\).

This is done so that Ops can easily be "chained" together, one after the other. If you have an Op for \(f\) and an Op for \(g\), you can compute the gradient of \(f\) knowing that the result target is \(g \circ f\).

See this guide for a detailed look on writing ops manually on your own numerical functions.

Note that end users should probably never be required to construct an Op explicitly this way. Instead, libraries should provide carefuly pre-constructed ones, or provide ways to generate them automatically (like op1, op2, and op3 here).

For examples of Ops implemented from scratch, see the implementations of +., -., recipOp, sinOp, etc.

See Numeric.Backprop.Op for a mini-tutorial on using Rec and 'Rec Identity'.

newtype Op as a

An Op as a describes a differentiable function from as to a.

For example, a value of type

Op '[Int, Bool] Double

is a function from an Int and a Bool, returning a Double. It can be differentiated to give a gradient of an Int and a Bool if given a total derivative for the Double. If we call Bool \(2\), then, mathematically, it is akin to a:

\[ f : \mathbb{Z} \times 2 \rightarrow \mathbb{R} \]

See runOp, gradOp, and gradOpWith for examples on how to run it, and Op for instructions on creating it.

It is simpler to not use this type constructor directly, and instead use the op2, op1, op2, and op3 helper smart constructors.

See Numeric.Backprop.Op for a mini-tutorial on using Rec and 'Rec Identity'.

To use an Op with the backprop library, see liftOp, liftOp1, liftOp2, and liftOp3.

Constructors

Op

Construct an Op by giving a function creating the result, and also a continuation on how to create the gradient, given the total derivative of a.

See the module documentation for Numeric.Backprop.Op for more details on the function that this constructor and Op expect.

Fields

• runOpWith :: Rec Identity as -> (a, a -> Rec Identity as)

  Run the function that the Op encodes, returning a continuation to compute the gradient, given the total derivative of a. See documentation for Numeric.Backprop.Op for more information.

Instances

(RecApplicative * as, AllConstrained * Floating as, AllConstrained * Fractional as, AllConstrained * Num as, Floating a) => Floating (Op as a)

Methods

pi :: Op as a

exp :: Op as a -> Op as a

log :: Op as a -> Op as a

sqrt :: Op as a -> Op as a

(**) :: Op as a -> Op as a -> Op as a

logBase :: Op as a -> Op as a -> Op as a

sin :: Op as a -> Op as a

cos :: Op as a -> Op as a

tan :: Op as a -> Op as a

asin :: Op as a -> Op as a

acos :: Op as a -> Op as a

atan :: Op as a -> Op as a

sinh :: Op as a -> Op as a

cosh :: Op as a -> Op as a

tanh :: Op as a -> Op as a

asinh :: Op as a -> Op as a

acosh :: Op as a -> Op as a

atanh :: Op as a -> Op as a

log1p :: Op as a -> Op as a

expm1 :: Op as a -> Op as a

log1pexp :: Op as a -> Op as a

log1mexp :: Op as a -> Op as a

(RecApplicative * as, AllConstrained * Num as, Fractional a) => Fractional (Op as a)

Methods

(/) :: Op as a -> Op as a -> Op as a

recip :: Op as a -> Op as a

fromRational :: Rational -> Op as a

(RecApplicative * as, AllConstrained * Num as, Num a) => Num (Op as a)

Methods

(+) :: Op as a -> Op as a -> Op as a

(-) :: Op as a -> Op as a -> Op as a

(*) :: Op as a -> Op as a -> Op as a

negate :: Op as a -> Op as a

abs :: Op as a -> Op as a

signum :: Op as a -> Op as a

fromInteger :: Integer -> Op as a

Tuple Types

Rec, from the vinyl library (in Data.Vinyl.Core) is a heterogeneous list/tuple type, which allows you to tuple together multiple values of different types and operate on them generically.

A Rec f '[a, b, c] contains an f a, an f b, and an f c, and is constructed by consing them together with :& (using RNil as nil):

Identity "hello" :& Identity True :& Identity 7.8 :& RNil    :: Rec I '[String, Bool, Double]
Const "hello" :& Const "world" :& Const "ok" :& RNil  :: Rec (C String) '[a, b, c]
Proxy :& Proxy :& Proxy :& RNil           :: Rec Proxy '[a, b, c]

So, in general:

x :: f a
y :: f b
z :: f c
x :& y :& z :& RNil :: Rec f '[a, b, c]

data Rec u (a :: u -> *) (b :: [u]) :: forall u. (u -> *) -> [u] -> * where

A record is parameterized by a universe u, an interpretation f and a list of rows rs. The labels or indices of the record are given by inhabitants of the kind u; the type of values at any label r :: u is given by its interpretation f r :: *.

Constructors

RNil :: Rec u a ([] u)

(:&) :: Rec u a ((:) u r rs) infixr 7

Instances

TestCoercion u f => TestCoercion [u] (Rec u f)

Methods

testCoercion :: f a -> f b -> Maybe (Coercion (Rec u f) a b)

TestEquality u f => TestEquality [u] (Rec u f)

Methods

testEquality :: f a -> f b -> Maybe ((Rec u f :~: a) b)

Eq (Rec u f ([] u))

Methods

(==) :: Rec u f [u] -> Rec u f [u] -> Bool

(/=) :: Rec u f [u] -> Rec u f [u] -> Bool

(Eq (f r), Eq (Rec a f rs)) => Eq (Rec a f ((:) a r rs))

Methods

(==) :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Bool

(/=) :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Bool

Ord (Rec u f ([] u))

Methods

compare :: Rec u f [u] -> Rec u f [u] -> Ordering

(<) :: Rec u f [u] -> Rec u f [u] -> Bool

(<=) :: Rec u f [u] -> Rec u f [u] -> Bool

(>) :: Rec u f [u] -> Rec u f [u] -> Bool

(>=) :: Rec u f [u] -> Rec u f [u] -> Bool

max :: Rec u f [u] -> Rec u f [u] -> Rec u f [u]

min :: Rec u f [u] -> Rec u f [u] -> Rec u f [u]

(Ord (f r), Ord (Rec a f rs)) => Ord (Rec a f ((:) a r rs))

Methods

compare :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Ordering

(<) :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Bool

(<=) :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Bool

(>) :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Bool

(>=) :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Bool

max :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs)

min :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs)

RecAll u f rs Show => Show (Rec u f rs)

Records may be shown insofar as their points may be shown. reifyConstraint is used to great effect here.

Methods

showsPrec :: Int -> Rec u f rs -> ShowS

show :: Rec u f rs -> String

showList :: [Rec u f rs] -> ShowS

Semigroup (Rec u f ([] u))

Methods

(<>) :: Rec u f [u] -> Rec u f [u] -> Rec u f [u]

sconcat :: NonEmpty (Rec u f [u]) -> Rec u f [u]

stimes :: Integral b => b -> Rec u f [u] -> Rec u f [u]

(Monoid (f r), Monoid (Rec a f rs)) => Semigroup (Rec a f ((:) a r rs))

Methods

(<>) :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs)

sconcat :: NonEmpty (Rec a f ((a ': r) rs)) -> Rec a f ((a ': r) rs)

stimes :: Integral b => b -> Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs)

Monoid (Rec u f ([] u))

Methods

mempty :: Rec u f [u]

mappend :: Rec u f [u] -> Rec u f [u] -> Rec u f [u]

mconcat :: [Rec u f [u]] -> Rec u f [u]

(Monoid (f r), Monoid (Rec a f rs)) => Monoid (Rec a f ((:) a r rs))

Methods

mempty :: Rec a f ((a ': r) rs)

mappend :: Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs) -> Rec a f ((a ': r) rs)

mconcat :: [Rec a f ((a ': r) rs)] -> Rec a f ((a ': r) rs)

Storable (Rec u f ([] u))

Methods

sizeOf :: Rec u f [u] -> Int

alignment :: Rec u f [u] -> Int

peekElemOff :: Ptr (Rec u f [u]) -> Int -> IO (Rec u f [u])

pokeElemOff :: Ptr (Rec u f [u]) -> Int -> Rec u f [u] -> IO ()

peekByteOff :: Ptr b -> Int -> IO (Rec u f [u])

pokeByteOff :: Ptr b -> Int -> Rec u f [u] -> IO ()

peek :: Ptr (Rec u f [u]) -> IO (Rec u f [u])

poke :: Ptr (Rec u f [u]) -> Rec u f [u] -> IO ()

(Storable (f r), Storable (Rec a f rs)) => Storable (Rec a f ((:) a r rs))

Methods

sizeOf :: Rec a f ((a ': r) rs) -> Int

alignment :: Rec a f ((a ': r) rs) -> Int

peekElemOff :: Ptr (Rec a f ((a ': r) rs)) -> Int -> IO (Rec a f ((a ': r) rs))

pokeElemOff :: Ptr (Rec a f ((a ': r) rs)) -> Int -> Rec a f ((a ': r) rs) -> IO ()

peekByteOff :: Ptr b -> Int -> IO (Rec a f ((a ': r) rs))

pokeByteOff :: Ptr b -> Int -> Rec a f ((a ': r) rs) -> IO ()

peek :: Ptr (Rec a f ((a ': r) rs)) -> IO (Rec a f ((a ': r) rs))

poke :: Ptr (Rec a f ((a ': r) rs)) -> Rec a f ((a ': r) rs) -> IO ()

Running

Pure

runOp :: Num a => Op as a -> Rec Identity as -> (a, Rec Identity as)

Run the function that an Op encodes, to get the resulting output and also its gradient with respect to the inputs.

>>> gradOp' (op2 (*)) (3 :& 5 :& RNil)
(15, 5 :& 3 :& RNil)

evalOp :: Op as a -> Rec Identity as -> a

Run the function that an Op encodes, to get the result.

>>> runOp (op2 (*)) (3 :& 5 :& RNil)
15

gradOp :: Num a => Op as a -> Rec Identity as -> Rec Identity as

Run the function that an Op encodes, and get the gradient of the output with respect to the inputs.

>>> gradOp (op2 (*)) (3 :& 5 :& RNil)
5 :& 3 :& RNil
-- the gradient of x*y is (y, x)

gradOp o xs = gradOpWith o xs 1

gradOpWith

Arguments

:: Op as a

Op to run

-> Rec Identity as

Inputs to run it with

-> a

The total derivative of the result.

-> Rec Identity as

The gradient

Get the gradient function that an Op encodes, with a third argument expecting the total derivative of the result.

See the module documentaiton for Numeric.Backprop.Op for more information.

Creation

op0 :: a -> Op '[] a

Create an Op that takes no inputs and always returns the given value.

There is no gradient, of course (using gradOp will give you an empty tuple), because there is no input to have a gradient of.

>>> runOp (op0 10) RNil
(10, RNil)

For a constant Op that takes input and ignores it, see opConst and opConst'.

opConst :: forall as a. (AllConstrained Num as, RecApplicative as) => a -> Op as a

An Op that ignores all of its inputs and returns a given constant value.

>>> gradOp' (opConst 10) (1 :& 2 :& 3 :& RNil)
(10, 0 :& 0 :& 0 :& RNil)

idOp :: Op '[a] a

An Op that just returns whatever it receives. The identity function.

idOp = opIso id id

opLens :: Num a => Lens' a b -> Op '[a] b

An Op that extracts a value from an input value using a Lens'.

Warning: This is unsafe! It assumes that it extracts a specific value unchanged, with derivative 1, so will break for things that numerically manipulate things before returning them.

Giving gradients directly

op1 :: (a -> (b, b -> a)) -> Op '[a] b

Create an Op of a function taking one input, by giving its explicit derivative. The function should return a tuple containing the result of the function, and also a function taking the derivative of the result and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R} \rightarrow \mathbb{R}\cr y &= f(x)\cr z &= g(y) } \]

Then the derivative \( \frac{dz}{dx} \), it would be:

\[ \frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx} \]

If our Op represents \(f\), then the second item in the resulting tuple should be a function that takes \(\frac{dz}{dy}\) and returns \(\frac{dz}{dx}\).

As an example, here is an Op that squares its input:

square :: Num a => Op '[a] a
square = op1 $ \x -> (x*x, \d -> 2 * d * x
                     )

Remember that, generally, end users shouldn't directly construct Ops; they should be provided by libraries or generated automatically.

op2 :: (a -> b -> (c, c -> (a, b))) -> Op '[a, b] c

Create an Op of a function taking two inputs, by giving its explicit gradient. The function should return a tuple containing the result of the function, and also a function taking the derivative of the result and return the derivative of the input.

If we have

\[ \eqalign{ f &: \mathbb{R}^2 \rightarrow \mathbb{R}\cr z &= f(x, y)\cr k &= g(z) } \]

Then the gradient \( \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> \) would be:

\[ \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> = \left< \frac{dk}{dz} \frac{\partial z}{dx}, \frac{dk}{dz} \frac{\partial z}{dy} \right> \]

If our Op represents \(f\), then the second item in the resulting tuple should be a function that takes \(\frac{dk}{dz}\) and returns \( \left< \frac{\partial k}{dx}, \frac{\partial k}{dx} \right> \).

As an example, here is an Op that multiplies its inputs:

mul :: Num a => Op '[a, a] a
mul = op2' $ \x y -> (x*y, \d -> (d*y, x*d)
                     )

Remember that, generally, end users shouldn't directly construct Ops; they should be provided by libraries or generated automatically.

op3 :: (a -> b -> c -> (d, d -> (a, b, c))) -> Op '[a, b, c] d

Create an Op of a function taking three inputs, by giving its explicit gradient. See documentation for op2 for more details.

From Isomorphisms

opCoerce :: Coercible a b => Op '[a] b

An Op that coerces an item into another item whose type has the same runtime representation.

>>> gradOp' opCoerce (Identity 5) :: (Int, Identity Int)
(5, Identity 1)

opCoerce = opIso coerced coerce

opTup :: Op as (Rec Identity as)

An Op that takes as and returns exactly the input tuple.

>>> gradOp' opTup (1 :& 2 :& 3 :& RNil)
(1 :& 2 :& 3 :& RNil, 1 :& 1 :& 1 :& RNil)

opIso :: (a -> b) -> (b -> a) -> Op '[a] b

An Op that runs the input value through an isomorphism.

Warning: This is unsafe! It assumes that the isomorphisms themselves have derivative 1, so will break for things like exp & log. Basically, don't use this for any "numeric" isomorphisms.

opIso2 :: (a -> b -> c) -> (c -> (a, b)) -> Op '[a, b] c

An Op that runs the two input values through an isomorphism. Useful for things like constructors. See opIso for caveats.

Since: 0.1.4.0

opIso3 :: (a -> b -> c -> d) -> (d -> (a, b, c)) -> Op '[a, b, c] d

An Op that runs the three input values through an isomorphism. Useful for things like constructors. See opIso for caveats.

Since: 0.1.4.0

opIsoN :: (Rec Identity as -> b) -> (b -> Rec Identity as) -> Op as b

An Op that runs the input value through an isomorphism between a tuple of values and a value. See opIso for caveats.

In Numeric.Backprop.Op since version 0.1.2.0, but only exported from Numeric.Backprop since version 0.1.3.0.

Since: 0.1.2.0

No gradient

noGrad1 :: (a -> b) -> Op '[a] b

Create an Op with no gradient. Can be evaluated with evalOp, but will throw a runtime exception when asked for the gradient.

Can be used with BVar with liftOp1, and evalBP will work fine. gradBP and backprop will also work fine if the result is never used in the final answer, but will throw a runtime exception if the final answer depends on the result of this operation.

Useful if your only API is exposed through backprop. Just be sure to tell your users that this will explode when finding the gradient if the result is used in the final result.

Since: 0.1.3.0

noGrad :: (Rec Identity as -> b) -> Op as b

Create an Op with no gradient. Can be evaluated with evalOp, but will throw a runtime exception when asked for the gradient.

Can be used with BVar with liftOp, and evalBP will work fine. gradBP and backprop will also work fine if the result is never used in the final answer, but will throw a runtime exception if the final answer depends on the result of this operation.

Useful if your only API is exposed through backprop. Just be sure to tell your users that this will explode when finding the gradient if the result is used in the final result.

Since: 0.1.3.0

Manipulation

composeOp

Arguments

:: (AllConstrained Num as, RecApplicative as)

=> Rec (Op as) bs

Rec of Ops taking as and returning different b in bs

-> Op bs c

OpM taking eac of the bs from the input Rec.

-> Op as c

Composed Op

Compose Ops together, like sequence for functions, or liftAN.

That is, given an Op as b1, an Op as b2, and an Op as b3, it can compose them with an Op '[b1,b2,b3] c to create an Op as c.

composeOp1 :: (AllConstrained Num as, RecApplicative as) => Op as b -> Op '[b] c -> Op as c

Convenient wrapper over composeOp for the case where the second function only takes one input, so the two Ops can be directly piped together, like for ..

(~.) :: (AllConstrained Num as, RecApplicative as) => Op '[b] c -> Op as b -> Op as c infixr 9

Convenient infix synonym for (flipped) composeOp1. Meant to be used just like .:

f :: Op '[b]   c
g :: Op '[a,a] b

f ~. g :: Op '[a, a] c

Utility

Numeric Ops

Built-in ops for common numeric operations.

Note that the operators (like +.) are meant to be used in prefix form, like:

liftOp2 (.+) v1 v2

(+.) :: Num a => Op '[a, a] a

Op for addition

(-.) :: Num a => Op '[a, a] a

Op for subtraction

(*.) :: Num a => Op '[a, a] a

Op for multiplication

negateOp :: Num a => Op '[a] a

Op for negation

absOp :: Num a => Op '[a] a

Op for absolute value

signumOp :: Num a => Op '[a] a

Op for signum

(/.) :: Fractional a => Op '[a, a] a

Op for division

recipOp :: Fractional a => Op '[a] a

Op for multiplicative inverse

expOp :: Floating a => Op '[a] a

Op for exp

logOp :: Floating a => Op '[a] a

Op for the natural logarithm

sqrtOp :: Floating a => Op '[a] a

Op for square root

(**.) :: Floating a => Op '[a, a] a

Op for exponentiation

logBaseOp :: Floating a => Op '[a, a] a

Op for logBase

sinOp :: Floating a => Op '[a] a

Op for sine

cosOp :: Floating a => Op '[a] a

Op for cosine

tanOp :: Floating a => Op '[a] a

Op for tangent

asinOp :: Floating a => Op '[a] a

Op for arcsine

acosOp :: Floating a => Op '[a] a

Op for arccosine

atanOp :: Floating a => Op '[a] a

Op for arctangent

sinhOp :: Floating a => Op '[a] a

Op for hyperbolic sine

coshOp :: Floating a => Op '[a] a

Op for hyperbolic cosine

tanhOp :: Floating a => Op '[a] a

Op for hyperbolic tangent

asinhOp :: Floating a => Op '[a] a

Op for hyperbolic arcsine

acoshOp :: Floating a => Op '[a] a

Op for hyperbolic arccosine

atanhOp :: Floating a => Op '[a] a

Op for hyperbolic arctangent