Copyright  (c) Justin Le 2018 

License  BSD3 
Maintainer  justin@jle.im 
Stability  experimental 
Portability  nonportable 
Safe Haskell  None 
Language  Haskell2010 
Provides the Op
type and combinators, which represent differentiable
functions/operations on values, and are used internally by the library
to perform backpropagation.
Users of the library can ignore this module for the most part. Library
authors defining backpropagatable primitives for their functions are
recommend to simply use op0
, op1
, op2
, op3
, which are
reexported in Numeric.Backprop. However, authors who want more
options in defining their primtive functions might find some of these
functions useful.
Note that if your entire function is a single nonbranching composition
of functions, Op
and its utility functions alone are sufficient to
differentiate/backprop. However, this happens rarely in practice.
To use these Op
s with the backprop library, they can be made to work
with BVar
s using liftOp
, liftOp1
, liftOp2
, and liftOp3
.
If you are writing a library, see
https://backprop.jle.im/06equippingyourlibrary.html for a guide for
equipping your library with backpropatable operations using Op
s.
See also this guide for writing Ops manually on your own numerical functions.
Synopsis
 newtype Op as a = Op {}
 data Rec (a :: u > *) (b :: [u]) :: forall u. (u > *) > [u] > * where
 runOp :: Num a => Op as a > Rec Identity as > (a, Rec Identity as)
 evalOp :: Op as a > Rec Identity as > a
 gradOp :: Num a => Op as a > Rec Identity as > Rec Identity as
 gradOpWith :: Op as a > Rec Identity as > a > Rec Identity as
 op0 :: a > Op '[] a
 opConst :: forall as a. RPureConstrained Num as => a > Op as a
 idOp :: Op '[a] a
 opLens :: Num a => Lens' a b > Op '[a] b
 op1 :: (a > (b, b > a)) > Op '[a] b
 op2 :: (a > b > (c, c > (a, b))) > Op '[a, b] c
 op3 :: (a > b > c > (d, d > (a, b, c))) > Op '[a, b, c] d
 opCoerce :: Coercible a b => Op '[a] b
 opTup :: Op as (Rec Identity as)
 opIso :: (a > b) > (b > a) > Op '[a] b
 opIso2 :: (a > b > c) > (c > (a, b)) > Op '[a, b] c
 opIso3 :: (a > b > c > d) > (d > (a, b, c)) > Op '[a, b, c] d
 opIsoN :: (Rec Identity as > b) > (b > Rec Identity as) > Op as b
 noGrad1 :: (a > b) > Op '[a] b
 noGrad :: (Rec Identity as > b) > Op as b
 composeOp :: forall as bs c. RPureConstrained Num as => Rec (Op as) bs > Op bs c > Op as c
 composeOp1 :: RPureConstrained Num as => Op as b > Op '[b] c > Op as c
 (~.) :: RPureConstrained Num as => Op '[b] c > Op as b > Op as c
 (+.) :: Num a => Op '[a, a] a
 (.) :: Num a => Op '[a, a] a
 (*.) :: Num a => Op '[a, a] a
 negateOp :: Num a => Op '[a] a
 absOp :: Num a => Op '[a] a
 signumOp :: Num a => Op '[a] a
 (/.) :: Fractional a => Op '[a, a] a
 recipOp :: Fractional a => Op '[a] a
 expOp :: Floating a => Op '[a] a
 logOp :: Floating a => Op '[a] a
 sqrtOp :: Floating a => Op '[a] a
 (**.) :: Floating a => Op '[a, a] a
 logBaseOp :: Floating a => Op '[a, a] a
 sinOp :: Floating a => Op '[a] a
 cosOp :: Floating a => Op '[a] a
 tanOp :: Floating a => Op '[a] a
 asinOp :: Floating a => Op '[a] a
 acosOp :: Floating a => Op '[a] a
 atanOp :: Floating a => Op '[a] a
 sinhOp :: Floating a => Op '[a] a
 coshOp :: Floating a => Op '[a] a
 tanhOp :: Floating a => Op '[a] a
 asinhOp :: Floating a => Op '[a] a
 acoshOp :: Floating a => Op '[a] a
 atanhOp :: Floating a => Op '[a] a
Implementation
Op
s contain information on a function as well as its gradient, but
provides that information in a way that allows them to be "chained".
For example, for a function
\[ f : \mathbb{R}^n \rightarrow \mathbb{R} \]
We might want to apply a function \(g\) to the result we get, to get our "final" result:
\[ \eqalign{ y &= f(\mathbf{x})\cr z &= g(y) } \]
Now, we might want the gradient \(\nabla z\) with respect to \(\mathbf{x}\), or \(\nabla_\mathbf{x} z\). Explicitly, this is:
\[ \nabla_\mathbf{x} z = \left< \frac{\partial z}{\partial x_1}, \frac{\partial z}{\partial x_2}, \ldots \right> \]
We can compute that by multiplying the total derivative of \(z\) with respect to \(y\) (that is, \(\frac{dz}{dy}\)) with the gradient of \(f\)) itself:
\[ \eqalign{ \nabla_\mathbf{x} z &= \frac{dz}{dy} \left< \frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots \right>\cr \nabla_\mathbf{x} z &= \frac{dz}{dy} \nabla_\mathbf{x} y } \]
So, to create an
with the Op
as aOp
constructor, you give
a function that returns a tuple, containing:
 An
a
: The result of the function  An
a > Rec Identity as
: A function that, when given \(\frac{dz}{dy}\), returns the total gradient \(\nabla_z \mathbf{x}\).
This is done so that Op
s can easily be "chained" together, one after
the other. If you have an Op
for \(f\) and an Op
for \(g\), you can
compute the gradient of \(f\) knowing that the result target is
\(g \circ f\).
See this guide for a detailed look on writing ops manually on your own numerical functions.
Note that end users should probably never be required to construct an
Op
explicitly this way. Instead, libraries should provide
carefuly preconstructed ones, or provide ways to generate them
automatically (like op1
, op2
, and op3
here).
For examples of Op
s implemented from scratch, see the implementations
of +.
, .
, recipOp
, sinOp
, etc.
See Numeric.Backprop.Op for a minitutorial on using Rec
and
'Rec Identity'.
An
describes a differentiable function from Op
as aas
to a
.
For example, a value of type
Op
'[Int, Bool] Double
is a function from an Int
and a Bool
, returning a Double
. It can
be differentiated to give a gradient of an Int
and a Bool
if given
a total derivative for the Double
. If we call Bool
\(2\), then,
mathematically, it is akin to a:
\[ f : \mathbb{Z} \times 2 \rightarrow \mathbb{R} \]
See runOp
, gradOp
, and gradOpWith
for examples on how to run it,
and Op
for instructions on creating it.
It is simpler to not use this type constructor directly, and instead use
the op2
, op1
, op2
, and op3
helper smart constructors.
See Numeric.Backprop.Op for a minitutorial on using Rec
and
'Rec Identity'.
To use an Op
with the backprop library, see liftOp
, liftOp1
,
liftOp2
, and liftOp3
.
Op  Construct an See the module documentation for Numeric.Backprop.Op for more
details on the function that this constructor and 
Instances
(RPureConstrained Num as, Floating a) => Floating (Op as a) Source #  
(RPureConstrained Num as, Fractional a) => Fractional (Op as a) Source #  
(RPureConstrained Num as, Num a) => Num (Op as a) Source #  
Tuple Types
Rec
, from the vinyl library
(in Data.Vinyl.Core) is a heterogeneous list/tuple type, which allows
you to tuple together multiple values of different types and operate on
them generically.
A
contains an Rec
f '[a, b, c]f a
, an f b
, and an f c
, and
is constructed by consing them together with :&
(using RNil
as nil):
Identity
"hello":&
Identity True :& Identity 7.8 :& RNil ::Rec
I
'[String, Bool, Double]Const
"hello" :& Const "world" :& Const "ok" :& RNil ::Rec
(C
String) '[a, b, c]Proxy
:& Proxy :& Proxy :& RNil ::Rec
Proxy
'[a, b, c]
So, in general:
x :: f a y :: f b z :: f c x :& y :& z :& RNil :: Rec f '[a, b, c]
data Rec (a :: u > *) (b :: [u]) :: forall u. (u > *) > [u] > * where #
A record is parameterized by a universe u
, an interpretation f
and a
list of rows rs
. The labels or indices of the record are given by
inhabitants of the kind u
; the type of values at any label r :: u
is
given by its interpretation f r :: *
.
Instances
RecElem (Rec :: (a > *) > [a] > *) (r :: a) (r' :: a) (r ': rs :: [a]) (r' ': rs :: [a]) Z  
Defined in Data.Vinyl.Lens type RecElemFCtx Rec f :: Constraint #  
(RIndex r (s ': rs) ~ S i, RecElem (Rec :: (a > *) > [a] > *) r r' rs rs' i) => RecElem (Rec :: (a > *) > [a] > *) (r :: a) (r' :: a) (s ': rs :: [a]) (s ': rs' :: [a]) (S i)  
Defined in Data.Vinyl.Lens type RecElemFCtx Rec f :: Constraint #  
RecSubset (Rec :: (k > *) > [k] > *) ([] :: [k]) (ss :: [k]) ([] :: [Nat])  
Defined in Data.Vinyl.Lens type RecSubsetFCtx Rec f :: Constraint #  
(RElem r ss i, RSubset rs ss is) => RecSubset (Rec :: (k > *) > [k] > *) (r ': rs :: [k]) (ss :: [k]) (i ': is)  
Defined in Data.Vinyl.Lens type RecSubsetFCtx Rec f :: Constraint #  
TestCoercion f => TestCoercion (Rec f :: [u] > *)  
Defined in Data.Vinyl.Core  
TestEquality f => TestEquality (Rec f :: [u] > *)  
Defined in Data.Vinyl.Core  
Eq (Rec f ([] :: [u]))  
(Eq (f r), Eq (Rec f rs)) => Eq (Rec f (r ': rs))  
Ord (Rec f ([] :: [u]))  
Defined in Data.Vinyl.Core  
(Ord (f r), Ord (Rec f rs)) => Ord (Rec f (r ': rs))  
Defined in Data.Vinyl.Core compare :: Rec f (r ': rs) > Rec f (r ': rs) > Ordering # (<) :: Rec f (r ': rs) > Rec f (r ': rs) > Bool # (<=) :: Rec f (r ': rs) > Rec f (r ': rs) > Bool # (>) :: Rec f (r ': rs) > Rec f (r ': rs) > Bool # (>=) :: Rec f (r ': rs) > Rec f (r ': rs) > Bool # max :: Rec f (r ': rs) > Rec f (r ': rs) > Rec f (r ': rs) # min :: Rec f (r ': rs) > Rec f (r ': rs) > Rec f (r ': rs) #  
(RMap rs, ReifyConstraint Show f rs, RecordToList rs) => Show (Rec f rs)  Records may be shown insofar as their points may be shown.

Semigroup (Rec f ([] :: [u]))  
(Semigroup (f r), Semigroup (Rec f rs)) => Semigroup (Rec f (r ': rs))  
Monoid (Rec f ([] :: [u]))  
(Monoid (f r), Monoid (Rec f rs)) => Monoid (Rec f (r ': rs))  
Storable (Rec f ([] :: [u]))  
Defined in Data.Vinyl.Core  
(Storable (f r), Storable (Rec f rs)) => Storable (Rec f (r ': rs))  
Defined in Data.Vinyl.Core sizeOf :: Rec f (r ': rs) > Int # alignment :: Rec f (r ': rs) > Int # peekElemOff :: Ptr (Rec f (r ': rs)) > Int > IO (Rec f (r ': rs)) # pokeElemOff :: Ptr (Rec f (r ': rs)) > Int > Rec f (r ': rs) > IO () # peekByteOff :: Ptr b > Int > IO (Rec f (r ': rs)) # pokeByteOff :: Ptr b > Int > Rec f (r ': rs) > IO () #  
type RecElemFCtx (Rec :: (a > *) > [a] > *) (f :: a > *)  
Defined in Data.Vinyl.Lens  
type RecElemFCtx (Rec :: (a > *) > [a] > *) (f :: a > *)  
Defined in Data.Vinyl.Lens  
type RecSubsetFCtx (Rec :: (k > *) > [k] > *) (f :: k > *)  
Defined in Data.Vinyl.Lens  
type RecSubsetFCtx (Rec :: (k > *) > [k] > *) (f :: k > *)  
Defined in Data.Vinyl.Lens 
Running
Pure
runOp :: Num a => Op as a > Rec Identity as > (a, Rec Identity as) Source #
Run the function that an Op
encodes, to get the resulting output and
also its gradient with respect to the inputs.
>>>
gradOp' (op2 (*)) (3 :& 5 :& RNil)
(15, 5 :& 3 :& RNil)
evalOp :: Op as a > Rec Identity as > a Source #
Run the function that an Op
encodes, to get the result.
>>>
runOp (op2 (*)) (3 :& 5 :& RNil)
15
gradOp :: Num a => Op as a > Rec Identity as > Rec Identity as Source #
Run the function that an Op
encodes, and get the gradient of the
output with respect to the inputs.
>>>
gradOp (op2 (*)) (3 :& 5 :& RNil)
5 :& 3 :& RNil  the gradient of x*y is (y, x)
gradOp
o xs =gradOpWith
o xs 1
:: Op as a 

> Rec Identity as  Inputs to run it with 
> a  The total derivative of the result. 
> Rec Identity as  The gradient 
Get the gradient function that an Op
encodes, with a third argument
expecting the total derivative of the result.
See the module documentaiton for Numeric.Backprop.Op for more information.
Creation
Create an Op
that takes no inputs and always returns the given
value.
There is no gradient, of course (using gradOp
will give you an empty
tuple), because there is no input to have a gradient of.
>>>
runOp (op0 10) RNil
(10, RNil)
For a constant Op
that takes input and ignores it, see opConst
and
opConst'
.
opConst :: forall as a. RPureConstrained Num as => a > Op as a Source #
An Op
that ignores all of its inputs and returns a given constant
value.
>>>
gradOp' (opConst 10) (1 :& 2 :& 3 :& RNil)
(10, 0 :& 0 :& 0 :& RNil)
Giving gradients directly
op1 :: (a > (b, b > a)) > Op '[a] b Source #
Create an Op
of a function taking one input, by giving its explicit
derivative. The function should return a tuple containing the result of
the function, and also a function taking the derivative of the result
and return the derivative of the input.
If we have
\[ \eqalign{ f &: \mathbb{R} \rightarrow \mathbb{R}\cr y &= f(x)\cr z &= g(y) } \]
Then the derivative \( \frac{dz}{dx} \), it would be:
\[ \frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx} \]
If our Op
represents \(f\), then the second item in the resulting
tuple should be a function that takes \(\frac{dz}{dy}\) and returns
\(\frac{dz}{dx}\).
As an example, here is an Op
that squares its input:
square :: Num a =>Op
'[a] a square =op1
$ \x > (x*x, \d > 2 * d * x )
Remember that, generally, end users shouldn't directly construct Op
s;
they should be provided by libraries or generated automatically.
op2 :: (a > b > (c, c > (a, b))) > Op '[a, b] c Source #
Create an Op
of a function taking two inputs, by giving its explicit
gradient. The function should return a tuple containing the result of
the function, and also a function taking the derivative of the result
and return the derivative of the input.
If we have
\[ \eqalign{ f &: \mathbb{R}^2 \rightarrow \mathbb{R}\cr z &= f(x, y)\cr k &= g(z) } \]
Then the gradient \( \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> \) would be:
\[ \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> = \left< \frac{dk}{dz} \frac{\partial z}{dx}, \frac{dk}{dz} \frac{\partial z}{dy} \right> \]
If our Op
represents \(f\), then the second item in the resulting
tuple should be a function that takes \(\frac{dk}{dz}\) and returns
\( \left< \frac{\partial k}{dx}, \frac{\partial k}{dx} \right> \).
As an example, here is an Op
that multiplies its inputs:
mul :: Num a =>Op
'[a, a] a mul =op2'
$ \x y > (x*y, \d > (d*y, x*d) )
Remember that, generally, end users shouldn't directly construct Op
s;
they should be provided by libraries or generated automatically.
From Isomorphisms
opTup :: Op as (Rec Identity as) Source #
An Op
that takes as
and returns exactly the input tuple.
>>>
gradOp' opTup (1 :& 2 :& 3 :& RNil)
(1 :& 2 :& 3 :& RNil, 1 :& 1 :& 1 :& RNil)
opIsoN :: (Rec Identity as > b) > (b > Rec Identity as) > Op as b Source #
An Op
that runs the input value through an isomorphism between
a tuple of values and a value. See opIso
for caveats.
In Numeric.Backprop.Op since version 0.1.2.0, but only exported from Numeric.Backprop since version 0.1.3.0.
Since: backprop0.1.2.0
No gradient
noGrad1 :: (a > b) > Op '[a] b Source #
Create an Op
with no gradient. Can be evaluated with evalOp
, but
will throw a runtime exception when asked for the gradient.
Can be used with BVar
with liftOp1
, and evalBP
will work fine.
gradBP
and backprop
will also work fine if the result is never used
in the final answer, but will throw a runtime exception if the final
answer depends on the result of this operation.
Useful if your only API is exposed through backprop. Just be sure to tell your users that this will explode when finding the gradient if the result is used in the final result.
Since: backprop0.1.3.0
noGrad :: (Rec Identity as > b) > Op as b Source #
Create an Op
with no gradient. Can be evaluated with evalOp
, but
will throw a runtime exception when asked for the gradient.
Can be used with BVar
with liftOp
, and evalBP
will work fine.
gradBP
and backprop
will also work fine if the result is never used
in the final answer, but will throw a runtime exception if the final
answer depends on the result of this operation.
Useful if your only API is exposed through backprop. Just be sure to tell your users that this will explode when finding the gradient if the result is used in the final result.
Since: backprop0.1.3.0
Manipulation
composeOp1 :: RPureConstrained Num as => Op as b > Op '[b] c > Op as c Source #
(~.) :: RPureConstrained Num as => Op '[b] c > Op as b > Op as c infixr 9 Source #
Convenient infix synonym for (flipped) composeOp1
. Meant to be used
just like .
:
f ::Op
'[b] c g ::Op
'[a,a] b f~.
g :: Op '[a, a] c