| Copyright | (c) Justin Le 2018 |
|---|---|
| License | BSD3 |
| Maintainer | justin@jle.im |
| Stability | experimental |
| Portability | non-portable |
| Safe Haskell | None |
| Language | Haskell2010 |
Numeric.Backprop.Op
Contents
Description
Provides the Op type and combinators, which represent differentiable
functions/operations on values, and are used internally by the library
to perform back-propagation.
Users of the library can ignore this module for the most part. Library
authors defining backpropagatable primitives for their functions are
recommend to simply use op0, op1, op2, op3, which are
re-exported in Numeric.Backprop. However, authors who want more
options in defining their primtive functions might find some of these
functions useful.
Note that if your entire function is a single non-branching composition
of functions, Op and its utility functions alone are sufficient to
differentiate/backprop. However, this happens rarely in practice.
To use these Ops with the backprop library, they can be made to work
with BVars using liftOp, liftOp1, liftOp2, and liftOp3.
If you are writing a library, see
https://backprop.jle.im/06-equipping-your-library.html for a guide for
equipping your library with backpropatable operations using Ops.
See also this guide for writing Ops manually on your own numerical functions.
Synopsis
- newtype Op as a = Op {}
- data Rec (a :: u -> Type) (b :: [u]) :: forall u. (u -> Type) -> [u] -> Type where
- runOp :: Num a => Op as a -> Rec Identity as -> (a, Rec Identity as)
- evalOp :: Op as a -> Rec Identity as -> a
- gradOp :: Num a => Op as a -> Rec Identity as -> Rec Identity as
- gradOpWith :: Op as a -> Rec Identity as -> a -> Rec Identity as
- op0 :: a -> Op '[] a
- opConst :: forall as a. RPureConstrained Num as => a -> Op as a
- idOp :: Op '[a] a
- opLens :: Num a => Lens' a b -> Op '[a] b
- op1 :: (a -> (b, b -> a)) -> Op '[a] b
- op2 :: (a -> b -> (c, c -> (a, b))) -> Op '[a, b] c
- op3 :: (a -> b -> c -> (d, d -> (a, b, c))) -> Op '[a, b, c] d
- opCoerce :: Coercible a b => Op '[a] b
- opTup :: Op as (Rec Identity as)
- opIso :: (a -> b) -> (b -> a) -> Op '[a] b
- opIso2 :: (a -> b -> c) -> (c -> (a, b)) -> Op '[a, b] c
- opIso3 :: (a -> b -> c -> d) -> (d -> (a, b, c)) -> Op '[a, b, c] d
- opIsoN :: (Rec Identity as -> b) -> (b -> Rec Identity as) -> Op as b
- noGrad1 :: (a -> b) -> Op '[a] b
- noGrad :: (Rec Identity as -> b) -> Op as b
- composeOp :: forall as bs c. RPureConstrained Num as => Rec (Op as) bs -> Op bs c -> Op as c
- composeOp1 :: RPureConstrained Num as => Op as b -> Op '[b] c -> Op as c
- (~.) :: RPureConstrained Num as => Op '[b] c -> Op as b -> Op as c
- (+.) :: Num a => Op '[a, a] a
- (-.) :: Num a => Op '[a, a] a
- (*.) :: Num a => Op '[a, a] a
- negateOp :: Num a => Op '[a] a
- absOp :: Num a => Op '[a] a
- signumOp :: Num a => Op '[a] a
- (/.) :: Fractional a => Op '[a, a] a
- recipOp :: Fractional a => Op '[a] a
- expOp :: Floating a => Op '[a] a
- logOp :: Floating a => Op '[a] a
- sqrtOp :: Floating a => Op '[a] a
- (**.) :: Floating a => Op '[a, a] a
- logBaseOp :: Floating a => Op '[a, a] a
- sinOp :: Floating a => Op '[a] a
- cosOp :: Floating a => Op '[a] a
- tanOp :: Floating a => Op '[a] a
- asinOp :: Floating a => Op '[a] a
- acosOp :: Floating a => Op '[a] a
- atanOp :: Floating a => Op '[a] a
- sinhOp :: Floating a => Op '[a] a
- coshOp :: Floating a => Op '[a] a
- tanhOp :: Floating a => Op '[a] a
- asinhOp :: Floating a => Op '[a] a
- acoshOp :: Floating a => Op '[a] a
- atanhOp :: Floating a => Op '[a] a
Implementation
Ops contain information on a function as well as its gradient, but
provides that information in a way that allows them to be "chained".
For example, for a function
\[ f : \mathbb{R}^n \rightarrow \mathbb{R} \]
We might want to apply a function \(g\) to the result we get, to get our "final" result:
\[ \eqalign{ y &= f(\mathbf{x})\cr z &= g(y) } \]
Now, we might want the gradient \(\nabla z\) with respect to \(\mathbf{x}\), or \(\nabla_\mathbf{x} z\). Explicitly, this is:
\[ \nabla_\mathbf{x} z = \left< \frac{\partial z}{\partial x_1}, \frac{\partial z}{\partial x_2}, \ldots \right> \]
We can compute that by multiplying the total derivative of \(z\) with respect to \(y\) (that is, \(\frac{dz}{dy}\)) with the gradient of \(f\)) itself:
\[ \eqalign{ \nabla_\mathbf{x} z &= \frac{dz}{dy} \left< \frac{\partial y}{\partial x_1}, \frac{\partial y}{\partial x_2}, \ldots \right>\cr \nabla_\mathbf{x} z &= \frac{dz}{dy} \nabla_\mathbf{x} y } \]
So, to create an Op as aOp constructor, you give
a function that returns a tuple, containing:
- An
a: The result of the function - An
a -> Rec Identity as: A function that, when given \(\frac{dz}{dy}\), returns the total gradient \(\nabla_z \mathbf{x}\).
This is done so that Ops can easily be "chained" together, one after
the other. If you have an Op for \(f\) and an Op for \(g\), you can
compute the gradient of \(f\) knowing that the result target is
\(g \circ f\).
See this guide for a detailed look on writing ops manually on your own numerical functions.
Note that end users should probably never be required to construct an
Op explicitly this way. Instead, libraries should provide
carefuly pre-constructed ones, or provide ways to generate them
automatically (like op1, op2, and op3 here).
For examples of Ops implemented from scratch, see the implementations
of +., -., recipOp, sinOp, etc.
See Numeric.Backprop.Op for a mini-tutorial on using Rec and
'Rec Identity'.
An Op as aas to a.
For example, a value of type
Op '[Int, Bool] Double
is a function from an Int and a Bool, returning a Double. It can
be differentiated to give a gradient of an Int and a Bool if given
a total derivative for the Double. If we call Bool \(2\), then,
mathematically, it is akin to a:
\[ f : \mathbb{Z} \times 2 \rightarrow \mathbb{R} \]
See runOp, gradOp, and gradOpWith for examples on how to run it,
and Op for instructions on creating it.
It is simpler to not use this type constructor directly, and instead use
the op2, op1, op2, and op3 helper smart constructors.
See Numeric.Backprop.Op for a mini-tutorial on using Rec and
'Rec Identity'.
To use an Op with the backprop library, see liftOp, liftOp1,
liftOp2, and liftOp3.
Constructors
| Op | Construct an See the module documentation for Numeric.Backprop.Op for more
details on the function that this constructor and |
Instances
| (RPureConstrained Num as, Floating a) => Floating (Op as a) Source # | |
| (RPureConstrained Num as, Fractional a) => Fractional (Op as a) Source # | |
| (RPureConstrained Num as, Num a) => Num (Op as a) Source # | |
Tuple Types
Rec, from the vinyl library
(in Data.Vinyl.Core) is a heterogeneous list/tuple type, which allows
you to tuple together multiple values of different types and operate on
them generically.
A Rec f '[a, b, c]f a, an f b, and an f c, and
is constructed by consing them together with :& (using RNil as nil):
Identity"hello":&Identity True :& Identity 7.8 :& RNil ::RecI'[String, Bool, Double]Const"hello" :& Const "world" :& Const "ok" :& RNil ::Rec(CString) '[a, b, c]Proxy:& Proxy :& Proxy :& RNil ::RecProxy'[a, b, c]
So, in general:
x :: f a y :: f b z :: f c x :& y :& z :& RNil :: Rec f '[a, b, c]
data Rec (a :: u -> Type) (b :: [u]) :: forall u. (u -> Type) -> [u] -> Type where #
A record is parameterized by a universe u, an interpretation f and a
list of rows rs. The labels or indices of the record are given by
inhabitants of the kind u; the type of values at any label r :: u is
given by its interpretation f r :: *.
Constructors
| RNil :: forall u (a :: u -> Type) (b :: [u]). Rec a ([] :: [u]) | |
| (:&) :: forall u (a :: u -> Type) (b :: [u]) (r :: u) (rs :: [u]). !(a r) -> !(Rec a rs) -> Rec a (r ': rs) infixr 7 |
Instances
| RecElem (Rec :: (a -> Type) -> [a] -> Type) (r :: a) (r' :: a) (r ': rs :: [a]) (r' ': rs :: [a]) Z | |
Defined in Data.Vinyl.Lens Associated Types type RecElemFCtx Rec f :: Constraint # | |
| (RIndex r (s ': rs) ~ S i, RecElem (Rec :: (a -> Type) -> [a] -> Type) r r' rs rs' i) => RecElem (Rec :: (a -> Type) -> [a] -> Type) (r :: a) (r' :: a) (s ': rs :: [a]) (s ': rs' :: [a]) (S i) | |
Defined in Data.Vinyl.Lens Associated Types type RecElemFCtx Rec f :: Constraint # | |
| RecSubset (Rec :: (k -> Type) -> [k] -> Type) ([] :: [k]) (ss :: [k]) ([] :: [Nat]) | |
Defined in Data.Vinyl.Lens Associated Types type RecSubsetFCtx Rec f :: Constraint # | |
| (RElem r ss i, RSubset rs ss is) => RecSubset (Rec :: (k -> Type) -> [k] -> Type) (r ': rs :: [k]) (ss :: [k]) (i ': is) | |
Defined in Data.Vinyl.Lens Associated Types type RecSubsetFCtx Rec f :: Constraint # | |
| TestCoercion f => TestCoercion (Rec f :: [u] -> Type) | |
Defined in Data.Vinyl.Core | |
| TestEquality f => TestEquality (Rec f :: [u] -> Type) | |
Defined in Data.Vinyl.Core | |
| Eq (Rec f ([] :: [u])) | |
| (Eq (f r), Eq (Rec f rs)) => Eq (Rec f (r ': rs)) | |
| Ord (Rec f ([] :: [u])) | |
Defined in Data.Vinyl.Core | |
| (Ord (f r), Ord (Rec f rs)) => Ord (Rec f (r ': rs)) | |
Defined in Data.Vinyl.Core Methods compare :: Rec f (r ': rs) -> Rec f (r ': rs) -> Ordering # (<) :: Rec f (r ': rs) -> Rec f (r ': rs) -> Bool # (<=) :: Rec f (r ': rs) -> Rec f (r ': rs) -> Bool # (>) :: Rec f (r ': rs) -> Rec f (r ': rs) -> Bool # (>=) :: Rec f (r ': rs) -> Rec f (r ': rs) -> Bool # max :: Rec f (r ': rs) -> Rec f (r ': rs) -> Rec f (r ': rs) # min :: Rec f (r ': rs) -> Rec f (r ': rs) -> Rec f (r ': rs) # | |
| (RMap rs, ReifyConstraint Show f rs, RecordToList rs) => Show (Rec f rs) | Records may be shown insofar as their points may be shown.
|
| Generic (Rec f ([] :: [u])) | |
| Generic (Rec f rs) => Generic (Rec f (r ': rs)) | |
| Semigroup (Rec f ([] :: [u])) | |
| (Semigroup (f r), Semigroup (Rec f rs)) => Semigroup (Rec f (r ': rs)) | |
| Monoid (Rec f ([] :: [u])) | |
| (Monoid (f r), Monoid (Rec f rs)) => Monoid (Rec f (r ': rs)) | |
| Storable (Rec f ([] :: [u])) | |
Defined in Data.Vinyl.Core | |
| (Storable (f r), Storable (Rec f rs)) => Storable (Rec f (r ': rs)) | |
Defined in Data.Vinyl.Core Methods sizeOf :: Rec f (r ': rs) -> Int # alignment :: Rec f (r ': rs) -> Int # peekElemOff :: Ptr (Rec f (r ': rs)) -> Int -> IO (Rec f (r ': rs)) # pokeElemOff :: Ptr (Rec f (r ': rs)) -> Int -> Rec f (r ': rs) -> IO () # peekByteOff :: Ptr b -> Int -> IO (Rec f (r ': rs)) # pokeByteOff :: Ptr b -> Int -> Rec f (r ': rs) -> IO () # | |
| (ReifyConstraint Backprop f rs, RMap rs, RApply rs, IsoXRec f rs) => Backprop (XRec f rs) Source # | Since: 0.2.6.3 |
| (ReifyConstraint Backprop f rs, RMap rs, RApply rs) => Backprop (Rec f rs) Source # | Since: 0.2.6.3 |
| type RecElemFCtx (Rec :: (a -> Type) -> [a] -> Type) (f :: a -> Type) | |
Defined in Data.Vinyl.Lens | |
| type RecElemFCtx (Rec :: (a -> Type) -> [a] -> Type) (f :: a -> Type) | |
Defined in Data.Vinyl.Lens | |
| type RecSubsetFCtx (Rec :: (k -> Type) -> [k] -> Type) (f :: k -> Type) | |
Defined in Data.Vinyl.Lens | |
| type RecSubsetFCtx (Rec :: (k -> Type) -> [k] -> Type) (f :: k -> Type) | |
Defined in Data.Vinyl.Lens | |
| type Rep (Rec f (r ': rs)) | |
Defined in Data.Vinyl.Core type Rep (Rec f (r ': rs)) = C1 (MetaCons ":&" (InfixI RightAssociative 7) False) (S1 (MetaSel (Nothing :: Maybe Symbol) NoSourceUnpackedness SourceStrict DecidedStrict) (Rec0 (f r)) :*: S1 (MetaSel (Nothing :: Maybe Symbol) NoSourceUnpackedness NoSourceStrictness DecidedLazy) (Rep (Rec f rs))) | |
| type Rep (Rec f ([] :: [u])) | |
Defined in Data.Vinyl.Core | |
Running
Pure
runOp :: Num a => Op as a -> Rec Identity as -> (a, Rec Identity as) Source #
Run the function that an Op encodes, to get the resulting output and
also its gradient with respect to the inputs.
>>>gradOp' (op2 (*)) (3 :& 5 :& RNil)(15, 5 :& 3 :& RNil)
evalOp :: Op as a -> Rec Identity as -> a Source #
Run the function that an Op encodes, to get the result.
>>>runOp (op2 (*)) (3 :& 5 :& RNil)15
gradOp :: Num a => Op as a -> Rec Identity as -> Rec Identity as Source #
Run the function that an Op encodes, and get the gradient of the
output with respect to the inputs.
>>>gradOp (op2 (*)) (3 :& 5 :& RNil)5 :& 3 :& RNil -- the gradient of x*y is (y, x)
gradOpo xs =gradOpWitho xs 1
Arguments
| :: Op as a |
|
| -> Rec Identity as | Inputs to run it with |
| -> a | The total derivative of the result. |
| -> Rec Identity as | The gradient |
Get the gradient function that an Op encodes, with a third argument
expecting the total derivative of the result.
See the module documentaiton for Numeric.Backprop.Op for more information.
Creation
Create an Op that takes no inputs and always returns the given
value.
There is no gradient, of course (using gradOp will give you an empty
tuple), because there is no input to have a gradient of.
>>>runOp (op0 10) RNil(10, RNil)
For a constant Op that takes input and ignores it, see opConst and
opConst'.
opConst :: forall as a. RPureConstrained Num as => a -> Op as a Source #
An Op that ignores all of its inputs and returns a given constant
value.
>>>gradOp' (opConst 10) (1 :& 2 :& 3 :& RNil)(10, 0 :& 0 :& 0 :& RNil)
Giving gradients directly
op1 :: (a -> (b, b -> a)) -> Op '[a] b Source #
Create an Op of a function taking one input, by giving its explicit
derivative. The function should return a tuple containing the result of
the function, and also a function taking the derivative of the result
and return the derivative of the input.
If we have
\[ \eqalign{ f &: \mathbb{R} \rightarrow \mathbb{R}\cr y &= f(x)\cr z &= g(y) } \]
Then the derivative \( \frac{dz}{dx} \), it would be:
\[ \frac{dz}{dx} = \frac{dz}{dy} \frac{dy}{dx} \]
If our Op represents \(f\), then the second item in the resulting
tuple should be a function that takes \(\frac{dz}{dy}\) and returns
\(\frac{dz}{dx}\).
As an example, here is an Op that squares its input:
square :: Num a =>Op'[a] a square =op1$ \x -> (x*x, \d -> 2 * d * x )
Remember that, generally, end users shouldn't directly construct Ops;
they should be provided by libraries or generated automatically.
op2 :: (a -> b -> (c, c -> (a, b))) -> Op '[a, b] c Source #
Create an Op of a function taking two inputs, by giving its explicit
gradient. The function should return a tuple containing the result of
the function, and also a function taking the derivative of the result
and return the derivative of the input.
If we have
\[ \eqalign{ f &: \mathbb{R}^2 \rightarrow \mathbb{R}\cr z &= f(x, y)\cr k &= g(z) } \]
Then the gradient \( \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> \) would be:
\[ \left< \frac{\partial k}{\partial x}, \frac{\partial k}{\partial y} \right> = \left< \frac{dk}{dz} \frac{\partial z}{dx}, \frac{dk}{dz} \frac{\partial z}{dy} \right> \]
If our Op represents \(f\), then the second item in the resulting
tuple should be a function that takes \(\frac{dk}{dz}\) and returns
\( \left< \frac{\partial k}{dx}, \frac{\partial k}{dx} \right> \).
As an example, here is an Op that multiplies its inputs:
mul :: Num a =>Op'[a, a] a mul =op2'$ \x y -> (x*y, \d -> (d*y, x*d) )
Remember that, generally, end users shouldn't directly construct Ops;
they should be provided by libraries or generated automatically.
From Isomorphisms
opTup :: Op as (Rec Identity as) Source #
An Op that takes as and returns exactly the input tuple.
>>>gradOp' opTup (1 :& 2 :& 3 :& RNil)(1 :& 2 :& 3 :& RNil, 1 :& 1 :& 1 :& RNil)
opIsoN :: (Rec Identity as -> b) -> (b -> Rec Identity as) -> Op as b Source #
An Op that runs the input value through an isomorphism between
a tuple of values and a value. See opIso for caveats.
In Numeric.Backprop.Op since version 0.1.2.0, but only exported from Numeric.Backprop since version 0.1.3.0.
Since: 0.1.2.0
No gradient
noGrad1 :: (a -> b) -> Op '[a] b Source #
Create an Op with no gradient. Can be evaluated with evalOp, but
will throw a runtime exception when asked for the gradient.
Can be used with BVar with liftOp1, and evalBP will work fine.
gradBP and backprop will also work fine if the result is never used
in the final answer, but will throw a runtime exception if the final
answer depends on the result of this operation.
Useful if your only API is exposed through backprop. Just be sure to tell your users that this will explode when finding the gradient if the result is used in the final result.
Since: 0.1.3.0
noGrad :: (Rec Identity as -> b) -> Op as b Source #
Create an Op with no gradient. Can be evaluated with evalOp, but
will throw a runtime exception when asked for the gradient.
Can be used with BVar with liftOp, and evalBP will work fine.
gradBP and backprop will also work fine if the result is never used
in the final answer, but will throw a runtime exception if the final
answer depends on the result of this operation.
Useful if your only API is exposed through backprop. Just be sure to tell your users that this will explode when finding the gradient if the result is used in the final result.
Since: 0.1.3.0
Manipulation
composeOp1 :: RPureConstrained Num as => Op as b -> Op '[b] c -> Op as c Source #
(~.) :: RPureConstrained Num as => Op '[b] c -> Op as b -> Op as c infixr 9 Source #
Convenient infix synonym for (flipped) composeOp1. Meant to be used
just like .:
f ::Op'[b] c g ::Op'[a,a] b f~.g :: Op '[a, a] c