Numeric.SGD.ParamSet

Documentation

class ParamSet a where

Class of types that can be treated as parameter sets. It provides basic element-wise operations (addition, multiplication, mapping) which are required to perform stochastic gradient descent. Many of the operations (add, mul, sub, div, etc.) have the same interpretation and follow the same laws (e.g. associativity) as the corresponding operations in Num and Fractional.

zero takes a parameter set as argument and "zero out"'s all its elements (as in the backprop library). This allows instances for Maybe, Map, etc., where the structure of the parameter set is dynamic. This leads to the following property:

add (zero x) x = x

However, zero does not have to obey (add (zero x) y = y).

A ParamSet can be also seen as a (structured) vector, hence pmap and norm_2. The latter is not strictly necessary to perform SGD, but it is useful to control the training process.

pmap should obey the following law:

pmap id x = x

If you leave the body of an instance declaration blank, GHC Generics will be used to derive instances if the type has a single constructor and each field is an instance of ParamSet.

Minimal complete definition

Nothing

Methods

pmap :: (Double -> Double) -> a -> a

Element-wise mapping

zero :: a -> a

Zero-out all elements

add :: a -> a -> a

Element-wise addition

sub :: a -> a -> a

Elementi-wise substruction

mul :: a -> a -> a

Element-wise multiplication

div :: a -> a -> a

Element-wise division

norm_2 :: a -> Double

L2 norm

pmap :: (Generic a, GPMap (Rep a)) => (Double -> Double) -> a -> a

Element-wise mapping

add :: (Generic a, GAdd (Rep a)) => a -> a -> a

Element-wise addition

sub :: (Generic a, GSub (Rep a)) => a -> a -> a

Elementi-wise substruction

mul :: (Generic a, GMul (Rep a)) => a -> a -> a

Element-wise multiplication

div :: (Generic a, GDiv (Rep a)) => a -> a -> a

Element-wise division

norm_2 :: (Generic a, GNorm2 (Rep a)) => a -> Double

L2 norm

Instances

ParamSet Double

Defined in Numeric.SGD.ParamSet

Methods

pmap :: (Double -> Double) -> Double -> Double

zero :: Double -> Double

add :: Double -> Double -> Double

sub :: Double -> Double -> Double

mul :: Double -> Double -> Double

div :: Double -> Double -> Double

norm_2 :: Double -> Double

ParamSet a => ParamSet (Maybe a)

Nothing represents a deactivated parameter set component. If Nothing is given as an argument to one of the ParamSet operations, the result is Nothing as well.

This differs from the corresponding instance in the backprop library, where Nothing is equivalent to `Just 0`. However, the implementation below seems to correspond adequately enough to the notion that a particular component is either active or not in both the parameter set and the gradient, hence it doesn't make sense to combine Just with Nothing.

Defined in Numeric.SGD.ParamSet

Methods

pmap :: (Double -> Double) -> Maybe a -> Maybe a

zero :: Maybe a -> Maybe a

add :: Maybe a -> Maybe a -> Maybe a

sub :: Maybe a -> Maybe a -> Maybe a

mul :: Maybe a -> Maybe a -> Maybe a

div :: Maybe a -> Maybe a -> Maybe a

norm_2 :: Maybe a -> Double

KnownNat n => ParamSet (R n)

Defined in Numeric.SGD.ParamSet

Methods

pmap :: (Double -> Double) -> R n -> R n

zero :: R n -> R n

add :: R n -> R n -> R n

sub :: R n -> R n -> R n

mul :: R n -> R n -> R n

div :: R n -> R n -> R n

norm_2 :: R n -> Double

(Ord k, ParamSet a) => ParamSet (Map k a)

A map with different parameter sets (of the same type) assigned to the individual keys.

When combining two maps with different sets of keys, only their intersection is preserved.

Defined in Numeric.SGD.ParamSet

Methods

pmap :: (Double -> Double) -> Map k a -> Map k a

zero :: Map k a -> Map k a

add :: Map k a -> Map k a -> Map k a

sub :: Map k a -> Map k a -> Map k a

mul :: Map k a -> Map k a -> Map k a

div :: Map k a -> Map k a -> Map k a

norm_2 :: Map k a -> Double

(KnownNat n, KnownNat m) => ParamSet (L n m)

Defined in Numeric.SGD.ParamSet

Methods

pmap :: (Double -> Double) -> L n m -> L n m

zero :: L n m -> L n m

add :: L n m -> L n m -> L n m

sub :: L n m -> L n m -> L n m

mul :: L n m -> L n m -> L n m

div :: L n m -> L n m -> L n m

norm_2 :: L n m -> Double