Spark.Core.Internal.Pruning

Methods to prune the computation graph.

Documentation

data NodeCacheStatus

The status of a node being computed.

On purpose, it does not store data. This is meant to be only the control plane of the compuations.

Constructors

NodeCacheRunning

NodeCacheError

NodeCacheSuccess

Instances

Eq NodeCacheStatus

Methods

(==) :: NodeCacheStatus -> NodeCacheStatus -> Bool

(/=) :: NodeCacheStatus -> NodeCacheStatus -> Bool

Show NodeCacheStatus

Methods

showsPrec :: Int -> NodeCacheStatus -> ShowS

show :: NodeCacheStatus -> String

showList :: [NodeCacheStatus] -> ShowS

data NodeCacheInfo

This structure describes the last time a node was observed by the controller, and the state it was in.

This information is used to do smart computation pruning, by assuming that the observables are kept by the Spark processes.

Constructors

NodeCacheInfo

Fields

• nciStatus :: !NodeCacheStatus
• nciComputation :: !ComputationID
• nciPath :: !NodePath

Instances

Eq NodeCacheInfo

Methods

(==) :: NodeCacheInfo -> NodeCacheInfo -> Bool

(/=) :: NodeCacheInfo -> NodeCacheInfo -> Bool

Show NodeCacheInfo

Methods

showsPrec :: Int -> NodeCacheInfo -> ShowS

show :: NodeCacheInfo -> String

showList :: [NodeCacheInfo] -> ShowS

type NodeCache = HashMap NodeId NodeCacheInfo

pruneGraph :: Show v => NodeCache -> (v -> NodeId) -> (v -> NodeCacheInfo -> v) -> Graph v StructureEdge -> Graph v StructureEdge

It assumes a compute graph, NOT a dependency dag.

pruneGraphDefault :: NodeCache -> Graph UntypedNode StructureEdge -> Graph UntypedNode StructureEdge

emptyNodeCache :: NodeCache