Control.Distributed.Task.Distribution.RunComputation

Defines a higher level interface to running calculations. Resolves HDFS input paths. |

Documentation

data MasterOptions

The definition of a distributed calculation.

Constructors

MasterOptions

Fields

_host :: String

the master hostname

_port :: Int

the master port

_taskSpec :: TaskSpec

the task logic

_dataSpecs :: DataSpec

which data to process

_resultSpec :: ResultSpec

how to process the result

data TaskSpec

Task logic definition, most modes expect task mode support, see RemoteExecutionSupport.

Constructors

SourceCodeSpec String

build the given string as module remotely (restrictions apply)

FullBinaryDeployment

run this binary as task

SerializedThunk (TaskInput -> TaskResult)

serialize the given function in the context of the given program, run both as task (restrictions apply)

ObjectCodeModuleDeployment (TaskInput -> TaskResult)

only transport some of the generated object code and relink remotely (restrictions apply) - the function here is ignored, it only forces the compilation of the contained module

data DataSpec

definition of input data

Constructors

SimpleDataSpec Int

simple test data, the path is configured, amount of files can be limited

HdfsDataSpec HdfsPath Int (Maybe String)

use given HDFS as starting directory, descend a number of directories from there and take all files starting with the filter prefix (if any given)

data ResultSpec

what to do with the result

Constructors

CollectOnMaster ([TaskResult] -> IO ())

process all results with the given method

StoreInHdfs String String

store the results in HDFS, in the given directory(1), with the given suffix (2), based on the input path.

Discard

do nothing, for testing purposes only

runMaster :: MasterOptions -> IO ()

Run a computation.