Safe Haskell | None |
---|---|
Language | Haskell2010 |
Defines a higher level interface to running calculations. Resolves HDFS input paths. |
- data MasterOptions = MasterOptions {
- _host :: String
- _port :: Int
- _taskSpec :: TaskSpec
- _dataSpecs :: DataSpec
- _resultSpec :: ResultSpec
- data TaskSpec
- data DataSpec
- data ResultSpec
- = CollectOnMaster ([TaskResult] -> IO ())
- | StoreInHdfs String String
- | Discard
- runMaster :: MasterOptions -> IO ()
Documentation
data MasterOptions Source
The definition of a distributed calculation.
MasterOptions | |
|
Task logic definition, most modes expect task mode support, see RemoteExecutionSupport.
SourceCodeSpec String | build the given string as module remotely (restrictions apply) |
FullBinaryDeployment | run this binary as task |
SerializedThunk (TaskInput -> TaskResult) | serialize the given function in the context of the given program, run both as task (restrictions apply) |
ObjectCodeModuleDeployment (TaskInput -> TaskResult) | only transport some of the generated object code and relink remotely (restrictions apply) - the function here is ignored, it only forces the compilation of the contained module |
definition of input data
SimpleDataSpec Int | simple test data, the path is configured, amount of files can be limited |
HdfsDataSpec HdfsPath Int (Maybe String) | use given HDFS as starting directory, descend a number of directories from there and take all files starting with the filter prefix (if any given) |
data ResultSpec Source
what to do with the result
CollectOnMaster ([TaskResult] -> IO ()) | process all results with the given method |
StoreInHdfs String String | store the results in HDFS, in the given directory(1), with the given suffix (2), based on the input path. |
Discard | do nothing, for testing purposes only |
runMaster :: MasterOptions -> IO () Source
Run a computation.