The cctools-workqueue package

[Tags: gpl, library]

This package provides a high-level Haskell interface to the WorkQueue library: http://www.nd.edu/~ccl/software/workqueue/

Work Queue is a framework for building large master-worker applications that span many computers including clusters, clouds, and grids. Work Queue applications are written in C, Perl, or Python using a simple API that allows users to define tasks, submit them to the queue, and wait for completion. Tasks are executed by a standard worker process that can run on any available machine. Each worker calls home to the master process, arranges for data transfer, and executes the tasks. Together, the master and worker handle a wide variety of failures, allowing for dynamically scalable and robust applications.

Example usage:

By default the master listens on 9123 on the localhost host. Start a worker:

 $ work_queue_worker -d all localhost 9123

Now we can execute the following master program:

 module Main where

 import Control.Distributed.CCTools.WorkQueue

 import Control.Applicative ((<$>))
 import qualified Data.ByteString.Char8 as BS (pack, putStrLn)
 import Foreign.C.String (newCStringLen)
 import Control.Monad (forM_)



 mktask :: Show a => a -> IO Task
 mktask v = do
   let script = BS.pack . unlines $ [
               "t=$(echo $RANDOM % 10 | bc)"
             , "sleep $t"
             , "echo " ++ show v
             ]
   t <- task $ cmd "bash script.sh"
   specifyBuffer t script (remote "script.sh") False
   specifyTag    t $ show v
   return t

 printStats :: WorkQueue -> IO ()
 printStats q = do
   s <- getStats q
   print $ map ($ s) [tasksRunning, tasksWaiting, tasksComplete]

 processResult :: WorkQueue -> Task -> IO ()
 processResult q r = do
   putStrLn $ "Got: " ++ show (tag r)
   BS.putStrLn . output $ r
   delete r

 main = do
   setDebugFlags [All]
   q  <- workqueue defaultQParams
   ts <- mapM mktask [1..10]
   forM_ ts (submit q)

   eventLoop q (seconds 1) printStats processResult

   putStrLn "Done!"

Properties

Versions3.6.1.0.1.0.0, 3.6.1.0.1.0.0.1
Dependenciesbindings-cctools (==3.6.1.0.1.*), bytestring, lens, monad-loops, unix
LicenseGPL-2
AuthorBadi' Abdul-Wahid
Maintainerabdulwahidc@gmail.com
CategoryDistributed Computing
Home pagehttp://bitbucket.org/badi/hs-cctools-workqueue
Source repositoryhead: git clone git@bitbucket.org:badi/hs-cctools-workqueue.git
Upload dateWed Jan 2 00:14:51 UTC 2013
Uploaded byBadiAbdulWahid
Downloads222 total (18 in last 30 days)

Modules

Flags

NameDescriptionDefault
warnTurn on warningsDisabled

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

Maintainers' corner

For package maintainers and hackage trustees