Control.Concurrent.Throttled

Handle concurrent fetches from a Foldable, throttled by the number of CPUs that the user has available.

The canonical function is throttle. Notice the type of function it expects as an argument:

(TQueue a -> a -> IO b)

This TQueue is an input queue derived from the given Foldable. This is exposed so that any concurrent action can dynamically grow the input.

The output is a TQueue of the result of each IO action. This be can fed to further concurrent operations, or drained into a list via:

import Control.Concurrent.STM (atomically)
import Control.Concurrent.STM.TQueue (TQueue, flushTQueue)

flush :: TQueue a -> IO [a]
flush = atomically . flushTQueue

Documentation

throttle :: Foldable f => (TQueue a -> a -> IO b) -> f a -> IO (TQueue b)

Concurrently traverse over some Foldable using 1 thread per CPU that the user has. The user's function is also passed the source TQueue, in case they wish to dynamically add work to it.

The order of elements of the original Foldable is not maintained.

throttle_ :: Foldable f => (TQueue a -> a -> IO b) -> f a -> IO ()

Like throttle, but doesn't store any output. This is more efficient than void . throttle.

throttleMaybe :: Foldable f => (TQueue a -> a -> IO (Maybe b)) -> f a -> IO (Either (TQueue b) (TQueue b))

Like throttle, but stop all threads safely if one finds a Nothing. The final TQueue contains as much completed work as the threads could manage before they completed / died. Matching on the Either will tell you which scenario occurred.

throttleMaybe_ :: Foldable f => (TQueue a -> a -> IO (Maybe b)) -> f a -> IO (Either () ())

Like throttleMaybe, but doesn't store any output. Pattern matching on the Either will still let you know if a failure occurred.