Stability | experimental |
---|---|
Portability | non-portable |
Safe Haskell | None |
Language | Haskell2010 |
A Dataloader is an extension of a Dataset and is primarily intended for compute-intensive, batch loading interfaces. When used with ImageFolder representations of Datasets, it shuffles the order of files to be loaded and leverages the async library when possible.
Concurrent loading primarily takes place in batchStream
. stream
exists
primarily to provide a unified API with training that is not batch-oriented.
Synopsis
- data Dataloader a b = Dataloader {}
- uniformIxline :: Dataset a -> GenIO -> IO (Vector Int)
- stream :: (MonadThrow io, MonadIO io) => Dataloader a b -> Stream (Of b) io ()
- batchStream :: (MonadThrow io, MonadIO io, NFData b) => Dataloader a b -> Stream (Of [b]) io ()
Documentation
data Dataloader a b Source #
Options for a data loading functions.
Dataloader | |
|
uniformIxline :: Dataset a -> GenIO -> IO (Vector Int) Source #
Generate a uniformly random index line from a dataset and a generator.
stream :: (MonadThrow io, MonadIO io) => Dataloader a b -> Stream (Of b) io () Source #
Stream a dataset in-memory, applying a transformation function.
batchStream :: (MonadThrow io, MonadIO io, NFData b) => Dataloader a b -> Stream (Of [b]) io () Source #
Stream batches of a dataset, concurrently processing each element
NOTE: Run with -threaded -rtsopts
to concurrently load data in-memory.