Torch.Data.Pipeline

Defining a Dataset

See the Vision module which implements the MNIST dataset for a good example of how to define a dataset.

Dataset

class Ord k => Dataset m dataset k sample | dataset -> m, dataset -> sample, dataset -> k where

The base dataset class. A dataset is capable of returning a sample for a given key, and every Dataset has a known set of keys.

Methods

getItem :: dataset -> k -> m sample

keys :: dataset -> Set k

Instances

Applicative m => Dataset (m :: Type -> Type) (MNIST m) Int (Tensor, Tensor)

Defined in Torch.Vision

Methods

getItem :: MNIST m -> Int -> m (Tensor, Tensor)

keys :: MNIST m -> Set Int

Applicative m => Dataset (m :: Type -> Type) (CachedDataset m sample) Int (sample :: Type)

Defined in Torch.Data.Utils

Methods

getItem :: CachedDataset m sample -> Int -> m sample

keys :: CachedDataset m sample -> Set Int

(KnownNat batchSize, KnownDevice device, Applicative m) => Dataset (m :: Type -> Type) (MNIST m device batchSize) Int ((Tensor device 'Float '[batchSize, 784], Tensor device 'Int64 '[batchSize]) :: Type)

Defined in Torch.Typed.Vision

Methods

getItem :: MNIST m device batchSize -> Int -> m (Tensor device 'Float '[batchSize, 784], Tensor device 'Int64 '[batchSize])

keys :: MNIST m device batchSize -> Set Int

data DatasetOptions

Dataset options used when loading datasets. Specify shuffling behavior, the number of threads to use, and the buffer size used to store retrieved samples in each thread.

Constructors

DatasetOptions

Fields

• dataBufferSize :: Int

  Max number of samples stored in each buffer at a given time.

• numWorkers :: Int

  Number of threads retrieving samples.

• shuffle :: Sample

  The ordering of samples streamed.

datasetOpts :: Int -> DatasetOptions

Default DatasetOptions. The Int parameter specifies the number of workers, and sets the buffer size equal to the number of workers. Sampling is sequential.

data Sample where

A Sample determines the ordering of samples streamed out of a dataset. You can either order sequentially, or supply a random generator to shuffle samples.

Constructors

Sequential :: Sample

Shuffle :: RandomGen g => g -> Sample

Dataloading

streamFromMap :: forall m dataset k sample r. (Dataset m dataset k sample, MonadIO m, MonadBaseControl IO m) => DatasetOptions -> dataset -> ContT r m (ListT m sample, Sample)

Return a stream of samples from the given dataset, along with a new Sample value. The returned stream contains every sample returned by getItem for every key in the set of keys associated with the given dataset. The returned Sample value returns an updated Sample value, this will be identical to the original Sample value if sampling is Sequential but will return a new random number generator if sampling is Shuffle.