Data.IDX.Conduit

Streaming decoders for the IDX format used in the MNIST handwritten digit recognition dataset [1].

Both sparse and dense decoders are provided. In either case, the range of the data is the same as the raw data (one unsigned byte per pixel).

Links

1) http://yann.lecun.com/exdb/mnist/

Labels

sourceIdxLabels

Arguments

:: MonadResource m

=> (ByteString -> Either e o)

parser for the labels, where the bytestring buffer contains exactly one unsigned byte

-> FilePath

filepath of uncompressed IDX labels file

-> ConduitT i (Either e o) m r

Outputs the labels corresponding to the data

mnistLabels :: ByteString -> Either String Int

Parser for the labels, can be plugged in as an argument to sourceIdxLabels

Data

Dense data buffers

sourceIdx

Arguments

:: MonadResource m

=> FilePath

filepath of uncompressed IDX data file

-> ConduitT a (Vector Word8) m ()

Outputs dense data buffers in the 0-255 range

In the case of MNIST dataset, 0 corresponds to the background of the image.

Sparse data buffers

sourceIdxSparse

Arguments

:: MonadResource m

=> FilePath

filepath of uncompressed IDX data file

-> ConduitT a (Sparse Word8) m ()

Outputs sparse data buffers (i.e without zero components)

This incurs at least one additional data copy of each vector, but the resulting vectors take up less space.

data Sparse a

Sparse buffer (containing only nonzero entries)

Instances

(Unbox a, Eq a) => Eq (Sparse a)

Defined in Data.IDX.Conduit

Methods

(==) :: Sparse a -> Sparse a -> Bool

(/=) :: Sparse a -> Sparse a -> Bool

(Show a, Unbox a) => Show (Sparse a)

Defined in Data.IDX.Conduit

Methods

showsPrec :: Int -> Sparse a -> ShowS

show :: Sparse a -> String

showList :: [Sparse a] -> ShowS

sBufSize :: Sparse a -> Int

total number of entries in the _dense_ buffer, i.e. including the zeros

sNzComponents :: Sparse a -> Vector (Int, a)

nonzero components, together with the linear index into their dense counterpart