Data.IDX

A package for reading and writing data in the IDX format. This data format is used for machine-learning data sets like the MINST database of handwritten digits (http://yann.lecun.com/exdb/mnist/)

Data types

data IDXData

Datatype for storing IDXData. Internally data is always stored either as Int or Double unboxed vectors. However when binary serialization is used, the data is serialized according to the IDXContentType.

Instances

Show IDXData

Binary IDXData

data IDXLabels

Instances

Binary IDXLabels

data IDXContentType where

A type to describe the content, according to IDX spec

Constructors

IDXUnsignedByte :: IDXContentType

IDXSignedByte :: IDXContentType

IDXShort :: IDXContentType

IDXInt :: IDXContentType

IDXFloat :: IDXContentType

IDXDouble :: IDXContentType

Instances

Show IDXContentType

Binary IDXContentType

Accessing data

idxType :: IDXData -> IDXContentType

Return the what type the data is stored in

idxDimensions :: IDXData -> Vector Int

Return an unboxed Vector of Int dimensions

isIDXReal :: IDXData -> Bool

Return wether the data in this IDXData value is stored as double values

isIDXIntegral :: IDXData -> Bool

Return wether the data in this IDXData value is stored as integral values

Raw data

idxDoubleContent :: IDXData -> Vector Double

Return contained doubles, if no doubles are contained convert the content to double by using fromIntegral. Data is stored like in a C-array, i.e. the last index changes first.

idxIntContent :: IDXData -> Vector Int

Return contained ints, if no ints are contained, convert content to ints by using round. Data is stored like in a C-array, i.e. the last index changes first.

Labeled data

labeledIntData :: IDXLabels -> IDXData -> Maybe [(Int, Vector Int)]

Partition a dataset and label each subpartition, return int values

labeledDoubleData :: IDXLabels -> IDXData -> Maybe [(Int, Vector Double)]

Partition a dataset and label each subpartition, return double values

IO / Serialization

IDXLabels

ByteString serialization

encodeIDXLabels :: IDXLabels -> ByteString

decodeIDXLabels :: ByteString -> Maybe IDXLabels

FileIO

encodeIDXLabelsFile :: IDXLabels -> FilePath -> IO ()

Read data from a file, return Nothing if something doesn't work

decodeIDXLabelsFile :: FilePath -> IO (Maybe IDXLabels)

Read labels from a file, return Nothing if something doesn't work

IDXData (e.g. images)

ByteString serialization

encodeIDX :: IDXData -> ByteString

decodeIDX :: ByteString -> Maybe IDXData

File IO

encodeIDXFile :: IDXData -> FilePath -> IO ()

decodeIDXFile :: FilePath -> IO (Maybe IDXData)