Data.Repa.Flow.IO.Bucket

Documentation

data Bucket

A bucket represents portion of a whole data-set on disk, and contains a file handle that points to the next piece of data to be read or written.

The bucket could be created from a portion of a single flat file, or be one file of a pre-split data set. The main advantage over a plain Handle is that a Bucket can represent a small portion of a single large file.

bucketLength :: Bucket -> Maybe Integer

Maximum length of the bucket, in bytes.

If Nothing then the length is indeterminate, which is used when writing to files.

openBucket :: FilePath -> IOMode -> IO Bucket

Open a file as a single bucket.

hBucket :: Handle -> IO Bucket

Wrap an existing file handle as a bucket.

Reading

fromFiles

Arguments

:: (Bulk l FilePath, Target l Bucket)

=> Array l FilePath

Files to open.

-> (Array l Bucket -> IO b)

Consumer.

-> IO b

Open some files as buckets and use them as Sources.

fromFiles' :: [FilePath] -> (Array B Bucket -> IO b) -> IO b

Like fromFiles, but take a list of file paths.

fromDir :: FilePath -> (Array B Bucket -> IO b) -> IO b

Open all the files in a directory as separate buckets.

This operation may fail with the same exceptions as getDirectoryContents.

fromSplitFile

Arguments

:: Int

Number of buckets.

-> (Word8 -> Bool)

Detect the end of a record.

-> FilePath

File to open.

-> (Array B Bucket -> IO b)

Consumer.

-> IO b

Open a file containing atomic records and split it into the given number of evenly sized buckets.

The records are separated by a special terminating charater, which the given predicate detects. The file is split cleanly on record boundaries, so we get a whole number of records in each bucket. As the records can be of varying size the buckets are not guaranteed to have exactly the same length, in either records or buckets, though we try to give them the approximatly the same number of bytes.

fromSplitFileAt

Arguments

:: Int

Number of buckets.

-> (Word8 -> Bool)

Detect the end of a record.

-> FilePath

File to open.

-> Integer

Starting offset.

-> (Array B Bucket -> IO b)

Consumer.

-> IO b

Like fromSplitFile but start at the given offset.

Writing

toFiles

Arguments

:: (Bulk l FilePath, Target l Bucket)

=> Array l FilePath

File paths.

-> (Array l Bucket -> IO b)

Consumer.

-> IO b

Open some files for writing as individual buckets and pass them to the given consumer.

toFiles' :: [FilePath] -> (Array B Bucket -> IO b) -> IO b

Like toFiles, but take a list of file paths.

toDir

Arguments

:: Int

Number of buckets to create.

-> FilePath

Path to directory.

-> (Array B Bucket -> IO b)

Consumer.

-> IO b

Create a new directory of the given name, containing the given number of buckets.

If the directory is named somedir then the files are named somedir/000000, somedir/000001, somedir/000002 and so on.

toDirs'

Arguments

:: Int

Number of buckets to create per directory.

-> [FilePath]

Paths to directories.

-> (Array (E B DIM2) Bucket -> IO b)

Consumer.

-> IO b

Given a list of directories, create those directories and open the given number of output files per directory.

In the resulting array of buckets, the outer dimension indexes each directory, and the inner one indexes each file in its directory.

For each directory somedir the files are named somedir/000000, somedir/000001, somedir/000002 and so on.

Bucket IO

bClose :: Bucket -> IO ()

Close a bucket, releasing the contained file handle.

bIsOpen :: Bucket -> IO Bool

Check if the bucket is currently open.

bAtEnd :: Bucket -> IO Bool

Check if the contained file handle is at the end of the bucket.

bSeek :: Bucket -> SeekMode -> Integer -> IO ()

Seek to a position with a bucket.

bGetArray :: Bucket -> Integer -> IO (Array F Word8)

Get some data from a bucket.

bPutArray :: Bucket -> Array F Word8 -> IO ()

Put some data in a bucket.