Data.Repa.Flow.Auto.IO

Read and write files.

The functions in this module are wrappers for the ones in Data.Repa.Flow.Default.SizedIO that use a default chunk size of 64kBytes and just call error if the source file appears corruped.

Documentation

defaultChunkSize :: Integer

The default chunk size of 64kBytes.

Buckets

module Data.Repa.Flow.IO.Bucket

Sourcing

sourceBytes :: Array B Bucket -> IO (Sources Word8)

Read data from some files, using the given chunk length.

sourceChars :: Array B Bucket -> IO (Sources Char)

Read 8-bit ASCII characters from some files, using the given chunk length.

sourceLines :: Array B Bucket -> IO (Sources (Array A Char))

Read complete lines of data from a text file, using the given chunk length. We read as many complete lines as will fit into each chunk.

• The trailing new-line characters are discarded.
• Data is read into foreign memory without copying it through the GHC heap.
• The provided file handle must support seeking, else you'll get an exception.
• Each file is closed the first time the consumer tries to pull a line from the associated stream when no more are available.

sourceRecords

Arguments

:: (Word8 -> Bool)

Detect the end of a record.

-> Array B Bucket

Input Buckets.

-> IO (Sources (Array A Word8))

Read complete records of data form a file, into chunks of the given length. We read as many complete records as will fit into each chunk.

The records are separated by a special terminating character, which the given predicate detects. After reading a chunk of data we seek the file to just after the last complete record that was read, so we can continue to read more complete records next time.

If we cannot fit at least one complete record in the chunk then perform the given failure action. Limiting the chunk length guards against the case where a large input file is malformed, as we won't try to read the whole file into memory.

• Data is read into foreign memory without copying it through the GHC heap.
• The provided file handle must support seeking, else you'll get an exception.
• Each file is closed the first time the consumer tries to pull a record from the associated stream when no more are available.

sourceTSV :: Array B Bucket -> IO (Sources (Array A (Array A Char)))

Read a file containing Tab-Separated-Values.

sourceCSV :: Array B Bucket -> IO (Sources (Array A (Array A Char)))

Read a file containing Comma-Separated-Values.

sourceFormatLn

Arguments

:: (Unpackable format, Target A (Value format))

=> Integer

Chunk length.

-> IO ()

Action when a line is too long.

-> IO (Array A Word8 -> IO ())

Action if we can't convert a value.

-> format

Format of each line.

-> Array B Bucket

Source buckets.

-> IO (Sources (Value format))

Read the lines of a text file, converting each line to values with the given format.

Sinking

sinkBytes :: Array B Bucket -> IO (Sinks Word8)

Write 8-bit bytes to some files.

sinkLines :: Array B Bucket -> IO (Sinks (Array A Char))

Write vectors of text lines to the given files handles.

sinkChars :: Array B Bucket -> IO (Sinks Char)

Write 8-bit ASCII characters to some files.

sinkFormatLn

Arguments

:: (Packable format, Bulk A (Value format), Show format)

=> format

Binary format for each value.

-> IO ()

Action when a value cannot be serialized.

-> Array B Bucket

Output buckets.

-> IO (Sinks (Value format))

Create sinks that convert values to some format and writes them to buckets.

> import Data.Repa.Flow           as F
> import Data.Repa.Convert.Format as F
> :{
  do let format = FixString ASCII 10 :*: Float64be :*: Int16be
     let vals   = listFormat format
                   [ "red"   :*: 5.3    :*: 100
                   , "green" :*: 2.8    :*: 93 
                   , "blue"  :*: 0.99   :*: 42 ]

     ss  <- F.fromList 1 vals
     out <- toFiles' ["colors.bin"] 
         $  sinkFormatLn format (error "convert failed")
     drainS ss out
  :}