Safe Haskell | None |
---|---|
Language | Haskell98 |
Read and write files.
The functions in this module are wrappers for the ones in
Data.Repa.Flow.Default.SizedIO that use a default chunk size of
64kBytes and just call error
if the source file appears corruped.
- defaultChunkSize :: Integer
- module Data.Repa.Flow.IO.Bucket
- sourceBytes :: Array B Bucket -> IO (Sources Word8)
- sourceChars :: Array B Bucket -> IO (Sources Char)
- sourceLines :: Array B Bucket -> IO (Sources (Array A Char))
- sourceRecords :: (Word8 -> Bool) -> Array B Bucket -> IO (Sources (Array A Word8))
- sourceTSV :: Array B Bucket -> IO (Sources (Array A (Array A Char)))
- sourceCSV :: Array B Bucket -> IO (Sources (Array A (Array A Char)))
- sourcePacked :: (Packable format, Target A (Value format)) => format -> IO () -> Array B Bucket -> IO (Sources (Value format))
- sinkBytes :: Array B Bucket -> IO (Sinks Word8)
- sinkLines :: Array B Bucket -> IO (Sinks (Array A Char))
- sinkChars :: Array B Bucket -> IO (Sinks Char)
- sinkPacked :: (Packable format, Bulk A (Value format)) => format -> IO () -> Array B Bucket -> IO (Sinks (Value format))
- toTable :: (Packable format, Bulk A (Value format)) => FilePath -> Int -> format -> IO () -> IO (Maybe (Sinks (Value format)))
- fromTable :: (Packable format, Target A (Value format)) => FilePath -> format -> IO () -> IO (Maybe (Sources (Value format)))
Documentation
defaultChunkSize :: Integer Source
The default chunk size of 64kBytes.
Buckets
module Data.Repa.Flow.IO.Bucket
Sourcing
sourceBytes :: Array B Bucket -> IO (Sources Word8) Source
Read data from some files, using the given chunk length.
sourceChars :: Array B Bucket -> IO (Sources Char) Source
Read 8-bit ASCII characters from some files, using the given chunk length.
sourceLines :: Array B Bucket -> IO (Sources (Array A Char)) Source
Read complete lines of data from a text file, using the given chunk length. We read as many complete lines as will fit into each chunk.
- The trailing new-line characters are discarded.
- Data is read into foreign memory without copying it through the GHC heap.
- The provided file handle must support seeking, else you'll get an exception.
- Each file is closed the first time the consumer tries to pull a line from the associated stream when no more are available.
:: (Word8 -> Bool) | Detect the end of a record. |
-> Array B Bucket | Input Buckets. |
-> IO (Sources (Array A Word8)) |
Read complete records of data form a file, into chunks of the given length. We read as many complete records as will fit into each chunk.
The records are separated by a special terminating character, which the given predicate detects. After reading a chunk of data we seek the file to just after the last complete record that was read, so we can continue to read more complete records next time.
If we cannot fit at least one complete record in the chunk then perform the given failure action. Limiting the chunk length guards against the case where a large input file is malformed, as we won't try to read the whole file into memory.
- Data is read into foreign memory without copying it through the GHC heap.
- The provided file handle must support seeking, else you'll get an exception.
- Each file is closed the first time the consumer tries to pull a record from the associated stream when no more are available.
sourceTSV :: Array B Bucket -> IO (Sources (Array A (Array A Char))) Source
Read a file containing Tab-Separated-Values.
sourceCSV :: Array B Bucket -> IO (Sources (Array A (Array A Char))) Source
Read a file containing Comma-Separated-Values.
:: (Packable format, Target A (Value format)) | |
=> format | Binary format for each value. |
-> IO () | Action when a value cannot be converted. |
-> Array B Bucket | Input buckets. |
-> IO (Sources (Value format)) |
Read packed binary data from some buckets and unpack the values
to some Sources
.
The following uses the colors.bin
file produced by the sinkPacked
example:
> import Data.Repa.Flow as F > import Data.Repa.Convert.Format as F > :{ do let format = FixString ASCII 10 :*: Float64be :*: Int16be ss <- fromFiles' ["colors.bin"] $ sourcePacked format (error "convert failed") toList1 0 ss :} ["red" :*: (5.3 :*: 100), "green" :*: (2.8 :*: 93), "blue" :*: (0.99 :*: 42)]
Sinking
sinkLines :: Array B Bucket -> IO (Sinks (Array A Char)) Source
Write vectors of text lines to the given files handles.
:: (Packable format, Bulk A (Value format)) | |
=> format | Binary format for each value. |
-> IO () | Action when a value cannot be serialized. |
-> Array B Bucket | Output buckets. |
-> IO (Sinks (Value format)) |
Create sinks that convert values to a packed binary format and writes them to some buckets.
> import Data.Repa.Flow as F > import Data.Repa.Convert.Format as F > :{ do let format = FixString ASCII 10 :*: Float64be :*: Int16be let vals = listFormat format [ "red" :*: 5.3 :*: 100 , "green" :*: 2.8 :*: 93 , "blue" :*: 0.99 :*: 42 ] ss <- F.fromList 1 vals out <- toFiles' ["colors.bin"] $ sinkPacked format (error "convert failed") drainS ss out :}