This module provides functionality for tokenizing text streams to differentiate between printed characters and structural elements such as newlines. Once tokenized, such text streams can be manipulated with the functions in this module.
- data TextStream a = TS [TextStreamEntity a]
- data TextStreamEntity a
- data Token a
- tokenize :: String -> a -> TextStream a
- serialize :: TextStream a -> String
- tokenLen :: Token a -> Int
- entityToken :: TextStreamEntity a -> Token a
- streamEntities :: TextStream a -> [TextStreamEntity a]
- truncateLine :: Int -> [Token a] -> [Token a]
- wrapStream :: Eq a => Int -> TextStream a -> TextStream a
- findLines :: [TextStreamEntity a] -> [[TextStreamEntity a]]
A text stream is a list of text stream entities. A text stream |combines structural elements of the text (e.g., newlines) with the |text itself (words, whitespace, etc.).
A text stream entity is either a token or a structural element.
The type of text tokens. These should consist of printable characters and NOT presentation characters (e.g., newlines). Each type of token should have as its contents a string of characters all of the same type. Tokens are generalized over an attribute type which can be used to annotate each token.
To and from strings
Tokenize a string and apply a default attribute to every token in the resulting text stream.
Given a text stream, serialize the stream to its original textual representation. This discards token attribute metadata.
Token from an entity or raises an exception if the entity
does not contain a token. Used primarily for convenience
transformations in which the parameter is known to be a token
Given a list of tokens, truncate the list so that its underlying string representation does not exceed the specified column width.
Given a text stream and a wrapping width, return a new
TextStream with newlines inserted in appropriate places to wrap
the text at the specified column. This function results in text
wrapped without leading or trailing whitespace on wrapped lines,
although it preserves leading whitespace in the text which was not
the cause of the wrapping transformation.