Control.Monad.Tokenizer.Class

A monad class for writing pure tokenizers in an imperative-looking way.

Main idea: You walk through the input string like a turtle, and everytime you find a token boundary, you call emit. If some specific kinds of tokens should be suppressed, you can discard them instead (or filter afterwards).

An concrete instance of this class is provided in Control.Monad.Tokenizer.

Monad class

class Monad m => MonadTokenizer m where

A monad for turtle tokenization.

Methods

walk :: m ()

Proceed to the next character

walkBack :: m ()

Walk back to the previous character, unless it was discarded/emitted.

peek :: m Char

Peek the current character

restore :: m ()

Restore the state after the last emit/discard.

emit :: m ()

Break at the current position and emit the scanned token

discard :: m ()

Break at the current position and discard the scanned token

isEOT :: m Bool

Have I reached the end of the input text?

lookAhead :: [Char] -> m Bool

Check if the next input chars agree with the given string

Instances

Tokenizable t => MonadTokenizer (Tokenizer t)

Defined in Control.Monad.Tokenizer

Methods

walk :: Tokenizer t ()

walkBack :: Tokenizer t ()

peek :: Tokenizer t Char

restore :: Tokenizer t ()

emit :: Tokenizer t ()

discard :: Tokenizer t ()

isEOT :: Tokenizer t Bool

lookAhead :: [Char] -> Tokenizer t Bool

untilEOT :: MonadTokenizer m => m () -> m ()

Repeat a given tokenizer as long as the end of text is not reached

Utilities

pop :: MonadTokenizer m => m Char

Peek the current character and proceed

walkWhile :: MonadTokenizer m => (Char -> Bool) -> m ()

Proceed as long as a given function succeeds

walkFold :: MonadTokenizer m => a -> (Char -> a -> Maybe a) -> m ()

Proceed as long as a given fold returns Just (generalization of walkWhile)

Text types

class Tokenizable t where

Text types that can be split by the Tokenizer monad. In this module, instances are provided for String, strict Text, and lazy Text. If you are dealing with ASCII ByteStrings, you can find instances in the modules Control.Monad.Tokenizer.Char8.Strict and Control.Monad.Tokenizer.Char8.Lazy

Methods

tnull :: t -> Bool

thead :: t -> Char

ttail :: t -> t

ttake :: Int -> t -> t

tdrop :: Int -> t -> t

tlower :: t -> t

Instances

Tokenizable ByteString

Assuming ASCII encoding

Defined in Control.Monad.Tokenizer.Char8.Lazy

Methods

tnull :: ByteString -> Bool

thead :: ByteString -> Char

ttail :: ByteString -> ByteString

ttake :: Int -> ByteString -> ByteString

tdrop :: Int -> ByteString -> ByteString

tlower :: ByteString -> ByteString

Tokenizable ByteString

Assuming ASCII encoding

Defined in Control.Monad.Tokenizer.Char8.Strict

Methods

tnull :: ByteString -> Bool

thead :: ByteString -> Char

ttail :: ByteString -> ByteString

ttake :: Int -> ByteString -> ByteString

tdrop :: Int -> ByteString -> ByteString

tlower :: ByteString -> ByteString

Tokenizable Text

Defined in Control.Monad.Tokenizer.Class

Methods

tnull :: Text -> Bool

thead :: Text -> Char

ttail :: Text -> Text

ttake :: Int -> Text -> Text

tdrop :: Int -> Text -> Text

tlower :: Text -> Text

Tokenizable Text

Defined in Control.Monad.Tokenizer.Class

Methods

tnull :: Text -> Bool

thead :: Text -> Char

ttail :: Text -> Text

ttake :: Int -> Text -> Text

tdrop :: Int -> Text -> Text

tlower :: Text -> Text

Tokenizable [Char]

Defined in Control.Monad.Tokenizer.Class

Methods

tnull :: [Char] -> Bool

thead :: [Char] -> Char

ttail :: [Char] -> [Char]

ttake :: Int -> [Char] -> [Char]

tdrop :: Int -> [Char] -> [Char]

tlower :: [Char] -> [Char]