Text.Tokenify

Tokenify is a module used for generating a tokenizer from a regex based grammar

Documentation

tokenize :: CharSeq s => Tokenizer s a -> s -> Either String (Seq a)

tokenize will transform a CharSeq into a sequence of tokens

matchHead :: CharSeq s => Regex s -> s -> Maybe (s, s, Int)

Attmpts to match the front of a CharSeq with a Regex, if succeful, it returns a tuple containing

• The matched CharSeq
• The remaining CharSeq
• The amount of characters consumed