ACE.Tokenizer

Tokenizer for ACE. Tokens retain source locations (line and column).

Documentation

tokenize :: Text -> Either String [Token]

Tokenize some complete ACE text.

tokenizer :: Parser ([Token], (Int, Int))

The tokenizer.

token :: (Int, Int) -> Parser (Token, (Int, Int))

Parse a token.

number :: (Int, Int) -> Parser (Token, (Int, Int))

Parse a number.

quotedString :: (Int, Int) -> Parser (Token, (Int, Int))

Parse a quoted string, "foobar".

period :: (Int, Int) -> Parser (Token, (Int, Int))

Parse a period ".".

comma :: (Int, Int) -> Parser (Token, (Int, Int))

Parse a comma ",".

questionMark :: (Int, Int) -> Parser (Token, (Int, Int))

Parse a question mark "?".

word :: (Int, Int) -> Parser (Token, (Int, Int))

Parse a word, which is any sequence of non-whitespace words containing none of the other token characters.

genitive :: (Int, Int) -> Parser (Maybe (Token, (Int, Int)))

Parse the Saxon genitive ' or 's. This is ran after parsing every token, but is expected to fail most of the time.

manyWithPos :: (Monad m, Alternative m) => ((t, t1) -> m (a, (t, t1))) -> ((t, t1) -> m (Maybe (a, (t, t1)))) -> (t, t1) -> m ([a], (t, t1))

Like many, but retains the current source position and supports postfix-parsing of the genitive apostrophe.

spaces :: (Int, Int) -> Parser (Int, Int)

Skip spaces (space, newline, tab (=4 spaces)) and keep positioning information up to date.