Hpp.Tokens

Tokenization breaks a String into pieces of whitespace, constants, symbols, and identifiers.

Documentation

data Token

Tokenization is words except the white space is tagged rather than discarded.

Constructors

Important String

Identifiers, symbols, and constants

Other String

White space, etc.

Instances

Eq Token

Ord Token

Show Token

detok :: Token -> String

Extract the contents of a Token.

isImportant :: Token -> Bool

True if the given Token is Important; False otherwise.

notImportant :: Token -> Bool

True if the given Token is not Important; False otherwise.

importants :: [Token] -> [String]

Return the contents of only Important (non-space) tokens.

trimUnimportant :: [Token] -> [Token]

Trim Other Tokens from both ends of a list of Tokens.

newLine :: Token -> Bool

Is a Token a newline character?

tokWords :: String -> [Token]

Break a String into space and non-whitespace runs.

skipLiteral :: ((String -> String) -> String -> r) -> String -> r

If you encounter a string literal, call this helper with a double-barreled continuation and the rest of your input. The continuation will expect the remainder of the string literal as the first argument, and the remaining input as the second argument.

splits :: (Char -> Bool) -> String -> [String]

splits isDelimiter str tokenizes str using isDelimiter as a delimiter predicate. Leading whitespace is also stripped from tokens.

validIdentifierChar :: Char -> Bool

Predicate on space characters based on something approximating valid identifier syntax. This is used to break apart non-space characters.

fixExponents :: [Token] -> [Token]

Something like 12E+FOO is a single pre-processor token, so FOO should not be macro expanded.

tokenize :: String -> [Token]

Break an input String into a sequence of Tokens. Warning: This may not exactly correspond to your target language's definition of a valid identifier!

detokenize :: [Token] -> String

Collapse a sequence of Tokens back into a String. detokenize . tokenize == id.