Language.Haskell.Lexer
- type PosToken = (Token, (Pos, String))
- data Token
- = Varid
- | Conid
- | Varsym
- | Consym
- | Reservedid
- | Reservedop
- | Specialid
- | IntLit
- | FloatLit
- | CharLit
- | StringLit
- | Qvarid
- | Qconid
- | Qvarsym
- | Qconsym
- | Special
- | Whitespace
- | NestedCommentStart
- | NestedComment
- | LiterateComment
- | Commentstart
- | Comment
- | ErrorToken
- | GotEOF
- | TheRest
- | ModuleName
- | ModuleAlias
- | Layout
- | Indent Int
- | Open Int
- lexerPass0 :: String -> [PosToken]
- lexerPass0' :: Pos -> String -> [PosToken]
- lexerPass1 :: String -> [PosToken]
- rmSpace :: [PosToken] -> [PosToken]
- layoutPre :: [PosToken] -> [PosToken]
- data Pos = Pos {}
- simpPos :: Pos -> (Int, Int)
- startPos :: Pos
- nextPos :: Pos -> String -> Pos
- nextPos1 :: Pos -> Char -> Pos
Documentation
Haskell token classifications:
Constructors
| Varid | Variable |
| Conid | Constructor |
| Varsym | Variable operator |
| Consym | Constructor operator |
| Reservedid | Reserved keyword |
| Reservedop | Reserved operator |
| Specialid | |
| IntLit | Integral numeric literal |
| FloatLit | Fractional numeric literal |
| CharLit | Character literal |
| StringLit | String literal |
| Qvarid | Qualified variable |
| Qconid | Qualified constructor |
| Qvarsym | Qualified variable operator |
| Qconsym | Qualified constructor operator |
| Special | |
| Whitespace | White space |
| NestedCommentStart | Internal: causes a call to an external function |
| NestedComment | A nested comment ({- ... -}) |
| LiterateComment | Not handled by the lexer |
| Commentstart | Dashes |
| Comment | The stuff after the dashes |
| ErrorToken | |
| GotEOF | |
| TheRest | |
| ModuleName | |
| ModuleAlias | recognized in a later pass |
| Layout | for implicit braces |
| Indent Int | <n>, to preceed first token on each line |
| Open Int | {n}, after let, where, do or of, if not followed by a "{" |
lexerPass0 :: String -> [PosToken]Source
Tokenize and add position information. Preserves white space, and does not insert extra tokens due to layout.
lexerPass0' :: Pos -> String -> [PosToken]Source
Same as lexerPass0, except that it uses the given start position.
lexerPass1 :: String -> [PosToken]Source
The function lexerPass1 handles the part of lexical analysis that
can be done independently of the parser---the tokenization and the
addition of the extra layout tokens <n> and {n}, as specified in
section 9.3 of the revised Haskell 98 Report.
rmSpace :: [PosToken] -> [PosToken]Source
Remove token that are not meaningful (e.g., white space and comments).
layoutPre :: [PosToken] -> [PosToken]Source
This is an implementation of Haskell layout, as specified in section 9.3 of the revised Haskell 98 report. This preprocessor inserts the extra <n> and {n} tokens.
The posisiotn within a file.