Language.Python.Version3.Lexer

Lexical analysis for Python version 3.x programs. See: http://docs.python.org/3.1/reference/lexical_analysis.html.

Lexical analysis

lex

Arguments

:: String

The input stream (python source code).

-> String

The name of the python source (filename or input device).

-> Either ParseError [Token]

An error or a list of tokens.

Parse a string into a list of Python Tokens, or return an error.

lexOneToken

Arguments

:: String

The input stream (python source code).

-> String

The name of the python source (filename or input device).

-> Either ParseError (Token, String)

An error or the next token and the rest of the input after the token.

Try to lex the first token in an input string. Return either a parse error or a pair containing the next token and the rest of the input after the token.

lexer :: P [Token]

Lex a sequence of tokens.

initLexState :: String -> String -> ParseState