Copyright | © 2018–present Mark Karpov |
---|---|
License | BSD 3 clause |
Maintainer | Mark Karpov <markkarpov92@gmail.com> |
Stability | experimental |
Portability | portable |
Safe Haskell | Safe-Inferred |
Language | GHC2021 |
Documentation
Token types that are used as tags to mark spans of source code.
KeywordTok | Keyword |
PragmaTok | Pragmas |
SymbolTok | Symbols (punctuation that is not an operator) |
VariableTok | Variable name (term level) |
ConstructorTok | Data/type constructor |
OperatorTok | Operator |
CharTok | Character |
StringTok | String |
IntegerTok | Integer |
RationalTok | Rational number |
CommentTok | Comment (including Haddocks) |
SpaceTok | Space filling |
OtherTok | Something else? |
The start and end positions of a span. The arguments of the data constructor contain in order:
- Line number of start position of a span
- Column number of start position of a span
- Line number of end position of a span
- Column number of end position of a span
Since: 0.0.2.0
tokenizeHaskell :: Text -> Maybe [(Token, Text)] Source #
Tokenize Haskell source code. If the code cannot be parsed, return
Nothing
. Otherwise return the original input tagged by Token
s.
Nothing
is rarely returned, if ever, because it looks like the lexer is
capable of interpreting almost any text as a stream of GHC tokens.
The parser does not require the input source code to form a valid Haskell
program, so as long as the lexer can decompose your input (most of the
time), it'll return something in Just
.
tokenizeHaskellLoc :: Text -> Maybe [(Token, Loc)] Source #
Similar to tokenizeHaskell
, but instead of Text
chunks provides
locations of corresponding spans in the given input stream.
Since: 0.0.2.0