Safe Haskell | None |
---|
The module provides functions for dictionary-driven analysis of the input text.
Documentation
A token is an element of the analysis result.
tokenize :: Text -> [Either Text Other]Source
Perform simple tokenization -- spaces and punctuation are treated as token ending markers.