| Safe Haskell | None |
|---|
NLP.HistPL.Analyse
Description
The module provides functions for dictionary-driven analysis of the input text.
Documentation
A token is an element of the analysis result.
Constructors
| Token | |
A punctuation or a space.
tokenize :: Text -> [Either Text Other]Source
Perform simple tokenization -- spaces and punctuation are treated as token ending markers.