Glider.NLP.Tokenizer

This module contains functions which parses text into tokens. tokens are not normalized. If you need all tokens from the document then check function tokenize. If you need only words (na dots, numbers etc.) then check function getWords.

Documentation

data Token

Token type

Constructors

Word Text

Number Text

Punctuation Char

Symbol Char

Whitespace

Unknown Char

Instances

Eq Token

Show Token

foldCase :: [Text] -> [Text]

Convert all words to the same case

getWords :: [Token] -> [Text]

Exctract all words from tokens

 getWords "one two." == ["one", "two"] 

tokenize :: Text -> [Token]

Split text into tokens

 tokenize "one two." == [Word "one", Whitespace, Word "two", "Separator "."] 

wordParser :: Parser

Parse word

numberParser :: Parser

Parse number

punctuationParser :: Parser

Parse punctuation

symbolParser :: Parser

Parse symbol

spaceParser :: Parser

Parse whitespaces

allParser :: Parser

Apply all parsers to the input. Return result from the first which will parse correctly given text.