Glider.NLP.Tokenizer

This module contains functions which parses text into tokens. Tokens are not normalized. If you need all tokens from the document then use function "tokenize". If you need only words (no dots, numbers etc.) then check function "getWords".

Documentation

data Token

Token type

Constructors

Word Text

Number Text

Punctuation Char

Symbol Char

Whitespace

Unknown Char

Instances

Eq Token

Methods

(==) :: Token -> Token -> Bool

(/=) :: Token -> Token -> Bool

Show Token

Methods

showsPrec :: Int -> Token -> ShowS

show :: Token -> String

showList :: [Token] -> ShowS

foldCase :: [Text] -> [Text]

Convert all words to the same case

getWords :: [Token] -> [Text]

Extract all words from tokens

getWords "one two." == ["one", "two"] 

tokenize :: Text -> [Token]

Split text into tokens

tokenize "one two." == [Word "one", Whitespace, Word "two", "Separator "."] 

wordParser :: Parser

Parse word

numberParser :: Parser

Parse number

punctuationParser :: Parser

Parse punctuation

symbolParser :: Parser

Parse symbol

spaceParser :: Parser

Parse whitespaces

allParser :: Parser

Apply all parsers to the input. Return result from the first which will parse correctly given text.