NLP.Tokenize.Text

NLP Tokenizer, adapted to use Text instead of Strings from the tokenize package.

Documentation

newtype EitherList a b

The EitherList is a newtype-wrapped list of Eithers.

Constructors

E

Fields

unE :: [Either a b]

Instances

Monad (EitherList a)

type Tokenizer = Text -> EitherList Text Text

A Tokenizer is function which takes a list and returns a list of Eithers (wrapped in a newtype). Right Strings will be passed on for processing to tokenizers down the pipeline. Left Strings will be passed through the pipeline unchanged. Use a Left String in a tokenizer to protect certain tokens from further processing (e.g. see the uris tokenizer).

tokenize :: Text -> [Text]

Split string into words using the default tokenizer pipeline

run :: Tokenizer -> Text -> [Text]

Run a tokenizer

defaultTokenizer :: Tokenizer

whitespace :: Tokenizer

Split string on whitespace. This is just a wrapper for Data.List.words

uris :: Tokenizer

Detect common uris and freeze them

punctuation :: Tokenizer

Split off initial and final punctuation

finalPunctuation :: Tokenizer

Split off word-final punctuation

initialPunctuation :: Tokenizer

Split off word-initial punctuation

contractions :: Tokenizer

Split common contractions off and freeze them. | Currently deals with: 'm, 's, 'd, 've, 'll

negatives :: Tokenizer

Split words ending in n't, and freeze n't