úÎ+0)G      Safe-Inferred 4The EitherList is a newtype-wrapped list of Eithers.ÿTA Tokenizer is function which takes a list and returns a list of Eithers (wrapped in a newtype). Right Texts will be passed on for processing to tokenizers down the pipeline. Left Texts will be passed through the pipeline unchanged. Use a Left Texts in a tokenizer to protect certain tokens from further processing (e.g. see the c tokenizer). You can define your own custom tokenizer pipelines by chaining tokenizers together:=Split string into words using the default tokenizer pipeline Run a tokenizer"Detect common uris and freeze them'Split off initial and final punctuation  Split off word-final punctuation "Split off word-initial punctuation ÜSplit tokens on transitions between punctuation and non-punctuation characters. This tokenizer is not included in defaultTokenizer pipeline because dealing with word-internal punctuation is quite application specific. *Split words ending in n't, and freeze n't \Split common contractions off and freeze them. | Currently deals with: 'm, 's, 'd, 've, 'llFSplit string on whitespace. This is just a wrapper for Data.List.words  !    ! Safe-Inferred 4The EitherList is a newtype-wrapped list of Eithers.ÿYA Tokenizer is function which takes a list and returns a list of Eithers (wrapped in a newtype). Right Strings will be passed on for processing to tokenizers down the pipeline. Left Strings will be passed through the pipeline unchanged. Use a Left String in a tokenizer to protect certain tokens from further processing (e.g. see the c tokenizer). You can define your own custom tokenizer pipelines by chaining tokenizers together:=Split string into words using the default tokenizer pipeline Run a tokenizer"Detect common uris and freeze them'Split off initial and final punctuation Split off word-final punctuation"Split off word-initial punctuationÜSplit tokens on transitions between punctuation and non-punctuation characters. This tokenizer is not included in defaultTokenizer pipeline because dealing with word-internal punctuation is quite application specific.*Split words ending in n't, and freeze n't \Split common contractions off and freeze them. | Currently deals with: 'm, 's, 'd, 've, 'llFSplit string on whitespace. This is just a wrapper for Data.List.words"#$"#$ Safe-Inferred%          tokenize-0.2.2NLP.Tokenize.TextNLP.Tokenize.String NLP.Tokenize EitherListEunE TokenizertokenizerundefaultTokenizeruris punctuationfinalPunctuationinitialPunctuationallPunctuation negatives contractions whitespacehyphensunwrapexamples$fMonadEitherList