NLP.Nerf.Tokenize

The module implements the tokenization used within Nerf and some other tokenization-related stuff.

Tokenization

tokenize :: String -> [String]

Tokenize sentence using the default tokenizer.

Synchronization

class Word a where

A class of objects which can be converted to String.

Methods

word :: a -> String

Instances

Word String

Word Text

Word Text

moveNEs :: (Word b, Word c) => NeForest a b -> [c] -> NeForest a c

Synchronize named entities with tokenization represented by the second function argument. Of course, both arguments should relate to the same sentence.