NLP.Nerf.Tokenize

The module implements the tokenization used within Nerf and some other tokenization-related stuff.

Tokenization

tokenize :: String -> [String]

Tokenize sentence using the default tokenizer.

Synchronization

class Word a

A class of objects with size.

Instances

Word String

Word Text

Word Text

moveNEs :: Word b => NeForest a b -> [b] -> NeForest a b

Synchronize named entities with tokenization represented by the second function argument. Of course, both arguments should relate to the same sentence.