NLP.Nerf.Tokenize

The module implements the tokenization used within Nerf and some other tokenization-related stuff.

Tokenization

tokenize :: String -> [String]

Tokenize sentence using the default tokenizer.

Synchronization

class Word a where

A class of objects which can be converted to String.

Methods

word :: a -> String

Instances

Word Text

Defined in NLP.Nerf.Tokenize

Methods

word :: Text -> String

Word Text

Defined in NLP.Nerf.Tokenize

Methods

word :: Text -> String

Word String

Defined in NLP.Nerf.Tokenize

Methods

word :: String -> String

sync

Arguments

:: (Word b, Word c)

=> NeForest a b

NE forest

-> [c]

New tokenization

-> NeForest a c

Resulting NE forest

Synchronize the list of NE trees with the new tokenization.