nerf-0.5.3: Nerf, the named entity recognition tool based on linear-chain CRFs

Safe HaskellSafe-Inferred
LanguageHaskell98

NLP.Nerf.Tokenize

Contents

Description

The module implements the tokenization used within Nerf and some other tokenization-related stuff.

Synopsis

Tokenization

tokenize :: String -> [String] Source

Tokenize sentence using the default tokenizer.

Synchronization

class Word a where Source

A class of objects which can be converted to String.

Methods

word :: a -> String Source

Instances

sync Source

Arguments

:: (Word b, Word c) 
=> NeForest a b

NE forest

-> [c]

New tokenization

-> NeForest a c

Resulting NE forest

Synchronize the list of NE trees with the new tokenization.