The ngrams-loader package

[Tags: library, mit, program]

Ngrams loader based on http:www.ngrams.info format


[Skip to ReadMe]

Properties

Versions0.1.0.0, 0.1.0.1
Change logNone available
Dependenciesattoparsec (==0.11.1.*), base (==4.6.*), machines (>=0.2.5 && <0.3), mtl, ngrams-loader, parseargs (==0.1.5.*), resourcet (>=0.4.3 && <0.5), sqlite-simple (>=0.4.5 && <0.5), text (>=0.11 && <1.2) [details]
LicenseMIT
CopyrightCopyright (C) 2014 Yorick Laupa
AuthorYorick Laupa
MaintainerYorick Laupa <yo.eight@gmail.com>
CategoryData
Home pagehttp://github.com/YoEight/ngrams-loader
Bug trackerhttps://github.com/YoEight/ngrams-loader/issues
Source repositoryhead: git clone git://github.com/YoEight/ngrams-loader.git
Executablesngrams-loader
UploadedTue Mar 25 09:40:50 UTC 2014 by YorickLaupa
DistributionsNixOS:0.1.0.1
Downloads292 total (15 in last 30 days)
Votes
0 []
StatusDocs available [build log]
Successful builds reported [all 1 reports]

Modules

[Index]

Downloads

Maintainers' corner

For package maintainers and hackage trustees

Readme for ngrams-loader-0.1.0.1

ngrams-loader

Ngrams loader based on http://www.ngrams.info format

Build Status

Installation

Supposed you have at least cabal 1.18 installed

$ cabal sandbox init
$ cabal install --only-dependencies
$ cabal configure
$ cabal install

-- program located in ~/.cabal-sandbox/bin

Usage

usage: ngrams-loader [options] <n-grams file> <SQLite file>
  [-2,--bigram]     Parses bigrams
  [-3,--trigram]    Parses trigrams
  [-4,--quadgram]   Parses 4-grams
  [-5,--pentagram]  Parses 5-grams
  [-c,--create]     Creates table before inserts
  <n-grams file>    N-grams file
  <SQLite file>     SQlite db file

Example

ngrams-loader --bigram --create w2.txt bigram.db

It parses each line of w2.txt as a bigram, create bigram table before performing inserts and saves everything in bigram.db

Figures

Specs

ngrams-loader --bigram --create w2.txt bigram.db gets

real    0m16.244s
user    0m15.597s
sys   0m0.143s

Sql Schemas

Bigram

create table bigrams(
  frequence int,
  word1 varchar(100),
  word2 varchar(100)
);

Trigram

create table tridgrams(
  frequence int,
  word1 varchar(100),
  word2 varchar(100),
  word3 varchar(100)
);

4-gram

create table quadgrams(
  frequence int,
  word1 varchar(100),
  word2 varchar(100),
  word3 varchar(100),
  word4 varchar(100)
);

5-gram

create table pentagrams(
  frequence int,
  word1 varchar(100),
  word2 varchar(100),
  word3 varchar(100),
  word4 varchar(100),
  word5 varchar(100)
);