The htsn-import package

[Tags:deprecated, gpl, program, test]


htsn-import [OPTIONS] [FILES]

The Sports Network offers an XML feed containing various sports news and statistics. Our sister program htsn is capable of retrieving the feed and saving the individual XML documents contained therein. But what to do with them?

The purpose of htsn-import is to take these XML documents and get them into something we can use, a relational database management system (RDBMS), i.e. "a SQL database". The structure of relational database, is, well, relational, and the feed XML is not. So there is some work to do before the data can be inserted.

First, we must parse the XML. Each supported document type (see below) has a full pickle/unpickle implementation ("pickle" is simply a synonym for serialize here). That means that we parse the entire document into a data structure, and if we pickle (serialize) that data structure, we get the exact same XML document tha we started with.

This is important for two reasons. First, it serves as a second level of validation. The first validation is performed by the XML parser, but if that succeeds and unpicking fails, we know that something is fishy. Second, we don't ever want to be surprised by some new element or attribute showing up in the XML. The fact that we can unpickle the whole thing now means that we won't be surprised in the future.

The aforementioned feature is especially important because we automatically migrate the database schema every time we import a document. If you attempt to import a "newsxml.dtd" document, all database objects relating to the news will be created if they do not exist. We don't want the schema to change out from under us without warning, so it's important that no XML be parsed that would result in a different schema than we had previously. Since we can pickle/unpickle everything already, this should be impossible.

Examples and usage documentation are available in the man page.


Versions 0.0.1, 0.0.2, 0.0.4, 0.0.5, 0.0.6, 0.0.7, 0.0.8, 0.0.9, 0.1.0, 0.1.1, 0.1.2, 0.1.3, 0.2.0, 0.2.1, 0.2.2, 0.2.3, 0.2.4
Dependencies base (>=4.6 && <5), cmdargs (>=0.10.6), configurator (>=0.2), containers (>=0.5), directory (>=1.2), filepath (>=1.3), fixed-vector-hetero (>=0.3), groundhog (>=0.7), groundhog-postgresql (>=0.7), groundhog-sqlite (>=0.7), groundhog-th (>=0.7), hslogger (>=1.2), htsn-common (>=0.0.1), hxt (>=9.3), MissingH (>=1.2), old-locale (>=1), split (>=0.2), tasty (>=0.8), tasty-hunit (>=0.8), time (>=1.4), transformers (>=0.3), tuple (>=0.2) [details]
License GPL-3
Author Michael Orlitzky
Maintainer Michael Orlitzky <>
Category Utils
Source repository head: git clone -b master
Uploaded Tue Mar 10 11:15:42 UTC 2015 by MichaelOrlitzky
Distributions NixOS:0.2.4
Downloads 2886 total (33 in the last 30 days)
0 []
Status Docs not available [build log]
Last success reported on 2015-05-20 [all 8 reports]
Hackage Matrix CI


Maintainer's Corner

For package maintainers and hackage trustees