The archiver package
archiver is a daemon which will process a specified text file, each line of which is a URL, and will (randomly) one by one request that the URLs be archived or spidered by http://www.webcitation.org, http://www.archive.org, and http://www.wikiwix.com for future reference. (One may optionally specify an arbitrary sh command like wget to download URLs locally.)
Because the interface is a simple text file, this can be combined with other scripts; for example, a script using Sqlite to extract visited URLs from Firefox, or a program extracting URLs from Pandoc documents. (See http://www.gwern.net/Archiving%20URLs.)
For explanation of the derivation of the code in Network.URL.Archiver, see http://www.gwern.net/haskell/Wikipedia%20Archive%20Bot.
Properties
| Versions | 0.1, 0.2, 0.3, 0.3.1, 0.4, 0.5, 0.5.1, 0.6.0, 0.6.1, 0.6.2, 0.6.2.1 |
|---|---|
| Dependencies | base (==4.*), bytestring, containers, curl, HTTP, network, process, random [details] |
| License | BSD3 |
| Author | Gwern |
| Maintainer | Gwern <gwern@gwern.net> |
| Stability | provisional |
| Category | Documentation, Network |
| Source repository | head: git clone git://github.com/gwern/archiver-bot.git |
| Uploaded | Fri Jan 3 19:44:15 UTC 2014 by GwernBranwen |
| Distributions | NixOS:0.6.2.1 |
| Downloads | 2482 total (27 in the last 30 days) |
| Votes | |
| Status | Docs available [build log] Successful builds reported [all 1 reports] |
Downloads
- archiver-0.6.2.1.tar.gz [browse] (Cabal source package)
- Package description (included in the package)