http-conduit-downloader: HTTP downloader tailored for web-crawler needs.

[ bsd3, library, web ] [ Propose Tags ]

HTTP/HTTPS downloader built on top of http-conduit and used in https://bazqux.com crawler.

  • Handles all possible http-conduit exceptions and returns human readable error messages.

  • Handles some web server bugs (returning deflate data instead of gzip, invalid gzip encoding).

  • Uses OpenSSL instead of tls package since it doesn't handle all sites.

  • Ignores invalid SSL sertificates.

  • Receives data in 32k blocks internally to reduce memory fragmentation on many parallel downloads.

  • Download timeout.

  • Total download size limit.

  • Returns HTTP headers for subsequent redownloads and handles 'Not modified' results.

  • Can be used with external DNS resolver (hsdns-cache for example).

Modules

[Index]

Downloads

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.0.9, 1.0.10, 1.0.11, 1.0.12, 1.0.13, 1.0.14, 1.0.15, 1.0.16, 1.0.17, 1.0.18, 1.0.19, 1.0.20, 1.0.21, 1.0.22, 1.0.23, 1.0.24, 1.0.25, 1.0.30, 1.0.31, 1.0.32, 1.0.33, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5
Dependencies base (>=4 && <5), bytestring, conduit, connection, data-default, HsOpenSSL, http-client, http-conduit (>=2.0.0.4), http-types, lifted-base, mtl, network, old-locale, resourcet, time, zlib [details]
License BSD-3-Clause
Author Vladimir Shabanov <vshabanoff@gmail.com>
Maintainer Vladimir Shabanov <vshabanoff@gmail.com>
Category Web
Home page https://github.com/bazqux/http-conduit-downloader
Source repo head: git clone https://github.com/bazqux/http-conduit-downloader
Uploaded by VladimirShabanov at 2014-03-19T17:55:30Z
Distributions
Reverse Dependencies 2 direct, 0 indirect [details]
Downloads 22207 total (43 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Successful builds reported [all 1 reports]