http-conduit-downloader: HTTP downloader tailored for web-crawler needs.

[ bsd3, library, web ] [ Propose Tags ]

HTTP/HTTPS downloader built on top of http-conduit and used in http://bazqux.com crawler.

  • Handles all possible http-conduit exceptions and returns human readable error messages.

  • Handles some web server bugs (no persistent connections on HTTP/1.1, returning deflate data instead of gzip)

  • Ignores invalid SSL sertificates.

  • Receives data in 32k blocks internally to reduce memory fragmentation on many parallel downloads.

  • Download timeout.

  • Total download size limit.

  • Returns HTTP headers for subsequent redownloads and handles 'Not modified' results.

  • Can be used with external DNS resolver (hsdns-cache for example).

Modules

[Index]

Downloads

Note: This package has metadata revisions in the cabal description newer than included in the tarball. To unpack the package including the revisions, use 'cabal get'.

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 1.0.0, 1.0.1, 1.0.2, 1.0.3, 1.0.4, 1.0.5, 1.0.6, 1.0.7, 1.0.8, 1.0.9, 1.0.10, 1.0.11, 1.0.12, 1.0.13, 1.0.14, 1.0.15, 1.0.16, 1.0.17, 1.0.18, 1.0.19, 1.0.20, 1.0.21, 1.0.22, 1.0.23, 1.0.24, 1.0.25, 1.0.30, 1.0.31, 1.0.32, 1.0.33, 1.1.0, 1.1.1, 1.1.2, 1.1.3, 1.1.4, 1.1.5
Dependencies base (>=4 && <5), bytestring, conduit, data-default, http-conduit (>=1.9), http-types, lifted-base, mtl, network (<3), resourcet, tls, zlib [details]
License BSD-3-Clause
Copyright Vladimir Shabanov 2013
Author Vladimir Shabanov <vshabanoff@gmail.com>
Maintainer Vladimir Shabanov <vshabanoff@gmail.com>
Revised Revision 1 made by Bodigrim at 2024-02-07T20:06:13Z
Category Web
Home page https://github.com/bazqux/http-conduit-downloader
Source repo head: git clone https://github.com/bazqux/http-conduit-downloader
Uploaded by VladimirShabanov at 2013-03-11T12:19:08Z
Distributions
Reverse Dependencies 2 direct, 0 indirect [details]
Downloads 22207 total (43 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs uploaded by user
Build status unknown [no reports yet]