use of HTTP needs overhauling

[duncan]

We're not using HTTP the protocol or HTTP the package effectively

• As I understand it, the HTTP lib buffers the entire http download in memory rather than returning it lazily, so we cannot stream it to disk. It's also all done using String.
• We do not check (and nor does the HTTP lib) for truncated downloads. That is where the Content-Length header differs from the actual length.
• We do not cache the ETag so we cannot optimise the case of re-downloading an unchanged index.
• We make a new tcp connection for each download, rather than doing all downloads within a single browse session.
• We are not doing downloads concurrent with package builds.

We should probably use a dedicated thread for downloads. It should serialise downloads and use a single browse session. Even without support for streaming the body directly we could at least write files atomically, checking length header. And also cache the ETag.

[duncan]

See also #448.

Note that HTTP-4000 allows the use of ByteString and I think allows lazy downloads.

[duncan]

We're now using HTTP-4000.x so download is a bit faster but we still have all the other issues. The other issues are however completely covered by ticket #448.