Ticket #338 (closed enhancement: duplicate)
use of HTTP needs overhauling
|Reported by:||duncan||Owned by:|
We're not using HTTP the protocol or HTTP the package effectively
- As I understand it, the HTTP lib buffers the entire http download in memory rather than returning it lazily, so we cannot stream it to disk. It's also all done using String.
- We do not check (and nor does the HTTP lib) for truncated downloads. That is where the Content-Length header differs from the actual length.
- We do not cache the ETag so we cannot optimise the case of re-downloading an unchanged index.
- We make a new tcp connection for each download, rather than doing all downloads within a single browse session.
- We are not doing downloads concurrent with package builds.
We should probably use a dedicated thread for downloads. It should serialise downloads and use a single browse session. Even without support for streaming the body directly we could at least write files atomically, checking length header. And also cache the ETag.