http-conduit-1.8.7.1: HTTP client package with conduit interface and HTTPS support.

Safe HaskellNone

Network.HTTP.Conduit

Contents

Description

This module contains everything you need to initiate HTTP connections. If you want a simple interface based on URLs, you can use simpleHttp. If you want raw power, http is the underlying workhorse of this package. Some examples:

 -- Just download an HTML document and print it.
 import Network.HTTP.Conduit
 import qualified Data.ByteString.Lazy as L

 main = simpleHttp "http://www.haskell.org/" >>= L.putStr

This example uses interleaved IO to write the response body to a file in constant memory space.

 import Data.Conduit.Binary (sinkFile)
 import Network.HTTP.Conduit
 import qualified Data.Conduit as C

 main :: IO ()
 main = do
      request <- parseUrl "http://google.com/"
      withManager $ \manager -> do
          Response _ _ _ src <- http request manager
          src C.$$+- sinkFile "google.html"

The following headers are automatically set by this module, and should not be added to requestHeaders:

  • Content-Length

Note: In previous versions, the Host header would be set by this module in all cases. Starting from 1.6.1, if a Host header is present in requestHeaders, it will be used in place of the header this module would have generated. This can be useful for calling a server which utilizes virtual hosting.

Any network code on Windows requires some initialization, and the network library provides withSocketsDo to perform it. Therefore, proper usage of this library will always involve calling that function at some point. The best approach is to simply call them at the beginning of your main function, such as:

 import Network.HTTP.Conduit
 import qualified Data.ByteString.Lazy as L
 import Network (withSocketsDo)

 main = withSocketsDo
      $ simpleHttp "http://www.haskell.org/" >>= L.putStr

 Cookies are implemented according to RFC 6265.

Note that by default, the functions in this package will throw exceptions for non-2xx status codes. If you would like to avoid this, you should use checkStatus, e.g.:

 import Data.Conduit.Binary (sinkFile)
 import Network.HTTP.Conduit
 import qualified Data.Conduit as C
 import Network

 main :: IO ()
 main = withSocketsDo $ do
      request' <- parseUrl "http://www.yesodweb.com/does-not-exist"
      let request = request' { checkStatus = \_ _ -> Nothing }
      res <- withManager $ httpLbs request
      print res

Synopsis

Perform a request

simpleHttp :: MonadIO m => String -> m ByteStringSource

Download the specified URL, following any redirects, and return the response body.

This function will throwIO an HttpException for any response with a non-2xx status code (besides 3xx redirects up to a limit of 10 redirects). It uses parseUrl to parse the input. This function essentially wraps httpLbs.

Note: Even though this function returns a lazy bytestring, it does not utilize lazy I/O, and therefore the entire response body will live in memory. If you want constant memory usage, you'll need to use the conduit package and http directly.

Note: This function creates a new Manager. It should be avoided in production code.

httpLbs :: (MonadBaseControl IO m, MonadResource m) => Request m -> Manager -> m (Response ByteString)Source

Download the specified Request, returning the results as a Response.

This is a simplified version of http for the common case where you simply want the response data as a simple datatype. If you want more power, such as interleaved actions on the response body during download, you'll need to use http directly. This function is defined as:

httpLbs = lbsResponse <=< http

Even though the Response contains a lazy bytestring, this function does not utilize lazy I/O, and therefore the entire response body will live in memory. If you want constant memory usage, you'll need to use conduit packages's Source returned by http.

Note: Unlike previous versions, this function will perform redirects, as specified by the redirectCount setting.

http :: (MonadResource m, MonadBaseControl IO m) => Request m -> Manager -> m (Response (ResumableSource m ByteString))Source

The most low-level function for initiating an HTTP request.

The first argument to this function gives a full specification on the request: the host to connect to, whether to use SSL, headers, etc. Please see Request for full details. The second argument specifies which Manager should be used.

This function then returns a Response with a Source. The Response contains the status code and headers that were sent back to us, and the Source contains the body of the request. Note that this Source allows you to have fully interleaved IO actions during your HTTP download, making it possible to download very large responses in constant memory. You may also directly connect the returned Source into a Sink, perhaps a file or another socket.

Note: Unlike previous versions, this function will perform redirects, as specified by the redirectCount setting.

Datatypes

data Proxy Source

Define a HTTP proxy, consisting of a hostname and port number.

Constructors

Proxy 

Fields

proxyHost :: ByteString

The host name of the HTTP proxy.

proxyPort :: Int

The port number of the HTTP proxy.

data RequestBody m Source

When using one of the RequestBodySource / RequestBodySourceChunked constructors, you must ensure that the Source can be called multiple times. Usually this is not a problem.

The RequestBodySourceChunked will send a chunked request body, note that not all servers support this. Only use RequestBodySourceChunked if you know the server you're sending to supports chunked request bodies.

Instances

Show (RequestBody m)

Since 1.8.7

Monad m => Monoid (RequestBody m)

Since 1.8.7

data Response body Source

A simple representation of the HTTP response created by lbsConsumer.

Instances

Functor Response

Since 1.1.2.

Typeable1 Response 
Eq body => Eq (Response body) 
Show body => Show (Response body) 

Request

data Request m Source

All information on how to connect to a host and what should be sent in the HTTP request.

If you simply wish to download from a URL, see parseUrl.

The constructor for this data type is not exposed. Instead, you should use either the def method to retrieve a default instance, or parseUrl to construct from a URL, and then use the records below to make modifications. This approach allows http-conduit to add configuration options without breaking backwards compatibility.

For example, to construct a POST request, you could do something like:

 initReq <- parseUrl "http://www.example.com/path"
 let req = initReq
             { method = "POST"
             }

For more information, please see http://www.yesodweb.com/book/settings-types.

Instances

def :: Default a => a

The default value for this type.

method :: Request m -> MethodSource

HTTP request method, eg GET, POST.

secure :: Request m -> BoolSource

Whether to use HTTPS (ie, SSL).

clientCertificates :: Request m -> [(X509, Maybe PrivateKey)]Source

SSL client certificates

path :: Request m -> ByteStringSource

Everything from the host to the query string.

requestHeaders :: Request m -> RequestHeadersSource

Custom HTTP request headers

As already stated in the introduction, the Content-Length and Host headers are set automatically by this module, and shall not be added to requestHeaders.

Moreover, the Accept-Encoding header is set implicitly to gzip for convenience by default. This behaviour can be overridden if needed, by setting the header explicitly to a different value. In order to omit the Accept-Header altogether, set it to the empty string "". If you need an empty Accept-Header (i.e. requesting the identity encoding), set it to a non-empty white-space string, e.g. " ". See RFC 2616 section 14.3 for details about the semantics of the Accept-Header field. If you request a content-encoding not supported by this module, you will have to decode it yourself (see also the decompress field).

Note: Multiple header fields with the same field-name will result in multiple header fields being sent and therefore it's the responsibility of the client code to ensure that the rules from RFC 2616 section 4.2 are honoured.

proxy :: Request m -> Maybe ProxySource

Optional HTTP proxy.

socksProxy :: Request m -> Maybe SocksConfSource

Optional SOCKS proxy.

rawBody :: Request m -> BoolSource

If True, a chunked and/or gzipped body will not be decoded. Use with caution.

decompress :: Request m -> ContentType -> BoolSource

Predicate to specify whether gzipped data should be decompressed on the fly (see alwaysDecompress and browserDecompress). Default: browserDecompress.

redirectCount :: Request m -> IntSource

How many redirects to follow when getting a resource. 0 means follow no redirects. Default value: 10.

checkStatus :: Request m -> Status -> ResponseHeaders -> Maybe SomeExceptionSource

Check the status code. Note that this will run after all redirects are performed. Default: return a StatusCodeException on non-2XX responses.

responseTimeout :: Request m -> Maybe IntSource

Number of microseconds to wait for a response. If Nothing, will wait indefinitely. Default: 5 seconds.

Manager

data Manager Source

Keeps track of open connections for keep-alive. If possible, you should share a single Manager between multiple threads and requests.

newManager :: ManagerSettings -> IO ManagerSource

Create a Manager. You must manually call closeManager to shut it down.

Creating a new Manager is an expensive operation, you are advised to share a single Manager between requests instead.

closeManager :: Manager -> IO ()Source

Close all connections in a Manager. Afterwards, the Manager can be reused if desired.

withManager :: (MonadIO m, MonadBaseControl IO m, MonadThrow m, MonadUnsafeIO m) => (Manager -> ResourceT m a) -> m aSource

Create a new manager, use it in the provided function, and then release it.

This function uses the default manager settings. For more control, use withManagerSettings.

withManagerSettings :: (MonadIO m, MonadBaseControl IO m, MonadThrow m, MonadUnsafeIO m) => ManagerSettings -> (Manager -> ResourceT m a) -> m aSource

Create a new manager with provided settings, use it in the provided function, and then release it.

Settings

data ManagerSettings Source

Settings for a Manager. Please use the def function and then modify individual settings.

managerConnCount :: ManagerSettings -> IntSource

Number of connections to a single host to keep alive. Default: 10.

managerCheckCerts :: ManagerSettings -> CertificateStore -> ByteString -> [X509] -> IO CertificateUsageSource

Check if the server certificate is valid. Only relevant for HTTPS.

managerCertStore :: ManagerSettings -> IO CertificateStoreSource

Load up the certificate store. By default uses the system store.

Defaults

defaultCheckCerts :: CertificateStore -> ByteString -> [X509] -> IO CertificateUsageSource

Check certificates using the operating system's certificate checker.

Cookies

data CookieJar Source

Instances

updateCookieJarSource

Arguments

:: Response a

Response received from server

-> Request m

Request which generated the response

-> UTCTime

Value that should be used as "now"

-> CookieJar

Current cookie jar

-> (CookieJar, Response a)

(Updated cookie jar with cookies from the Response, The response stripped of any "Set-Cookie" header)

This applies receiveSetCookie to a given Response

receiveSetCookieSource

Arguments

:: SetCookie

The SetCookie the cookie jar is receiving

-> Request m

The request that originated the response that yielded the SetCookie

-> UTCTime

Value that should be used as "now"

-> Bool

Whether or not this request is coming from an "http" source (not javascript or anything like that)

-> CookieJar

Input cookie jar to modify

-> CookieJar

Updated cookie jar

This corresponds to the algorithm described in Section 5.3 "Storage Model" This function consists of calling generateCookie followed by insertCheckedCookie. Use this function if you plan to do both in a row. generateCookie and insertCheckedCookie are only provided for more fine-grained control.

generateCookieSource

Arguments

:: SetCookie

The SetCookie we are encountering

-> Request m

The request that originated the response that yielded the SetCookie

-> UTCTime

Value that should be used as "now"

-> Bool

Whether or not this request is coming from an "http" source (not javascript or anything like that)

-> Maybe Cookie

The optional output cookie

Turn a SetCookie into a Cookie, if it is valid

insertCheckedCookieSource

Arguments

:: Cookie

The SetCookie the cookie jar is receiving

-> CookieJar

Input cookie jar to modify

-> Bool

Whether or not this request is coming from an "http" source (not javascript or anything like that)

-> CookieJar

Updated (or not) cookie jar

Insert a cookie created by generateCookie into the cookie jar (or not if it shouldn't be allowed in)

insertCookiesIntoRequestSource

Arguments

:: Request m

The request to insert into

-> CookieJar

Current cookie jar

-> UTCTime

Value that should be used as "now"

-> (Request m, CookieJar)

(Ouptut request, Updated cookie jar (last-access-time is updated))

This applies the computeCookieString to a given Request

computeCookieStringSource

Arguments

:: Request m

Input request

-> CookieJar

Current cookie jar

-> UTCTime

Value that should be used as "now"

-> Bool

Whether or not this request is coming from an "http" source (not javascript or anything like that)

-> (ByteString, CookieJar)

(Contents of a "Cookie" header, Updated cookie jar (last-access-time is updated))

This corresponds to the algorithm described in Section 5.4 "The Cookie Header"

evictExpiredCookiesSource

Arguments

:: CookieJar

Input cookie jar

-> UTCTime

Value that should be used as "now"

-> CookieJar

Filtered cookie jar

This corresponds to the eviction algorithm described in Section 5.3 "Storage Model"

Utility functions

parseUrl :: Failure HttpException m => String -> m (Request m')Source

Convert a URL into a Request.

This defaults some of the values in Request, such as setting method to GET and requestHeaders to [].

Since this function uses Failure, the return monad can be anything that is an instance of Failure, such as IO or Maybe.

applyBasicAuth :: ByteString -> ByteString -> Request m -> Request mSource

Add a Basic Auth header (with the specified user name and password) to the given Request. Ignore error handling:

applyBasicAuth user pass $ fromJust $ parseUrl url

addProxy :: ByteString -> Int -> Request m -> Request mSource

Add a proxy to the Request so that the Request when executed will use the provided proxy.

lbsResponse :: Monad m => Response (ResumableSource m ByteString) -> m (Response ByteString)Source

Convert a Response that has a Source body to one with a lazy ByteString body.

getRedirectedRequest :: Request m -> ResponseHeaders -> Int -> Maybe (Request m)Source

If a request is a redirection (status code 3xx) this function will create a new request from the old request, the server headers returned with the redirection, and the redirection code itself. This function returns Nothing if the code is not a 3xx, there is no location header included, or if the redirected response couldn't be parsed with parseUrl.

If a user of this library wants to know the url chain that results from a specific request, that user has to re-implement the redirect-following logic themselves. An example of that might look like this:

 myHttp req man = E.catch (runResourceT $ http req' man >> return [req'])
                    (\ (StatusCodeException status headers) -> do
                        l <- myHttp (fromJust $ nextRequest status headers) man
                        return $ req' : l)
     where req' = req { redirectCount = 0 }
           nextRequest status headers = getRedirectedRequest req' headers $ W.statusCode status

Decompression predicates

alwaysDecompress :: ContentType -> BoolSource

Always decompress a compressed stream.

browserDecompress :: ContentType -> BoolSource

Decompress a compressed stream unless the content-type is 'application/x-tar'.

Request bodies

Network.HTTP.Conduit.MultipartFormData provides an API for building form-data request bodies.

urlEncodedBody :: Monad m => [(ByteString, ByteString)] -> Request m' -> Request mSource

Add url-encoded parameters to the Request.

This sets a new requestBody, adds a content-type request header and changes the method to POST.

Exceptions

data HttpException Source

Constructors

StatusCodeException Status ResponseHeaders 
InvalidUrlException String String 
TooManyRedirects [Response ByteString]

List of encountered responses containing redirects in reverse chronological order; including last redirect, which triggered the exception and was not followed.

UnparseableRedirect (Response ByteString)

Response containing unparseable redirect.

TooManyRetries 
HttpParserException String 
HandshakeFailed 
OverlongHeaders 
ResponseTimeout 
FailedConnectionException String Int

host/port

ExpectedBlankAfter100Continue 
InvalidStatusLine ByteString 
InvalidHeader ByteString