Network.Craze

Craze is a small module for performing multiple similar HTTP GET requests in parallel. This is performed through the raceGet function, which will perform all the requests and pick the first successful response that passes a certain check, menaing that the parallel requests are essentially racing against each other.

What is the usefulness of this?

If you are dealing with data source or API that is very unreliable (high latency, random failures) and there are no limitations or consequences on perfoming significantly more requests, then performing multiple requests (through direct connections, proxies, VPNs) may increase the chances of getting a successful response faster and more reliably.

However, if using a different data source or transport is a possibility, it is potentially a better option that this approach.

Examples:

Performing two parallel GET requests against https://chromabits.com and returning the status code of the first successful one:

>>> :{
 let racer = (Racer
               { racerProviders =
                   [ return defaultProviderOptions
                   , return defaultProviderOptions
                   ]
               , racerHandler = return . respStatus
               , racerChecker = (200 ==)
               , racerDebug = False
               } :: Racer [(String, String)] ByteString Int)
 in (raceGet racer "https://chromabits.com" >>= print)
:}
Just 200

Types

type RacerHandler headerTy bodyTy a = CurlResponse_ headerTy bodyTy -> IO a

A RacerHandler is simply a function for transforming a response after it is received. The handler is only applied to successful requests before they are checked by the RacerChecker.

This is primarily for extracting or parsing a CurlResponse_ before doing any further work. The type returned by the handler will be used as the input of the checker and will be the return type of raceGet.

type RacerChecker a = a -> Bool

A function that computes whether or not a result is valid or not. Successful responses that do not pass the checker are discarded.

This should help filtering out successful responses that do not, for some reason, have the expected result (e.g. Random content changes, Rate Limitting, etc).

data Racer headerTy bodyTy a

A record describing the rules for racing requests.

Constructors

Racer

Fields

racerHandler :: RacerHandler headerTy bodyTy a

racerChecker :: RacerChecker a

racerProviders :: [RacerProvider]

On a Racer, each RaceProvider represents a separate client configuration. When performing a race, each provider will be used to spwan a client and perform a request. This allows one to control the number of requests performed and with which CurlOptions.

racerDebug :: Bool

When set to True, debugging messages will be written to stdout.

Instances

Default (Racer [(String, String)] ByteString ByteString)

data ProviderOptions

Options for a specific provider.

Constructors

ProviderOptions

Fields

poOptions :: [CurlOption]

Options to pass down to Curl.

poDelay :: Maybe Int

Number of microseconds to delay the request by.

poTag :: Text

A tag to identify this type provider.

Instances

Show ProviderOptions

Default ProviderOptions

data RacerResult a

The result of a racing operation. This can be used to collect statistics on which providers win more often, etc.

Functions

defaultRacer :: Racer [(String, String)] ByteString ByteString

A Racer with some default values.

Note: The handler will extract the response body as a ByteString and ignore everything else, hence the type:

Racer [(String, String)] ByteString ByteString

If this is not the desired behavior, or if the response should be parsed or processed, you should use the Racer constructor directly and provide all fields.

defaultProviderOptions :: ProviderOptions

A default set of options for a provider.

raceGet :: (Eq a, CurlHeader ht, CurlBuffer bt) => Racer ht bt a -> URLString -> IO (Maybe a)

Perform a GET request on the provided URL using all providers in parallel.

Rough summary of the algorithm:

• Start all requests

• Wait for a request to finish.

  • If the request is successful, apply the handler on it.

    • If the result of the handler passes the checker, cancel all other requests, and return the result.
    • If the check fails, go back to waiting for another request to finish.
  • If the request fails, go back to waiting for another request to finish.

raceGetResult :: (Eq a, CurlHeader ht, CurlBuffer bt) => Racer ht bt a -> URLString -> IO (RacerResult a)

Same as raceGet, but returns a RacerResult which contains more information about the race performed.

Providers

simple :: [CurlOption] -> IO ProviderOptions

A simple provider. It does not delay requests.

simpleTagged :: [CurlOption] -> Text -> IO ProviderOptions

Like simple, but with a tag for identification.

delayed :: [CurlOption] -> Int -> IO ProviderOptions

A provider which will delay a request by the provided number of microseconds.

delayedTagged :: [CurlOption] -> Int -> Text -> IO ProviderOptions

Like delayed, but with a tag for identification.

Results

rrResponse :: RacerResult a -> Maybe a

rrWinner :: RacerResult a -> Maybe ProviderOptions

rrProviders :: RacerResult a -> [RacerProvider]