SecondTransfer

This library implements enough of the HTTP/2 to build compliant HTTP/2 servers. It also implements enough of HTTP/1.1 so you can actually use it to build polyglot web-servers.

For HTTP/2, frame encoding and decoding is done with Kazu Yamamoto's http2 package. This library just takes care of making sense of sent and received frames.

You can find more detailed information about this library at the page https://www.httptwo.com/second-transfer/.

The library

• Is concurrent, meaning that you can use amazing Haskell lightweight threads to process the requests.
• Obeys HTTP2 flow control aspects, when talking HTTP2.
• And gives you freedom to (ab)use the HTTP/2 protocol in all the ways envisioned by the standard. In particular you should be able to process streaming requests (long uploads in POST or PUT requests) and to deliver streaming responses. You should even be able to do both simultaneously.

Setting up TLS for HTTP/2 correctly is a shore, so I have bundled here the TLS setup logic. Before you read any further, ATTENTION: enable always the threaded ghc runtime in your final programs if you want TLS to work.

Here is how you create a very basic HTTP/2 webserver:

import SecondTransfer(
    CoherentWorker
    , Footers
    , DataAndConclusion
    , tlsServeWithALPN
    , http2Attendant
    , http11Attendant
    )
import SecondTransfer.Sessions(
      makeSessionsContext
    , defaultSessionsConfig
    )

import Data.Conduit


saysHello :: DataAndConclusion
saysHello = do 
    -- The data in each yield will be automatically split across multiple 
    -- data frames if needed, so you can yield a large block of contents here
    -- if you wish. 
    -- If you do multiple yields, no data will be left buffered between them, 
    -- so that you can for example implement a chat client in a single HTTP/2 stream.
    -- Pity browsers hardly support that.
    yield "Hello world!"
    -- No footers
    return []


helloWorldWorker :: CoherentWorker
helloWorldWorker (_request_headers, _maybe_post_data) = do 
    dropIncomingData _maybe_post_data
    return (
        [
            (":status", "200")
        ],
        [], -- No pushed streams
        saysHello
        )


-- For this program to work, it should be run from the top of 
-- the developement directory.
main = do 
    sessions_context <- makeSessionsContext defaultSessionsConfig
    let 
        http2_attendant = http2Attendant sessions_context helloWorldWorker
        http11_attendant = http11Attendant sessions_context helloWorldWorker
    tlsServeWithALPN
        "tests/support/servercert.pem"   -- Server certificate
        "tests/support/privkey.pem"      -- Certificate private key
        "127.0.0.1"                      -- On which interface to bind
        [
            ("no-protocol", http11_attendant), -- The first protocol in the list is used when 
                                               -- when no ALPN negotiation happens, and the
                                               -- name is really a filler.
            ("h2-14", http2_attendant),    -- Protocols present in the ALPN negotiation
            ("h2",    http2_attendant),    -- they may be slightly different, but for this 
                                           -- test it doesn't matter.

            ("http1.1", http11_attendant) -- Let's talk HTTP1.1 if everything else fails.
        ]
        8000 

CoherentWorker is the type of the basic callback function that you need to implement. The callback is used to handle all requests to the server on a given negotiated ALPN protocol. If you need routing functionality (and you most certainly will need it), you need to build that functionality inside the callback.

The above program uses a test certificate by a fake certificate authority. The certificate is valid for the server name ("authority", in HTTP/2 lingo) www.httpdos.com. So, in order for the above program to run, you probably need to add an alias to your /etc/hosts file. You also need very up-to-date versions of OpenSSL (I'm using OpenSSL 1.0.2) to be compliant with the cipher suites demanded by HTTP/2. The easiest way to test the above program is using a fairly recent version of curl. If everything is allright, you should be able to do:

   $ curl -k --http2 https://www.httpdos.com:8000/
   Hello world!

Types related to coherent workers

A coherent worker is an abstraction that can dance at the tune of HTTP2 and HTTP1.1. That is, it should be able to take headers from a request first, then a source of data coming in the request (for example, POST data). Even before exhausting the source, the coherent worker can post the response headers and its source for the response data. A coherent worker can also present streams to push to the client.

type Headers = [Header]

List of headers. The first part of each tuple is the header name (be sure to conform to the HTTP/2 convention of using lowercase) and the second part is the headers contents. This list needs to include the special :method, :scheme, :authority and :path pseudo-headers for requests; and :status (with a plain numeric value represented in ascii digits) for responses.

type HeaderName = ByteString

The name part of a header

type HeaderValue = ByteString

The value part of a header

type Header = (HeaderName, HeaderValue)

The complete header

type Request = (Headers, Maybe InputDataStream)

A request is a set of headers and a request body.... which will normally be empty, except for POST and PUT requests. But this library enforces none of that.

type Footers = FinalizationHeaders

Finalization headers

type CoherentWorker = Request -> IO PrincipalStream

Main type of this library. You implement one of these for your server. Basically this is a callback that the library calls as soon as it has all the headers of a request. For GET requests that's the entire request basically, but for POST and PUT requests this is just before the data starts arriving to the server.

It is important that you consume the data in the cases where there is an input stream, otherwise the memory is lost for the duration of the request, and a malicious client can use that.

Also, notice that when handling requests your worker can be interrupted with an asynchronous exception of type StreamCancelledException, if the peer cancels the stream

type PrincipalStream = (Headers, PushedStreams, DataAndConclusion)

You use this type to answer a request. The Headers are thus response headers and they should contain the :status pseudo-header. The PushedStreams is a list of pushed streams...(I don't thaink that I'm handling those yet)

type PushedStreams = [IO PushedStream]

A list of pushed streams. Notice that a list of IO computations is required here. These computations only happen when and if the streams are pushed to the client. The lazy nature of Haskell helps to avoid unneeded computations if the streams are not going to be sent to the client.

type PushedStream = (Headers, Headers, DataAndConclusion)

A pushed stream, represented by a list of request headers, a list of response headers, and the usual response body (which may include final footers (not implemented yet)).

type DataAndConclusion = ConduitM () ByteString IO Footers

A source-like conduit with the data returned in the response. The return value of the conduit is a list of footers. For now that list can be anything (even bottom), I'm not handling it just yet.

type InputDataStream = Source IO ByteString

This is a Source conduit (see Haskell Data.Conduit library from Michael Snoyman) that you can use to retrieve the data sent by the client piece-wise.

type FinalizationHeaders = Headers

Finalization headers. If you don't know what they are, chances are that you don't need to worry about them for now. The support in this library for those are at best sketchy.

How to couple bidirectional data channels to sessions

type Attendant = PushAction -> PullAction -> CloseAction -> IO ()

A function which takes three arguments: the first one says how to send data (on a socket or similar transport), and the second one how to receive data on the transport. The third argument encapsulates the sequence of steps needed for a clean shutdown.

You can implement one of these to let somebody else supply the push, pull and close callbacks. For example, tlsServeWithALPN will supply these arguments to an Attendant.

Attendants encapsulate all the session book-keeping functionality, which for HTTP/2 is quite complicated. You use the functions http**Attendant to create one of these from a CoherentWorker.

This library supplies two of such Attendant factories, http11Attendant for HTTP 1.1 sessions, and http2Attendant for HTTP/2 sessions.

http11Attendant :: SessionsContext -> CoherentWorker -> Attendant

Session attendant that speaks HTTP/1.1

http2Attendant :: SessionsContext -> CoherentWorker -> Attendant

The type of this function is equivalent to:

     http2Attendant :: CoherentWorker -> PushAction -> PullAction -> CloseAction ->  IO ()

Given a CoherentWorker, this function wraps it with flow control, multiplexing, and state maintenance needed to run an HTTP/2 session.

Notice that this function is using HTTP/2 over TLS. We haven't implemented yet a session handling mechanism for HTTP/1.1 .

High level OpenSSL functions.

Use these functions to create your TLS-compliant HTTP/2 server in a snap.

tlsServeWithALPN

Arguments

:: FilePath

Path to a certificate the server is going to use to identify itself. Bear in mind that multiple domains can be served from the same HTTP/2 TLS socket, so please create the HTTP/2 certificate accordingly. Also, currently this function only accepts paths to certificates or certificate chains in .pem format.

-> FilePath

Path to the key of your certificate.

-> String

Name of the network interface where you want to start your server

-> [(String, Attendant)]

List of protocol names and the corresponding Attendant to use for each. This way you can serve both HTTP/1.1 over TLS and HTTP/2 in the same socket. When no ALPN negotiation is present during the negotiation, the first protocol in this list is used.

-> Int

Port to open to listen for connections.

-> IO ()

Simple function to open

tlsServeWithALPNAndFinishOnRequest

Arguments

:: FilePath

-> FilePath

Same as for tlsServeWithALPN

-> String

Same as for tlsServeWithALPN

-> [(String, Attendant)]

Same as for tlsServeWithALPN

-> Int

Same as for tlsServeWithALPN

-> MVar FinishRequest

Finish request, write a value here to finish serving

-> IO ()

Interruptible version of tlsServeWithALPN. Use the extra argument to ask the server to finish: you pass an empty MVar and when you want to finish you just populate it.

data TLSLayerGenericProblem

Exceptions inheriting from IOProblem. This is thrown by the OpenSSL subsystem to signal that the connection was broken or that otherwise there was a problem at the SSL layer.

Constructors

TLSLayerGenericProblem String

Instances

Show TLSLayerGenericProblem

Exception TLSLayerGenericProblem

data FinishRequest

Singleton type. Used in conjunction with an MVar. If the MVar is full, the fuction tlsServeWithALPNAndFinishOnRequest knows that it should finish at its earliest convenience and call the CloseAction for any open sessions.

Constructors

FinishRequest

dropIncomingData :: Maybe InputDataStream -> IO ()

If you are not processing the potential POST input in a request, use this consumer to drop the data to oblivion. Otherwise it will remain in an internal queue until the client closes the stream, and if the client doesn't want to do so....

Logging

The library uses hslogger for its logging. Since logging is expensive, most of the instrumentation needs to be activated at compile time by activating the "debug" flag. And then you need to configure the loggers. The function enableConsoleLogging configures them to output a lot of information to standard output.

enableConsoleLogging :: IO ()

Activates logging to terminal