Copyright	(c) Alexey Radkov 2020
License	BSD-style
Maintainer	alexey.radkov@gmail.com
Stability	experimental
Portability	non-portable (requires Template Haskell)
Safe Haskell	None
Language	Haskell2010

NgxExport.Tools.Subrequest

Contents

Making HTTP subrequests
Getting full response data from HTTP subrequests
Forwarding full response data to the client

Description

Easy HTTP subrequests from the more extra tools collection for nginx-haskell-module.

Making HTTP subrequests

Using asynchronous variable handlers and services together with the HTTP client from Network.HTTP.Client allows making HTTP subrequests easily. This module provides such functionality by exporting asynchronous variable handlers makeSubrequest and makeSubrequestWithRead, and functions makeSubrequest and makeSubrequestWithRead to build custom handlers.

Below is a simple example.

File test_tools_extra_subrequest.hs

{-# LANGUAGE TemplateHaskell #-}

module TestToolsExtraSubrequest where

import           NgxExport
import           NgxExport.Tools
import           NgxExport.Tools.Subrequest

import           Data.ByteString (ByteString)
import qualified Data.ByteString.Lazy as L

makeRequest :: ByteString -> Bool -> IO L.ByteString
makeRequest = const . makeSubrequest

ngxExportSimpleService 'makeRequest $ PersistentService $ Just $ Sec 10

Handler makeRequest will be used in a periodical service which will retrieve data from a specified URI every 10 seconds.

File nginx.conf

user                    nobody;
worker_processes        2;

events {
    worker_connections  1024;
}

http {
    default_type        application/octet-stream;
    sendfile            on;

    haskell load /var/lib/nginx/test_tools_extra_subrequest.so;

    upstream backend {
        server 127.0.0.1:8020;
    }

    haskell_run_service simpleService_makeRequest $hs_service_httpbin
            '{"uri": "http://httpbin.org"}';

    haskell_var_empty_on_error $hs_subrequest;

    server {
        listen       8010;
        server_name  main;
        error_log    /tmp/nginx-test-haskell-error.log;
        access_log   /tmp/nginx-test-haskell-access.log;

        location / {
            haskell_run_async makeSubrequest $hs_subrequest
                    '{"uri": "http://127.0.0.1:8010/proxy",
                      "headers": [["Custom-Header", "$arg_a"]]}';

            if ($hs_subrequest = '') {
                echo_status 404;
                echo "Failed to perform subrequest";
                break;
            }

            echo -n $hs_subrequest;
        }

        location /proxy {
            allow 127.0.0.1;
            deny all;
            proxy_pass http://backend;
        }

        location /httpbin {
            echo $hs_service_httpbin;
        }
    }

    server {
        listen       8020;
        server_name  backend;

        location / {
            set $custom_header $http_custom_header;
            echo "In backend, Custom-Header is '$custom_header'";
        }
    }
}

Configurations of subrequests are defined via JSON objects which contain URI and other relevant data such as HTTP method, request body and headers. In this configuration we are running a periodical service which gets contents of httpbin.org every 10 seconds, and doing a subrequest to a virtual server backend on every request to location /. In this subrequest, an HTTP header Custom-Header is sent to the backend with value equal to the value of argument a from the client request's URI.

It is worth noting that making HTTP subrequests to the own Nginx service (e.g. via 127.0.0.1) allows for leveraging well-known advantages of Nginx such as load-balancing via upstreams as it is happening in this example.

A simple test

$ curl -s 'http://localhost:8010/httpbin' | head
<!DOCTYPE html>
<html lang="en">

<head>
    <meta charset="UTF-8">
    <title>httpbin.org</title>
    <link href="https://fonts.googleapis.com/css?family=Open+Sans:400,700|Source+Code+Pro:300,600|Titillium+Web:400,600,700"
        rel="stylesheet">
    <link rel="stylesheet" type="text/css" href="/flasgger_static/swagger-ui.css">
    <link rel="icon" type="image/png" href="/static/favicon.ico" sizes="64x64 32x32 16x16" />

$ curl 'http://localhost:8010/?a=Value'
In backend, Custom-Header is 'Value'

Let's do a nasty thing. By injecting a comma into the argument a we shall break JSON parsing.

$ curl -D- 'http://localhost:8010/?a=Value"'
HTTP/1.1 404 Not Found
Server: nginx/1.17.9
Date: Mon, 30 Mar 2020 14:42:42 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive

Failed to perform subrequest

makeSubrequest Source #

Arguments

:: ByteString	Subrequest configuration
-> IO ByteString

Makes an HTTP request.

This is the core function of the makeSubrequest handler. From perspective of an Nginx request, it spawns a subrequest, hence the name. However, this function can also be used to initiate an original HTTP request from a service handler.

Accepts a JSON object representing an opaque type SubrequestConf. The object may contain 5 fields: method (optional, default is GET), uri (mandatory), body (optional, default is an empty value), headers (optional, default is an empty array), and timeout (optional, default is the default response timeout of the HTTP manager which is normally 30 seconds, use value {"tag": "Unset"} to disable response timeout completely).

Examples of subrequest configurations:

{"uri": "http://example.com/", "timeout": {"tag": "Sec", "contents": 10}}

{"uri": "http://127.0.0.1/subreq", "method": "POST", "body": "some value"}

{"uri": "http://127.0.0.1/subreq",
    "headers": [["Header1", "Value1"], ["Header2", "Value2"]]}

Returns the response body if HTTP status of the response is 2xx, otherwise throws an error. To avoid leakage of error messages into variable handlers, put the corresponding variables into the list of directive haskell_var_empty_on_error.

makeSubrequestWithRead Source #

Arguments

:: ByteString	Subrequest configuration
-> IO ByteString

Makes an HTTP request.

Behaves exactly as makeSubrequest except it parses Haskell terms representing SubrequestConf with read. Exported on the Nginx level by handler makeSubrequestWithRead.

An example of a subrequest configuration:

SubrequestConf { srMethod = ""
               , srUri = "http://127.0.0.1/subreq"
               , srBody = ""
               , srHeaders = [("Header1", "Value1"), ("Header2", "Value2")]
               , srResponseTimeout = ResponseTimeout (Sec 10)
               }

Notice that unlike JSON parsing, fields of SubrequestConf are not omittable and must be listed in the order shown in the example. Empty srMethod implies GET.

Getting full response data from HTTP subrequests

Handlers makeSubrequest and makeSubrequestWithRead return response body of subrequests skipping the response status and headers. To retrieve full data from a response, use another pair of asynchronous variable handlers and functions: makeSubrequestFull and makeSubrequestFullWithRead, and makeSubrequestFull and makeSubrequestFullWithRead respectively.

Unlike the simple body handlers, there is no sense of using the corresponding variables directly as they are binary encoded values. Instead, the response status, headers and the body must be extracted using handlers extractStatusFromFullResponse, extractHeaderFromFullResponse, and extractBodyFromFullResponse which are based on functions of the same name. Handler extractRequestStatusFromFullResponse and the corresponding function can be used to extract the request status: 0 means that the request was completed without technical errors, 1 means that the request failed due to connection errors including timeouts, and 2 means that the request failed due to other errors.

Let's extend our example with these handlers.

File test_tools_extra_subrequest.hs does not have any changes as we are going to use exported handlers only.

File nginx.conf: new location /full in server main

        location /full {
            haskell_run_async makeSubrequestFull $hs_subrequest
                    '{"uri": "http://127.0.0.1:$arg_p/proxy",
                      "headers": [["Custom-Header", "$arg_a"]]}';

            haskell_run extractStatusFromFullResponse $hs_subrequest_status
                    $hs_subrequest;

            haskell_run extractHeaderFromFullResponse $hs_subrequest_header
                    subrequest-header|$hs_subrequest;

            haskell_run extractBodyFromFullResponse $hs_subrequest_body
                    $hs_subrequest;

            if ($hs_subrequest_status = 400) {
                echo_status 400;
                echo "Bad request";
                break;
            }

            if ($hs_subrequest_status = 500) {
                echo_status 500;
                echo "Internal server error while making subrequest";
                break;
            }

            if ($hs_subrequest_status = 502) {
                echo_status 502;
                echo "Backend unavailable";
                break;
            }

            if ($hs_subrequest_status != 200) {
                echo_status 404;
                echo "Subrequest status: $hs_subrequest_status";
                break;
            }

            echo    "Subrequest status: $hs_subrequest_status";
            echo    "Subrequest-Header: $hs_subrequest_header";
            echo -n "Subrequest body: $hs_subrequest_body";
        }

Now we can recognize HTTP response statuses of subrequests and handle them differently. We also can read a response header Subrequest-Header.

File nginx.conf: new response header Subrequest-Header in location / of server backend

            add_header Subrequest-Header "This is response from subrequest";

A simple test

$ curl -D- 'http://localhost:8010/full/?a=Value"'
HTTP/1.1 400 Bad Request
Server: nginx/1.17.9
Date: Sat, 04 Apr 2020 12:44:36 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive

Bad request

Good. Now we see that adding a comma into a JSON field is a bad request.

$ curl -D- 'http://localhost:8010/full/?a=Value'
HTTP/1.1 500 Internal Server Error
Server: nginx/1.17.9
Date: Sat, 04 Apr 2020 12:47:11 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive

Internal server error while making subrequest

This is also good. Now we are going to define port of the backend server via argument $arg_p. Skipping this makes URI look unparsable (http://127.0.0.1:/) which leads to the error.

$ curl -D- 'http://localhost:8010/full/?a=Value&p=8020'
HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Sat, 04 Apr 2020 12:52:03 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive

Subrequest status: 200
Subrequest-Header: This is response from subrequest
Subrequest body: In backend, Custom-Header is 'Value'

Finally, we are getting a good response with all the response data decoded correctly.

Let's try another port.

$ curl -D- 'http://localhost:8010/full/?a=Value&p=8021'
HTTP/1.1 502 Bad Gateway
Server: nginx/1.17.9
Date: Sat, 04 Apr 2020 12:56:02 GMT
Content-Type: application/octet-stream
Transfer-Encoding: chunked
Connection: keep-alive

Backend unavailable

Good. There is no server listening on port 8021.

makeSubrequestFull Source #

Arguments

:: ByteString	Subrequest configuration
-> IO ByteString

Makes an HTTP request.

The same as makeSubrequest except it returns a binary encoded response data whose parts must be extracted by handlers made of extractRequestStatusFromFullResponse, extractStatusFromFullResponse, extractHeaderFromFullResponse, and extractBodyFromFullResponse. It also does not throw errors when HTTP status of the response is not 2xx. Exported on the Nginx level by handler makeSubrequestFull.

makeSubrequestFullWithRead Source #

Arguments

:: ByteString	Subrequest configuration
-> IO ByteString

Makes an HTTP request.

The same as makeSubrequestWithRead except it returns a binary encoded response data whose parts must be extracted by handlers made of extractRequestStatusFromFullResponse, extractStatusFromFullResponse, extractHeaderFromFullResponse, and extractBodyFromFullResponse. It also does not throw errors when HTTP status of the response is not 2xx. Exported on the Nginx level by handler makeSubrequestFullWithRead.

extractRequestStatusFromFullResponse Source #

Arguments

:: ByteString	Encoded HTTP response
-> ByteString

Extracts the request status from an encoded response.

Must be used to extract response data encoded by makeSubrequestFull or makeSubrequestFullWithRead. Exported on the Nginx level by handler extractRequestStatusFromFullResponse.

The request status may be one of the following values:

0 — the request was completed without technical errors,
1 — the request failed due to connection errors including timeouts,
2 — the request failed due to other errors.

extractStatusFromFullResponse Source #

Arguments

:: ByteString	Encoded HTTP response
-> ByteString

Extracts the HTTP status from an encoded response.

Must be used to extract response data encoded by makeSubrequestFull or makeSubrequestFullWithRead. Exported on the Nginx level by handler extractStatusFromFullResponse.

extractHeaderFromFullResponse Source #

Arguments

:: ByteString	Encoded HTTP response
-> ByteString

Extracts a specified header from an encoded response.

Must be used to extract response data encoded by makeSubrequestFull or makeSubrequestFullWithRead. Exported on the Nginx level by handler extractHeaderFromFullResponse.

Expects that the encoded response data is attached after the name of the header and a vertical bar such as Header-Name|$hs_body. The lookup of the header name is case-insensitive. Returns an empty value if the header was not found.

extractBodyFromFullResponse Source #

Arguments

:: ByteString	Encoded HTTP response
-> ByteString

Extracts the body from an encoded response.

Must be used to extract response data encoded by makeSubrequestFull or makeSubrequestFullWithRead. Exported on the Nginx level by handler extractBodyFromFullResponse.

Forwarding full response data to the client

Data encoded in the full response can be translated to ContentHandlerResult and forwarded downstream to the client in directive haskell_content. Handler fromFullResponse performs such a translation. Not all response headers are allowed being forwarded downstream, and thus the handler deletes response headers with names listed in set notForwardableResponseHeaders as well as all headers with names starting with X-Accel- before sending the response to the client. The set of not forwardable response headers can be customized in function contentFromFullResponse.

Let's forward responses in location /full when argument proxy in the client request's URI is equal to yes.

File nginx.conf: forward responses from location /full

            if ($arg_proxy = yes) {
                haskell_content fromFullResponse $hs_subrequest;
                break;
            }

A simple test

$ curl -D- 'http://localhost:8010/full/?a=Value&p=8020&proxy=yes'
HTTP/1.1 200 OK
Server: nginx/1.17.9
Date: Fri, 24 Jul 2020 13:14:33 GMT
Content-Type: application/octet-stream
Content-Length: 37
Connection: keep-alive
Subrequest-Header: This is response from subrequest

In backend, Custom-Header is 'Value'

notForwardableResponseHeaders :: HashSet HeaderName Source #

Default set of not forwardable response headers.

HTTP response headers that won't be forwarded to the client in handler fromFullResponse. The set contains Connection, Content-Length, Date, Keep-Alive, Last-Modified, Server, Transfer-Encoding, and Content-Type headers (the latter gets reset in the handler's result value). If this set is not satisfactory, then handler fromFullResponse must be replaced with a custom handler based on contentFromFullResponse with a customized set of not forwardable response headers.

contentFromFullResponse Source #

Arguments

:: HashSet HeaderName	Set of not forwardable response headers
-> Bool	Do not forward X-Accel-... response headers
-> ByteString	Encoded HTTP response
-> ContentHandlerResult

Translates encoded response to ContentHandlerResult.

The translated data can be forwarded to the client by a simple handler based on this function in directive haskell_content. Handler fromFullResponse forwards the response to the client after deleting headers listed in set notForwardableResponseHeaders and headers with names starting with X-Accel-.