`{FI      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~                                  ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H  $IJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklIJKLMNOPQRSTWYZ[IJKLJKLMNNOPPQRRSTTWYZ[ portable experimental!dominic.steinitz@blueyonder.co.uk mnopqrBCut up a string into 72 char lines, each line terminated by CRLF. stuvwmrvwmrvwportable experimental!dominic.steinitz@blueyonder.co.ukxy Take [Octet] and return [Octet] according to the standard. H The length of the result is always 16 octets or 128 bits as required  by the standard. xyxynon-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>?Streams should make layering of TLS protocol easier in future,  they allow reading/$writing to files etc for debugging, + they allow use of protocols other than TCP/IP  and they allow customisation. (Instances of this class should not trim - the input in any way, e.g. leave LF on line 3 endings etc. Unless that is exactly the behaviour ) you want from your twisted instances ;) >This is the type returned by many exported network functions.        non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>zcrlf* is our beloved two-char line terminator. {sp% lets you save typing one character. |split delim ls# splits a list into two parts, the delim occurring % at the head of the second list. If delim isn't in ls, Nothing is  returned. }trim str. removes leading and trailing whitespace from str. ~ trimL str+ removes leading whitespace (as defined by )  from str.  trimL str, removes trailing whitespace (as defined by )  from str. splitMany delim ls removes the delimiter delim from ls. readsOne f def str tries to  str , taking $ the first result and passing it to f . If the   doesn't succeed, return def. z{|}~z{|}~non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>EDThis class allows us to write generic header manipulation functions  for both Request and Response data types. HTTP Header Name type: 0 Why include this at all? I have some reasons / 1) prevent spelling errors of header names, 5 2) remind everyone of what headers are available, 4 3) might speed up searches for specific headers. Arguments against: / 1) makes customising header names laborious  2) increases code volume. +Long discussions can be had on this topic! 1Allows for unrecognised or experimental headers. $MIME entity headers (for sub-parts)  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGThe Header data type pairs header names & values. HI4Header constructor as a function, hiding above rep. JKLM0Inserts a header with the given name and value.  Allows duplicate header names. 6Adds the new header only if no previous header shares  the same name. )Removes old headers with duplicate name. Inserts multiple headers. N)Gets a list of headers with a particular . O<Lookup presence of specific HeaderName in a list of Headers 3 Returns the value from the first matching header. PQC !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQCGHFEDCBA@?>=<;:9876543210/.-,+*)('&%$#"! IJKMNLOPQC4FEDCBA@?>=<;:9876543210/.-,+*)('&%$#"!  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHHIJKLMNOPQnon-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>RSTUVWXYZ[\]^_`abThe  BufferType3 class encodes, in a mixed-mode way, the interface A that the library requires to operate over data embedded in HTTP ? requests and responses. That is, we use explicit dictionaries ; for the operations, but overload the name of these dicts. cdefRSTUVWXYZ[\]^_`abcdefbcRSTUVWXYZ[\]^_`adefRSTUVWXYZ[\]^_`aSTUVWXYZ[\]^_`abccdefnon-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>dgRequestNormalizer1 is the shape of a (pure) function that rewrites & a request into some normalized form. hNormalizeRequestOptions' brings together the various defaulting/normalization options  over ~}s. Use & for the standard selection of option ijklmnopqrstAn HTTP Response.  The : instance of this type is used for message serialisation, B which means no body data is output, additionally the output will D show an HTTP version of 1.1 instead of the actual version returned  by a server. uvwxyz RequestData. contains the head of a HTTP request; method, ! its URL along with the auxillary/supporting header data. { ResponseData* contains the head of a response payload; < HTTP response code, accompanying text description + header  fields. |/For easy pattern matching, HTTP response codes xyz are  represented as (x,y,z). }An HTTP Request.  The : instance of this type is used for message serialisation, % which means no body data is output. ~might need changing in future  1) to support  uri in OPTIONS request + 2) transparent support for both relative  &% absolute uris, although this should  already work (leave scheme & host parts empty). +The HTTP request method, to be used in the ~} object. = We are missing a few of the stranger methods, but these are - not really necessary until we add full TLS. #Parse the authority part of a URL.  RFC 1732, section 3.1:  4 //<user>:<password>@<host>:<port>/<url-path> @ Some or all of the parts "<user>:<password>@", ":<password>", 0 ":<port>", and "/<url-path>" may be excluded. 'mkRequest method uri' constructs a well formed 8 request for the given HTTP method and URI. It does not ; normalize the URI for the request _nor_ add the required : Host: header. That is done either explicitly by the user 8 or when requests are normalized prior to transmission. getRequestVersion req& returns the HTTP protocol version of  the request req. If Nothing, the default  can be assumed. setRequestVersion v req% returns a new request, identical to  req#, but with its HTTP version set to v. getResponseVersion rsp& returns the HTTP protocol version of  the response rsp. If Nothing, the default  can be  assumed. setResponseVersion v rsp& returns a new response, identical to  rsp#, but with its HTTP version set to v.  getAuth req9 fishes out the authority portion of the URL in a request's Host  header. normalizeRequest opts req- is the entry point to use to normalize your K request prior to transmission (or other use.) Normalization is controlled  via the NormalizeRequestOptions record. normalizeUserAgent ua x req augments the request req with  a User-Agent: ua header if req doesn't already have a  a  User-Agent: set. !normalizeConnectionClose opts req sets the header Connection: close # to indicate one-shot behavior iff  normDoClose is True. i.e., it then  _replaces_ any an existing  Connection: header in req. normalizeHostURI forProxy req" rewrites your request to have it G follow the expected formats by the receiving party (proxy or server.) 4Used when we know exactly how many bytes to expect. 'Used when nothing about data is known, . Unfortunately waiting for a socket closure ' causes bad behaviour. Here we just ( take data once and give up the rest. A necessary feature of HTTP/1.1 , Also the only transfer variety likely to  return any footers. 2Maybe in the future we will have a sensible thing 4 to do here, at that time we might want to change  the name. DRemove leading crlfs then call readTillEmpty2 (not required by RFC) 'Read lines until an empty line (CRLF), - also accepts a connection close as end of  input, which is not an HTTP/1.1 compliant * thing to do - so probably indicates an  error condition.  catchIO a hD handles IO action exceptions throughout codebase; version-specific  tweaks better go here. Ughijklmnopqrstuvwxyz{|}~U}~tuvwxynsrqpo{|zhijklmgUghijklmijklmnsrqpoopqrstuvwxyuvwxyz{|}~~ non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>,The - newtype is a wrapper that allows us to make F connections an instance of the Stream class, without ghc extensions. D While this looks sort of like a generic reference to the transport = layer it is actually TCP specific, which can be seen in the  implementation of the 'Stream Connection' instance. openTCPPort uri port' establishes a connection to a remote  host, using / which possibly queries the DNS system, hence # may trigger a network connection. 4Checks both that the underlying Socket is connected 0 and that the connection peer matches the given ( host name (which is recorded locally). non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>!Allows stream logging. Refer to  below. Wraps a stream with logging I/O. 7 The first argument is a filename which is opened in  AppendMode. non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>)Exception handler for socket operations. non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>  simpleHTTP: transmits a resource across a non-persistent connection. simpleHTTP_debug debugFile req behaves like  , but logs ' the HTTP operation via the debug file  debugFile. Like *, but acting on an already opened stream. sendHTTP hStream httpRequest transmits  httpRequest over  hStreamH, but does not alter the status of the connection, nor request it to be % closed upon receiving the response. *sendHTTP_notify hStream httpRequest action behaves like , but  lets you supply an IO action3 to execute once the request has been successfully L transmitted over the connection. Useful when you want to set up tracing of + request transmission and its performance. receiveHTTP hStream reads a ~} from the  hStream  respondHTTP hStream httpResponse transmits an HTTP ut over  the  hStream+. It could be used to implement simple web 2 server interactions, performing the dual role to .  non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com> simpleHTTP req transmits the ~} req by opening a direct, non-persistent $ connection to the HTTP server that req+ is destined for, followed by transmitting ' it and gathering up the response as a  . Prior to sending the request,  it is normalized (via &). If you have to mediate the request R via an HTTP proxy, you will have to normalize the request yourself. Or switch to  using Network.Browser instead.  Examples: 7 simpleHTTP (getRequest "http://hackage.haskell.org/") < simpleHTTP (getRequest "http://hackage.haskell.org:8012/")  Identical to *, but acting on an already opened stream. sendHTTP hStream httpRequest transmits  httpRequest (after normalization) over  hStreamH, but does not alter the status of the connection, nor request it to be % closed upon receiving the response. *sendHTTP_notify hStream httpRequest action behaves like , but  lets you supply an IO action3 to execute once the request has been successfully L transmitted over the connection. Useful when you want to set up tracing of + request transmission and its performance. receiveHTTP hStream reads a ~} from the  hStream  respondHTTP hStream httpResponse transmits an HTTP ut over  the  hStream+. It could be used to implement simple web 2 server interactions, performing the dual role to . getRequest urlString* is convenience constructor for basic GET ~}s. If   urlString isn';t a syntactically valid URL, the function raises an error. postRequest urlString% is convenience constructor for POST ~}s. If   urlString isn';t a syntactically valid URL, the function raises an error. getResponseBody response4 takes the response of a HTTP requesting action and $ tries to extricate the body of the ut response. If the request action / returned an error, an IO exception is raised.  !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQghijklmnopqrstuvwxyz{|}~ non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com> FSimple way to transmit a resource across a non-persistent connection. Like *, but acting on an already opened stream. HReceive and parse a HTTP request from the given Stream. Should be used ! for server side interactions. HVery simple function, send a HTTP response over the given stream. This 9 could be improved on to use different transfer types.   non-portable (not tested) experimental)Sigbjorn Finne <sigbjorn.finne@gmail.com>3 is the enumerated list of events that the browser 8 internals will report to a user-defined event handler.  BrowserEvent; is the event record type that a user-defined handler, set  via E5, will be passed. It indicates various state changes * encountered in the processing of a given , along with timestamps  at which they occurred.  RequestState8 is an internal tallying type keeping track of various I per-connection counters, like the number of authorization attempts and  forwards we've gone through. number of 401 responses so far number of redirects so far number of retries so far !whether to pre-empt 401 response  BrowserAction' is the IO monad, but carrying along a .  BrowserState1 is the (large) record type tracking the current  settings of the browser. Proxy6 specifies if a proxy should be used for the request. +Use the proxy given. Should be of the form http://host:port, host,  host:port, or  http://host . Optional $ to authenticate with the proxy as. Don't use a proxy.  Authority5 specifies the HTTP Authentication method to use for  a given domain/realm; Basic or Digest.       Algorithm# controls the digest algorithm to, MD5 or  MD5Session. Cookie6 is the Haskell representation of HTTP cookie values. 3 See its relevant specs for authoritative details.  !"#$%defaultCookieFilter* is the initial cookie acceptance filter. % It welcomes them all into the store :-) &userCookieFilter* is a handy acceptance filter, asking the  user if he/3she is willing to accept an incoming cookie before  adding it to the store. /Serialise a Cookie for inclusion in a request. ' addCookie c: adds a cookie to the browser state, removing duplicates. (setCookies cookies& replaces the set of cookies known to  the browser to cookies). Useful when wanting to restore cookies  used across 6 invocations. ) getCookies- returns the current set of cookies known to  the browser. *setCookieFilter fn& sets the cookie acceptance filter to fn. +getCookieFilter/ returns the current cookie acceptance filter. headerChallenge base www_auth tries to convert the WWW-Authenticate header  www_auth into a   value. 0Return authorities for a given domain and path.  Assumes dom is lower case ,getAuthorities return the current set of  Authoritys known  to the browser. -./getAuthorityGen) returns the current authority generator 0setAuthorityGen genAct sets the auth generator to genAct. 1setAllowBasicAuth onOff enables/$disables HTTP Basic Authentication. 2setMaxAuthAttempts mbMax4 sets the maximum number of authentication attempts  to do. If Nothing, rever to default max. 3getMaxAuthAttempts+ returns the current max auth attempts. If Nothing,  the browser's default is used. 4setMaxErrorRetries mbMax( sets the maximum number of attempts at  transmitting a request. If Nothing, rever to default max. 5getMaxErrorRetries2 returns the current max number of error retries. 3Retrieve a likely looking authority for a Request. *Asking the user to respond to a challenge 5Generating a credentials value from an Authority, in 7 the context of a specific request. If a client nonce 1 was to be used then this function might need to ( be of type ... -> BrowserAction String 6 browse act% is the toplevel action to perform a .  Example use: %browse (request (getRequest yourURL)). Default browser state.. Alter browser state 7getBrowserState, returns the current browser config. Useful  for restoring state across s. 8withBrowserAction st act performs act with  st. 9Lifts an IO action into the  monad. : setErrHandler! sets the IO action to call when 4 the browser reports running errors. To disable any  such, set it to const (return ()). ; setErrHandler! sets the IO action to call when : the browser chatters info on its running. To disable any  such, set it to const (return ()). <=>setAllowRedirects onOff toggles the willingness to : follow redirects (HTTP responses with 3xx status codes). ?getAllowRedirects9 returns current setting of the do-chase-redirects flag. @setMaxRedirects maxCount- sets the maxiumum number of forwarding hops L we are willing to jump through. A no-op if the count is negative; if zero, I the max is set to whatever default applies. Notice that setting the max  redirects count does not+ enable following of redirects itself; use  > to do so. AgetMaxRedirects9 returns the current setting for the max-redirect count.  If Nothing, the Network.Browser's default is used. B setProxy p will disable proxy usage if p is NoProxy.  If p is Proxy proxyURL mbAuth, then proxyURL is interpreted > as the URL of the proxy to use, possibly authenticating via   information in mbAuth. CgetProxy% returns the current proxy settings. DsetDebugLog mbFile turns off debug logging iff mbFile  is Nothing . If set to  Just fStem, logs of browser activity " is appended to files of the form fStem-url-authority, i.e.,  fStem8 is just the prefix for a set of log files, one per host/ authority. EsetEventHandler onBrowserEvent configures event handling.  If onBrowserEvent is Nothing , event handling is turned off;  setting it to  Just onEv causes the onEv IO action to be ? notified of browser events during the processing of a request  by the Browser pipeline. @The default number of hops we are willing not to go beyond for  request forwardings. ?The default number of error retries we are willing to perform. BThe default maximum HTTP Authentication attempts we will make for  a single request. Frequest httpRequest tries to submit the ~}  httpRequest + to some HTTP server (possibly going via a proxy, see B.) G Upon successful delivery, the URL where the response was fetched from  is returned along with the ut itself. =Internal helper function, carrying along per-request counts. -The internal request handling state machine. BMaximum number of open connections we are willing to have active. GHuriDefaultTo a b1 returns a URI that is consistent with the first  argument URI a, when read in the context of the second URI b. B If the second argument is not sufficient context for determining  a full URI then anarchy reins. _      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHc6F78>?@A       ,-. /014523*+%& !"#$)(':;EBCD<=9GH]             !"#$ !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGH !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWWXYZ[\]^_`aabcdefghijklmnopqrstuvvwxyz{|}~                                                      ! " # $ % & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C C D D E E F F G H I J K L M N O P Q R S T U V W X Y Z [ \ ] ^ _ ` a b c d e f g h i_jklmnopqrstupvwxypz{p|}~                               _             K                   HTTP-4000.0.5Network.StreamNetwork.HTTP.HeadersNetwork.BufferTypeNetwork.HTTP.Base Network.TCPNetwork.StreamDebuggerNetwork.StreamSocketNetwork.HTTP.HandleStream Network.HTTPNetwork.HTTP.StreamNetwork.BrowserNetwork.HTTP.MD5AuxNetwork.HTTP.Base64Network.HTTP.MD5Network.HTTP.UtilsStreamreadLine readBlock writeBlockcloseResult ConnError ErrorMisc ErrorParse ErrorClosed ErrorReset failParsefailWithbindEfmapE HasHeaders getHeaders setHeaders HeaderName HdrCustomHdrContentTransferEncodingHdrLastModified HdrExpiresHdrETagHdrContentTypeHdrContentRange HdrContentMD5HdrContentLocationHdrContentLengthHdrContentLanguageHdrContentEncodingHdrContentBaseHdrAllowHdrWWWAuthenticate HdrWarningHdrVary HdrTrailerHdrTE HdrSetCookie HdrServer HdrRetryAfter HdrPublicHdrProxyAuthenticate HdrLocationHdrAge HdrUserAgent HdrRefererHdrRangeHdrProxyAuthorizationHdrMaxForwardsHdrIfUnmodifiedSince HdrIfRangeHdrIfNoneMatch HdrIfMatchHdrIfModifiedSinceHdrHostHdrFrom HdrExpect HdrCookieHdrAuthorizationHdrAcceptLanguageHdrAcceptEncodingHdrAcceptCharset HdrAcceptHdrVia HdrUpgradeHdrTransferEncoding HdrPragmaHdrDate HdrConnectionHdrCacheControlHeadermkHeader insertHeaderinsertHeaderIfMissing replaceHeader insertHeadersretrieveHeaders findHeader lookupHeader parseHeadersBufferOpbuf_hGetbuf_hGetContentsbuf_hPut buf_hGetLine buf_empty buf_append buf_concat buf_fromStr buf_toStrbuf_snoc buf_splitAtbuf_spanbuf_isLineTerm buf_isEmpty BufferType bufferOpsstrictBufferOp lazyBufferOpstringBufferOpRequestNormalizerNormalizeRequestOptions normDoClose normForProxy normUserAgent normCustomsResponseNextStep DieHorribly ExpectEntityDoneRetryContinueResponserspCode rspReason rspHeadersrspBody RequestData ResponseData ResponseCodeRequestrqURIrqMethod rqHeadersrqBody HTTPResponse HTTPRequestResponse_StringRequest_String RequestMethodCustomCONNECTTRACEOPTIONSDELETEPOSTGETPUTHEAD URIAuthorityuserpasswordhostportparseURIAuthorityuriToAuthorityStringuriAuthToString uriAuthPort reqURIAuth httpVersionlibUAdefaultUserAgentdefaultGETRequestdefaultGETRequest_ mkRequestparseRequestHeadparseResponseHeadgetRequestVersionsetRequestVersiongetResponseVersionsetResponseVersion matchResponse urlDecode urlEncode urlEncodeVarsgetAuthnormalizeRequestURIdefaultNormalizeRequestOptionsnormalizeRequestsplitRequestURInormalizeHostHeader findConnCloselinearTransferhopefulTransferchunkedTransferuglyDeathTransferreadTillEmpty1readTillEmpty2catchIOcatchIO_responseParseErrorHStream openStream StreamHooks hook_readLinehook_readBlockhook_writeBlock hook_close HandleStream ConnectionhstreamToConnection nullHookssetStreamHooks openTCPPortopenTCPConnection isConnectedToisTCPConnectedToStreamDebugger debugStreamdebugByteStreamhandleSocketErrormyrecv simpleHTTPsimpleHTTP_debug simpleHTTP_sendHTTPsendHTTP_notify receiveHTTP respondHTTP getRequest postRequestgetResponseBodyFormFormVarBrowserEventTypeResponseFinish ResponseEnd RequestSentReuseConnectionOpenConnection BrowserEventbrowserTimestampbrowserRequestIDbrowserRequestURIbrowserEventType RequestID BrowserAction BrowserStateProxyNoProxy Authority AuthDigestauNonce auAlgorithmauDomainauOpaqueauQop AuthBasicauRealm auUsername auPasswordauSite Challenge ChalDigestchDomainchNoncechOpaquechStale chAlgorithmchQop ChalBasicchRealmQop QopAuthIntQopAuth Algorithm AlgMD5sessAlgMD5CookieMkCookieckDomainckNameckValueckPath ckComment ckVersiondefaultCookieFilteruserCookieFilter addCookie setCookies getCookiessetCookieFiltergetCookieFiltergetAuthoritiessetAuthorities addAuthoritygetAuthorityGensetAuthorityGensetAllowBasicAuthsetMaxAuthAttemptsgetMaxAuthAttemptssetMaxErrorRetriesgetMaxErrorRetriesbrowsegetBrowserStatewithBrowserStateioAction setErrHandler setOutHandlerouterrsetAllowRedirectsgetAllowRedirectssetMaxRedirectsgetMaxRedirectssetProxygetProxy setDebugLogsetEventHandlerrequest formToRequest uriDefaultToMD5get_nextlen_padfinishedWordListBoolListStrABCDRotationXYZZord64rotLmd5md5smd5imd5_main md5_do_block md5_roundmd5_inner_functionmd5_fmd5_gmd5_hmd5_i magic_numbersroundsabcd_to_stringdisplay_32bits_as_hexabcd_to_integerrev_numstring_to_word32sbools_to_word32slength_to_charsOctet encodeArray int4_char3 char3_int4enc1chop72 quadrupletsencdcdencodedecodehashcrlfspsplittrimtrimLbase GHC.UnicodeisSpacetrimRsplitByreadsOne Text.Readread headerMap parseHeaderGHC.ShowShowGHC.Num* pURIAuthority pUserInfobefore orNothing rqMethodMaptoBufOps withVersiongetHttpVersionsetHttpVersiondropHttpVersion isHttpVersionnormalizeUserAgentnormalizeConnectionClosenormalizeHostURIchunkedTransferCConn ConnClosedMkConnconnSock connHandle connBuffer connInputconnHost connHooksgetRef connHooks'network-2.2.1.7 Network.BSD getHostByNameopenTCPConnection_closeConnection readBlockBS readLineBS writeBlockBScloseItbufferGetBlockbufferPutBlockbufferReadLine onNonClosedDoDbgdebugStreamHooksreadBlockSocketreadLineSocketwriteBlockSocketsendMainswitchResponsegetResponseHead headerName ifChunkeddebug httpLogFile RequestState reqDenies reqRedirects reqRetries reqStopOnDenyBAliftBSbsErrbsOut bsCookiesbsCookieFilterbsAuthorityGen bsAuthoritiesbsAllowRedirectsbsAllowBasicAuthbsMaxRedirectsbsMaxErrorRetriesbsMaxAuthAttemptsbsConnectionPoolbsProxybsDebugbsEvent bsRequestIDcookieToHeaderheaderToCookies getCookiesForheaderToChallenge getAuthFor pickChallengeanticipateChallengechallengeToAuthority withAuthoritystringToOctetsoctetsToString base64encodekddefaultBrowserStatealterBSgetBS newRequestnullRequestStatebuildBrowserEvent reportEventdefaultMaxRetriesdefaultMaxErrorRetriesdefaultMaxAuthAttemptsrequest' spaceappend dorequestupdateConnectionPool maxPoolSize quotedstringword