Network.Shpider.Options

This module provides all the settable options in shpider.

Documentation

stayOnDomain :: Bool -> Shpider ()

Setting this to True will forbid you to download and sendForm to any site which isn't on the domain shared by the url given in setStartPage.

setTimeOut :: Long -> Shpider ()

Set the CurlTimeout option. Requests will TimeOut after this number of seconds.

setStartPage :: String -> Shpider ()

Set the start page of your shpidering antics. The start page must be an absolute URL, if not, this will raise an error.

getStartPage :: Shpider String

Return the starting URL, as set by setStartPage

onlyDownloadHtml :: Bool -> Shpider ()

If onlyDownloadHtml is True, then during download, shpider will make a HEAD request to see if the content type is text/html or application/xhtml+xml, and only if it is, then it will make a GET request.

setCurrentPage :: Page -> Shpider ()

Set the given page as the currentPage.

getCurrentPage :: Shpider Page

Return the current page

keepTrack :: Shpider ()

When keepTrack is set, shpider will remember the pages which have been visited.

addCurlOpts :: [CurlOption] -> Shpider ()

Add CURL options to Shpider

setCurlOpts :: [CurlOption] -> Shpider ()

Set Shpider's CURL options from scratch

setThrottle :: Maybe Int -> Shpider ()

Set download throttling, so that subsequent calls to download or sendForm block, making sure at least N micro-seconds pass. Passing a Nothing would disable any throttling.