This module provides all the settable options in shpider.
Set the CurlTimeout option. Requests will TimeOut after this number of seconds.
Set the start page of your shpidering antics. The start page must be an absolute URL, if not, this will raise an error.
If onlyDownloadHtml is True, then during
download, shpider will make a HEAD request to see if the content type is text/html or application/xhtml+xml, and only if it is, then it will make a GET request.