Network.Shpider

This module exposes the main functionality of shpider It allows you to quickly write crawlers, and for simple cases even without reading the page source eg.

 runShpider $ do
    download "http://hackage.haskell.org/packages/archive/pkg-list.html"
    l : _ <- getLinksByText "shpider"
    download $ linkAddress l

Documentation

module Network.Shpider.Code

module Network.Shpider.State

module Network.Shpider.URL

module Network.Shpider.Options

module Network.Shpider.Forms

module Network.Shpider.Links

Crawl Functions

download :: String -> Shpider (ShpiderCode, Page)

Fetch whatever is at this address, and attempt to parse the content into a Page. Return the status code with the parsed content.

sendForm :: Form -> Shpider (ShpiderCode, Page)

Send a form to the URL specified in its action attribute

Basic Parsing/Decision Support

getLinksByText :: String -> Shpider [Link]

Get all links which match this text.

getLinksByTextRegex :: String -> Shpider [Link]

Get all links whose text matches this regex.

getLinksByAddressRegex :: String -> Shpider [Link]

Get all links whose address matches this regex.

getFormsByAction :: String -> Shpider [Form]

Get all forms whose action matches the given action

getFormsHasAction :: (String -> Bool) -> Shpider [Form]

currentLinks :: Shpider [Link]

Return the links on the current page.

currentForms :: Shpider [Form]

Return the forms on the current page.

Utilities

parsePage :: String -> String -> Shpider Page

Parse a given URL and source html into the Page datatype. This will set the current page.

isAuthorizedDomain :: String -> Shpider Bool

If stayOnDomain has been set to true, then isAuthorizedDomain returns True if the given URL is on the domain and false otherwise. If stayOnDomain has not been set to True, then it returns True.

withAuthorizedDomain :: String -> Shpider (ShpiderCode, Page) -> Shpider (ShpiderCode, Page)

withAuthorizedDomain will execute the function if the url given is an authorized domain. See isAuthorizedDomain.

haveVisited :: String -> Shpider Bool

if keepTrack has been set, then haveVisited will return True if the given URL has been visited.