This is another hand-written lexer, this time for the Xtract command-language. The entry point is lexXtract. You don't normally need to use this module directly - the lexer is called automatically by the parser. (We only expose this interface for debugging purposes.)
The Xtract command language is very like the XPath specification.
First argument is a transformer for pattern strings, e.g. map toLower, but only applying to parts of the pattern not in quotation marks. (Needed to canonicalise HTML where tags are case-insensitive, but attribute values are case sensitive.)