treat byte order mark as zero-width whitespace

[igloo]

The U+FEFF ZERO WIDTH NO-BREAK SPACE Unicode character, better known as BYTE ORDER MARK (BOM), currently gives a lexical error:

$ printf '\xEF\xBB\xBF\nz = "str"\n' > z.hs
$ ghci z.hs
GHCi, version 6.8.0.20070927: http://www.haskell.org/ghc/  :? for help
Loading package base ... linking ... done.

z.hs:1:0: lexical error at character '\65279'
Failed, modules loaded: none.
Prelude> Leaving GHCi.

The character is only in categories Other and Format, not Space, but I think we should lex it as whitespace anyway (with zero width for the purposes of the layout rule). Ideally Haskell' would do likewise.

[guest]

Please note, that according to  http://en.wikipedia.org/wiki/Byte_Order_Mark the character U+FEFF is considered a BOM only if it appears as the first character of a file. In the context of UTF-8 it simply serves to identify the encoding. So there's no need to lex it as space, only to ignore it as the first character of a source file. Thanks.

[Porges]

Can I vote for this also? Some editors insist upon inserting the BOM when working with UTF-8 source, and this bug is highly annoying.

[simonmar]

I'm on it

[simonmar]

Fixed:

Fri Nov 30 10:11:00 GMT 2007  Simon Marlow <simonmar@microsoft.com>
  * FIX #1744: ignore the byte-order mark at the beginning of a file

[igloo]

Merged