hsc2hs and utf-8

[TaruKarttunen]

hsc2hs is broken on a source file containing certain utf-8 characters in the comments.

Attached is an example module.

The probable culprit is:

    showCChar c    = ['\\',
                      intToDigit (ord c `quot` 64),
                      intToDigit (ord c `quot` 8 `mod` 8),
                      intToDigit (ord c          `mod` 8)]

in hsc2hs code.

[TaruKarttunen]

Not sure if the encoding got preserved.

It is simply:

oz:HsOpenSSL$ od -t x1 A.hsc 0000000 2d 2d 20 e3 82 a8 0a 0000007

japanese: U+30A8 エ e3 82 a8 KATAKANA LETTER E

[simonmar]

Fixed:

Fri Mar 19 21:56:57 GMT 2010  Simon Marlow <marlowsd@gmail.com>
  * Process the file in binary mode so we pass though UTF-8 (#3837)
  Maybe strictly speaking it would be better to encode/decode UTF-8, but
  it would be a fiddle and I don't think it really matters for hsc2hs.

Mon Mar 22 13:55:55 GMT 2010  Simon Marlow <marlowsd@gmail.com>
  * add a comment

[igloo]

Merged.

[awson]

Now it's broken more badly. For example:

#{test З}

where З here is UTF-8 coded char 1047, gives "<stdout>: commitBuffer: invalid argument (character is not in the code page)" error. I believe any UTF-8 coded char > 127 gives this error.

[simonmar]

Fixed:

Wed Dec 22 01:14:55 PST 2010  Simon Marlow <marlowsd@gmail.com>
  * write the output file in binary mode (#3837)

[igloo]

Merged.