Make the IO/Handle system encoding aware.

[sjanssen]

Make the IO/Handle system encoding aware.

The IO system in the base library has no concept of character encodings. Because of this, many Haskell programs resort to custom made hacks to perform input and output (like the UTF8.lhs module in half a dozen projects).

The goals of this project:

• Extend Handles with encoding information.
• Make the standard IO functions use this encoding information to interpret the raw IO stream as Haskell Strings and Chars.
• Implement a few encodings (UTF-8, Latin1, etc.)

Interested Mentors

• Kido Takahiro (shelarcy) <shelarcy@…>
• Simon Marlow <simonmarhaskell@…>

Interested Students

• Spencer Janssen (sjanssen) <sjanssen@…>
• Brandon Wilson (xelxebar) <bmw.stx@…>

[duncan]

This is something we do need to do. Simon Marlow has a suggested design for the internals using iconv. The project should also consider the API.

[shelarcy]

MinGW doesn't include iconv. But Windows has encoding APIs.

•  MSDN'S Unicode and Character Sets information
  • use getACP, to get environment's encoding informantion.
  • use MultiByteToWideChar? and WideCharToMultiByte?, to convert between encoding and Unicode charactor set.
    • GHCi uses Haskell binding function  stringToUnicode, to support system encoding in console.

And Mac OS X also has encoding API in NSString class.

•  http://developer.apple.com/documentation/Cocoa/Reference/Foundation/Classes/NSString_Class/Reference/NSString.html

So, I think we must add configuration to know that "Does system have iconv library?", and "Does system have other encoding support API?".

Without OS environment, we can't implement and test all platform. So, we don't expect that for GSoC project. But I think we don't forgot configuration mechanism for future.

[dons]

The semi-standard solution for the past 6 months has been to use the utf8-string library,

 http://hackage.haskell.org/cgi-bin/hackage-scripts/package/utf8-string

A relevant background reference.

The other major reference is the handle/IO thread on the mailing list from February

[duncan]

The standalone GNU iconv lib builds on all interesting systems, including windows so there is no problem with systems that do not have iconv. Of course if a platform has a preferred native api then that might be preferable but it does not prevent using iconv in the general case.

[simonmar]

Already done, just needs porting to Windows.