with-utf8: Get your IO right on the first try

[ io, library, mpl, program ] [ Propose Tags ] [ Report a vulnerability ]

This minimalistic library helps you navigate the world of text encodings avoiding invalid argument (invalid byte sequence) and invalid argument (invalid character) in runtime.

See this blog post for why this library exists and what exactly it does.

The two most important modules are:


[Skip to Readme]

Downloads

Note: This package has metadata revisions in the cabal description newer than included in the tarball. To unpack the package including the revisions, use 'cabal get'.

Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees

Candidates

Versions [RSS] 1.0.0.0, 1.0.1.0, 1.0.2.0, 1.0.2.1, 1.0.2.2, 1.0.2.3, 1.0.2.4, 1.1.0.0
Change log CHANGELOG.md
Dependencies base (>=4.10 && <4.21), directory (>=1.2.5.0 && <1.4), filepath (>=1.0 && <1.6), process (>=1.0.1.1 && <1.7), safe-exceptions (>=0.1 && <0.2), text (>=0.7 && <2.2), th-env (>=0.1.0.0 && <0.2) [details]
License MPL-2.0
Copyright 2020 Serokell
Author Kirill Elagin <kirelagin@serokell.io>
Maintainer Kirill Elagin <kirelagin@serokell.io>
Revised Revision 1 made by gromak at 2024-07-08T11:42:58Z
Category IO
Home page https://github.com/serokell/haskell-with-utf8#readme
Bug tracker https://github.com/serokell/haskell-with-utf8/issues
Source repo head: git clone https://github.com/serokell/haskell-with-utf8
Uploaded by gromak at 2024-01-17T09:43:14Z
Distributions Arch:1.1.0.0, LTSHaskell:1.1.0.0, NixOS:1.0.2.4, Stackage:1.1.0.0
Reverse Dependencies 10 direct, 5 indirect [details]
Executables utf8-troubleshoot
Downloads 5886 total (103 in the last 30 days)
Rating 2.0 (votes: 1) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2024-01-17 [all 1 reports]

Readme for with-utf8-1.1.0.0

[back to package description]

with-utf8

Get your IO right on the first try.

Reading files in Haskell is trickier than it could be due to the non-obvious interactions between file encodings and system locale. This library is meant to make it easy once and for all by providing “defaults” that make more sense in the modern world.

See this blog post for more details on why this library needs to exists and an explanation of some of the opinionated decisions it is based on.

Use

See the documentation on Hackage for details, this is a quick summary.

Step 1: Get it

The library is on Hackage, go ahead and add it to the dependencies of your project.

Step 2: Wrap your main

Import withUtf8 from Main.Utf8 and wrap it around your main:

import Main.Utf8 (withUtf8)

main :: IO ()
main = withUtf8 $
  {- ... your main function ... -}

This will make sure that if your program reads something from stdin or outputs something to stdout/stderr, it will not fail with a runtime error due to encoding issues.

Step 3: Read files using UTF-8

If you are going to read a text file (to be precise, if you are going to open a file in text mode), you’ll probably use withFile, openFile, or readFile. Grab the first two from System.IO.Utf8 or the latter from Data.Text.IO.Utf8. Starting from text-2.1, Data.Text.IO.Utf8 is available in the text package itself, hence this module in with-utf8 is now deprecated.

Note: it is best to import these modules qualified.

Note: there is no System.IO.Utf8.readFile because it’s 2024 and you should not read Strings from files.

All these functions will make sure that the content will be treated as if it was encoded in UTF-8.

If, for some reason, you really need to use withFile/openFile from base, or you got your file handle from somewhere else, wrap the code that works with it in a call to withHandle from System.IO.Utf8:

import qualified System.IO as IO
import qualified System.IO.Utf8 as Utf8

doSomethingWithAFile :: IO.Handle -> IO ()
doSomethingWithAFile h = Utf8.withhandle h $ do
    {- ... work with the file ... -}

Step 4: Write files using UTF-8

When writing a file either open it using withFile/openFile from System.IO.Utf8 or write to it directly with writeFile from Data.Text.IO.Utf8. Starting from text-2.1, Data.Text.IO.Utf8 is available in the text package itself, hence this module in with-utf8 is now deprecated.

Note: it is best to import these modules qualified.

Note: there is no System.IO.Utf8.writeFile.

If, for some reason, you really need to use withFile/openFile from base, do the same as in the previous step.

Troubleshooting

Locales are pretty straightforward, but some people might have their terminals misconfigured for various reasons. To help troubleshoot any potential issues, this package comes with a tool called utf8-troubleshoot.

This tool outputs some basic information about locale settings in the OS and what they end up being mapped to in Haskell. If you are looking for help, please, provide the output of this tool, or if you are helping someone, ask them to run this tool and provide the output.

Contributing

If you encounter any issues when using this library or have improvement ideas, please open report in issue on GitHub. You are also very welcome to submit pull request, if you feel like doing so.

License

MPL-2.0 © Serokell