Data.Text.Utf8

This module provides functions that allow treating Text values as series of UTF-8 code units instead of characters. Currently, it also contains a stub Text type which treats its internal byte array as UTF-8 encoded. We use this as a placeholder until we can use text-2.

Documentation

type CodePoint = Char

A Unicode code point.

type CodeUnit = Word8

A UTF-8 code unit is a byte. A Unicode code point can be encoded as up to four code units.

newtype CodeUnitIndex

An index into the raw UTF-8 data of a Text. This is not the code point index as conventionally accepted by Text, so we wrap it to avoid confusing the two. Incorrect index manipulation can lead to surrogate pairs being sliced, so manipulate indices with care. This type is also used for lengths.

Constructors

CodeUnitIndex

Fields

• codeUnitIndex :: Int

Instances

Bounded CodeUnitIndex

Defined in Data.Text.Utf8

Methods

minBound :: CodeUnitIndex

maxBound :: CodeUnitIndex

Eq CodeUnitIndex

Defined in Data.Text.Utf8

Methods

(==) :: CodeUnitIndex -> CodeUnitIndex -> Bool

(/=) :: CodeUnitIndex -> CodeUnitIndex -> Bool

Num CodeUnitIndex

Defined in Data.Text.Utf8

Methods

(+) :: CodeUnitIndex -> CodeUnitIndex -> CodeUnitIndex

(-) :: CodeUnitIndex -> CodeUnitIndex -> CodeUnitIndex

(*) :: CodeUnitIndex -> CodeUnitIndex -> CodeUnitIndex

negate :: CodeUnitIndex -> CodeUnitIndex

abs :: CodeUnitIndex -> CodeUnitIndex

signum :: CodeUnitIndex -> CodeUnitIndex

fromInteger :: Integer -> CodeUnitIndex

Ord CodeUnitIndex

Defined in Data.Text.Utf8

Methods

compare :: CodeUnitIndex -> CodeUnitIndex -> Ordering

(<) :: CodeUnitIndex -> CodeUnitIndex -> Bool

(<=) :: CodeUnitIndex -> CodeUnitIndex -> Bool

(>) :: CodeUnitIndex -> CodeUnitIndex -> Bool

(>=) :: CodeUnitIndex -> CodeUnitIndex -> Bool

max :: CodeUnitIndex -> CodeUnitIndex -> CodeUnitIndex

min :: CodeUnitIndex -> CodeUnitIndex -> CodeUnitIndex

Show CodeUnitIndex

Defined in Data.Text.Utf8

Methods

showsPrec :: Int -> CodeUnitIndex -> ShowS

show :: CodeUnitIndex -> String

showList :: [CodeUnitIndex] -> ShowS

Generic CodeUnitIndex

Defined in Data.Text.Utf8

Associated Types

type Rep CodeUnitIndex :: Type -> Type

Methods

from :: CodeUnitIndex -> Rep CodeUnitIndex x

to :: Rep CodeUnitIndex x -> CodeUnitIndex

Hashable CodeUnitIndex

Defined in Data.Text.Utf8

Methods

hashWithSalt :: Int -> CodeUnitIndex -> Int

hash :: CodeUnitIndex -> Int

ToJSON CodeUnitIndex

Defined in Data.Text.Utf8

Methods

toJSON :: CodeUnitIndex -> Value

toEncoding :: CodeUnitIndex -> Encoding

toJSONList :: [CodeUnitIndex] -> Value

toEncodingList :: [CodeUnitIndex] -> Encoding

FromJSON CodeUnitIndex

Defined in Data.Text.Utf8

Methods

parseJSON :: Value -> Parser CodeUnitIndex

parseJSONList :: Value -> Parser [CodeUnitIndex]

NFData CodeUnitIndex

Defined in Data.Text.Utf8

Methods

rnf :: CodeUnitIndex -> ()

type Rep CodeUnitIndex

Defined in Data.Text.Utf8

type Rep CodeUnitIndex = D1 ('MetaData "CodeUnitIndex" "Data.Text.Utf8" "alfred-margaret-1.1.2.0-XY9PIx1ODVHYQ5PowYWXi" 'True) (C1 ('MetaCons "CodeUnitIndex" 'PrefixI 'True) (S1 ('MetaSel ('Just "codeUnitIndex") 'NoSourceUnpackedness 'NoSourceStrictness 'DecidedLazy) (Rec0 Int)))

data Text

Constructors

Text

A placeholder data type for UTF-8 encoded text until we can use text-2.0.

Fields

• !ByteArray

  Underlying array encoded using UTF-8.

• !Int

  Starting position of the UTF-8 sequence in bytes.

• !Int

  Length of the UTF-8 sequence in bytes.

Instances

Eq Text

Defined in Data.Text.Utf8

Methods

(==) :: Text -> Text -> Bool

(/=) :: Text -> Text -> Bool

Ord Text

Defined in Data.Text.Utf8

Methods

compare :: Text -> Text -> Ordering

(<) :: Text -> Text -> Bool

(<=) :: Text -> Text -> Bool

(>) :: Text -> Text -> Bool

(>=) :: Text -> Text -> Bool

max :: Text -> Text -> Text

min :: Text -> Text -> Text

Show Text

Defined in Data.Text.Utf8

Methods

showsPrec :: Int -> Text -> ShowS

show :: Text -> String

showList :: [Text] -> ShowS

IsString Text

Defined in Data.Text.Utf8

Methods

fromString :: String -> Text

Hashable Text

Defined in Data.Text.Utf8

Methods

hashWithSalt :: Int -> Text -> Int

hash :: Text -> Int

ToJSON Text

Defined in Data.Text.Utf8

Methods

toJSON :: Text -> Value

toEncoding :: Text -> Encoding

toJSONList :: [Text] -> Value

toEncodingList :: [Text] -> Encoding

FromJSON Text

Defined in Data.Text.Utf8

Methods

parseJSON :: Value -> Parser Text

parseJSONList :: Value -> Parser [Text]

NFData Text

Defined in Data.Text.Utf8

Methods

rnf :: Text -> ()

lengthUtf8 :: Text -> CodeUnitIndex

The return value of this function is not really an index. However the signature is supposed to make it clear that the length is returned in terms of code units, not code points.

lowerCodePoint :: Char -> Char

Lower-Case a UTF-8 codepoint. Uses toLowerAscii for ASCII and toLower otherwise.

lowerUtf8 :: Text -> Text

toLowerAscii :: Char -> Char

Lower-case the ASCII code points A-Z and leave the rest of ASCII intact.

unicode2utf8 :: (Ord a, Num a, Bits a) => a -> [a]

Convert a Unicode Code Point c into a list of UTF-8 code units (bytes).

unpackUtf8 :: Text -> [CodeUnit]

Decoding

Functions that turns code unit sequences into code point sequences.

decode2 :: CodeUnit -> CodeUnit -> CodePoint

Decode 2 UTF-8 code units into their code point. The given code units should have the following format:

┌───────────────┬───────────────┐
│1 1 0 x x x x x│1 0 x x x x x x│
└───────────────┴───────────────┘

decode3 :: CodeUnit -> CodeUnit -> CodeUnit -> CodePoint

Decode 3 UTF-8 code units into their code point. The given code units should have the following format:

┌───────────────┬───────────────┬───────────────┐
│1 1 1 0 x x x x│1 0 x x x x x x│1 0 x x x x x x│
└───────────────┴───────────────┴───────────────┘

decode4 :: CodeUnit -> CodeUnit -> CodeUnit -> CodeUnit -> CodePoint

Decode 4 UTF-8 code units into their code point. The given code units should have the following format:

┌───────────────┬───────────────┬───────────────┬───────────────┐
│1 1 1 1 0 x x x│1 0 x x x x x x│1 0 x x x x x x│1 0 x x x x x x│
└───────────────┴───────────────┴───────────────┴───────────────┘

decodeUtf8 :: [CodeUnit] -> [CodePoint]

Decode a list of UTF-8 code units into a list of code points.

stringToByteArray :: String -> ByteArray

Indexing

Text can be indexed by code units or code points. A CodePoint is a 21-bit Unicode code point and can consist of up to four code units. A CodeUnit is a single byte.

indexCodeUnit :: Text -> CodeUnitIndex -> CodeUnit

Get the code unit at the given CodeUnitIndex. Performs bounds checking.

unsafeIndexCodePoint :: Text -> CodeUnitIndex -> (CodeUnitIndex, CodePoint)

Does exactly the same thing as unsafeIndexCodePoint', but on Text values.

unsafeIndexCodePoint' :: ByteArray -> CodeUnitIndex -> (CodeUnitIndex, CodePoint)

Decode a code point at the given CodeUnitIndex. Returns garbage if there is no valid code point at that position. Does not perform bounds checking. See decode2, decode3 and decode4 for the expected format of multi-byte code points.

unsafeIndexCodeUnit :: Text -> CodeUnitIndex -> CodeUnit

unsafeIndexCodeUnit' :: ByteArray -> CodeUnitIndex -> CodeUnit

Slicing Functions

unsafeCutUtf8 and unsafeSliceUtf8 are used to retrieve slices of Text values. unsafeSliceUtf8 begin length returns a substring of length length starting at begin. unsafeSliceUtf8 begin length returns a tuple of the "surrounding" substrings.

They satisfy the following property:

let (prefix, suffix) = unsafeCutUtf8 begin length t
in concat [prefix, unsafeSliceUtf8 begin length t, suffix] == t

The following diagram visualizes the relevant offsets for begin = CodeUnitIndex 2, length = CodeUnitIndex 6 and t = "BCDEFGHIJKL".

 off                 off+len
  │                     │
  ▼                     ▼
──┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬─┬──
 A│B│C│D│E│F│G│H│I│J│K│L│M│N
──┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴─┴──
      ▲           ▲
      │           │
 off+begin   off+begin+length

unsafeSliceUtf8 begin length t == "DEFGHI"
unsafeCutUtf8 begin length t == ("BC", "JKL")

The shown array is open at each end because in general, t may be a slice as well.

WARNING: As their name implies, these functions are not (necessarily) bounds-checked. Use at your own risk.

unsafeCutUtf8

Arguments

:: CodeUnitIndex

Starting position of substring.

-> CodeUnitIndex

Length of substring.

-> Text

Initial string.

-> (Text, Text)

unsafeSliceUtf8 :: CodeUnitIndex -> CodeUnitIndex -> Text -> Text

General Functions

These functions are available in text as well and should be removed once this library moves to text-2. You should be able to use these by doing import qualified Data.Text.Utf8 as Text just like you would with text.

NOTE: The Text instances for Show, Eq, Ord, IsString, FromJSON, ToJSON and Hashable in this file also fall in this category.

concat :: [Text] -> Text

TODO: Inefficient placeholder implementation.

dropWhile :: (Char -> Bool) -> Text -> Text

See dropWhile.

null :: Text -> Bool

Checks whether a text is the empty string.

readFile :: FilePath -> IO Text

See readFile. TODO: Uses readFile and loops through each byte individually. Use copyPtrToMutableByteArray here if possible.

replicate :: Int -> Text -> Text

TODO: Inefficient placeholder implementation. See replicate

indices :: Text -> Text -> [Int]

TODO: Inefficient placeholder implementation. This function implements very basic string search. It's text counterpart is indices, which implements the Boyer-Moore algorithm. Since we have this function only to check whether our own Boyer-Moore implementation works, it would not make much sense to implement it using the same algorithm. Once we can use text-2, we can compare our implementation to the official text one which presumably works.

isInfixOf :: Text -> Text -> Bool

TODO: Inefficient placeholder implementation.

pack :: String -> Text

TODO: Inefficient placeholder implementation.

unpack :: Text -> String

TODO: Inefficient placeholder implementation.