Data.Text

A time and space-efficient implementation of Unicode text using packed Word16 arrays. Suitable for performance critical use, both in terms of large data quantities and high speed.

This module is intended to be imported qualified, to avoid name clashes with Prelude functions, e.g.

 import qualified Data.Text as T

Fusion

Most of the functions in this module are subject to fusion, meaning that a pipeline of such functions will usually allocate at most one Text value.

Types

data Text

A space efficient, packed, unboxed Unicode text type.

Instances

Eq Text

Ord Text

Read Text

Show Text

Typeable Text

IsString Text

Monoid Text

Creation and elimination

pack :: String -> Text

O(n) Convert a String into a Text. Subject to fusion.

unpack :: Text -> String

O(n) Convert a Text into a String. Subject to fusion.

singleton :: Char -> Text

O(1) Convert a character into a Text. Subject to fusion.

empty :: Text

O(1) The empty Text.

Basic interface

cons :: Char -> Text -> Text

O(n) Adds a character to the front of a Text. This function is more costly than its List counterpart because it requires copying a new array. Subject to fusion.

snoc :: Text -> Char -> Text

O(n) Adds a character to the end of a Text. This copies the entire array in the process, unless fused. Subject to fusion.

append :: Text -> Text -> Text

O(n) Appends one Text to the other by copying both of them into a new Text. Subject to fusion.

uncons :: Text -> Maybe (Char, Text)

O(1) Returns the first character and rest of a Text, or Nothing if empty. Subject to fusion.

head :: Text -> Char

O(1) Returns the first character of a Text, which must be non-empty. Subject to fusion.

last :: Text -> Char

O(1) Returns the last character of a Text, which must be non-empty. Subject to fusion.

tail :: Text -> Text

O(1) Returns all characters after the head of a Text, which must be non-empty. Subject to fusion.

init :: Text -> Text

O(1) Returns all but the last character of a Text, which must be non-empty. Subject to fusion.

null :: Text -> Bool

O(1) Tests whether a Text is empty or not. Subject to fusion.

length :: Text -> Int

O(n) Returns the number of characters in a Text. Subject to fusion.

Transformations

map :: (Char -> Char) -> Text -> Text

O(n) map f t is the Text obtained by applying f to each element of t. Subject to fusion.

intercalate :: Text -> [Text] -> Text

O(n) The intercalate function takes a Text and a list of Texts and concatenates the list after interspersing the first argument between each element of the list.

intersperse :: Char -> Text -> Text

O(n) The intersperse function takes a character and places it between the characters of a Text. Subject to fusion.

transpose :: [Text] -> [Text]

O(n) The transpose function transposes the rows and columns of its Text argument. Note that this function uses pack, unpack, and the list version of transpose, and is thus not very efficient.

reverse :: Text -> Text

O(n) Reverse the characters of a string. Subject to fusion.

replace

Arguments

:: Text

Text to search for

-> Text

Replacement text

-> Text

Input text

-> Text

O(m)*O(n) Replace every occurrence of one substring with another.

Case conversion

When case converting Text values, do not use combinators like map toUpper to case convert each character of a string individually, as this gives incorrect results according to the rules of some writing systems. The whole-string case conversion functions from this module, such as toUpper, obey the correct case conversion rules. As a result, these functions may map one input character to two or three output characters. For examples, see the documentation of each function.

toCaseFold :: Text -> Text

O(n) Convert a string to folded case. This function is mainly useful for performing caseless (also known as case insensitive) string comparisons.

A string x is a caseless match for a string y if and only if:

toCaseFold x == toCaseFold y

The result string may be longer than the input string, and may differ from applying toLower to the input string. For instance, the Armenian small ligature "ﬓ" (men now, U+FB13) is case folded to the sequence "մ" (men, U+0574) followed by "ն" (now, U+0576), while the Greek "µ" (micro sign, U+00B5) is case folded to "μ" (small letter mu, U+03BC) instead of itself.

toLower :: Text -> Text

O(n) Convert a string to lower case, using simple case conversion. The result string may be longer than the input string. For instance, "İ" (Latin capital letter I with dot above, U+0130) maps to the sequence "i" (Latin small letter i, U+0069) followed by " ̇" (combining dot above, U+0307).

toUpper :: Text -> Text

O(n) Convert a string to upper case, using simple case conversion. The result string may be longer than the input string. For instance, the German "ß" (eszett, U+00DF) maps to the two-letter sequence "SS".

Justification

justifyLeft :: Int -> Char -> Text -> Text

O(n) Left-justify a string to the given length, using the specified fill character on the right. Subject to fusion. Examples:

 justifyLeft 7 'x' "foo"    == "fooxxxx"
 justifyLeft 3 'x' "foobar" == "foobar"

justifyRight :: Int -> Char -> Text -> Text

O(n) Right-justify a string to the given length, using the specified fill character on the left. Examples:

 justifyRight 7 'x' "bar"    == "xxxxbar"
 justifyRight 3 'x' "foobar" == "foobar"

center :: Int -> Char -> Text -> Text

O(n) Center a string to the given length, using the specified fill character on either side. Examples:

 center 8 'x' "HS" = "xxxHSxxx"

Folds

foldl :: (b -> Char -> b) -> b -> Text -> b

O(n) foldl, applied to a binary operator, a starting value (typically the left-identity of the operator), and a Text, reduces the Text using the binary operator, from left to right. Subject to fusion.

foldl' :: (b -> Char -> b) -> b -> Text -> b

O(n) A strict version of foldl. Subject to fusion.

foldl1 :: (Char -> Char -> Char) -> Text -> Char

O(n) A variant of foldl that has no starting value argument, and thus must be applied to a non-empty Text. Subject to fusion.

foldl1' :: (Char -> Char -> Char) -> Text -> Char

O(n) A strict version of foldl1. Subject to fusion.

foldr :: (Char -> b -> b) -> b -> Text -> b

O(n) foldr, applied to a binary operator, a starting value (typically the right-identity of the operator), and a Text, reduces the Text using the binary operator, from right to left. Subject to fusion.

foldr1 :: (Char -> Char -> Char) -> Text -> Char

O(n) A variant of foldr that has no starting value argument, and thust must be applied to a non-empty Text. Subject to fusion.

Special folds

concat :: [Text] -> Text

O(n) Concatenate a list of Texts. Subject to fusion.

concatMap :: (Char -> Text) -> Text -> Text

O(n) Map a function over a Text that results in a Text, and concatenate the results. Subject to fusion.

Note: if in concatMap f t, f is defined in terms of fusible functions, it will also be fusible.

any :: (Char -> Bool) -> Text -> Bool

O(n) any p t determines whether any character in the Text t satisifes the predicate p. Subject to fusion.

all :: (Char -> Bool) -> Text -> Bool

O(n) all p t determines whether all characters in the Text t satisify the predicate p. Subject to fusion.

maximum :: Text -> Char

O(n) maximum returns the maximum value from a Text, which must be non-empty. Subject to fusion.

minimum :: Text -> Char

O(n) minimum returns the minimum value from a Text, which must be non-empty. Subject to fusion.

Construction

Scans

scanl :: (Char -> Char -> Char) -> Char -> Text -> Text

O(n) scanl is similar to foldl, but returns a list of successive reduced values from the left. Subject to fusion.

 scanl f z [x1, x2, ...] == [z, z `f` x1, (z `f` x1) `f` x2, ...]

Note that

 last (scanl f z xs) == foldl f z xs.

scanl1 :: (Char -> Char -> Char) -> Text -> Text

O(n) scanl1 is a variant of scanl that has no starting value argument. Subject to fusion.

 scanl1 f [x1, x2, ...] == [x1, x1 `f` x2, ...]

scanr :: (Char -> Char -> Char) -> Char -> Text -> Text

O(n) scanr is the right-to-left dual of scanl.

 scanr f v == reverse . scanl (flip f) v . reverse

scanr1 :: (Char -> Char -> Char) -> Text -> Text

O(n) scanr1 is a variant of scanr that has no starting value argument. Subject to fusion.

Accumulating maps

mapAccumL :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text)

O(n) Like a combination of map and foldl. Applies a function to each element of a Text, passing an accumulating parameter from left to right, and returns a final Text.

mapAccumR :: (a -> Char -> (a, Char)) -> a -> Text -> (a, Text)

The mapAccumR function behaves like a combination of map and foldr; it applies a function to each element of a Text, passing an accumulating parameter from right to left, and returning a final value of this accumulator together with the new Text.

Generation and unfolding

replicate :: Int -> Text -> Text

O(n*m) replicate n t is a Text consisting of the input t repeated n times. Subject to fusion.

replicateChar :: Int -> Char -> Text

O(n) replicateChar n c is a Text of length n with c the value of every element. Subject to fusion.

unfoldr :: (a -> Maybe (Char, a)) -> a -> Text

O(n), where n is the length of the result. The unfoldr function is analogous to the List unfoldr. unfoldr builds a Text from a seed value. The function takes the element and returns Nothing if it is done producing the Text, otherwise Just (a,b). In this case, a is the next Char in the string, and b is the seed value for further production. Subject to fusion.

unfoldrN :: Int -> (a -> Maybe (Char, a)) -> a -> Text

O(n) Like unfoldr, unfoldrN builds a Text from a seed value. However, the length of the result should be limited by the first argument to unfoldrN. This function is more efficient than unfoldr when the maximum length of the result is known and correct, otherwise its performance is similar to unfoldr. Subject to fusion.

Substrings

Breaking strings

take :: Int -> Text -> Text

O(n) take n, applied to a Text, returns the prefix of the Text of length n, or the Text itself if n is greater than the length of the Text. Subject to fusion.

drop :: Int -> Text -> Text

O(n) drop n, applied to a Text, returns the suffix of the Text of length n, or the empty Text if n is greater than the length of the Text. Subject to fusion.

takeWhile :: (Char -> Bool) -> Text -> Text

O(n) takeWhile, applied to a predicate p and a Text, returns the longest prefix (possibly empty) of elements that satisfy p. Subject to fusion.

dropWhile :: (Char -> Bool) -> Text -> Text

O(n) dropWhile p t returns the suffix remaining after takeWhile p t. Subject to fusion.

dropWhileEnd :: (Char -> Bool) -> Text -> Text

O(n) dropWhileEnd p t returns the prefix remaining after dropping characters that fail the predicate p from the end of t. Subject to fusion. Examples:

 dropWhileEnd (=='.') "foo..." == "foo"

dropAround :: (Char -> Bool) -> Text -> Text

O(n) dropAround p t returns the substring remaining after dropping characters that fail the predicate p from both the beginning and end of t. Subject to fusion.

strip :: Text -> Text

O(n) Remove leading and trailing white space from a string. Equivalent to:

 dropAround isSpace

stripStart :: Text -> Text

O(n) Remove leading white space from a string. Equivalent to:

 dropWhile isSpace

stripEnd :: Text -> Text

O(n) Remove trailing white space from a string. Equivalent to:

 dropWhileEnd isSpace

splitAt :: Int -> Text -> (Text, Text)

O(n) splitAt n t returns a pair whose first element is a prefix of t of length n, and whose second is the remainder of the string. It is equivalent to (take n t, drop n t).

span :: (Char -> Bool) -> Text -> (Text, Text)

O(n) span, applied to a predicate p and text t, returns a pair whose first element is the longest prefix (possibly empty) of t of elements that satisfy p, and whose second is the remainder of the list.

break :: (Char -> Bool) -> Text -> (Text, Text)

O(n) break is like span, but the prefix returned is over elements that fail the predicate p.

group :: Text -> [Text]

O(n) Group characters in a string by equality.

groupBy :: (Char -> Char -> Bool) -> Text -> [Text]

O(n) Group characters in a string according to a predicate.

inits :: Text -> [Text]

O(n) Return all initial segments of the given Text, shortest first.

tails :: Text -> [Text]

O(n) Return all final segments of the given Text, longest first.

Breaking into many substrings

Splitting functions in this library do not perform character-wise copies to create substrings; they just construct new Texts that are slices of the original.

split

Arguments

:: Text

Text to split on

-> Text

Input text

-> [Text]

O(m)*O(n) Break a Text into pieces separated by the first Text argument, consuming the delimiter. An empty delimiter is invalid, and will cause an error to be raised.

Examples:

 split "\r\n" "a\r\nb\r\nd\r\ne" == ["a","b","d","e"]
 split "aaa"  "aaaXaaaXaaaXaaa"  == ["","X","X","X",""]
 split "x"    "x"                == ["",""]

and

 intercalate s . split s         == id
 split (singleton c)             == splitWith (==c)

splitTimes

Arguments

:: Int

Maximum number of times to split

-> Text

Text to split on

-> Text

Input text

-> [Text]

O(m)*O(n) Break a Text into pieces at most k times, treating the first Text argument as the delimiter to break on, and consuming the delimiter. The last element of the list contains the remaining text after the number of times to split has been reached. A value of zero or less for k causes no splitting to occur. An empty delimiter is invalid, and will cause an error to be raised.

Examples:

 splitTimes 0   "//"  "a//b//c"   == ["a//b//c"]
 splitTimes 2   ":"   "a:b:c:d:e" == ["a","b","c:d:e"]
 splitTimes 100 "???" "a????b"    == ["a","?b"]

and

 intercalate s . splitTimes k s   == id

splitTimesEnd

Arguments

:: Int

Maximum number of times to split

-> Text

Text to split on

-> Text

Input text

-> [Text]

O(m)*O(n) Break a Text into pieces at most k times, like splitTimes, but start from the end of the input and work towards the start.

Examples:

 splitTimes 2    "::" "a::b::c::d::e" == ["a","b","c::d::e"]
 splitTimesEnd 2 "::" "a::b::c::d::e" == ["a::b::c","d","e"]

splitWith :: (Char -> Bool) -> Text -> [Text]

O(n) Splits a Text into components delimited by separators, where the predicate returns True for a separator element. The resulting components do not contain the separators. Two adjacent separators result in an empty component in the output. eg.

 splitWith (=='a') "aabbaca" == ["","","bb","c",""]
 splitWith (=='a') ""        == [""]

chunksOf :: Int -> Text -> [Text]

O(n) Splits a Text into components of length k. The last element may be shorter than the other chunks, depending on the length of the input. Examples:

 chunksOf 3 "foobarbaz"   == ["foo","bar","baz"]
 chunksOf 4 "haskell.org" == ["hask","ell.","org"]

Breaking into lines and words

lines :: Text -> [Text]

O(n) Breaks a Text up into a list of Texts at newline Chars. The resulting strings do not contain newlines.

words :: Text -> [Text]

O(n) Breaks a Text up into a list of words, delimited by Chars representing white space.

unlines :: [Text] -> Text

O(n) Portably breaks a Text up into a list of Texts at line boundaries.

A line boundary is considered to be either a line feed, a carriage return immediately followed by a line feed, or a carriage return. This accounts for both Unix and Windows line ending conventions, and for the old convention used on Mac OS 9 and earlier.

O(n) Joins lines, after appending a terminating newline to each.

unwords :: [Text] -> Text

O(n) Joins words using single space characters.

Predicates

isPrefixOf :: Text -> Text -> Bool

O(n) The isPrefixOf function takes two Texts and returns True iff the first is a prefix of the second. This function is subject to fusion.

isSuffixOf :: Text -> Text -> Bool

O(n) The isSuffixOf function takes two Texts and returns True iff the first is a suffix of the second.

isInfixOf :: Text -> Text -> Bool

O(n) The isInfixOf function takes two Texts and returns True iff the first is contained, wholly and intact, anywhere within the second.

Searching

elem :: Char -> Text -> Bool

O(n) elem is the Text membership predicate.

filter :: (Char -> Bool) -> Text -> Text

O(n) filter, applied to a predicate and a Text, returns a Text containing those characters that satisfy the predicate.

find :: (Char -> Bool) -> Text -> Maybe Char

O(n) The find function takes a predicate and a Text, and returns the first element in matching the predicate, or Nothing if there is no such element.

partition :: (Char -> Bool) -> Text -> (Text, Text)

O(n) The partition function takes a predicate and a Text, and returns the pair of Texts with elements which do and do not satisfy the predicate, respectively; i.e.

 partition p t == (filter p t, filter (not . p) t)

Indexing

If you think of a Text value as an array of Char values (which it is not), you run the risk of writing inefficient code.

An idiom that is common in some languages is to find the numeric offset of a character or substring, then use that number to split or trim the searched string. With a Text value, this approach would require two O(n) operations: one to perform the search, and one to operate from wherever the search ended.

For example, suppose you have a string that you want to split on the substring "::", such as "foo::bar::quux". Instead of searching for the index of "::" and taking the substrings before and after that index, you would instead use splitTimes 1 ::.

index :: Text -> Int -> Char

O(n) Text index (subscript) operator, starting from 0.

findIndex :: (Char -> Bool) -> Text -> Maybe Int

O(n) The findIndex function takes a predicate and a Text and returns the index of the first element in the Text satisfying the predicate. Subject to fusion.

findIndices :: (Char -> Bool) -> Text -> [Int]

The findIndices function extends findIndex, by returning the indices of all elements satisfying the predicate, in ascending order. Subject to fusion.

elemIndex :: Char -> Text -> Maybe Int

O(n) The elemIndex function returns the index of the first element in the given Text which is equal to the query element, or Nothing if there is no such element. Subject to fusion.

elemIndices :: Char -> Text -> [Int]

O(n) The elemIndices function returns the index of every element in the given Text which is equal to the query element. Subject to fusion.

count :: Text -> Text -> Int

O(n*m) The count function returns the number of times the query string appears in the given Text. An empty query string is invalid, and will cause an error to be raised.

Zipping and unzipping

zip :: Text -> Text -> [(Char, Char)]

O(n) zip takes two Texts and returns a list of corresponding pairs of bytes. If one input Text is short, excess elements of the longer Text are discarded. This is equivalent to a pair of unpack operations.

zipWith :: (Char -> Char -> Char) -> Text -> Text -> Text

O(n) zipWith generalises zip by zipping with the function given as the first argument, instead of a tupling function.