h*      1.0.2 Safe-Inferred} Safe-Inferred "#%&'-<> tiktokenRank of this chunktiktoken/Rank of this chunk combined with the next chunktiktoken?This is basically the same thing as `Maybe Int` except with an  instance that treats  values as less than  valuestiktokenThis is an efficient internal representation of an encoding like  cl100k_base,  p50k_edit, or  o200k_basetiktoken Create an 5 from regular expression and an ordered set of tokenstiktokenParse an encoding from the  .tiktoken file formattiktokenAdd special tokens to a base tiktoken r50k_base tiktoken p50k_base tiktoken p50k_edit tiktoken cl100k_base tiktoken o200k_base tiktokenThis corresponds to the _byte_pair_merge function in the upstream tiktoken package: https://github.com/openai/tiktoken/blob/c0ba74c238d18b4824c25f3c27fc8698055b9a76/src/lib.rs#L18-L74&The intermediate data structure is an   instead of a  but other than that the algorithm is essentially identical.!tiktokenSplit a " into smaller "s, each of which are successive longest possible matches to the provided regular expression#tiktokenDivide up the input into coarse-grained chunks based on the provided splitting regular expression before doing the final byte pair encoding$tiktokenSplit a " into smaller "&s separated by the given separatortiktokenUse an  to tokenize a " into smaller " s and their associated ranks"This only fails if you provide an 3 that cannot rank all possible 1-byte sequences tiktokenUse an  to tokenize a " into smaller "s"This only fails if you provide an 3 that cannot rank all possible 1-byte sequences tiktokenUse an  to tokenize a " into ranks"This only fails if you provide an 3 that cannot rank all possible 1-byte sequences tiktokenCombine a sequence of " tokens back into a "This is just a synonym for Data.ByteString.% (no 9 necessary), provided solely for consistency/convenience. tiktoken(Convert a sequence of ranks back into a "This will fail if you supply any ranks which are not recognized by the .tiktokenRegular expression used for coarse-grained splitting of the inputtiktoken1The tokens in sequential order of their token IDstiktokenRegular expression used for coarse-grained splitting of the inputtiktokenThe contents of the  .tiktoken file!tiktokenRegex to matchtiktokenBytes to split into chunks$tiktoken Separatortiktoken" to separate     &      !"#$%&'()*+,-./0,123%tiktoken-1.0.2-8bP3kw6JoV7IEn3Wtil93uTiktokentiktokenPaths_tiktoken Data.VectorVectorEncodingtiktokenToEncodingaddSpecialTokens r50k_base p50k_base p50k_edit cl100k_base o200k_basetoTokensAndRankstoTokenstoRanks fromTokens fromRanks $fEqRanked $fOrdRanked$fGenericEncoding$fNFDataEncodingversion getBinDir getLibDir getDynLibDir getDataDir getLibexecDirgetDataFileName getSysconfDir$sel:rank:Chunk$sel:rank2:ChunkRankedghc-prim GHC.ClassesOrdUnrankedtokensToEncodingbytePairEncodecontainers-0.6.7Data.IntMap.InternalIntMapsplitUsingRegex*bytestring-0.12.1.0-7mxHsd9tX2kI40YX6YZhgSData.ByteString.Internal.Type ByteStringbytePairEncodeWithSplittingsplitOnSeparatorData.ByteStringconcat