h* 7     1.0.0 Safe-Inferred} Safe-Inferred "#%&'-<> tiktokenRank of this chunktiktoken/Rank of this chunk combined with the next chunktiktoken?This is basically the same thing as `Maybe Int` except with an  instance that treats  values as less than  valuestiktokenThis is an efficient internal representation of an encoding like  cl100k_base,  p50k_edit, or  o200k_basetiktoken Create an 5 from regular expression and an ordered set of tokenstiktokenParse an encoding from the  .tiktoken file formattiktokenAdd special tokens to a base tiktoken r50k_base tiktoken p50k_base tiktoken p50k_edit tiktoken cl100k_base tiktoken o200k_base tiktokenThis corresponds to the _byte_pair_merge function in the upstream tiktoken package: https://github.com/openai/tiktoken/blob/c0ba74c238d18b4824c25f3c27fc8698055b9a76/src/lib.rs#L18-L74&The intermediate data structure is an   instead of a ! but other than that the algorithm is essentially identical."tiktokenSplit a # into smaller #s, each of which are successive longest possible matches to the provided regular expression$tiktokenDivide up the input into coarse-grained chunks based on the provided splitting regular expression before doing the final byte pair encoding%tiktokenSplit a # into smaller #&s separated by the given separatortiktoken%Tokenizer that is special-token-aware tiktokenUse an  to tokenize a # into smaller #s"This only fails if you provide an 3 that cannot rank all possible 1-byte sequences tiktokenUse an  to tokenize a # into ranks"This only fails if you provide an 3 that cannot rank all possible 1-byte sequences tiktokenCombine a sequence of # tokens back into a #This is just a synonym for Data.ByteString.& (no 9 necessary), provided solely for consistency/convenience. tiktoken(Convert a sequence of ranks back into a #This will fail if you supply any ranks which are not recognized by the .tiktokenRegular expression used for coarse-grained splitting of the inputtiktoken1The tokens in sequential order of their token IDstiktokenRegular expression used for coarse-grained splitting of the inputtiktokenThe contents of the  .tiktoken file"tiktokenRegex to matchtiktokenBytes to split into chunks%tiktoken Separatortiktoken# to separate     '      !"#$%&'()*+,-./01-234$tiktoken-1.0.0-RVgUPpRfQrK6y1UQpyMoDTiktokentiktokenPaths_tiktokenEncodingtiktokenToEncodingaddSpecialTokens r50k_base p50k_base p50k_edit cl100k_base o200k_basetoTokensAndRankstoTokenstoRanks fromTokens fromRanks $fEqRanked $fOrdRanked$fGenericEncoding$fNFDataEncodingversion getBinDir getLibDir getDynLibDir getDataDir getLibexecDirgetDataFileName getSysconfDir$sel:rank:Chunk$sel:rank2:ChunkRankedghc-prim GHC.ClassesOrdUnrankedtokensToEncodingbytePairEncodecontainers-0.6.7Data.IntMap.InternalIntMap&vector-0.13.1.0-3iwp0ZpVFXd2eoMdapVrKn Data.VectorVectorsplitUsingRegex*bytestring-0.12.1.0-7mxHsd9tX2kI40YX6YZhgSData.ByteString.Internal.Type ByteStringbytePairEncodeWithSplittingsplitOnSeparatorData.ByteStringconcat