$       !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~  Safe-Inferred @The IsToken class captures the different token type in use. For ( GHC it represents the type returned by   , ( namely [(GHC.Located GHC.Token, String)] + For haskell-src-exts this is the reult of lexTokenStream , namely `[HSE.Loc HSE.Token]` @tokenLen returns the length of the string representation of the C token, not just the difference in the location, as the string may < have changed without the position being updated, e.g. in a  renaming 0Zero-length tokens, as appear in GHC as markers ACreate a stream of tokens from source, with first token start at  given location >Mark a token so that it can be use to trigger layout checking # later when the toks are retrieved  Construct a , from a Haskell AST and a stream of tokens. "2Match a SrcSpan, using a ForestLine as the marker &The length of the  span may have  changed due to  updated tokens. 0%Initial offset from token before the / stacked list of items, the (r,c) of the first A non-comment token, the (r,c) of the end of the last non-comment 7 token in the stacked list to be able to calculate the 6 (RowOffset,ColOffset) between the last token and the  start of the next item. 79An entry in the data structure for a particular srcspan. 8 Deleted has & * the source span has been deleted  * prior gap in lines + * the gap between this span end and the , start of the next in the fringe of the  tree. 9 Entry has , * the source span contained in this Node  * how the sub-elements nest 4 * the tokens for the SrcSpan if subtree is empty :>Identifies the tree carrying the main tokens, not any work in  progress or deleted ones >?Tokens that are ignored when determining the first non-comment  token in a span B.Extract an encoded ForestLine from a GHC line N  !"#$%&'()*+,-./0123456789:;<=>?@ABC   D  !"#$%&'()*+,-./0123456789:;<=>?@ABCD798 !:"#$%&'()654321.0/*-,+BCA ;<=>?@'   !"#$%&'()*-,+.0/123456798:;<=>?@ABC    Safe-InferredD?Split the token stream into three parts: the tokens before the B startPos, the tokens between startPos and endPos, and the tokens  after endPos. F Note: The startPos and endPos refer to the startPos of a token only. @ So a single token will have the same startPos and endPos  NO^^^^ LCSplit the given tokens into the ones that occur prior to the start ' of the list and ones that occur after MFSplit the given tokens to include the comments belonging to the span. N Get the start&3end location of t in the token stream, then extend A the start and end location to cover the preceding and following  comments. OKSplit a set of comment tokens into the ones that belong with the startLine ( and those that belong with the endLine YGets the version numbers ZGets the AST tree numbers [!Gets the SpanLengthChanged flags \0Checks if the version is zero in both positions ]"Checks if the version is non-zero ^&Checks if the AST version is non-zero _Checks if the version is zero b Puts a TreeId into a forestSpan c@Does the first span contain the second? Takes cognisance of the & various flags a ForestSpan can have. > NOTE: This function relies on the Eq instance for ForestLine  BExtract the start and end position of a span, without any leading  or trailing comments gBExtract the start and end position of a span, without any leading  or trailing comments h*ForestSpan version of GHC combineSrcSpans mStrip out the version markers o5Checks if the version is non-zero in either position p)Get the start and end position of a Tree 1 treeStartEnd :: Tree Entry -> (SimpPos,SimpPos) N treeStartEnd (Node (Entry sspan _) _) = (getGhcLoc sspan,getGhcLocEnd sspan) {&Neat 2-dimensional drawing of a tree. |(Neat 2-dimensional drawing of a forest. -Call drawTreeEntry on the entire token cache -Call drawTreeEntry on the entire token cache :Add a constant line and column offset to a span of tokens *Shift the whole token by the given offset NDEFGHIJKLMNOPQRS TUVWXYZ[\]^_`abcdef ghijklmnopqrstuvwxyz{|}~BDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~BDEFGHIJKXfMQRSWVLPyxNijmnzYZ[\]^_`abcdeklopqrstuvwhgTU{|}~ONDEFGHIJKLMNOPQRS TUVWXYZ[\]^_`abcdef ghijklmnopqrstuvwxyz{|}~ Safe-Inferred4CHow new SrcSpans should be inserted in the Token tree, relative to  the prior span Line and Col offset for $ start, num lines to add at the end 0 relative to the indent level of the prior line Line and Col offset for $ start, num lines to add at the end 0 relative to the indent level of the prior span Line offset and  absolute Col. Mainly  for forcing start at  left margin, number  of lines to add at  the end Start at the specified  line and col  Only a single space between the ' end of the prior span and the new one ;Keep track of when tokens are reversed, to avoid confusion  Initialise a ' from tokens only. Does not generate a & layout-aware tree due to missing AST "The primary data structure is the . This holds the  evolving forest of modified s. Each concrete 8 implementation should provide a function to generate a  # from its specific AST and tokens. 4Make a tree representing a particular set of tokens 4Make a tree representing a particular set of tokens OUtility function to either return True or throw an error to report the problem KCheck the invariant for the token cache. Returns list of any errors found.  Invariants: O 1. For each tree, either the rootLabel has a SrcSpan only, or the subForest /= []. [ 2a. The trees making up the subForest of a given node fully include the parent SrcSpan. D i.e. the leaves contain all the tokens for a given SrcSpan. ) 2b. The subForest is in SrcSpan order W 3. A given SrcSpan can only appear (or be included) in a single tree of the forest. X 4. The parent link for all sub-trees does exist, and actually points to the parent. 6 5. There are no nullForestSpan entries in the tree O NOTE: the tokens may extend before or after the SrcSpan, due to comments only I NOTE2: this will have to be revisited when edits to the tokens are made "Represent a tree in a more concise/ pretty way =Replace a single token in a token tree, without changing the  structure of the tree D NOTE: the GHC.SrcSpan may have been used to select the appropriate B forest in the first place, and is required to select the correct I span in the tree, due to the ForestLine annotations that may be present 5Removes a ForestSpan and its tokens from the forest. CAdd a new SrcSpan and Tokens after a given one in the token stream B and forest. This will be given a unique SrcSpan in return, which ' specifically indexes into the forest. CAdd new tokens after the given SrcSpan, constructing a new SrcSpan  in the process BFor a span about to be deleted, calculate the gap between the end E of the span being deleted and the start of the next one, at a token  level. BFor a span about to be deleted, calculate the gap between the end E of the span being deleted and the start of the next one, at a token  level. BReplace the tokens for a given SrcSpan with new ones. The SrcSpan < will be inserted into the tree if it is not already there. A If the SrcSpan changes size, replace the SrcSpan with a new one 1 (marked), and return it, as well as the old one : TODO: What about trailing comments? Preserve or replace? !Get the tokens preceding a given SrcSpan >Get the (possible cached) tokens for a given source span, and  cache their being fetched. C NOTE: The SrcSpan may be one introduced by HaRe, rather than GHC. >Get the (possible cached) tokens for a given source span, and  cache their being fetched. C NOTE: The SrcSpan may be one introduced by HaRe, rather than GHC. CStarting from a point in the zipper, retrieve all tokens backwards ) until the line changes for a non-comment/non-empty token or  beginning of file. ;Place the new tokens so that they are positioned correctly ! relative to the previous tokens Aindent the tree and tokens by the given offset, and sync the AST  to the tree too. =Open a zipper so that its focus has the given SrcSpan in its D subtree, or the location where the SrcSpan should go, if it is not  in the tree.  In the case of an 0% layout with the same SrcSpan below,  return that instead ERetrieve a path to the tree containing a ForestSpan from the forest, % inserting it if not already present ERetrieve a path to the tree containing a ForestSpan from the forest, & inserting it if not already present. > In the case where there is a nested series of spans as in an  0 layout, return the deepest one AInsert a ForestSpan into the forest, if it is not there already. A Assumes the forest was populated with the tokens containing the  ForestSpan already DTrue if the start of the second param lies in the span of the first ATrue if the start of the second param lies before the first, and  ends after or on the second BTrue if the end of the second param lies in the span of the first CSplit a given tree into a possibly empty part that lies before the B srcspan, the part that is wholly included in the srcspan and the ) part the lies outside of it at the end. =Open a zipper so that its focus has the given SrcSpan in its D subtree, or the location where the SrcSpan should go, if it is not  in the tree =Open a zipper so that its focus has the given SrcSpan in its D subtree, or the location where the SrcSpan should go, if it is not  in the tree @Open a zipper to a SrcSpan that has been added in the tree, and E thus does not necessarily fall in the logical hierarchy of the tree 7Insert a new node after the designated one in the tree 2Open a zipper so that its focus is the given node - NOTE: the node must already be in the tree 2Open a zipper so that its focus is the given node - NOTE: the node must already be in the tree CReplace a single token in the token stream by a new token, without  adjusting the layout. C Note1: does not re-align, else other later replacements may fail. E Note2: must keep original end col, to know what the inter-token gap  was when re-aligning ETransfer the location information from the first param to the second @Retrieve all the tokens at the leaves of the tree, in order. No D adjustments are made to address gaps or re-alignment of the tokens Merge adjacent Deleted entries  >sort out line numbering so that they are always monotonically  increasing. @Some tokens are marked if they belong to identifiers which have C been renamed. When the renaming takes place, no layout adjustment E is done. This function adjusts the spacing for the rest of the line B to match as far as possible the original spacing, except for the  name change. !FReplace any ForestLine flags already in a SrcSpan with the given ones Strip out the version markers ?Get the indent of the line before, taking into account in-line  'where', 'let', 'in' and 'do' tokens =Convert a string into a set of Haskell tokens, following the E given position, with each line indented by a given column offset if  required  TODO: replace 'colOffset withFirstLineIndent' with a Maybe Int ++AZ++ >Convert a string into a set of Haskell tokens. It has default I position and offset, since it will be stitched into place in TokenUtils `"Preceding location for new tokens New tokens to be added ,updated TokenCache and SrcSpan location for ! the new tokens in the TokenTree "#$%&'Updated forest, removed span The forest to update "The new span comes after this one Existing span for the tokens ,The new tokens belonging to the new SrcSpan *Unique SrcSpan allocated in the forest to $ identify this span in its position TokenTree to be modified "Preceding location for new tokens New tokens to be added +updated TokenTree and SrcSpan location for ! the new tokens in the TokenTree The AST (or fragment) Existing token tree $(signed) number of columns to indent/dedent Updated AST and tokens ()*+, -./0!12GNiGNiZ"#$%&'()*+, -./0!12NoneLoad a file using 'haskell-src-exts'$ to provide both the AST and tokens Load a file using 'haskell-src-exts' to provide both the AST and ? tokens, using a custom loading mode allowing extensions to be  enabled 3@Return the length of the token, using the string representation < where possible as it may have changed during a refactoing. 4)Bottom-up synthesis of a data structure; < 1st argument z is the initial element for the synthesis; = 2nd argument o is for reduction of results from subterms; K 3rd argument f updates the synthesised data according to the given term 5,This instance is the purpose of this module 67839:;<=>4?5@AB67839:;<=>4?5@AB Safe-InferredNone(The main data structure for this module 5This line needs to be grouped with the next in terms 8 of layout, so any column offsets need to be propagated CJWe need this otherwise the SemiGroup instance for SimpSpan cause problems <The value that bubbles up. This is the Span occupied by the D subtree, together with a string representation of the subtree. The 0 origin of the string is the start of the span. D Combine the U1 annotations as they propagate from the leafs to = be cached at the root of the tree. This is the heart of the  DualTree functionality 2CEFGHIDJKLMNOPQR"" CEFGHIDJKLMNOPQRNoneSTUVWXYZ[\]^_`abcdefSTUVWXYZ[\]^_`abcdefNonegConstruct the initial  for use by  h'gets the (row,col) of the start of the  GHC.SrcSpan , or (-1,-1)  if there is an GHC.UnhelpfulSpan i%gets the (row,col) of the end of the  GHC.SrcSpan , or (-1,-1)  if there is an GHC.UnhelpfulSpan jShow a GHC API structure /Compose a new token using the given arguments. kAUsed as a marker in the filename part of the SrcSpan on modified = tokens, to trigger re-alignment when retrieving the tokens. l>Mark a token so that it can be use to trigger layout checking # later when the toks are retrieved m&Does a token have the file mark in it nAGet the current length of the token, based on its String, rather  than the original span length. FReplace any ForestLine flags already in a SrcSpan with the given ones H TODO ++AZ++ : should not be required, convert to SimSpan then use that Create a new name token. If useQual then use the qualified  name, if it exists. B The end position is not changed, so the eventual realignment can 4 know what the difference in length in the token is o)Bottom-up synthesis of a data structure; < 1st argument z is the initial element for the synthesis; = 2nd argument o is for reduction of results from subterms; K 3rd argument f updates the synthesised data according to the given term pKChecks whether the current item is undesirable for analysis in the current  AST Stage. qCheck the Typeable1 items r:sanitize the GHC ParsedSource by purging it of undefineds ) These can occur in (at least one place)  HsCmdTop - last field sBottom-up transformation t%Top-down version of everywhereStaged u#Staged variation of SYB.everything E The stage must be provided to avoid trying to modify elements which 5 may not be present at all stages of AST processing.  Note: Top-down order v@Traverse the parsed source looking for points requiring layout, > and insert them into the LayoutTree at the appropriate point w3Extract the layout-sensitive parts of the GHC AST. gxyz{|}~hijklmnopqrstuvw  gxyz{|}~hijklmnopqrstuvw None     !"#$%&'()*+,-./00123456789:;<=>?@ABCBDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstuvwxyz{|}~      !"#$%&'()*+,-./0123456789:;<=>?@ABCDEFGHIJKLMNOPQRSTUVWXYZ[\]^_`abcdefghijklmnopqrstu;vwxyz{|b}~FGh_`ahaskell-token-utils-0.0.0.6!Language.Haskell.TokenUtils.Types!Language.Haskell.TokenUtils.Utils&Language.Haskell.TokenUtils.TokenUtils&Language.Haskell.TokenUtils.HSE.Layout"Language.Haskell.TokenUtils.Layout$Language.Haskell.TokenUtils.DualTree"Language.Haskell.TokenUtils.Pretty&Language.Haskell.TokenUtils.GHC.LayoutGHCgetRichTokenStreamLanguage.Haskell.TokenUtils.API Outputableppr LayoutTree TokenLayoutHasLocgetLoc getLocEndgetSpanputSpanIsTokentokenLen isCommentisEmpty mkZeroTokenisDoisElseisInisLetisOfisThenisWhere tokenToStringshowTokenStreamlexStringToTokens markTokenisMarked Allocatable allocTokens TokenCacheTKtkCache tkLastTreeIdTreeIdTId ForestSpan ForestPos ForestLineflSpanLengthChangedflTreeSelectorflInsertVersionflLine EndOffset FromAlignColSameLineNoneLayoutNoChangeAboveSimpSpanSimpPosColRow ColOffset RowOffsetEntryDeletedmainTid notWhiteSpaceisWhiteSpaceOrIgnored isIgnoredisIgnoredNonComment isWhereOrLetshowFriendlyToksforestLenChangedMaskghcLineToForestLineforestLineToGhcLine splitToksgheadglastgtail gfromJust addEndOffsetscalcLastTokenPos makeOffsetsplitToksForListsplitToksIncCommentsstartEndLocIncComments'divideComments placeAbove makeGroupmakeGroupLayoutmakeSpanFromTrees getStartLoc getEndLoc subTreeOnlymkGroupmakeLeafforestSpanVersionsforestSpanAstVersionsforestSpanLenChangedFlagsforestSpanVersionNotSetforestPosVersionSetforestPosAstVersionSetforestPosVersionNotSetforestSpanLenChangedforestPosLenChangedtreeIdIntoForestSpan spanContainsinsertVersionsInForestSpaninsertLenChangedInForestSpanmakeLeafFromToksnonCommentSpan combineSpanssimpPosToForestSpanss2fforestSpanFromEntryputForestSpanInEntryforestSpanToSimpPosf2ssforestSpanVersionSet treeStartEndgroupTokensByLinetokenRowtokenCol tokenColEndtokenPos tokenPosEndsrcPosToSimpPosstrip allocListtreeIdFromForestSpan drawTreeEntrydrawForestEntry showLayoutdrawTreeCompactshowForestSpandrawTreeWithToksdrawTokenCachedrawTokenCacheDetailedaddOffsetToToksdecorateincreaseSrcSpan Positioning PlaceIndent PlaceOffset PlaceAbsCol PlaceAbsolute PlaceAdjacent ReversedToksRTinitTokenCacheinitTokenCacheLayoutputToksInCachereplaceTokenInCacheremoveToksFromCachegetTokensFromCachegetTokensNoIntrosFromCachegetTokensBeforeFromCacheaddTokensAfterSpanInCache reverseToks unReverseToks reversedToksmkTreeFromTokensmkTreeFromSpanTokensnullForestSpan nullForestPos invariantshowTreeshowToksgetTreeFromCachereplaceTreeInCachereplaceTokenForSrcSpan removeSrcSpanaddNewSrcSpanAndToksAfterplaceToksForSpan limitPrevToksaddToksAfterSrcSpan calcEndGapupdateTokensForSrcSpangetTokensBefore getTokensForgetTokensForNoIntrosretrievePrevLineToks reIndentToksindentDeclToksaddOffsetToSpan getSrcSpanFor insertSrcSpan splitSubToks containsStartcontainsMiddle containsEnd splitSubtreeopenZipperToSpanopenZipperToSpanOrigopenZipperToSpanAddedgetTreeSpansAsListinsertNodeAfterreplaceTokNoReAlign matchTokenPosretrieveTokensInterim reAlignMarkedreAlignOneLineinsertVersionsInSrcSpaninsertLenChangedInSrcSpanforestSpanToGhcPosgetIndentOffset splitOnNewLn newLnTokentokenise basicTokeniseTuTokenCTloadFileloadFileWithModetemplateHaskellModess2ss2ssretrieveTokens SourceTreePrimPDeletedPToksAnnotASubtreeADeletedAnnLineOptOGroupONoneSource SWasAddedSAdded SOriginal Alignment AVerticalANoneLineUpUDeletedTransformationTAbove DeletedSpanrenderLayoutTreerenderLinesFromLayoutTreeretrieveLinesFromLayoutTree retrieveLinesrenderSourceTree renderLineslayoutTreeToSourceTreeshowPpr GhcPosTokenmkToken nullSrcSpangs2ff2gsgs2ssss2gsinsertForestLineInSrcSpan showSrcSpan showSrcSpanF newNameTokTL isWhiteSpaceforestLineMaskforestVersionMaskforestTreeMaskforestVersionShiftforestTreeShift$fOrdForestLine$fShowForestLine$fEqForestLinenonCommentSpanLayoutnullSpan getTreeLoctoksOnSameLine drawEntrydrawTreeCompact'drawTreeWithToks' showLevel $fHasLoc(,) $fHasLocEntry $fEqEntry $fOrdTree invariantOk prettyshowcalcPriorAndEndGapopenZipperToSpanDeepgetSrcSpanForDeepopenZipperToNodeopenZipperToNodeDeep mergeDeletesmonotonicLineToksinsertForestLineInSpanforestSpanStart forestSpanEnd prettyToks putTidInTreetreeIdIntoTreestash doSplitTreemkTreeListFromTokensgetTreeSpansAsList'getChildrenAsZretrieveTokens'goMonotonicLineToksapplyOffsetToTreeShallowaddOffsetToForestSpanstripForestLinesonSameLn newLinesToken hseTokenLen synthesizel$fAllocatableModuleLocmergeToksAndCommentss2ff2shseTokenToStringhseShowTokenStreamhseLexStringToTokens showToken'hseAllocTokens allocTokens'$fMonoidLayout$fHasLocSrcSpan $fHasLocLoc $fIsTokenLoc DtSimpSpan combineUpsDtmconcatlfs2smkUpmkLinesFromToksadjustForDeleted calcDeltamkSpan$fActionTransformationUp$fSemigroupTransformation $fSemigroupUp$fSemigroupDtSimpSpan $fShowLine$fOutputableDtSimpSpansafeShowTokenStream$fOutputable(,)$fOutputable[]$fOutputableInt$fOutputableBool$fOutputableForestLine$fOutputableLineOpt$fOutputableSource$fOutputableLine$fOutputableNonEmpty$fOutputableDeletedSpan$fOutputableAlignment$fOutputableUp$fOutputableAnnot$fOutputableEndOffset$fOutputableTransformation$fOutputablePrim$fOutputableDUALTreeNE$fOutputableDUALTreeU$fOutputableDUALTreeinitTokenLayout getGhcLoc getGhcLocEndshowGhc tokenFileMark ghcMarkToken ghcIsMarked ghcTokenLencheckItemStagecheckItemStage2sanitizeeverywhereStagedeverywhereStaged'everythingStaged addLayoutghcAllocTokens allocDecls allocTyClD allocInstD allocDerivD allocValD allocSigD allocDefD allocForD allocWarningD allocAnnD allocRuleD allocVectD allocSpliceD allocDocDallocQuasiQuoteDallocLTyClDecl allocMatches allocGRHSsallocPatallocRhs allocStmtallocParStmtBlock allocExpr allocDoExprallocExprListComp allocCmdTopallocHsRecordBindsallocHsRecFieldallocLocalBindsstartPosForList endPosForList allocBindallocSigallocArithSeqInfo allocType allocInstDeclallocLFamInstDeclallocTupArgList allocLocatedallocTyVarBndrsallocTyVarBndr allocHsTyDefn allocConDeclallocHsConDeclDetailsallocConDeclFieldListallocConDeclFieldallocLBangType allocHsKind allocCTypeallocHsContextallocInterleavedListsshimmakeMixedListEntryallocMixedListgetLocatedStart getLocatedEndghcSpanStartEnd ghcTokenColghcTokenColEnd ghcTokenRow tokenSrcSpan ghcIsEmpty ghcZeroTokeng2ss2g ghcGetSpan ghcPutSpan ghcIsWhereghcIsLet ghcIsElse ghcIsThenghcIsOfghcIsDoghcIsIn ghcIsCommentghcSrcSpanToForestSpanforestSpanToGhcSrcSpanghcAllocTokens'allocTokensSrcSpanssynthesizelStagedcheckItemStage1ghcLexStringToTokenslexStringToRichTokens$fShowGenLocated$fHasLocGenLocated $fIsToken(,)$fAllocatableGenLocated(,)$fOutputableToken$fOutputableLayout$fOutputableEntry$fOutputableTree