!{D      !"#$%&'()*+,-./0123456789:;<=>?@ABC List-based dataframe(c) Marco Zocca (2018-2019) BSD-styleocramz fripost org experimentalGHCNone,456=?M  heidiA  is a list of rows.heidi Retain n rowsheidi Drop n rowsheidi This generalizes the list-based  function. heidiLeft-associative scan heidiRight-associative scan heidi  takes row comparison function and a list and returns a list of lists such that the concatenation of the result is equal to the argument. Moreover, each sublist in the result contains only elements that satisfy the comparison. heidi Produce a Vector of rows heidiProduce a Frame from a Vector of rows &Core.Data.Row internal bits and pieces(c) Marco Zocca (2018-2019) BSD-styleocramz fripost org experimentalGHCNone2456MDheidiKey exceptions DEFComposable decoding of terms(c) Marco Zocca (2019)MITocramz fripost org experimentalGHCNone4MGheidiWe can decouple lookup and value conversion and have distinct error behaviour. Multiple value decoding functions can be combined via the Applicative and Alternative instance.Note : GZ is called Kleisli in base.Control.Arrow; among other things it has a Profunctor instance.HheidiRun a decoding functionIheidi Construct a G from a monadic arrow.JheidiLeft-to-right composition @(>>>) :: Monad m => Decode m a b -> Decode m b c -> Decode m a cKheidiThis instance is copied from Kleisli (defined in Control.Arrow)GHIJA sparse dataframe(c) Marco Zocca (2018-2019) BSD-styleocramz fripost org experimentalGHCNone,456=?M2LheidiA L is a non-empty list of rows.MheidiTake the first row of a L$head (fromList [row0, row1]) == row0TrueNheidiTake the first n rows of a FrameOheidiDrop the first n rows of a FramePheidi0Construct a table given a non-empty list of rows/(head <$> fromNEList [row0, row1]) == Just row0True fromNEList []NothingQheidiJConstruct a table given a list of rows. Crashes if the input list is emptyRheidi,Zip two frames with a row combining functionSheidi Filters a Lx according to a predicate. Returns Nothing only if the resulting table is empty (i.e. if no rows satisfy the predicate).Theidi This generalizes the list-based S function.Uheidi Filter a L by decoding row values.UThis is an intermediate function that doesn't require fixing the row type within the L.NB: a G returning VH can be declared via its Functor, Applicative and Alternative instances.WheidiLeft-associative scanXheidiRight-associative scanYheidiY takes row comparison function and a list and returns a list of lists such that the concatenation of the result is equal to the argument. Moreover, each sublist in the result contains only elements that satisfy the comparison. ZheidiZv uses a comparison function that Maybe returns a Bool. This is useful when used in conjuction with lookup-based logic.[heidiO(n)& Count the number of rows in the table numRows t04\heidi Produce a Vector of rows]heidiProduce a Frame from a Vector of rowsUheidi Row decoderLMNOPQRSUWXY[\],Generic 1-hot encoding of enumeration types (c) Marco Zocca (2019)MITocramz fripost org experimentalGHCNone7>X<^heidi1-hot encoded vector.XThis representation is used to encode categorical variables as points in a vector space._heidi#Dimensionality of the ambient space`heidi index of '1'aheidi Construct a ^5 encoding from generic datatype and value information2mkOH (gdatatypeInfo (Proxy :: Proxy C)) (gfrom C2)OH {ohDim = 3, ohIx = 1}bheidi/Embedding dimension of the 1-hot encoded vectorcheidiActive (hot#) index of the 1-hot encoded vectordheidiCreate a one-hot vector^abcd'Generic encoding of algebraic datatypes(c) Marco Zocca (2019)MITocramz fripost org experimentalGHCNone&'278=>?X_[,6heidiPrimitive typesxNB : this is just a convenience for unityping the dataframe contents, but it should not be exposed to the library users eheidifgheidihiheidijkheidilmheidinoheidipqheidirsheidituheidivwheidixyheidiVzheidi{|heidi}~heidiheidiheidiheidiheidi1-hot encoding of an enum valueheidi Type errorsheidi2Typeclass for types which have a generic encoding.NOTE: if your type has a 8 instance you just need to declare an empty instance of  for it.example: data A = A Int Char deriving ( ) instance  A heidi.Internal representation of encoded ADTs valuessThe first String parameter contains the type name at the given level, the second contains the type constructor nameheidi recursionheidi1-hot encoding of an enumheidiprimitive typesheidiA (type, constructor) name pairheidiHFlatten a value into a 1-layer hashmap, via the value's generic encodingheidiFlatten a value into a ", via the value's generic encodingheidi Type nameheidiType constructorheidi:Create a fake TC with the given string as type constructorheidi3Create a fake TC with the given string as type nameheidiFold a Q into a 1-layer hashmap indexed by the input value's (type, constructor) metadataheidiFold a  into a 1-layer : indexed by the input value's (type, constructor) metadataheidi take 3 labels["_0","_1","_2"]heidiExtract an IntheidiExtract an Int8heidiExtract an Int16heidiExtract an Int32heidiExtract an Int64heidiExtract a WordheidiExtract a Word8heidiExtract a Word16heidiExtract a Word32heidiExtract a Word64heidiExtract a BoolheidiExtract a FloatheidiExtract a DoubleheidiExtract a ScientificheidiExtract a CharheidiExtract a StringheidiExtract a Text stringheidiExtract a OneHot valueheidi$Helper function for decoding into a .Segikmoqsuwyz|~ Noneaoheidi;Decode any real numerical value (integer, double, float or ) into a Doubleheidi;Decode any real numerical value (integer, double, float or ) into a ScientificheidiDecode a string ( or Text ) into a TextheidiDecode a string ( or Text) into a StringheidiDecode a one-hot value( ,A sparse dataframe row, based on GenericTrie(c) Marco Zocca (2018-2019) BSD-styleocramz fripost org experimentalGHCNone 4567SX_,heidiA  type is internally a Trie:Fast random accessFast set operationsSupports missing elementsheidiFocus on a given columnheidi : a Lens' that takes a key prefix and relates a row having lists as keys and the subset of columns corresponding to keys having that prefixheidi4Focus on all elements that share a common key prefixe.g.  >>> :t k ->   (eachPrefixed k . 6) (GT.TrieKey k, Eq k) => [k] -> Row [k] VP -> [Bool] heidiWExtract all elements that share a common key prefix into a monoidal value (e.g. a list)heidiHelper for filtering Framese.g.#:t \k -> keep (text k) (== "hello")* :: GT.TrieKey k => k -> Row k VP -> Boolheidi Decode a V from the given column indexheidi Decode a f from the given column indexheidi Decode a { from the given column indexheidi Decode a } from the given column indexheidi Decode a  from the given column index heidi Decode a  from the given column index!heidi Decode a  from the given column index"heidi Decode a  from the given column index#heidi Decode a ^ from the given column index$heidi Construct a " from a list of key-element pairs.(lookup 3 (rowFromList [(3,'a'),(4,'b')])Just 'a'(lookup 6 (rowFromList [(3,'a'),(4,'b')])Nothingheidi Construct a  from a trie (unsafe).heidi An empty row%heidi,Access the key-value pairs contained in the &heidi/Lookup the value stored at a given key in a row lookup 0 row0Just 'a' lookup 1 row0Nothing'heidiGCompares for ordering two rows by the values indexed at a specific key.8Returns Nothing if the key is not present in either row.(heidi:Compares two rows by the values indexed at a specific key.8Returns Nothing if the key is not present in either row.)heidi9Compares two rows by the values indexed at a set of keys.6Returns Nothing if a key in either row is not present.*heidi0Returns an empty row if the argument is Nothing.+heidiList the keys of a given row keys row0[0,3],heidi+Takes the union of a Foldable container of s and discards the values-heidiReturns a new ) that doesn't have a given key-value pair.heidiProduce a new 5 such that its keys do _not_ belong to a certain set./heidiCMap over all elements with a function of both the key and the value0heidiLFilter a row by applying a predicate to its keys and corresponding elements.ANB : filtering _retains_ the elements that satisfy the predicate.1heidiLRetains the entries for which the given list is a prefix of the indexing key2heidi Partition a b into two new ones, such as the elements that satisfy the predicate will end up in the _left_ row.3heidiUses 2 internally4heidiXRetains the entries for which the given item appears at any position in the indexing keyheidi=Insert a key-value pair into a row and return the updated onekeys $ insert 2 'y' row0[0,2,3]heidivInsert a key-value pair into a row and return the updated one, or updates the value by using the combination function.5heidi5Fold over a row with a function of both key and value6heidi Traverse a 2 using a function of both the key and the element.7heidiSet union of two rowskeys $ union row0 row1 [0,1,3,666]8heidi@Set union of two rows, using a combining function for equal keys9heidiSet intersection of two rows:heidiHSet intersections of two rows, using a combining function for equal keys;heidiLooks up a key from a row and applies a predicate to its value (if this is found). If no value is found at that key the function returns False.7This function is meant to be used as first argument to .elemSatisfies (== 'a') 0 row0TrueelemSatisfies (== 'a') 42 row0False<heidiInline synonym for ;heidi%key prefix of the columns of interestheidi%key prefix of the columns of interestheidi%key prefix of the columns of interestheidie.g. a predicate-heidi Key to remove1heidi key prefix2heidi predicate3heidi key prefix- !"#$%&'()*+,-./0123456789:;<&GenericTrie-based dataframe algorithms(c) Marco Zocca (2018-2019) BSD-styleocramz fripost org experimentalGHCNone =heidi7Merge two frames by taking the set union of the columns>heidi>a moves column names into a "key" column, gathering the column values into a single "value" columnheidi"gather one row into a list of rows?heidi?~ moves the unique values of a key column into the column names, spreading the values of a value column across the new columns.heidiWspread1 creates a single row from multiple ones that share a subset of key-value pairs.@heiditGROUP BY : given a key and a table that uses it, split the table in multiple tables, one per value taken by the key.1numRows <$> (HM.lookup "129" $ groupBy "id.0" t0)Just 2AheidixLEFT (OUTER) JOIN : given two dataframes and one key from each, compute the left outer join using the keys as relations.BheidilINNER JOIN : given two dataframes and one key from each, compute the inner join using the keys as relations.head t0,[("id.0","129"),("qty","1"),("item","book")]head t1 [("id.1","129"),("price","100")]$head $ innerJoin "id.0" "id.1" t0 t1K[("id.1","129"),("id.0","129"),("qty","5"),("item","book"),("price","100")]heidi("build" phase of the hash-join algorithmFor a given key kQ and a set of frame rows, populates a hashmap from the _values_ corresponding to k to the corresponding rows.=heidiElement combination function>heidiset of keys to gatherheidi "key" keyheidi "value" keyheidiinput dataframeheidirow to look intoheidi "key" keyheidi "value" key?heidi "key" keyheidi "value" keyheidiinput dataframe@heidiKey to group byheidi'A 'Frame (GTR.Row k v) can be used hereBheidiKey into the first tableheidiKey into the second tableheidiFirst dataframeheidiSecond dataframe=>?@AB=?>@BA None_1Populate dataframes with generically-encoded data(c) Marco Zocca (2019)MITocramz fripost org experimentalGHCNone7_Cheidi Populate a Frame* with the generic encoding of the row dataFor example, a list of records having two fields each will produce a dataframe with two columns, having the record field names as column labels. *data P1 = P1 Int Char deriving (Eq, Show,  ) instance  P1 data P2 = P2 { p2i :: Int, p2c :: Char } deriving (Eq, Show, Generic) instance Heidi P2 data Q = Q (Maybe Int) (Either Double Char) deriving (Eq, Show, Generic) instance Heidi Q encode [P1 42 'z']QFrame {tableRows = [([TC "P1" "_0"],VPInt 42),([TC "P1" "_1"],VPChar 'z')] :| []}encode [P2 42 'z']SFrame {tableRows = [([TC "P2" "p2c"],VPChar 'z'),([TC "P2" "p2i"],VPInt 42)] :| []} Test using  and  record fields :6encode [Q (Just 42) (Left 1.2), Q Nothing (Right 'b')]Frame {tableRows = [([TC "Q" "_0",TC "Maybe" "Just"],VPInt 42),([TC "Q" "_1",TC "Either" "Left"],VPDouble 1.2)] :| [[([TC "Q" "_1",TC "Either" "Right"],VPChar 'b')]]},NB: as the last example above demonstrates, X values are not inserted in the rows, which can be used to encode missing data features.heidi Populate a RowB with a generic encoding of the input value (generic-trie backend)Ctidy data in Haskell(c) Marco Zocca (2018-2020) BSD-styleocramz fripost org experimentalGHCNoneӭC  !"#$%&'()*+,-./0123456789:;<>?@ABCCC  ?>@BA $%+-014.23&<;*()'789:/5,6 !"#None4567_>heidiZCopy the contents of a list-indexed Row into a tree-shaped structure (for pretty-printing)unfold [("aa", 41), ("ab", 42)]H[('a',[('a',[('a',41)]),('b',42)])] -- FIXME why 3 levels and not 2 ?!?heidi,Copy a single list-indexed value into a treeinsert M.empty "abc" 42[('a',[('b',[('c',42)])])]heidi(union of the set of keys across all rowsheidi dataframeheidi column key   !"#$% & ' ( ) * + , - . / 0 1 2 3 4 5 6 7 8 9 : ; < = > ? @ A B C D E F G H I J K L MNOPQRSTUVWXYZ[\]^_`abcdefghijk`almnopqnorsnotunovw`axynz{|nz}~nznz`a`a`ann! - n   n n  n  "heidi-0.0.0-DfPtGUuKGqKBthOvzb5i7rHeidi'Heidi.Data.Frame.Algorithms.GenericTrieCore.Data.Frame.ListCore.Data.Row.InternalData.Generics.DecodeCore.Data.FrameData.Generics.Encode.OneHotData.Generics.Encode.InternalData.Generics.CodecHeidi.Data.Row.GenericTrie Lens.MicrotoListOf/Heidi.Data.Frame.Algorithms.GenericTrie.GenericCore.Data.Frame.GenericGGenericCore.Data.Frame.PrettyPrintFrameheadtakedropzipWith frameFromListnumRowsfilterfilterAscanlscanr groupWithtoVector fromVectorVPTCtcTyNtcTyConmkTyConmkTyNRowatatPrefix eachPrefixed foldPrefixedkeepboolintfloatdoublecharstringtext scientificoneHot rowFromListtoListlookupcompareByLookup eqByLookup eqByLookups maybeEmptykeyskeysOnlydelete deleteMany mapWithKey filterWithKeyfilterWithKeyPrefixpartitionWithKeypartitionWithKeyPrefixfilterWithKeyAny foldWithKeytraverseWithKeyunion unionWith intersectionintersectionWith elemSatisfies!: unionColsWith gatherWith spreadWithgroupBy leftOuterJoin innerJoinencodeKeyErrorAlreadyPresentKeyErrorMissingKeyErrorDecode runDecodemkDecode>>>$fCategoryTYPEDecode fromNEListfromList filterDecodeghc-prim GHC.TypesBool groupWithMOneHotohDimohIxmkOH onehotDimonehotIxoneHotVVPIntIntVPInt8baseGHC.IntInt8VPInt16Int16VPInt32Int32VPInt64Int64VPWordWordVPWord8GHC.WordWord8VPWord16Word16VPWord32Word32VPWord64Word64VPBoolVPFloatFloatVPDoubleDouble VPScientific)scientific-0.3.6.2-HDoLkL8YC3O28Jlh4HmWvSData.Scientific ScientificVPCharCharVPStringGHC.BaseStringVPText text-1.2.3.1Data.Text.InternalTextVPOH TypeError GHC.GenericsValVRecVEnumVPrim gflattenHM gflattenGT)generic-trie-0.3.1-Ec9akeJLU5tEOEAka4EGo1Data.GenericTrie.InternalTrie flattenHM flattenGTlabelsgetIntgetInt8getInt16getInt32getInt64getWordgetWord8 getWord16 getWord32 getWord64getBoolgetFloat getDouble getScientificgetChar getStringgetText getOneHotdecodeM(exceptions-0.10.4-622ySnQapwMAU2KS1exqorControl.Monad.Catch MonadThrow_vpInt_vpBool_vpFloat _vpDouble _vpScientific_vpChar _vpString_vpText _vpOneHotIntCastE Int8CastE Int16CastE Int32CastE Int64CastE WordCastE Word8CastE Word16CastE Word32CastE Word64CastE BoolCastE FloatCastE DoubleCastEScientificCastE CharCastE StringCastE TextCastE OneHotCastEvpBoolvpCharvpDoublevpFloatvpIntvpOneHot vpScientificvpStringvpTextgetIntMgetInt8M getInt16M getInt32M getInt64MgetWordM getWord8M getWord16M getWord32M getWord64MgetBoolM getFloatM getDoubleMgetScientificMgetCharM getStringMgetTextM getOneHotMrealM scientificMtextMstringMoneHotMint8int16int32int64wordword8word16word32word64mkRowemptyinsert insertWithGHC.List insertManygather1spread1hjBuild keyToValue valueToKey GHC.MaybeMaybe Data.EitherEitherNothinggToRowGTunfoldallKeys columnBoxMMbMlarr0arr1box1box2 groupSort groupSortBy