| Safe Haskell | None |
|---|---|
| Language | Haskell2010 |
Spark.Core.Internal.Projections
Contents
Description
Defines some projections operations over dataframes, observables, and columns. This allows users for a fairly natural manipulation of data.
- type family ProjectReturn from proj where ...
- class Project from proj
- (//) :: forall from proj. Project from proj => from -> proj -> ProjectReturn from proj
- (/-) :: forall from. Project from Text => from -> Text -> ProjectReturn from Text
- _1 :: FixedProjection1
- _2 :: FixedProjection2
- data StaticColProjection from to = StaticColProjection {
- _staticProj :: SQLType from -> Try (FieldPath, SQLType to)
- data DynamicColProjection
- unsafeStaticProjection :: forall from to. HasCallStack => SQLType from -> String -> StaticColProjection from to
- dynamicProjection :: String -> DynamicColProjection
Documentation
type family ProjectReturn from proj where ... Source #
Equations
| ProjectReturn DataFrame DynamicColProjection = DynColumn | |
| ProjectReturn DataFrame (StaticColProjection from to) = DynColumn | |
| ProjectReturn DataFrame Text = DynColumn | |
| ProjectReturn DynColumn DynamicColProjection = DynColumn | |
| ProjectReturn DynColumn Text = DynColumn | |
| ProjectReturn (Dataset (x1, x2)) FixedProjection1 = Column (x1, x2) x1 | |
| ProjectReturn (Dataset (x1, x2)) FixedProjection2 = Column (x1, x2) x2 | |
| ProjectReturn (Dataset x) DynamicColProjection = DynColumn | |
| ProjectReturn (Dataset x) (StaticColProjection x y) = Column x y | |
| ProjectReturn (Dataset x) Text = DynColumn |
class Project from proj Source #
Instances
(//) :: forall from proj. Project from proj => from -> proj -> ProjectReturn from proj Source #
The projector operation.
This is the general projection operation in Spark. It lets you extract columns from datasets or dataframes, or sub-observables from observables.
TODO(kps) put an example here.
(/-) :: forall from. Project from Text => from -> Text -> ProjectReturn from Text Source #
The projector operation for string.
This is the general projection operation in Spark. It lets you extract columns from datasets or dataframes, or sub-observables from observables.
Because of a Haskell limitation, this operator is different for strings.
TODO(kps) put an example here.
Developer functions
data StaticColProjection from to Source #
The class of static projections that are guaranteed to succeed by using the type system.
from is the type of the dataset (which is also a typed dataset) to is the type of the final column.
Constructors
| StaticColProjection | |
Fields
| |
Instances
| Project DataFrame (StaticColProjection a b) Source # | |
| Project (Dataset a) (StaticColProjection a b) Source # | |
data DynamicColProjection Source #
The class of projections that require some runtime introspection to confirm that the projection is valid.
unsafeStaticProjection Source #
Arguments
| :: HasCallStack | |
| => SQLType from | The start type |
| -> String | The name of a field assumed to be found in the start type. This only has to be valid for Spark purposes, not internal Haskell representation. |
| -> StaticColProjection from to |
Lets the users define their own static projections.
Throws an error if the type cannot be found, so should be used with caution.
String has to be used because of type inferrence issues