Spark.Core.Context

This module defines session objects that act as entry points to spark.

There are two ways to interact with Spark: using an explicit state object, or using the default state object (interactive session).

While the interactive session is the most convenient, it should not be used for more than quick experimentations. Any complex code should use the SparkSession and SparkState objects.

Documentation

data SparkSessionConf

The configuration of a remote spark session in Karps.

Constructors

SparkSessionConf

Fields

• confEndPoint :: !Text

  The URL of the end point.

• confPort :: !Int

  The port used to configure the end point.

• confPollingIntervalMillis :: !Int

  (internal) the polling interval

• confRequestedSessionName :: !Text

  (optional) the requested name of the session. This name must obey a number of rules: - it must consist in alphanumerical and -,_: [a-zA-Z0-9-_] - if it already exists on the server, it will be reconnected to

  The default value is "" (a new random context name will be chosen).

• confUseNodePrunning :: !Bool

  If enabled, attempts to prune the computation graph as much as possible.

  This option is useful in interactive sessions when long chains of computations are extracted. This forces the execution of only the missing parts. The algorithm is experimental, so disabling it is a safe option.

  Disabled by default.

Instances

Show SparkSessionConf

Methods

showsPrec :: Int -> SparkSessionConf -> ShowS

show :: SparkSessionConf -> String

showList :: [SparkSessionConf] -> ShowS

data SparkSession

A session in Spark. Encapsualates all the state needed to communicate with Spark and to perfor some simple optimizations on the code.

Instances

Show SparkSession

Methods

showsPrec :: Int -> SparkSession -> ShowS

show :: SparkSession -> String

showList :: [SparkSession] -> ShowS

type SparkState a = SparkStateT IO a

Represents the state of a session and accounts for the communication with the server.

data SparkInteractiveException

The exception thrown when a request cannot be completed in an interactive session.

Instances

Show SparkInteractiveException

Methods

showsPrec :: Int -> SparkInteractiveException -> ShowS

show :: SparkInteractiveException -> String

showList :: [SparkInteractiveException] -> ShowS

Exception SparkInteractiveException

Methods

toException :: SparkInteractiveException -> SomeException

fromException :: SomeException -> Maybe SparkInteractiveException

displayException :: SparkInteractiveException -> String

class FromSQL a

Instances

FromSQL Bool

Methods

_cellToValue :: Cell -> TryS Bool

FromSQL Double

Methods

_cellToValue :: Cell -> TryS Double

FromSQL Int

Methods

_cellToValue :: Cell -> TryS Int

FromSQL Text

Methods

_cellToValue :: Cell -> TryS Text

FromSQL DataTypeRepr

Methods

_cellToValue :: Cell -> TryS DataTypeRepr

FromSQL DataTypeElementRepr

Methods

_cellToValue :: Cell -> TryS DataTypeElementRepr

FromSQL Cell

Methods

_cellToValue :: Cell -> TryS Cell

FromSQL a => FromSQL [a]

Methods

_cellToValue :: Cell -> TryS [a]

FromSQL a => FromSQL (Maybe a)

Methods

_cellToValue :: Cell -> TryS (Maybe a)

(FromSQL a1, FromSQL a2) => FromSQL (a1, a2)

Methods

_cellToValue :: Cell -> TryS (a1, a2)

defaultConf :: SparkSessionConf

The default configuration if the Karps server is being run locally.

executeCommand1 :: forall a. FromSQL a => LocalData a -> SparkState (Try a)

Executes a command: - performs the transforms and the optimizations in the pure state - sends the computation to the backend - waits for the terminal nodes to reach a final state - commits the final results to the state

If any failure is detected that is internal to Karps, it returns an error. If the error comes from an underlying library (http stack, programming failure), an exception may be thrown instead.

executeCommand1' :: UntypedLocalData -> SparkState (Try Cell)

computationStats :: ComputationID -> SparkState BatchComputationResult

Exposed for debugging

createSparkSessionDef :: SparkSessionConf -> IO ()

Creates a spark session that will be used as the default session.

If a session already exists, an exception will be thrown.

closeSparkSessionDef :: IO ()

Closes the default session. The default session is empty after this call completes.

NOTE: This does not currently clear up the resources! It is a stub implementation used in testing.

currentSessionDef :: IO (Maybe SparkSession)

computationStatsDef :: ComputationID -> IO BatchComputationResult

exec1Def :: FromSQL a => LocalData a -> IO a

Executes a command using the default spark session.

This is the most unsafe way of running a command: it executes a command using the default spark session, and throws an exception if any error happens.

exec1Def' :: LocalFrame -> IO Cell

execStateDef :: SparkState a -> IO a

Runs the computation described in the state transform, using the default Spark session.

Will throw an exception if no session currently exists.