Spark.Core.Internal.ContextIOInternal

Documentation

returnPure :: forall a. SparkStatePure a -> SparkState a

createSparkSession :: MonadLoggerIO m => SparkSessionConf -> m SparkSession

Creates a new Spark session.

This session is unique, and it will not try to reconnect to an existing session.

createSparkSession' :: SparkSessionConf -> IO SparkSession

Convenience function for simple cases that do not require monad stacks.

executeCommand1 :: forall a. FromSQL a => LocalData a -> SparkState (Try a)

Executes a command: - performs the transforms and the optimizations in the pure state - sends the computation to the backend - waits for the terminal nodes to reach a final state - commits the final results to the state

If any failure is detected that is internal to Karps, it returns an error. If the error comes from an underlying library (http stack, programming failure), an exception may be thrown instead.

executeCommand1' :: UntypedLocalData -> SparkState (Try Cell)

checkDataStamps :: [HdfsPath] -> SparkState [(HdfsPath, Try DataInputStamp)]

Given a list of paths, checks each of these paths on the file system of the given Spark cluster to infer the status of these resources.

The primary role of this function is to check how recent these resources are compared to some previous usage.

updateSourceInfo :: ComputeGraph -> SparkState (Try ComputeGraph)

Exposed for debugging

createComputation :: ComputeGraph -> SparkState (Try Computation)

Exposed for debugging

computationStats :: ComputationID -> SparkState BatchComputationResult

Exposed for debugging