hlivy: Client library for the Apache Livy REST API.

This is a package candidate release! Here you can preview how this package release will appear once published to the main package index (which can be accomplished via the 'maintain' link below). Please note that once a package has been published to the main package index it cannot be undone! Please consult the package uploading documentation for more information.




[Skip to ReadMe]


Versions1.0.0, 1.0.0, 1.0.1
Change logCHANGELOG.md
Dependenciesaeson (>=1.4.2 && <1.5), base (>=4.5 && <5), bytestring (>= && <0.11), exceptions (==0.10.*), http-client (>=0.5.14 && <0.6), http-types (>=0.12.2 && <0.13), lens (>=4.17 && <5.0), mtl (>=2.2.2 && <2.3), resourcet (>=1.2.2 && <1.3), text (>= && <1.3), transformers (>=0.5.5 && <0.6), unordered-containers (>=0.2.9 && <0.3) [details]
Copyright2019 Earnest Research
AuthorDaniel Donohue <ddonohue@earnestresearch.com>
MaintainerDaniel Donohue <ddonohue@earnestresearch.com>
CategoryDistributed Computing
Home pagehttps://github.com/EarnestResearch/hlivy
Source repositoryhead: git clone https://github.com/EarnestResearch/hlivy.git
UploadedMon Feb 4 18:49:01 UTC 2019 by ddonohue




Maintainers' corner

For package maintainers and hackage trustees

Readme for hlivy-1.0.0

[back to package description]



hlivy is a Haskell library that provides bindings to the Apache Livy REST API, which enables one to easily launch Spark applications -- either in an interactive or batch fashion -- via HTTP requests to the Livy server running on the master node of a Spark cluster.


Start with

import Network.Livy

which brings all functionality into scope. In particular, this exposes a monad Livy that has all the capabilites required to run Livy actions with runLivy. Generally, the format of a Livy action follows the pattern

send $ basicRequestObject requiredArg1 requiredArg2
     & requestLens1 ?~ optionalArg1
     & requestLens2 ?~ optionalArg2

This action is ran simply:

let req = basicRequestObject requiredArg1 requiredArg2
        & requestLens1 ?~ optionalArg1
        & requestLens2 ?~ optionalArg2
resp <- runLivy env $ send req

where env is a suitable environment. Concretely, if one wanted to create an interactive session, one would do something like this:

λ => import Network.Livy
λ => -- Create a default environment
λ => env <- newEnv "localhost" 8998
λ => resp <- runLivy env $ send createSession

The response body, in this case a CreateSessionResponse, should contain the the Session just created.

With this Session at hand, one can run "statements" -- snippets of Spark Scala, PySpark, SparkR, or SparkSQL -- in the given session.

λ => req = runStatement (SessionId 0) "val x = 1 + 1; println(x)" SparkSession
λ => resp <- runLivy env $ send req

This response object, in this case a RunStatementResponse, contains the information needed to check on the status of the statement or retrieve results if available.

Batch actions are organized in the Network.Livy.Client.Batch module, and are used similarly:

λ => import Control.Lens
λ => -- Application JAR in HDFS
λ => req = createBatch "/user/hadoop/my-app.jar"
λ => resp <- runLivy env (send req & cbClassName ?~ "com.company.my_app" ?~ cbExecutorCores ?~ 4)

See examples for more example use.