porcupine-core: Express portable, composable and reusable data tasks and pipelines

[ arrows, combinators, control, data, json, library, mit, numerical, pipes, program, streaming ] [ Propose Tags ]

porcupine is centered around the PTask datatype, which represents a computation that will request access to some resources (both in read and write modes) and require some options (parameters, with docstrings and default values). PTasks are composable both sequentially and in parallel, into a pipeline of tasks. The resources and parameters are organized in a tree which will be automatically exposed to the outside world. This makes the pipeline self-documented, and makes it so any option or file required at some point by any task can be visualized and setremapped (via a combination of a YAMLJSON config file and command-line arguments) before the pipeline will run. That means that the PTasks are completely agnostic of their data inputs, and that new data sources can be added without having to change any of the tasks' logic or even their types. This is done via the LocationAccessor typeclass. porcupine-core provides only access to local files (via resourcet), other location accessors will be in separate packages. See for instance the <https://hackage.haskell.org/package/porcupine-http porcupine-http> package to access HTTP urls. PTasks also provide caching thanks to the funflow package. See the README at https://github.com/tweag/porcupine#README.md and the examples in porcupine-core package.

Versions [faq] 0.1.0.0, 0.1.0.1
Dependencies aeson, aeson-pretty, attoparsec, base (>=4.10 && <5), binary, binary-orphans, bytestring, cassava, clock, conduit, containers, contravariant, data-default, deepseq, directory, docrecords (==0.1.*), filepath, foldl, formatting, funflow (>=1.5.0), hashable, hvega, katip (==0.8.*), lens, log-domain, monad-bayes, monad-control, mtl, optparse-applicative, path, porcupine-core, profunctors, reader-soup (==0.1.*), resourcet, safe-exceptions, store, streaming, streaming-bytestring, streaming-conduit, streaming-utils, template-haskell, temporary, text, transformers, transformers-base, unix, unliftio-core, unordered-containers, url, vector, vinyl, yaml, zlib [details]
License MIT
Copyright 2018 EURL Tweag, NovaDiscovery
Author
Maintainer Yves Parès <yves.pares@tweag.io>
Category Data, Arrows, Combinators, Control, Pipes, Streaming, Numerical, JSON
Home page https://github.com/tweag/porcupine#readme
Bug tracker https://github.com/tweag/porcupine/issues
Source repo head: git clone https://github.com/tweag/porcupine
Uploaded by YvesPares at Wed Oct 9 10:16:26 UTC 2019
Distributions NixOS:0.1.0.1
Executables example2, example1, example0.1, example0, example-radon
Downloads 112 total (25 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs not available [build log]
All reported builds failed as of 2019-10-09 [all 2 reports]

Modules

  • Control
    • Arrow
      • Control.Arrow.FoldA
  • Data
    • Data.Locations
      • Data.Locations.Accessors
      • Data.Locations.FunflowRemoteCache
      • Data.Locations.Loc
      • Data.Locations.LocVariable
      • Data.Locations.LocationTree
      • Data.Locations.LogAndErrors
      • Data.Locations.Mappings
      • Data.Locations.SerializationMethod
      • Data.Locations.VirtualFile
    • Data.Representable
  • Porcupine
    • Porcupine.Foldl
    • Porcupine.Run
    • Porcupine.Serials
    • Porcupine.Tasks
    • Porcupine.VFiles
  • Streaming
    • Streaming.TaskPipelineUtils
  • System
    • System.ClockHelpers
    • System.TaskPipeline
      • System.TaskPipeline.CLI
      • System.TaskPipeline.Caching
      • System.TaskPipeline.ConfigurationReader
      • System.TaskPipeline.Logger
      • System.TaskPipeline.Options
      • System.TaskPipeline.PTask
        • System.TaskPipeline.PTask.Internal
      • System.TaskPipeline.PorcupineTree
      • System.TaskPipeline.Repetition
        • System.TaskPipeline.Repetition.Foldl
        • System.TaskPipeline.Repetition.Internal
        • System.TaskPipeline.Repetition.Streaming
      • System.TaskPipeline.Run
      • System.TaskPipeline.VirtualFileAccess

Downloads

Maintainer's Corner

For package maintainers and hackage trustees