porcupine-core: Express portable, composable and reusable data tasks and pipelines

[ arrows, combinators, control, data, json, library, mit, numerical, pipes, program, streaming ] [ Propose Tags ]

porcupine is centered around the PTask datatype, which represents a computation that will request access to some resources (both in read and write modes) and require some options (parameters, with docstrings and default values). PTasks are composable both sequentially and in parallel, into a pipeline of tasks. The resources and parameters are organized in a tree which will be automatically exposed to the outside world. This makes the pipeline self-documented, and makes it so any option or file required at some point by any task can be visualized and set or remapped (via a combination of a YAML or JSON config file and command-line arguments) before the pipeline will run. That means that the PTasks are completely agnostic of their data inputs, and that new data sources can be added without having to change any of the tasks' logic or even their types. This is done via the LocationAccessor typeclass. porcupine-core provides only access to local files (via resourcet), other location accessors will be in separate packages. See for instance the https://hackage.haskell.org/package/porcupine-http package to access HTTP urls. PTasks also provide caching thanks to the funflow package. See the README at https://github.com/tweag/porcupine#README.md and the examples in porcupine-core package.

Versions [faq] 0.1.0.0, 0.1.0.1
Dependencies aeson, aeson-pretty, attoparsec, base (>=4.10 && <5), binary, binary-orphans, bytestring, cassava, clock, conduit, containers, contravariant, data-default, deepseq, directory, docrecords (==0.1.*), filepath, foldl, formatting, funflow (>=1.5.0), hashable, hvega, katip (==0.8.*), lens, log-domain, monad-bayes, monad-control, mtl, optparse-applicative, path, porcupine-core, profunctors, reader-soup (==0.1.*), resourcet, safe-exceptions, store, streaming, streaming-bytestring, streaming-conduit, streaming-utils, template-haskell, temporary, text, transformers, transformers-base, unix, unliftio-core, unordered-containers, url, vector, vinyl, yaml, zlib [details]
License MIT
Copyright 2018 EURL Tweag, NovaDiscovery
Author
Maintainer Yves Parès <yves.pares@tweag.io>
Category Data, Arrows, Combinators, Control, Pipes, Streaming, Numerical, JSON
Home page https://github.com/tweag/porcupine#readme
Bug tracker https://github.com/tweag/porcupine/issues
Source repo head: git clone https://github.com/tweag/porcupine
Uploaded by YvesPares at Wed Oct 9 13:23:58 UTC 2019
Distributions NixOS:0.1.0.1
Executables example2, example1, example0.1, example0, example-radon
Downloads 56 total (56 in the last 30 days)
Rating (no votes yet) [estimated by rule of succession]
Your Rating
  • λ
  • λ
  • λ
Status Hackage Matrix CI
Docs not available [build log]
All reported builds failed as of 2019-10-09 [all 2 reports]

Modules

  • Control
    • Arrow
      • Control.Arrow.FoldA
  • Data
    • Data.Locations
      • Data.Locations.Accessors
      • Data.Locations.FunflowRemoteCache
      • Data.Locations.Loc
      • Data.Locations.LocVariable
      • Data.Locations.LocationTree
      • Data.Locations.LogAndErrors
      • Data.Locations.Mappings
      • Data.Locations.SerializationMethod
      • Data.Locations.VirtualFile
    • Data.Representable
  • Porcupine
    • Porcupine.Foldl
    • Porcupine.Run
    • Porcupine.Serials
    • Porcupine.Tasks
    • Porcupine.VFiles
  • Streaming
    • Streaming.TaskPipelineUtils
  • System
    • System.ClockHelpers
    • System.TaskPipeline
      • System.TaskPipeline.CLI
      • System.TaskPipeline.Caching
      • System.TaskPipeline.ConfigurationReader
      • System.TaskPipeline.Logger
      • System.TaskPipeline.Options
      • System.TaskPipeline.PTask
        • System.TaskPipeline.PTask.Internal
      • System.TaskPipeline.PorcupineTree
      • System.TaskPipeline.Repetition
        • System.TaskPipeline.Repetition.Foldl
        • System.TaskPipeline.Repetition.Internal
        • System.TaskPipeline.Repetition.Streaming
      • System.TaskPipeline.Run
      • System.TaskPipeline.VirtualFileAccess

Flags

NameDescriptionDefaultType
usemonadbayesDisabledManual

Use -f <flag> to enable a flag, or -f -<flag> to disable that flag. More info

Downloads

Maintainer's Corner

For package maintainers and hackage trustees