persistent-postgresql-streaming: Memory-constant streaming of Persistent entities from PostgreSQL

[ bsd3, conduit, database, library ] [ Propose Tags ]

This library provides a set of APIs for performing queries on Persistent entities in constant memory, streaming the results using conduit.

The library relies on PostgreSQL-specific features to avoid loading all the results of a query into memory at once. This allows for accessing tables of millions of entities from Haskell without a memory blow-up.

[Skip to Readme]


Maintainer's Corner

Package maintainers

For package maintainers and hackage trustees


Versions [RSS]
Change log
Dependencies base (>=4.7 && <5), conduit (>=1.3 && <1.4), monad-logger (>= && <0.4), mtl (>=2 && <2.3), persistent (>=2.13.2 && <2.14), persistent-postgresql (>=2.13.2 && <2.14), postgresql-simple (>=0.6.1 && <0.7), resourcet (>=0.3 && <2), text (>=1 && <1.3), transformers (>=0.5 && <0.6) [details]
License BSD-3-Clause
Copyright (c) 2021 Supercede Ltd.
Author Isaac van Bakel
Category Conduit, Database
Home page
Source repo head: git clone
Uploaded by isaac_supercede at 2021-10-20T15:41:26Z
Distributions NixOS:
Reverse Dependencies 1 direct, 0 indirect [details]
Downloads 170 total (5 in the last 30 days)
Rating (no votes yet) [estimated by Bayesian average]
Your Rating
  • λ
  • λ
  • λ
Status Docs available [build log]
Last success reported on 2021-10-20 [all 1 reports]

Readme for persistent-postgresql-streaming-

[back to package description]


This library allows for memory-constant streaming of persistent entities from PostgreSQL databases.

This code makes use of the PostgreSQL-only cursors, which allow for batched access to the result set of a query at speeds comparable to loading all the results into memory at once.

See the main project README for more.

Streaming Persistent queries

The main function of this library is selectStream, which can be used in place of selectSource. selectSource runs in a ConduitT - consuming the conduit will pull results from the database in constant memory.


Why is selectStream so slow?

Have you configured PostgreSQL correctly? See the section in the README about it.