Haskell SuperBuffer
The superbuffer
packages was designed to efficiently build up bytestrings from IO
actions producing
smaller chunks. The goal was to reduce memory overhead as much as possible while still being as fast as possible.
In our use case, it reduced total memory usage of the program from 350 MB
(bytestring
builder) to 50 MB
(superbuffer
).
For speed see benchmarks below. Note that the speed heavily depends on a good choice of the initial buffer size and
the size of the chunks written. For small chunks the superbuffer
outperforms the bytestring
alternatives consistently. Note
that the library is currently not thread-safe, but it seems that it could be added (from outside) easily by making sure only one thread
at a time can call appendBuffer
(making the call slower).
Usage
{-# LANGUAGE OverloadedStrings #-}
module Example where
import Data.ByteString.SuperBuffer
import qualified Data.ByteString as BS
myBS :: IO BS.ByteString
myBS =
-- note: performance of superbuffer heavily depends on a
-- smart choice of the initial buffer size. Benchmark to
-- find what suits your needs.
withBuffer 1024 $ \buf ->
do appendBuffer buf "Hello "
appendBuffer buf "World!"
Benchmarks
benchmarking main/small/superbuffer (init=128 bytes)
time 22.20 ms (20.81 ms .. 24.69 ms)
0.973 R² (0.946 R² .. 0.992 R²)
mean 22.47 ms (21.70 ms .. 23.45 ms)
std dev 1.905 ms (1.515 ms .. 2.685 ms)
variance introduced by outliers: 38% (moderately inflated)
benchmarking main/small/superbuffer (init=4000 bytes)
time 19.92 ms (19.28 ms .. 20.58 ms)
0.996 R² (0.993 R² .. 0.998 R²)
mean 22.30 ms (21.56 ms .. 23.60 ms)
std dev 2.230 ms (1.324 ms .. 3.614 ms)
variance introduced by outliers: 46% (moderately inflated)
benchmarking main/small/superbuffer (init=8000 bytes)
time 22.72 ms (21.85 ms .. 23.92 ms)
0.994 R² (0.989 R² .. 0.997 R²)
mean 23.25 ms (22.73 ms .. 23.95 ms)
std dev 1.496 ms (1.007 ms .. 2.375 ms)
variance introduced by outliers: 24% (moderately inflated)
benchmarking main/small/superbuffer (init=16000 bytes)
time 23.28 ms (21.77 ms .. 25.14 ms)
0.978 R² (0.953 R² .. 0.994 R²)
mean 23.41 ms (22.79 ms .. 24.34 ms)
std dev 1.727 ms (1.209 ms .. 2.455 ms)
variance introduced by outliers: 29% (moderately inflated)
benchmarking main/small/superbuffer (init=20000000 bytes)
time 35.78 ms (34.57 ms .. 37.21 ms)
0.989 R² (0.972 R² .. 0.998 R²)
mean 36.45 ms (35.25 ms .. 37.34 ms)
std dev 2.167 ms (1.281 ms .. 3.677 ms)
variance introduced by outliers: 18% (moderately inflated)
benchmarking main/small/bytestring builder
time 31.34 ms (28.98 ms .. 33.54 ms)
0.987 R² (0.974 R² .. 0.999 R²)
mean 30.73 ms (29.54 ms .. 31.59 ms)
std dev 2.205 ms (1.699 ms .. 2.841 ms)
variance introduced by outliers: 28% (moderately inflated)
benchmarking main/small/bytestring fromChunks
time 26.35 ms (25.53 ms .. 27.40 ms)
0.996 R² (0.991 R² .. 0.999 R²)
mean 26.35 ms (25.88 ms .. 27.79 ms)
std dev 1.685 ms (675.3 μs .. 3.068 ms)
variance introduced by outliers: 25% (moderately inflated)
benchmarking main/small/bytestring concat
time 30.54 ms (29.83 ms .. 31.13 ms)
0.998 R² (0.997 R² .. 0.999 R²)
mean 31.45 ms (30.94 ms .. 33.55 ms)
std dev 1.658 ms (563.0 μs .. 3.317 ms)
variance introduced by outliers: 17% (moderately inflated)
benchmarking main/med/superbuffer (init=128 bytes)
time 20.98 ms (20.11 ms .. 21.97 ms)
0.992 R² (0.983 R² .. 0.997 R²)
mean 23.11 ms (22.53 ms .. 24.12 ms)
std dev 1.686 ms (1.020 ms .. 2.914 ms)
variance introduced by outliers: 29% (moderately inflated)
benchmarking main/med/superbuffer (init=40000 bytes)
time 21.62 ms (20.92 ms .. 22.43 ms)
0.996 R² (0.991 R² .. 0.998 R²)
mean 23.65 ms (23.06 ms .. 24.72 ms)
std dev 1.735 ms (1.044 ms .. 2.732 ms)
variance introduced by outliers: 29% (moderately inflated)
benchmarking main/med/superbuffer (init=80000 bytes)
time 20.65 ms (19.91 ms .. 21.35 ms)
0.994 R² (0.985 R² .. 0.998 R²)
mean 22.64 ms (22.09 ms .. 23.84 ms)
std dev 1.767 ms (971.0 μs .. 3.008 ms)
variance introduced by outliers: 33% (moderately inflated)
benchmarking main/med/superbuffer (init=160000 bytes)
time 23.35 ms (22.15 ms .. 24.56 ms)
0.992 R² (0.987 R² .. 0.998 R²)
mean 22.50 ms (22.15 ms .. 23.08 ms)
std dev 1.038 ms (639.2 μs .. 1.502 ms)
variance introduced by outliers: 14% (moderately inflated)
benchmarking main/med/superbuffer (init=20000000 bytes)
time 35.02 ms (31.20 ms .. 37.83 ms)
0.983 R² (0.970 R² .. 0.998 R²)
mean 32.02 ms (31.12 ms .. 33.32 ms)
std dev 2.159 ms (1.456 ms .. 3.304 ms)
variance introduced by outliers: 24% (moderately inflated)
benchmarking main/med/bytestring builder
time 22.03 ms (21.53 ms .. 22.48 ms)
0.998 R² (0.997 R² .. 0.999 R²)
mean 22.50 ms (22.06 ms .. 23.71 ms)
std dev 1.614 ms (467.3 μs .. 3.159 ms)
variance introduced by outliers: 29% (moderately inflated)
benchmarking main/med/bytestring fromChunks
time 23.88 ms (21.62 ms .. 26.14 ms)
0.975 R² (0.954 R² .. 0.998 R²)
mean 22.81 ms (22.25 ms .. 23.95 ms)
std dev 1.812 ms (784.9 μs .. 2.692 ms)
variance introduced by outliers: 33% (moderately inflated)
benchmarking main/med/bytestring concat
time 21.79 ms (21.28 ms .. 22.23 ms)
0.998 R² (0.997 R² .. 0.999 R²)
mean 22.52 ms (22.05 ms .. 23.70 ms)
std dev 1.646 ms (554.1 μs .. 3.042 ms)
variance introduced by outliers: 29% (moderately inflated)
benchmarking main/large/superbuffer (init=128 bytes)
time 25.97 ms (25.22 ms .. 26.62 ms)
0.994 R² (0.984 R² .. 0.999 R²)
mean 26.01 ms (25.18 ms .. 26.90 ms)
std dev 1.943 ms (1.274 ms .. 3.107 ms)
variance introduced by outliers: 30% (moderately inflated)
benchmarking main/large/superbuffer (init=400000 bytes)
time 21.51 ms (21.13 ms .. 21.88 ms)
0.999 R² (0.997 R² .. 1.000 R²)
mean 22.14 ms (21.84 ms .. 22.75 ms)
std dev 939.7 μs (407.0 μs .. 1.669 ms)
variance introduced by outliers: 14% (moderately inflated)
benchmarking main/large/superbuffer (init=800000 bytes)
time 21.76 ms (21.35 ms .. 22.34 ms)
0.997 R² (0.995 R² .. 0.999 R²)
mean 22.09 ms (21.77 ms .. 22.54 ms)
std dev 858.0 μs (532.7 μs .. 1.421 ms)
variance introduced by outliers: 14% (moderately inflated)
benchmarking main/large/superbuffer (init=1600000 bytes)
time 31.34 ms (30.55 ms .. 32.36 ms)
0.997 R² (0.992 R² .. 0.999 R²)
mean 30.42 ms (29.34 ms .. 31.35 ms)
std dev 2.043 ms (1.383 ms .. 3.018 ms)
variance introduced by outliers: 22% (moderately inflated)
benchmarking main/large/superbuffer (init=20000000 bytes)
time 38.79 ms (36.27 ms .. 41.72 ms)
0.983 R² (0.964 R² .. 0.995 R²)
mean 33.84 ms (32.32 ms .. 35.51 ms)
std dev 3.268 ms (2.816 ms .. 3.755 ms)
variance introduced by outliers: 36% (moderately inflated)
benchmarking main/large/bytestring builder
time 21.89 ms (21.35 ms .. 22.37 ms)
0.998 R² (0.995 R² .. 0.999 R²)
mean 23.02 ms (22.46 ms .. 24.92 ms)
std dev 2.169 ms (461.3 μs .. 4.197 ms)
variance introduced by outliers: 43% (moderately inflated)
benchmarking main/large/bytestring fromChunks
time 21.91 ms (21.49 ms .. 22.31 ms)
0.999 R² (0.998 R² .. 0.999 R²)
mean 22.47 ms (22.13 ms .. 23.63 ms)
std dev 1.230 ms (383.5 μs .. 2.326 ms)
variance introduced by outliers: 19% (moderately inflated)
benchmarking main/large/bytestring concat
time 22.12 ms (21.82 ms .. 22.42 ms)
0.999 R² (0.998 R² .. 1.000 R²)
mean 22.60 ms (22.27 ms .. 23.51 ms)
std dev 1.198 ms (404.6 μs .. 2.244 ms)
variance introduced by outliers: 19% (moderately inflated)