name: hsbencher version: 1.3.6 -- CHANGELOG: -- 1.0 : Initial release, new flexible benchmark format. -- 1.1 : Change interface to RunInPlace -- 1.1.1 : add defaultMainModifyConfig -- 1.2 : Significant interface changes. -- 1.2.1 : Hack to shorten tuples, annoying URL length problem. -- 1.3 : breaking change to Config/RunResult types -- 1.3.1 : bugfix -- 1.3.2 : change policy for timeouts and errors -- 1.3.4 : Add ability to prune/select benchmarks with commmand line args -- 1.3.6 : bugfix productivity for timeouts synopsis: Flexible benchmark runner for Haskell and non-Haskell benchmarks. description: Benchmark frameworks are usually very specific to the host language/environment. Hence they are usually about as reusable as compiler passes (that is, not). . Nevertheless, `hsbencher` is an attempt at a reusable benchmark framework. It knows fairly little about what the benchmarks do, and is mostly concerned with defining and iterating through configuration spaces (e.g. varying the number of threads), and managing the data that results. . Benchmark data is stored in simple text files, and optionally uploaded to Google Fusion Tables. -- TODO: Describe clusterbench functionality when it's ready. . `hsbencher` attempts to stradle the divide between language-specific and language-agnostic by having an extensible set of `BuildMethod`s. As shipped, `hsbencher` knows a little about cabal, ghc, and less about Make, but it can be taught more. . The general philosophy is to have benchmarks follow a simple protocol, for example printing out a line "SELFTIMED: 3.3s" if they wish to report their own timing. The focus is on benchmarks that run long enough to run in their own process. This is typical of parallelism benchmarks and different than the fine grained benchmarks that are well supported by "Criterion". . . `hsbencher` is used by creating a script or executable that imports `HSBencher` and provides a list of benchmarks, each of which is decorated with its parameter space. Below is a minimal example that creates a two-configuration parameter space: . @ import HSBencher main = defaultMainWithBechmarks . [ Benchmark \"bench1/bench1.cabal\" [\"1000\"] $ . Or [ Set NoMeaning (RuntimeParam \"+RTS -qa -RTS\") . , Set NoMeaning (RuntimeEnv \"HELLO\" \"yes\") ] ] @ . The output would appear as in this gist: . More examples can be found here: license: BSD3 license-file: LICENSE author: Ryan Newton maintainer: rrnewton@gmail.com copyright: (c) Ryan Newton 2013 category: Development build-type: Simple cabal-version: >=1.10 extra-source-files: README.md example/make_and_ghc/runit.sh example/make_and_ghc/benchmark.hs example/make_and_ghc/bench1/Makefile example/make_and_ghc/bench1/hello.c example/make_and_ghc/bench2/Hello.hs example/cabal/runit.sh example/cabal/benchmark.hs example/cabal/bench1/bench1.cabal example/cabal/bench1/Hello.hs Flag fusion description: Add support for Google Fusion Table upload of benchmark data. default: True Library source-repository head type: git location: https://github.com/rrnewton/HSBenchScaling.git exposed-modules: HSBencher HSBencher.App HSBencher.Types HSBencher.Config HSBencher.Logging HSBencher.Methods HSBencher.Utils HSBencher.MeasureProcess other-modules: Paths_hsbencher build-depends: -- base ==4.6.*, bytestring ==0.10.*, process ==1.1.*, directory ==1.2.*, filepath ==1.3.*, random ==1.0.*, -- unix ==2.6.*, containers ==0.5.*, time ==1.4.*, mtl ==2.1.*, async >= 2.0, base >= 4.5 && <= 4.7, bytestring, process, directory, filepath, random, unix, containers, time, mtl, async, hydra-print >= 0.1.0.3, io-streams >= 1.0, GenericPretty >= 1.2, http-conduit default-language: Haskell2010 if flag(fusion) { build-depends: handa-gdata >= 0.6.2 exposed-modules: HSBencher.Fusion cpp-options: -DFUSION_TABLES } -- [2013.05.28] This will come back later when the new ASCII benchmark file format is finished: ----------------------------------------------------------------------------------------------- -- Executable hsbencher -- main-is: Main.hs -- -- other-modules: -- build-depends: -- -- -- base >= 4.5, bytestring, process, directory, filepath, random, unix, containers, time, mtl, async, -- hydra-print >= 0.1.0.3, io-streams >= 1.0, -- GenericPretty >= 1.2 -- -- -- ghc-options: -threaded -- if flag(fusion) { -- build-depends: handa-gdata >= 0.6.2 -- cpp-options: -DFUSION_TABLES -- } -- default-language: Haskell2010 Test-suite test1 main-is: example/cabal/benchmark.hs type: exitcode-stdio-1.0 build-depends: -- base >= 4.5 && <= 4.7, bytestring, process, directory, filepath, random, unix, containers, time, mtl, async, hydra-print >= 0.1.0.3, io-streams >= 1.0, GenericPretty >= 1.2, http-conduit, hsbencher -- ghc-options: -threaded default-language: Haskell2010 if flag(fusion) { build-depends: handa-gdata >= 0.6.2 cpp-options: -DFUSION_TABLES } Test-suite test2 main-is: example/make_and_ghc/benchmark.hs type: exitcode-stdio-1.0 build-depends: -- base >= 4.5 && <= 4.7, bytestring, process, directory, filepath, random, unix, containers, time, mtl, async, hydra-print >= 0.1.0.3, io-streams >= 1.0, GenericPretty >= 1.2, http-conduit, hsbencher -- ghc-options: -threaded default-language: Haskell2010 if flag(fusion) { build-depends: handa-gdata >= 0.6.2 cpp-options: -DFUSION_TABLES }