Build multiple Cabal packages in parallel

[tibbe]

Many developers have multi-core machines but Cabal runs the build process in a single thread, only making use of one core. If the build process could be parallelized build times could be cut by perhaps a factor of 2-8, depending on the number of cores and opportunity of parallel execution available.

Task: Add support for building packages in parallel, via a set of command line flags (e.g. -j).

Sub-tasks:

• Keep a queue of available build tasks and schedule the tasks on multiple threads, while respecting task dependencies. Cabal already has an idea of what needs to be built in which order so you don't need to start from scratch.

• Adapt the program output (including logging) system so that parallel builds don't generate garbled output.

See this ticket for more information:  http://hackage.haskell.org/trac/hackage/ticket/447

Interested Student

• Mikhail Glushenkov <the.dead.shall.rise@…>
• Marco Túlio Gontijo e Silva <marcot@…>

[SamAnklesaria]

This project seems relatively modest for the several month period available. Could the task be generalized not only to parallelize cabal-install, but Cabal's build process itself as well?

[duncan]

Replying to SamAnklesaria:

This project seems relatively modest for the several month period available.

Yes, it is, we should throw in some related things.

Could the task be generalized not only to parallelize cabal-install, but Cabal's build process itself as well?

We did a GSoC project for that a couple years ago. It turned out to be bigger than a single summer's work.

[gregweber]

There was already a GSoC for this. Hackage ticket has patches waiting. Can we close this proposal?

[refold]

This is already done.