Dynamically link GHCi on platforms that support it

[simonmar]

In 6.14.1 we should switch to shipping a dynamically linked GHCi binary, on those platforms for which dynamic linking is supported (currently Linux; MacOS X and Windows support is in progress).

Advantages

• The GHCi binary is smaller
• some packages don't need to be loaded on startup: lower memory use
• GHCi startup might be quicker (or it might not)
• some hacks due to having two copies of the base package are not necessary (see rts/Globals.c)
• We might save some space in the distributions.
• It takes us a step closer to not needing the RTS linker at all
• It takes us a step closer to using dynamic linking by default, which is where we want to go ultimately

Potential Issues

• Do we run into any problems with GHCi and the user program sharing the same stdin/stdout/stderr handles? Do we need to virtualise these explicitly in the GHCi front end?

• We cannot revert CAFs in packages that are shared by GHC and the user program. There are some old non-working hacks related to reverting CAFs when GHCi is dynamically linked (see KeepCAFsForGHCi) that need to be cleaned out. CAFs can only be reverted in code loaded by the RTS linker. We need to think about whether this is a necessary feature or not: we have never supported CAF reverting for interpreted code anyway. One reason to have it was so that you can recover after saying getContents at the GHCi prompt, but we can provide other ways to work around that.

• There will be installation/binary-dist issues to resolve; currently we don't install any dynamically-linked binaries.

Open questions

• Whether we continue to use the same binary for GHC and GHCi is an open question: it would be possible to provide a separate statically-linked GHC binary if performance of the dynamically-linked version was an issue.

• We might as well dynamically-link Haddock, and the other tools that come with GHC too.

[simonmar]

Some progress towards this:

Wed Apr 28 21:52:41 BST 2010  Simon Marlow <marlowsd@gmail.com>
  * Add $(GhcDynamic) knob, set to YES to get stage2 linked with -dynamic
  Default currently NO.
  
  Validate passed with GhcDynamic=YES on x86/Linux here.
  
  The compiler is currently slower on x86 when linked -dynamic,
  because the GC inner loop has been adversely affected by -fPIC, I'm
  looking into how to fix it.

The compiler really is slower when linked dynamically, at least on x86, because of the overhead of -fPIC - the GC is significantly slower. I looked into it but didn't find a way to fix it. We might have to ship separate GHC and GHCi binaries on x86; I expect this to be better on x86_64, but haven't measured it yet.

Currently, Cabal doesn't build packages dynamically by default, and this will become an accute issue when GHCi is linked dynamically. We need to do the package system changes needed to register each way separately, so that 'cabal install --enable-dynamic' works properly. Either that or Cabal has to enable dynamic by default.

GHCi is quicker to start up when dynamically linked: 0.1s vs. 0.23s.

[simonpj]

Bumping this to 7.2. We just aren't going to get this done for 7.0.

[simonmar]

(In #781) To fix this we need to use -fPIC by default on x86_64, which is also what we need for #3658. So let's plan to do this in 7.2, when we've thought through the implications of moving to dynamic linking by default.

[simonmar]

This has just become more important since we ran into an arbitrary limit in the OS X object file format:  http://www.haskell.org/pipermail/cvs-ghc/2010-September/055908.html.

Here's a rough plan of action:

• measure the performance impact of dynamic linking. If the impact is too much, fixing that is a blocking issue. We can't ship a separate static GHC binary, because GHC needs to use TH, so GHC itself has to be dynamic.
• make a build system switch to enable dynamic-by-default
• make GHC report whether it is a dynamic-default compiler, in the --info output.
• modify Cabal to use --enable-*-dynamic and --disable-library-vanilla, if GHC is dynamic.
• implement loading of object files by first calling 'gcc -shared' (or equiv. ld command line), and then dlopen(). Measure the performance impact of that. Figure out whether we need to cache the results of gcc -shared or regenerate them into $TMPDIR each time.
• enable dynamic-by-default for some platforms.

• Later: we need to track ways properly in Cabal and the package database.

[simonmar]

Furthermore:

• If the user mentions a .c file on the command line, we have to compile that with -fPIC too. The same goes for our _stub files.

[igloo]

Punting.