Quasiquoting without bloating

[rrnewton]

Quasiquoting is a compile-time feature, and sometimes can be useful for purposes as modest as multiline strings (see Hackage packages such as interpolatedstring-perl6).

However, I found that using quasiquoting would add about twelve megabytes to final executable size (on 6.12.1 and 6.13.20090607). It seems GHC links all of the libraries used at compile time into the final executable as well. Could GHC prune these?

Note: you can reproduce this effect with a simple hello world program like below:

    import Text.InterpolatedString.Perl6 (q)
    foo :: String
    foo = [$q|
       Well here is a
       multi-line string!
    |]
    main = putStrLn foo

[simonpj]

Indeed. See #4387 for discussion of the issue. Needs thought for 7.2.

Simon

[batterseapower]

#4387 suggests using the GCC module initialisation mechanism to initialise Haskell modules. This would prevent us from dragging in massive transitive module dependencies caused by just module initialisation.

I looked into this and made some discoveries:

• How initialisers are discovered is platform dependent :(
• Looking at the collect2 source code, on some platforms, it seems to happen by collect2 scanning the binary for certain well-known name prefixes (notably GLOBALI_). These are assembled into a constructor list which is walked and initialised by crtbegin.o (generated from GCC's crtstuff.c).
• On OS X, another mechanism seems to be used: initialisation function pointers are put into a MachO section of type mod_init_func_pointers with BFD name DATA.mod_init_func. I presume that the MachO ld then knows to collect all the od_init_func_pointers and generate some code to invoke these guys. On this platform, collect2 does not seem to invoke ld with crtbegin.o and crtend.o added to the link line.

There is some other information here:  http://www.emerson.emory.edu/services/gcc/html/Initialization.html

The most portable thing to do would be to just generate a C file with a function marked with the GCC-specific attribute((constructor)) that deals with initialisation. I think that in order to ensure that this is called before any of the symbols exported by the module proper, this function and the other symbols have to be in the same *object file*, so we need to do some extra work there.

The LLVM backend can skip all this and just set the @llvm.global_ctors list in the output code to contain the module initialisation function. See  http://llvm.org/docs/FAQ.html#iosinit

We can then link everything together with collect2 and win.

[batterseapower]

OK, well I had a crack at this over the weekend, but didn't get it done within my time budget. Maybe someone else can pick up the baton.

I will attach a totally broken partial implementation of the feature. Notes:

• I use the constructor attribute annotated with a *priority* to get some control over the initialisation order
  • hs_init is given a constructor attribute with a low priority 1000, so it is run before all the module initialisers
  • Priority 1500 is given to an initialisation function that does the first half of hs_add_root, setting up a cap and stack for the initalisers to run in
  • Priority 2000 is used for all generated module initialisers, which initialise only themselves (and NOT their dependencies)
  • Priority 2500 is given to a function that does the last half of hs_add_root to clean up after module initialisation
• A destructor attribute is used to ensure that hs_exit is called at shutdown

So far so good, except that:

• OS X GCC 4.2.1 is not recent enough to support constructor priorities (Cygwin GCC 4.3.4 does support them though) - not sure what to do here
• I got totally bogged down in DriverPipeline?.hs getting the _stubs.o object linked in to the corresponding .o. (I generate the constructor attributes for a module's init functions in the stub file). Linking them together into another .o with ld is necessary to ensure that the module initialisers are called if any of the module's exports are referenced.
  • hs_init needs to know the command line arguments, which aren't available to the initialiser functions in the normal way. I wrote some code to get them using the Windows API, but had to resort to a horrible hack for other platforms.

That is basically where the patch is. Because the driver pipeline is broken I haven't been able to give the initialisation strategy any serious testing. In particular, I'm worried that things might go wrong if the RTS assumes that modules are initialised in dependency order.

I think the treatment of foreign export stubs should be rethought in the light of this requirement: we should always generate the stub object in a temporary directory and just ld them together with the .o from the Haskell source to generate the final object. This is actually independently useful as e.g. external Haskell building tools don't have to worry about linking in stub object files themselves.

I do not grok the GHC top level enough to take this further without some serious extra time investment in reading the code.

[batterseapower]

OK, "darcs send" hangs when I try to create the patches (ugghhhh....), so I will attach the output of "darcs diff" instead.

[batterseapower]

Broken quarter-implementation of the feature

[simonmar]

I suggest not making hs_init into a constructor, at least initially. I don't think there will be any problems running the module initialisers before hs_init, since they only need to modify a few pointers to chain cost centres into lists. As you noticed, hs_init needs the command-line arguments, and in the case of a library we certainly don't know what arguments the client wants to use at initialisation time. If we don't have to run hs_init first, then we don't have to worry about constructor priorities too.

I've also been looking into merging the _stub.o files with the .o files, but got sidetracked into a larger refactoring when I noticed that it was going to be really hard to do this in DriverPipeline. I removed GhcMonad from all but the top-level GHC module, which should mean that it will be easier to plumb more information around in DriverPipeline. This patch should hit later today if I can get it validated.

I really hope that compiling these extra C fragments isn't going to add too much to compilation times...

[simonmar]

See #4519 for another report of this.

#3252 is an earlier ticket also asking to get rid of __stginit. Leaving both open because this ticket has lots of useful discussion.

[simonmar]

I'm looking into this.

[simonmar]

Done; see changeset:a52ff7619e8b7d74a9d933d922eeea49f580bca8