GHCi doesn't load weak symbols

[heatsink]

GHCi fails to load modules with weak symbols. The compiler, in contrast, has no trouble with them. The attached Cabal package demonstrates the problem. After building and installing:

:Prelude> :m +WeakTest
Prelude WeakTest> weak_test 0
Loading package WeakTest-0 ... linking ... <interactive>: /home/heatsink/.cabal/lib/WeakTest-0/ghc-6.10.3/HSWeakTest-0.o: unknown symbol `weak_test'

I encountered this problem while trying to build a package that contains C++ code. Because GCC generates weak symbols when compiling C++, libraries that contain C++ code will not work in GHCi. (Granted, this is not the only problem with mixing C++ and Haskell.)

[heatsink]

I investigated the linking rules for weak symbols in ELF,  http://www.skyfree.org/linux/references/ELF_Format.pdf . If I understand correctly, the linking precedence is:

global undefined < weak undefined < weak defined < global defined

The highest-precedence symbol is used for relocation. If it's "weak undefined", then the symbol gets an absolute address of NULL. If "weak defined", an arbitrary definition of the symbol is chosen. This should be do-able when loading a batch of object files.

Because libraries get loaded incrementally, not all situations can be handled properly. If a previously loaded weak symbol would get "promoted" to a higher precedence, GHCi will have to abort, as it currently does for conflicting global definitions.

[igloo]

Thanks for the report

[guest]

I encountered the same problem with LLVM recently. LLVM is also a C++ library with C interface. I did the following hack, without knowing what I'm really doing. I needed to build shared object files of the LLVM libraries anyway, in order to use them in GHCi. Then I built a big .so file containing all libraries (.a) and object files (.o) of LLVM. There the symbols seems to have been resolved.

# get many small object (.o) files out of the libraries (.a)
for src in /usr/lib/llvm/*.a; do ar -x $src ; done
# assemble those object files and the other ones of LLVM project to a big shared object file (.so)
gcc -shared -Wl,-soname,libLLVM.so.2.5 -o libLLVM.so.2.5 /usr/lib/llvm/LLVM*.o *.o
# provide the file with default name
ln -s libLLVM.so.2.5 libLLVM.so
# remove interim object files
rm *.o

I also had to replace the linker options -lLLVMSupport -lLLVMCore and so on by -lLLVM (the monolithic so-file) in the fields extraLibraries and ldOptions ~/.ghc/i386-linux-6.10.4/package.conf for the Haskell package 'llvm'.

I have no idea whether this hack would work for other libraries.

Btw. If you want to use LLVM in GHCi you may also have to respect the linker script problem described in #2615 and remove the pthread dependency in package.conf, too. With all those steps I could use LLVM in GHCi, but after a :reload, running an LLVM function causes GHCi to quit with:

ghci: JITEmitter.cpp:110: <unnamed>::JITResolver::JITResolver(llvm::JIT&): Assertion `TheJITResolver == 0 && "Multiple JIT resolvers?"' failed.

[hgolden]

Replying to heatsink:

GHCi fails to load modules with weak symbols.

I want to clarify the current situation. GHCi's linker loads modules with weak symbols. However, it doesn't put the weak symbols into the symbol table.

It would be easy to add weak symbols to the symbol table, but I'm not sure what the correct semantics should be. Perhaps weak symbols should be subordinated to global (STB_GLOBAL) and local (STB_LOCAL) symbols. Is this sufficient? (I haven't studied the documentation yet.)

Question: Is it possible to have the same weak symbol defined in more than one module? If so, how does the linker choose?

[heatsink]

It would be easy to add weak symbols to the symbol table, but I'm not sure what the correct semantics should be. Perhaps weak symbols should be subordinated to global (STB_GLOBAL) and local (STB_LOCAL) symbols. Is this sufficient? (I haven't studied the documentation yet.)

Weak symbols are never unified with local symbols. Probably it's an error to have a weak and a local symbol with the same name in the same file. I don't know what this means in terms of GHC's linker.

Question: Is it possible to have the same weak symbol defined in more than one module? If so, how does the linker choose?

Yes. AFAICT the choice is arbitrary. GCC's linker's choice depends on the order that files are listed on the command line. It is at least necessary to consider SHN_UNDEF when selecting a weak symbol (defined weak symbols take precedence over undefined weak symbols).

The following statement from the ELF document appears to be relevant, but I don't know what extracting archive members has to do with symbol resolution.

The link editor does not extract archive members to resolve undefined weak symbols.

[hgolden]

Replying to heatsink:

It would be easy to add weak symbols to the symbol table, but I'm not sure what the correct semantics should be. Perhaps weak symbols should be subordinated to global (STB_GLOBAL) and local (STB_LOCAL) symbols. Is this sufficient? (I haven't studied the documentation yet.)

Weak symbols are never unified with local symbols. Probably it's an error to have a weak and a local symbol with the same name in the same file. I don't know what this means in terms of GHC's linker.

I studied the documentation after writing my previous message. Local symbols stand alone. Global and weak symbols are in the same namespace, but global symbols take priority.

Question: Is it possible to have the same weak symbol defined in more than one module? If so, how does the linker choose?

Yes. AFAICT the choice is arbitrary. GCC's linker's choice depends on the order that files are listed on the command line. It is at least necessary to consider SHN_UNDEF when selecting a weak symbol (defined weak symbols take precedence over undefined weak symbols).

OK. Then I think the following approach would work:

1. Add local, global and the first occurrence of a weak symbol to the symbol table (if the global symbol doesn't exist already). It is allowable to have all 3 types of a single name in the symbol table.

2. When a local symbol is called for, return that. When a global symbol is called for, return the global symbol if it exists. If a global symbol doesn't exist, return a weak symbol if it exists.

I believe this can be implemented in the rts/Linker.c in a straightforward way. I will start working on this. If you disagree with my analysis of the semantics, please let me know what you want instead.

The following statement from the ELF document appears to be relevant, but I don't know what extracting archive members has to do with symbol resolution.

The link editor does not extract archive members to resolve undefined weak symbols.

I don't think this is relevant for the GHCi linker, since it only loads modules on demand and doesn't try to resolve undefined symbols at all.

[heatsink]

Replying to hgolden:

OK. Then I think the following approach would work: 1. Add local, global and the first occurrence of a weak symbol to the symbol table (if the global symbol doesn't exist already). It is allowable to have all 3 types of a single name in the symbol table. 2. When a local symbol is called for, return that. When a global symbol is called for, return the global symbol if it exists. If a global symbol doesn't exist, return a weak symbol if it exists.

There are three cases when a global symbol is called for.

1. If the global symbol is defined, return it.
2. Otherwise, if a weak symbol is defined in the table, return it.
3. Otherwise, if a weak symbol is in the table and marked "undefined", return NULL.
4. Otherwise, report an error because the symbol is undefined.

Weak undefined symbols can satisfy a need for a symbol, in contrast to global undefined symbols which cannot. Weak undefined symbols are sometimes used in the C/C++ domain as a way of allowing optional hooks or callbacks.

I believe this can be implemented in the rts/Linker.c in a straightforward way. I will start working on this. If you disagree with my analysis of the semantics, please let me know what you want instead.

There is one complication due to incremental linking. We could have a situation where a symbol's value changes as a result of module imports:

• load A.o which uses X (global undefined)
  and B.o which defines X = 1 (weak defined)
• call a function in A, which reveals that X has the value 1
• load C.o which defines X = 2 (global defined)
  and D.o which uses X (global undefined)
  Although X had the value 1 before, the global definition takes precedence over the weak definition so it now has the value 2
• call a function in D, which reveals that X has the value 2
• call a function in A, which reveals that X has the value 1 (inconsistent) or 2 (not referentially transparent)

I think that to avoid problems, it has to be detected whenever loading new modules would change a symbol's value. GHCi already detects redefined global symbols, so hopefully you can reuse that.

[hgolden]

Replying to heatsink:

There is one complication due to incremental linking. We could have a situation where a symbol's value changes as a result of module imports: * load A.o which uses X (global undefined)
and B.o which defines X = 1 (weak defined) * call a function in A, which reveals that X has the value 1 * load C.o which defines X = 2 (global defined)
and D.o which uses X (global undefined)
Although X had the value 1 before, the global definition takes precedence over the weak definition so it now has the value 2 * call a function in D, which reveals that X has the value 2 * call a function in A, which reveals that X has the value 1 (inconsistent) or 2 (not referentially transparent) I think that to avoid problems, it has to be detected whenever loading new modules would change a symbol's value. GHCi already detects redefined global symbols, so hopefully you can reuse that.

Here's my approach to that:

1. (Currently) redefining a global symbol is an error (that terminates the program). This wouldn't change. 2. When a weak symbol is defined and there is no defined global symbol, save the first defined weak symbol. (In other words, only save the first weak definition, if at all.) 3. When a weak symbol is "resolved" (actually called for in another module), it becomes a global symbol. The reason for this is that case 1 should follow if there is any global symbol defined later. This avoids the problems you have noted above. 4. If the weak symbol hasn't been "upgraded" to a global symbol by case 3, it will be superseded by a global symbol definition.

This approach is somewhat conservative, since a weak definition that hasn't been "used" (i.e., called or accessed), as opposed to "resolved," could be replaced by a global definition. However, to implement this would require some method of determining the first use of a symbol. (Perhaps this could be done, but I don't plan to implement it at present.)

To implement this, only one version of a global/weak symbol needs to be saved in the defined symbol table. (This revises my earlier statement that global and weak symbols with the same name would need to co-exist in the symbol table.)

[heatsink]

That resolution method [10] looks good to me.

[igloo]

See also #5435.

[akio]

I implemented a support for ELF weak symbols on Linux. The implementation follows hgolden's design above, except that it doesn't support undefined weak symbols. The change is implemented in 3 patches: the first one simplifies the code, the second one implements the support, and the third one adds a special symbol __dso_handle. This is not directly related to weak symbols, but is required to load C++ object files.

[hgolden]

Replying to akio:

I implemented a support for ELF weak symbols on Linux. The implementation follows hgolden's design above, except that it doesn't support undefined weak symbols. The change is implemented in 3 patches: the first one simplifies the code, the second one implements the support, and the third one adds a special symbol __dso_handle. This is not directly related to weak symbols, but is required to load C++ object files.

akio, it's great that you have done this, since I let it lie fallow. I have reassigned the ownership to you. You should also generate a test for your patch so it can be merged into the HEAD branch.

[akio]

Patch to the test suite

[akio]

The patches validate together on my Linux machine.

[batterseapower]

I got bit by this as well. My charsetdetect library (which links against a C++ library) won't load into ghci because it can't find dso_handle.

[akio]

Replying to igloo: Why is this blocked by #3658? Isn't the ability to load a static C++ object file independent to the ability to use the system linker?

[igloo]

Replying to akio:

Replying to igloo: Why is this blocked by #3658? Isn't the ability to load a static C++ object file independent to the ability to use the system linker?

We could fix it in our own linker, but I think it would be a waste of effort when we're removing our linker and using the system linker instead.

In HEAD we're already using the system linker on Elf platforms by default (except for FreeBSD, due to a bug in their system linker, but even there we will switch to the system linker once 7.8 is branched).