Context Navigation

← Previous Ticket
Next Ticket →

Ticket #2013 (closed bug: duplicate)

Opened 5 years ago

Last modified 5 years ago

ghci crash on startup: R_X86_64_32S relocation out of range.

Reported by:	mboes	Owned by:	simonmar
Priority:	high	Milestone:	6.10.1
Component:	GHCi	Version:	6.9
Keywords:		Cc:
Operating System:	FreeBSD	Architecture:	x86_64 (amd64)
Type of failure:		Difficulty:	Unknown
Test Case:		Blocked By:
Blocking:		Related Tickets:

Description

On freebsd-7.0 I have a working ghc-6.8.2 bootstrapped from a ghc-6.6.1 itself boostrapped from a 32-bit machine. I'd like to create a bindist for this platform, for which there are as yet no available binary distribution of ghc, but before that it would be nice if ghci worked. Currently I get the following error:

mboes@caracas:~/ > ghci
GHCi, version 6.8.2: http://www.haskell.org/ghc/  :? for help
ghc-6.8.2: internal error: R_X86_64_32S relocation out of range: (noname) = 0x802d779c0

    (GHC version 6.8.2 for x86_64_unknown_freebsd)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug
[1]    21394 abort      ghci

This bug has also been spotted before by Tim Newsham on ghc-6.8.1 on Freebsd-amd64 as well. Could this be related to #1562 or #1073 on Linux?

Attachments

linker-freebsd-amd64.patch (4.0 KB) - added by mboes 5 years ago.
2013.patch (74.6 KB) - added by simonmar 5 years ago.: new patch

Change History

Changed 5 years ago by mboes

Debug output of

./stage2/ghc-inplace --interactive +RTS -Dl 2>&1 | tee log

is at http://code.haskell.org/~mboes/log.bz2 (log too big for attachment on trac).

Note that FreeBSD, just like OpenBSD, does not define MAP_32BIT and MAP_ANONYMOUS. I see in Linker.c the following #define:

#if defined(x86_64_HOST_ARCH) && defined(MAP_32BIT)
#define EXTRA_MAP_FLAGS MAP_32BIT
#else
#define EXTRA_MAP_FLAGS 0
#endif

But then I see an attempt to resolve symbols allocated in high memory using a bounce sequence:

// On x86_64, 32-bit relocations are often used, which requires that
// we can resolve a symbol to a 32-bit offset.  However, shared
// libraries are placed outside the 2Gb area, which leaves us with a
// problem when we need to give a 32-bit offset to a symbol in a
// shared library.
// 
// For a function symbol, we can allocate a bounce sequence inside the
// 2Gb area and resolve the symbol to this.  The bounce sequence is
// simply a long jump instruction to the real location of the symbol.
//
// For data references, we're screwed.
//
...
static void*
x86_64_high_symbol( char *lbl, void *addr )
{
    x86_64_bounce *bounce;

    if ( x86_64_bounce_buffer == NULL || 
         x86_64_bb_next_off >= X86_64_BB_SIZE ) {
        x86_64_bounce_buffer = 
            mmap(NULL, X86_64_BB_SIZE * sizeof(x86_64_bounce), 
                 PROT_EXEC|PROT_READ|PROT_WRITE, 
                 MAP_PRIVATE|EXTRA_MAP_FLAGS|MAP_ANONYMOUS, -1, 0);
        if (x86_64_bounce_buffer == MAP_FAILED) {
            barf("x86_64_high_symbol: mmap failed");
        }
        x86_64_bb_next_off = 0;
    }
...

Since there is no MAP_32BIT on FreeBSD it would seem the above bounce sequence can get arbitrarily allocated in high memory. Could this be the problem? I'm way out of my depth here, so please forgive the daftness of this question, but on my system there are no 32bit libraries at all, only 64bit ones. Hence, could we do away with 32bit relocations entirely?

Hope this helps,

Mathieu

Changed 5 years ago by simonmar

See also Greg Wright's email in which he mentions exactly this problem: http://www.haskell.org/pipermail/glasgow-haskell-users/2007-April/012291.html

In the HEAD, the fragment of code mentioned above is now gone, in favour of a more general mechanism. You want the patch entitled "add PIC relocations for x86_64, and use a simpler hack in place of x86_64_high_symbol()" from HEAD.

However, this on its own won't be enough: when we allocate memory for objects with mmap(), we need to get memory in the low 2Gb, which is why we use MAP_32BIT on Linux. I'm not sure what the right way to do this on *BSD is, but you could try adding a hint address to the mmap() call, something in the low 2Gb, as Greg suggested in his mail above.

Changed 5 years ago by mboes

version changed from 6.8.1 to 6.9

The problem actually occurs before x86_64_high_symbol gets called. It occurs because in loadObj() oc->image gets mmapped elsewhere than in the low 2Gb to start with, causing do_Elf_Rela_relocations() to fail. Unfortunately I can't convince FreeBSD to mmap in the low 2Gb, even with a hint address.

The patch above looks like a good step forward, because as I understand the R_X86_64_PC32S and friend ELF relocations disappear by virtue of using PIC code. Trouble is though that ocAllocateSymbolExtras() won't compile on FreeBSD due to the use of the mremap() linuxism. mremap() doesn't exist on FreeBSD. This would have to be replaced by a pair of munmap(); mmap() calls, along with passing in the fd of the file being loaded to ocAllocateSymbolExtras(). I tried the simpler approach of #undef USE_MMAP and hence using stgReallocBytes() to map in the object instead and space for the GOT entries, along with a call to mprotect() to set the region executable. This however yields limited success: base links in fine (what's more in the low 2Gb!) but I get a segfault shortly after that:

$ ./stage2/ghc-inplace +RTS -Di -RTS --interactive
GHCi, version 6.9.20071215: http://www.haskell.org/ghc/  :? for help
Loading package base ... linking ... done.
Sp = 0x8029f4ef8   pc = 1      PUSH_G   0x802f7fff8
Sp = 0x8029f4ef0   pc = 3      PUSH_G   0x802f859c8
Sp = 0x8029f4ee8   pc = 5      PACK      2 words with itbl 0x802d48258
	Built Object 0x802a7d000 = CONSTR(0x802d48258(tag=1), 0x802f859c8, 0x802f7fff8)
Sp = 0x8029f4ef0   pc = 8      PUSH_L   0
Sp = 0x8029f4ee8   pc = 10      PUSH_APPLY_P
Sp = 0x8029f4ee0   pc = 11      PUSH_G   0x802f8a498
Sp = 0x8029f4ed8   pc = 13      SLIDE     3 down by 1
Sp = 0x8029f4ee0   pc = 16      ENTER

---------------------------------------------------------------
Evaluating: Object 0x802f8a498 = FUN/2(0x802db3618)
Sp = 0x8029f4ee8

RET_SMALL (0x1374330)
   stk[34] (0x8029f4ef0) = 0x802a7d000
Object 0x8029f4ef8 = UPDATE_FRAME(0x1372400,0x8026eb470)
RET_SMALL (0x1372570)
RET_SMALL (0x12af7a0)
   stk[29] (0x8029f4f18) = 0x8026eb651
RET_SMALL (0x12af7a0)
   stk[27] (0x8029f4f28) = 0x80292a6f1
RET_SMALL (0x12af7a0)
   stk[25] (0x8029f4f38) = 0x80292a701
RET_SMALL (0x12af7a0)
   stk[23] (0x8029f4f48) = 0x80292a711
RET_SMALL (0x136a810)
Object 0x8029f4f58 = CATCH_FRAME(0x136ac10,0x80292a722)
RET_SMALL (0x12d1fe8)
   stk[17] (0x8029f4f78) = 0x80292a730
RET_SMALL (0x136a6b8)
Object 0x8029f4f88 = CATCH_FRAME(0x136ac10,0x80292a749)
Object 0x8029f4fa0 = CATCH_FRAME(0x136ac10,0x80292a761)
Object 0x8029f4fb8 = CATCH_FRAME(0x136ac10,0x80292a779)
RET_SMALL (0x12c8dd8)
Object 0x8029f4fd8 = CATCH_FRAME(0x136ac10,0x16f8f30)
RET_SMALL (0x1370fe0)
Object 0x8029f4ff8 = STOP_FRAME(0x1370a90)



---------------------------------------------------------------
Returning: Object 0x802f8a498 = FUN/2(0x802db3618)
Sp = 0x8029f4ee8

RET_SMALL (0x1374330)
   stk[34] (0x8029f4ef0) = 0x802a7d000
Object 0x8029f4ef8 = UPDATE_FRAME(0x1372400,0x8026eb470)
RET_SMALL (0x1372570)
RET_SMALL (0x12af7a0)
   stk[29] (0x8029f4f18) = 0x8026eb651
RET_SMALL (0x12af7a0)
   stk[27] (0x8029f4f28) = 0x80292a6f1
RET_SMALL (0x12af7a0)
   stk[25] (0x8029f4f38) = 0x80292a701
RET_SMALL (0x12af7a0)
   stk[23] (0x8029f4f48) = 0x80292a711
RET_SMALL (0x136a810)
Object 0x8029f4f58 = CATCH_FRAME(0x136ac10,0x80292a722)
RET_SMALL (0x12d1fe8)
   stk[17] (0x8029f4f78) = 0x80292a730
RET_SMALL (0x136a6b8)
Object 0x8029f4f88 = CATCH_FRAME(0x136ac10,0x80292a749)
Object 0x8029f4fa0 = CATCH_FRAME(0x136ac10,0x80292a761)
Object 0x8029f4fb8 = CATCH_FRAME(0x136ac10,0x80292a779)
RET_SMALL (0x12c8dd8)
Object 0x8029f4fd8 = CATCH_FRAME(0x136ac10,0x16f8f30)
RET_SMALL (0x1370fe0)
Object 0x8029f4ff8 = STOP_FRAME(0x1370a90)


Sp = 0x8029f4ef8   pc = 1      PUSH_G   0x802f7fff8
Sp = 0x8029f4ef0   pc = 3      PUSH_G   0x802f85a40
Sp = 0x8029f4ee8   pc = 5      PACK      2 words with itbl 0x802d48258
	Built Object 0x802a3c000 = CONSTR(0x802d48258(tag=1), 0x802f85a40, 0x802f7fff8)
Sp = 0x8029f4ef0   pc = 8      PUSH_L   0
Sp = 0x8029f4ee8   pc = 10      PUSH_APPLY_P
Sp = 0x8029f4ee0   pc = 11      PUSH_G   0x802f8a498
Sp = 0x8029f4ed8   pc = 13      SLIDE     3 down by 1
Sp = 0x8029f4ee0   pc = 16      ENTER

---------------------------------------------------------------
Evaluating: Object 0x802f8a498 = FUN/2(0x802db3618)
Sp = 0x8029f4ee8

RET_SMALL (0x1374330)
   stk[34] (0x8029f4ef0) = 0x802a3c000
Object 0x8029f4ef8 = UPDATE_FRAME(0x1372400,0x802693bb8)
RET_SMALL (0x1372570)
RET_SMALL (0x12af7a0)
   stk[29] (0x8029f4f18) = 0x802693d99
RET_SMALL (0x12af7a0)
   stk[27] (0x8029f4f28) = 0x802a3a641
RET_SMALL (0x12af7a0)
   stk[25] (0x8029f4f38) = 0x80292a701
RET_SMALL (0x12af7a0)
   stk[23] (0x8029f4f48) = 0x80292a711
RET_SMALL (0x136a810)
Object 0x8029f4f58 = CATCH_FRAME(0x136ac10,0x80292a722)
RET_SMALL (0x12d1fe8)
   stk[17] (0x8029f4f78) = 0x80292a730
RET_SMALL (0x136a6b8)
Object 0x8029f4f88 = CATCH_FRAME(0x136ac10,0x80292a749)
Object 0x8029f4fa0 = CATCH_FRAME(0x136ac10,0x80292a761)
Object 0x8029f4fb8 = CATCH_FRAME(0x136ac10,0x80292a779)
RET_SMALL (0x12c8dd8)
Object 0x8029f4fd8 = CATCH_FRAME(0x136ac10,0x16f8f30)
RET_SMALL (0x1370fe0)
Object 0x8029f4ff8 = STOP_FRAME(0x1370a90)



---------------------------------------------------------------
Returning: Object 0x802f8a498 = FUN/2(0x802db3618)
Sp = 0x8029f4ee8

RET_SMALL (0x1374330)
   stk[34] (0x8029f4ef0) = 0x802a3c000
Object 0x8029f4ef8 = UPDATE_FRAME(0x1372400,0x802693bb8)
RET_SMALL (0x1372570)
RET_SMALL (0x12af7a0)
   stk[29] (0x8029f4f18) = 0x802693d99
RET_SMALL (0x12af7a0)
   stk[27] (0x8029f4f28) = 0x802a3a641
RET_SMALL (0x12af7a0)
   stk[25] (0x8029f4f38) = 0x80292a701
RET_SMALL (0x12af7a0)
   stk[23] (0x8029f4f48) = 0x80292a711
RET_SMALL (0x136a810)
Object 0x8029f4f58 = CATCH_FRAME(0x136ac10,0x80292a722)
RET_SMALL (0x12d1fe8)
   stk[17] (0x8029f4f78) = 0x80292a730
RET_SMALL (0x136a6b8)
Object 0x8029f4f88 = CATCH_FRAME(0x136ac10,0x80292a749)
Object 0x8029f4fa0 = CATCH_FRAME(0x136ac10,0x80292a761)
Object 0x8029f4fb8 = CATCH_FRAME(0x136ac10,0x80292a779)
RET_SMALL (0x12c8dd8)
Object 0x8029f4fd8 = CATCH_FRAME(0x136ac10,0x16f8f30)
RET_SMALL (0x1370fe0)
Object 0x8029f4ff8 = STOP_FRAME(0x1370a90)


Sp = 0x8029f4ef8   pc = 1      PUSH_G   0x802f7fff8
Sp = 0x8029f4ef0   pc = 3      PUSH_G   0x802f85960
Sp = 0x8029f4ee8   pc = 5      PACK      2 words with itbl 0x802d48258
	Built Object 0x802a40000 = CONSTR(0x802d48258(tag=1), 0x802f85960, 0x802f7fff8)
Sp = 0x8029f4ef0   pc = 8      PUSH_L   0
Sp = 0x8029f4ee8   pc = 10      PUSH_APPLY_P
Sp = 0x8029f4ee0   pc = 11      PUSH_G   0x802f8a498
Sp = 0x8029f4ed8   pc = 13      SLIDE     3 down by 1
Sp = 0x8029f4ee0   pc = 16      ENTER

---------------------------------------------------------------
Evaluating: Object 0x802f8a498 = FUN/2(0x802db3618)
Sp = 0x8029f4ee8

RET_SMALL (0x1374330)
   stk[34] (0x8029f4ef0) = 0x802a40000
Object 0x8029f4ef8 = UPDATE_FRAME(0x1372400,0x8026bdca0)
RET_SMALL (0x1372570)
RET_SMALL (0x12af7a0)
   stk[29] (0x8029f4f18) = 0x8026bde81
RET_SMALL (0x12af7a0)
   stk[27] (0x8029f4f28) = 0x802a3f2b1
RET_SMALL (0x12af7a0)
   stk[25] (0x8029f4f38) = 0x80292a701
RET_SMALL (0x12af7a0)
   stk[23] (0x8029f4f48) = 0x80292a711
RET_SMALL (0x136a810)
Object 0x8029f4f58 = CATCH_FRAME(0x136ac10,0x80292a722)
RET_SMALL (0x12d1fe8)
   stk[17] (0x8029f4f78) = 0x80292a730
RET_SMALL (0x136a6b8)
Object 0x8029f4f88 = CATCH_FRAME(0x136ac10,0x80292a749)
Object 0x8029f4fa0 = CATCH_FRAME(0x136ac10,0x80292a761)
Object 0x8029f4fb8 = CATCH_FRAME(0x136ac10,0x80292a779)
RET_SMALL (0x12c8dd8)
Object 0x8029f4fd8 = CATCH_FRAME(0x136ac10,0x16f8f30)
RET_SMALL (0x1370fe0)
Object 0x8029f4ff8 = STOP_FRAME(0x1370a90)



---------------------------------------------------------------
Returning: Object 0x802f8a498 = FUN/2(0x802db3618)
Sp = 0x8029f4ee8

RET_SMALL (0x1374330)
   stk[34] (0x8029f4ef0) = 0x802a40000
Object 0x8029f4ef8 = UPDATE_FRAME(0x1372400,0x8026bdca0)
RET_SMALL (0x1372570)
RET_SMALL (0x12af7a0)
   stk[29] (0x8029f4f18) = 0x8026bde81
RET_SMALL (0x12af7a0)
   stk[27] (0x8029f4f28) = 0x802a3f2b1
RET_SMALL (0x12af7a0)
   stk[25] (0x8029f4f38) = 0x80292a701
RET_SMALL (0x12af7a0)
   stk[23] (0x8029f4f48) = 0x80292a711
RET_SMALL (0x136a810)
Object 0x8029f4f58 = CATCH_FRAME(0x136ac10,0x80292a722)
RET_SMALL (0x12d1fe8)
   stk[17] (0x8029f4f78) = 0x80292a730
RET_SMALL (0x136a6b8)
Object 0x8029f4f88 = CATCH_FRAME(0x136ac10,0x80292a749)
Object 0x8029f4fa0 = CATCH_FRAME(0x136ac10,0x80292a761)
Object 0x8029f4fb8 = CATCH_FRAME(0x136ac10,0x80292a779)
RET_SMALL (0x12c8dd8)
Object 0x8029f4fd8 = CATCH_FRAME(0x136ac10,0x16f8f30)
RET_SMALL (0x1370fe0)
Object 0x8029f4ff8 = STOP_FRAME(0x1370a90)

[1]    60850 segmentation fault  ./stage2/ghc-inplace --interactive +RTS -Di -RTS

Could this due to stgRealloc'ed region being still non-executable, despite the mprotect() call? Should I instead stick to mmap()'ing the object and work out some of avoiding mremap()?

Many thanks,

Mathieu

Changed 5 years ago by simonmar

Setting milestone to 6.8.3 as we would like GHCi to work properly on FreeBSD/x86-64.

I don't have any good suggestions yet. Object code that is not compiled with -fPIC must be all linked together in the same 2Gb segment of the address space, because 32-bit relative addressing is used throughout. We use the small memory model by default, like gcc - I haven't investigated using larger memory models, but in any case -fPIC would be better than moving to another memory model as it is already implemented.

Unfortunately -fPIC doesn't really solve the problem - even objects compiled with -fPIC assume that other objects in the same package are within the same 2Gb segment. So we really do need a way to map memory in the low 2Gb. As far as I can tell, the only way to do this when MAP_32BIT is not supported is with the hint address - perhaps look at the memory map of the running process, and figure out a suitable place to request memory? It might be necessary to coordinate with MBlock.c to avoid requesting memory in the low 2Gb when we don't need it.

follow-up: ↓ 9 Changed 5 years ago by mboes

Good news: I have a working ghci loading base and giving me a prompt! This is done by adding MAP_FIXED to the EXTRA_MAP_FLAGS and giving a hint address, both in loadObj() and in x64_64_high_symbol(). Of course, ghci segfaults the minute you try to link in more packages, because new objects will overwrite the old ones in memory at the fixed address.

The upshot is that forcing 32-bit clean mmap()'s is the only issue needing solving for ghci on FreeBSD/amd64. All the bugs I used to see with 6.6.1 (eg. race conditions with writing characters to the screen) are now gone. The trick to solve this issue is to calculate a valid place in memory to mmap() objects in, large enough for the whole object, that does not conflict with anything else living in memory. Simon, any suggestions as to the best way to do that?

At the moment I'm loading the object at the 1Gb boundary, and looking around on the web that's the way that Xorg project are working around their need for a MAP_32BIT flag on NetBSD. It's an ugly hack, but I don't see any other way of dealing with a lack of MAP_32BIT. We could increment the load address by oc->fileSize + <space for jump islands> at each object load, so that all objects are somewhere above the 1Gb mark. Then create jump islands for each loaded object directly after it in memory. However, if we're loading so much in memory at fixed locations we'd have to be very sure that nothing else is allocated above the 1Gb mark before the linking. How feasible is this?

in reply to: ↑ 6 Changed 5 years ago by simonmar

Replying to wb.kloke:

Is it feasible to link packages like base statically to ghci to get ghci working for simple programs not loading modules? Or else, as base is already wired-in, is the loading of base necessary at all?

With static linking, the short answer is no (we have investigated various strategies and not found a way to do this reliably). The right way is to use dynamic linking, where the base package is a shared library, and can be used by both GHCi itself and the running program. Dynamic linking is mostly working, we're just fixing issues with installation and distributions.

in reply to: ↑ 7 Changed 5 years ago by simonmar

Replying to mboes:

Good news: I have a working ghci loading base and giving me a prompt! This is done by adding MAP_FIXED to the EXTRA_MAP_FLAGS and giving a hint address, both in loadObj() and in x64_64_high_symbol(). Of course, ghci segfaults the minute you try to link in more packages, because new objects will overwrite the old ones in memory at the fixed address. The upshot is that forcing 32-bit clean mmap()'s is the only issue needing solving for ghci on FreeBSD/amd64. All the bugs I used to see with 6.6.1 (eg. race conditions with writing characters to the screen) are now gone. The trick to solve this issue is to calculate a valid place in memory to mmap() objects in, large enough for the whole object, that does not conflict with anything else living in memory. Simon, any suggestions as to the best way to do that? At the moment I'm loading the object at the 1Gb boundary, and looking around on the web that's the way that Xorg project are working around their need for a MAP_32BIT flag on NetBSD. It's an ugly hack, but I don't see any other way of dealing with a lack of MAP_32BIT. We could increment the load address by oc->fileSize + <space for jump islands> at each object load, so that all objects are somewhere above the 1Gb mark. Then create jump islands for each loaded object directly after it in memory. However, if we're loading so much in memory at fixed locations we'd have to be very sure that nothing else is allocated above the 1Gb mark before the linking. How feasible is this?

This is encouraging. It looks like the solution is going to be dependent on the layout of memory on FreeBSD and hence might be fragile to future changes in that area, but it's better than nothing.

You want to look at the memory map of the running process to figure out where the best place to allocate memory is. On Linux the memory map is available in /proc/<pid>/maps for process <pid>.

Note that MBlock.c also allocates memory using mmap, you probably want to check that it isn't accidentally grabbing some of the sub-2Gb memory which would cause problems for the linker.

Changed 5 years ago by mboes

Hi,

I have a short patch for this, attached. I have a rather slow connection at the moment and downloading the stable darcs branch is going to be taking a few days at this rate so the patch is a unified diff relative to 6.8.2 release, rather than a darcs patch. Sorry about that!

On my system ghci never seems to be loaded with anything allocated in the 1-2GB segment. This patch makes ghci load all objects starting from 1GB growing upwards, one after the other. Loading will fail if there is anything in the way, but I haven't seen this to be a problem on my machine and besides linking happens very early in the lifetime of a ghci process. The jump islands for symbols in shared libs loaded in high mem are allocated starting from the 2GB mark, growing downwards. Let me know if you'd like to see any revisions to this patch.

Regards,

Mathieu

Note: See TracTickets for help on using tickets.

Download in other formats:

Context Navigation

Ticket #2013 (closed bug: duplicate)

ghci crash on startup: R_X86_64_32S relocation out of range.

Description

Attachments

Change History

Changed 5 years ago by simonmar

Changed 5 years ago by mboes

Changed 5 years ago by simonmar

Changed 5 years ago by mboes

Changed 5 years ago by simonmar

follow-up: ↓ 8 Changed 5 years ago by wb.kloke

follow-up: ↓ 9 Changed 5 years ago by mboes

in reply to: ↑ 6 Changed 5 years ago by simonmar

in reply to: ↑ 7 Changed 5 years ago by simonmar

Changed 5 years ago by mboes

Changed 5 years ago by mboes

Changed 5 years ago by simonmar

Changed 5 years ago by wb.kloke

Changed 5 years ago by simonmar

Changed 5 years ago by igloo

Changed 5 years ago by simonmar

Changed 5 years ago by simonmar

Changed 5 years ago by simonmar

Changed 5 years ago by simonmar

Changed 5 years ago by igloo

Changed 5 years ago by simonmar

Download in other formats: