integer overflow in rts/RtsUtils:heapOverflow()

[hvr]

When failing with a heap exhaustion, the RTS truncates the reported current maximum heap size modulo 2^{32, e.g.}

$ ghc +RTS -M4G -RTS -e 'sum [1..]'
Heap exhausted;
Current maximum heap size is 0 bytes (0 MB);
use `+RTS -M<size>' to increase it.

This is most probably due to OutOfHeapHook() already being called with the truncated value from heapOverflow():

void
heapOverflow(void)
{
    if (!heap_overflow)
    {
        /* don't fflush(stdout); WORKAROUND bug in Linux glibc */
        OutOfHeapHook(0/*unknown request size*/,
                      RtsFlags.GcFlags.maxHeapSize * BLOCK_SIZE);

        heap_overflow = rtsTrue;
    }
}

which multiplies RtsFlags.GcFlags.maxHeapSize and BLOCK_SIZE whose type are 32-bit unsigned ints, causing the result to be wrapped again into an unsigned int, whereas the result should be upcasted to a long unsigned int (which at least on 64bit archs would be equivalent to a C99 uint64_t)

[hvr]

Unless I've forgotten everything about C99, the following untested(!) modification should fix it:

rts/RtsUtils.c

@@ -130 +130 @@
     {
         /* don't fflush(stdout); WORKAROUND bug in Linux glibc */
         OutOfHeapHook(0/*unknown request size*/,
-                      RtsFlags.GcFlags.maxHeapSize * BLOCK_SIZE);
+                      (lnat)RtsFlags.GcFlags.maxHeapSize * BLOCK_SIZE);
 
         heap_overflow = rtsTrue;
     }

[igloo]

Thanks for the report and patch.

[simonmar]

Fixed (but I got the ticket number wrong in the commit log):

commit bf456a09f9ef68436db48eb5ea25193d3b2f2ed5

Author: Simon Marlow <marlowsd@gmail.com>
Date:   Thu Feb 2 10:28:34 2012 +0000

    avoid 32-bit integer overflow (#5831)

[pcapriotti]

Merged as 11b56ff206bfa99cee641a44d07831914f788c8a.