Incremental heap census

[ezyang]

At the moment, a heap census forces a major garbage collection, because a heap census is always traverses the entire heap, and the only time when there is no slop is right after a major GC. What would be nice is if we only carried out heap census on portions of the heap which were *immediately* GC'd; i.e. an incremental census.

The question, then, is how do we reconstruct full heap state from only a partial census? In particular, how do we know if an object that was previously present got collected or got promoted? If we always assume that it got collected (the naive implementation), a promoted object will disappear from the census, and only mysteriously reappear when we do a full census. The trick we want to employ is to save the starting pointer of all of the generations: while the entire generation is not slop free, the remainder of the first block and all of the later blocks are. I believe this is sufficient to reconstruct full heap state.

#4225 has related discussion.

[ezyang]

It appears to be a little more complicated than this: due to the fact that GHC will reuse todo blocks over minor collections, the first portion of a block will have slop, and the remainder of it will not.

[ezyang]

OK, so I have an implementation which adds a new field to block descriptors, "slopend", which records the division between sloppy / packed in the block. The trouble is, we don't actually have any space left in bdescr on 32-bit platforms...