lockFile: fd out of range

[guest]

My program was terminated with the following message:

internal error: lockFile: fd out of range
    (GHC version 6.6 for i386_unknown_linux)
    Please report this as a GHC bug:  http://www.haskell.org/ghc/reportabug

The reason appears to be the large number of simultaneously open files. If I limit the number of open descriptors to 1025 or lower, I get "openFile: resource exhausted (Too many open files" instead.

[igloo]

Can anyone give us a testcase for this please?

[simonmar]

See this thread:  http://www.haskell.org/pipermail/haskell-cafe/2005-October/011928.html

GHC uses FD_SETSIZE to determine the maximum number of file descriptors. Perhaps we should be using sysconf(_SC_OPEN_MAX) instead?

[simonmar]

Oh, even if we used sysconf(_SC_OPEN_MAX). the IO manager thread would probably crash because it is using FD_SETSIZE sized tables. What's the real story here - is FD_SETSIZE supposed to be large enough, or not?

[igloo]

According to  http://www.delorie.com/gnu/docs/glibc/libc_248.html FD_SETSIZE is the largest number of things that can be put into an fd_set. The number of open files can be higher (unlimited in the case of GNU HURD, although I'm unclear how you assign unlimited FDs).

We probably want a new datastructure that can cope with unlimited entries; punting for now.

[simonmar]

Fixed.

Patch to GHC:

Tue Nov 20 06:08:59 PST 2007  Simon Marlow <simonmar@microsoft.com>
  * Move file locking into the RTS, fixing #629, #1109

Patch to libraries/base:

Tue Nov 20 04:10:53 PST 2007  Simon Marlow <simonmar@microsoft.com>
  * Move file locking into the RTS, fixing #629, #1109

Patch to libraries/process:

Tue Nov 20 04:42:40 PST 2007  Simon Marlow <simonmar@microsoft.com>
  * catch up with changes to GHC.Handle.fdToHandle'

[simonmar]

One more patch to merge, in libraries/base:

Wed Nov 21 10:22:48 GMT 2007  Simon Marlow <simonmar@microsoft.com>
  * remove lockFile.h from install-includes

[igloo]

As discussed on IRC, some projects (e.g. darcs, I believe) use GHC.Handle.fdToHandle', so we shouldn't change its interface on the branch.

<JaffaCake> it's easy enough to make it compatible, I should probably do that

[simonmar]

Done. Revised list of patches to merge:

GHC:

Tue Nov 20 06:08:59 PST 2007  Simon Marlow <simonmar@microsoft.com>
  * Move file locking into the RTS, fixing #629, #1109

libraries/base:

Tue Nov 20 04:10:53 PST 2007  Simon Marlow <simonmar@microsoft.com>
  * Move file locking into the RTS, fixing #629, #1109

Fri Nov 30 13:51:22 GMT 2007  Simon Marlow <simonmar@microsoft.com>
  * restore fdToHandle' to avoid breaking clients (#1109)

Wed Nov 21 10:22:48 GMT 2007  Simon Marlow <simonmar@microsoft.com>
  * remove lockFile.h from install-includes

libraries/process can now work without modification on the stable branch, but I'll be updating it to use the new API in the HEAD.

[igloo]

All four merged