Replace use of select() in the I/O manager with epoll/kqueue/etc.

[simonmar]

Subject says it all. We've known about this problem for ever, but haven't got around to doing anything about it.

Here is a library that provides a unified API over the various event APIs:  http://monkey.org/~provos/libevent/.

See also  http://www.kegel.com/c10k.html.

1 week is an estimate for doing it properly on all relevant platforms, by someone who is familiar with the APIs and code.

[vs@…]

You can find my patches for a libeventified RTS for GHC 6.0 at:  http://www-i2.informatik.rwth-aachen.de/~stolz/Haskell/libevent/

[simonmar]

I should add that we'd prefer to do this using the I/O manager in the threaded runtime (libraries/base/GHC/Conc.lhs), rather than change the current select() machinery in the non-threaded runtime (ghc/rts/posix/Select.c). Volker's libevent patch is for the latter, I believe.

[guest]

i strongly prefer to see this in separate library which isn't part of RTS

[AudreyTang]

Should the libevent-based approach be investigated in the future,  http://software.schmorp.de/pkg/libev.html has a more efficient implementation of the same (actually emulated) API -- see  http://libev.schmorp.de/bench.html for benchmarks.

[bos]

I've started work on this, and have the threaded RTS's IO manager thread successfully using either poll or select, depending on the system.

This step alone is a big deal, because select has a low fixed limit on the number of file descriptors it can manage, usually 1,024. Using poll instead removes that limit.

There is a lot more work to do, but this seems like a good start.

[bos]

base.diff

[bos]

I've attached the current working patch to this ticket. It applies cleanly against both base 4.2.0.0 (as used in GHC 6.12.1) and the version used by HEAD.

Simon, I think this patch is worth applying on top of base-4.2.0.0 as is, since it removes that pesky dependency on select. This would be a nice fix to have in GHC 6.12.2.

For more substantial changes (use of epoll or kqueue), the amount of work involved is much higher.

[simonmar]

This does look like a step in the right direction. How does this compare in performance to the select version? Glancing at the patch it looks like building the fdset will be more expensive than before. There's one withForeignPtr per FD. The INLINEs on the C functions don't have any effect (unfortunately), even with -fvia-C.

[bos]

There is indeed one withForeignPtr per FD. ISTR I'd been told that was effectively free, so I didn't worry about it. Clearly, we could use a plain Ptr instead and just free the pollfd data by hand when shutting down.

I'll have benchmark numbers within a couple of days. I seem not to have the flu finally, so I have to do Real Work again.

[simonmar]

Replying to bos:

There is indeed one withForeignPtr per FD. ISTR I'd been told that was effectively free, so I didn't worry about it. Clearly, we could use a plain Ptr instead and just free the pollfd data by hand when shutting down.

You're right - withForeignPtr should be free, as long as the ForeignPtr is strict. Probably wouldn't hurt to glance at the Core and check it is being inlined away though.

[tibbe]

We've been working on a replacement for the I/O manager for a few weeks now. The latest code can be found at:

 http://github.com/tibbe/event

[kazu-yamamoto]

It seems to me that System.Event enables event-driven network programming on kqueue/epoll but does not enable thread programming (with Handle). Are there any plan to integrate the result of System.Event to GHC.Conc?

[tibbe]

Replying to kazu-yamamoto:

It seems to me that System.Event enables event-driven network programming on kqueue/epoll but does not enable thread programming (with Handle). Are there any plan to integrate the result of System.Event to GHC.Conc?

Yes. System.Event.Thread implements the needed functionality.

[kazu-yamamoto]

OK. Does patch to GHC.Conc already exist? If so, I would like to test it.

[tibbe]

Replying to kazu-yamamoto:

OK. Does patch to GHC.Conc already exist? If so, I would like to test it.

Not yet. Bryan said he would look into integrating the work. The whole event library have to be moved into the GHC repo. You can test it out right now. Have a look at the examples in the benchmarks directory e.g. PongServer?.

[kazu-yamamoto]

I have already looked the benchmarks directory. "dead-conn" cannot be complied due to link problems on Mac (Leopard). "pong-server" causes resource exhauseted. (I carefully unlimited descriptors.) % ./pong-server pong-server: accept: resource exhausted (Too many open files)

"event-http" and "static-http" works well. "event-http" is faster than "static-http".

Anyway, since I want to write thread code using Handle on epoll/kqueue, not event-driven code using Socket, I cannot use System.Event for my program at this moment.

[tibbe]

Replying to kazu-yamamoto:

I have already looked the benchmarks directory. "dead-conn" cannot be complied due to link problems on Mac (Leopard).

If you send me an email with the error I can try to reproduce it (I'm also on OS X).

"pong-server" causes resource exhauseted. (I carefully unlimited descriptors.) % ./pong-server pong-server: accept: resource exhausted (Too many open files)

There are a number of kernel settings that need to be changed to run some of these benchmarks. All of the settings are done automatically for you by the benchmark_shell.py script. It can be used like so:

sudo benchmarks/benchmark_shell.py benchmarks/pong-server

Anyway, since I want to write thread code using Handle on epoll/kqueue, not event-driven code using Socket, I cannot use System.Event for my program at this moment.

If you want to write production code you better wait for GHC 6.14 which is our goal for integrating the code into GHC.

If you want to play with the code using forkIO style concurrency (but using Sockets instead of Handles) you can you System.Event.Thread and benchmarks/EventSocket.hs.

[kazu-yamamoto]

If you send me an email with the error I can try to reproduce it (I'm also on OS X).

The error is attached.

There are a number of kernel settings that need to be changed to run some of these benchmarks. All of the settings are done automatically for you by the benchmark_shell.py script. It can be used like so:

Thanks!

If you want to play with the code using forkIO style concurrency (but using Sockets instead of Handles) you can you System.Event.Thread and benchmarks/EventSocket.hs.

I read it before. But I still want to use Handle for buffering.

The error: (Note I upgrade my Mac to Snow Leopard and GHC to 6.12.1)

% make
ranlib ../dist/build/libHSevent-0.1.a
ghc -funbox-strict-fields -package bytestring -ignore-package event -fno-ignore-asserts -Wall -fno-warn-orphans -fno-warn-missing-signatures -i../dist/build -package-name event-0.1  -package network -package network-bytestring -package attoparsec -package bytestring-show -package mtl -threaded -o dead-conn Args.o EventUtil.o EventSocket.o DeadConn.o ../dist/build/libHSevent-0.1.a
gcc -O2 -o deadconn deadconn.o
ld: warning: in deadconn.o, file is not of required architecture
Undefined symbols:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [deadconn] Error 1

[tibbe]

Replying to kazu-yamamoto:

The error: (Note I upgrade my Mac to Snow Leopard and GHC to 6.12.1)

% make
ranlib ../dist/build/libHSevent-0.1.a
ghc -funbox-strict-fields -package bytestring -ignore-package event -fno-ignore-asserts -Wall -fno-warn-orphans -fno-warn-missing-signatures -i../dist/build -package-name event-0.1  -package network -package network-bytestring -package attoparsec -package bytestring-show -package mtl -threaded -o dead-conn Args.o EventUtil.o EventSocket.o DeadConn.o ../dist/build/libHSevent-0.1.a
gcc -O2 -o deadconn deadconn.o
ld: warning: in deadconn.o, file is not of required architecture
Undefined symbols:
  "_main", referenced from:
      start in crt1.10.6.o
ld: symbol(s) not found
collect2: ld returned 1 exit status
make: *** [deadconn] Error 1

I'm not quite sure what this is, looks like a 32 vs 64-bit issue. If you don't need deadconn (which creates idle connections useful when benchmarking scaling properties) you can build the target you want using e.g. make pong-server.

[tibbe]

Done. It took a little longer than one week.

If you're experience incorrect behavior that you think is related to the new I/O manager please file a bug and assign it to me.

Sat Jul 24 12:23:55 CEST 2010  Johan Tibell <johan.tibell@gmail.com>
  * Integrate new I/O manager, with signal support

Tue Aug 10 10:22:48 CEST 2010  Simon Marlow <marlowsd@gmail.com>
  * Integrated new I/O manager
  (patch originally by Johan Tibell <johan.tibell@gmail.com>, minor merging by me)

[simonmar]

Thanks for closing the bug, and well done to both you and Bryan!