Re: Meeting notes

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

On Friday 26 March 2004 10:44 am, Joyce Park wrote:
> Hey guys,
>
> So we had a fabulous meeting Wednesday night, mostly focused on
> server improvements.  My notes are appended.  Old Mod-pubsubbers
> also had a chance to meet the fresh blood, Alex and Vishy and
> Henry, who are pushing the effort on behalf of Friendster.

It was really neat to meet everyone. I (for one) and very excited 
about the possiblity of getting modpubsub to be scalable enough to 
handle some of the loads we are discussing.

[snip]

> Performance
>   --memory
>   --long queues (delivery bug)
>   --select()

So to outline this problem a little bit more for the list, the 
traditional select() and poll() mechanisms available on POSIX systems 
require a main-loop iteration over a list of open file descriptors to 
determine if an action needs to happen for a given file descirptor. 
When that list becomes very long, that traversal becomes increasingly 
expensive, chewing up lots of CPU time that we (as app programmers) 
would like to be using to actually _do_ something.

Given that clients of modpubsub protocol servers generally require 
long-lived connections in order to acheive low-latency, modpubsub 
essentially forces worst-case behavior of select() and poll(). 
Fortunantly, OS manufacturers have recently come to our rescue with 
kqueue/kevent (FreeBSD and OS X) and epoll (Linux 2.6). These 
event-driven file descriptor change notification syscalls allow app 
programmers to register callbacks which are called when something 
happens, so apps can spend less time figuring out if they need to do 
something, and more time actually _doing_ things. This asynchronous 
approach is how we intend to get modpubsub servers supporting 10K 
concurrent clients. This also dove-tails into using the Twisted async 
framework for the Python server nicely.

>   --old event initial route population has a long pause
>   --HTTP 1.1

HTTP 1.1 will buy us chunking and reusable connections. These are both 
Good Things (TM).

>   --robust (lazy expiration)
>   --topics never die
>   --overhead for reaping
>
> Security
>   --SSL

It was noted that in REALLY BIG installations, SSL is still expensive, 
but might allow moderately sized sites/apps to allow HTTP 1.1 through 
proxies.

>   --authentication
>   --security

Security requirements seem to be one of the largest areas for 
improvement. It was decided that authentication and authorization 
shouldn't be handled in the server deamon itself, but rather that the 
server should expose sane "hooks" or proxy-points to allow an 
external security monitor service to give the thumbs up or down to a 
message. The discussion about Unix Domain Sockets sprung out of this 
approach to security, sinc we'll need REALLY fast IPC to get this 
working well.

>   --do_method=batch loses errors, error handling unspecified
>   --sequencing (events have to be delivered in order)
>   --auth/auth policy callbacks or hooks
>   --e.g. unix-domain-sockets

This point refers to a concept of talking the modpubsub HTTP protocol 
over a UDS and that a security monitor could then insert itself into 
the processing stream via this connection.

>   --disable introspection
>
> Latency
>
> Scalability
>   --10K connections/box

With fewer connections than this supported by a single server, the 
communications overhead for replication and failover looks to become 
overwhelming.

> Failover
>   --topic-based expiration policy

Since topics are events, events can expire their sub-events, and 
therefore topics can enforce sub-topic expiration. This seemed an 
elegant way of dealing with te reaping problem, but we don't have any 
experience with this in the wild. Time will tell if this design 
works.

>   --weak ref topic for route dest
>   --lease-extension for expiring objects

Topic creation events for existing topics will re-set their 
time-to-expire timestamps and then be dropped. This seems a good way 
to allow applications to do topic "keepalive".

I'm sure others will have better clarifications.

Regards

- -- 
Alex Russell
[email protected]   BD10 7AFC 87F6 63F9 1691 83FA 9884 3A15 AFC9 61B7
[email protected] F687 1964 1EF6 453E 9BD0 5148 A15D 1D43 AB92 9A46
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1.2.4 (Darwin)

iD8DBQFAZIrGoV0dQ6uSmkYRApRkAJ9+1Ls9Ztq7I18VtZweJKC6aQUFYACfetys
jYJPq7Pj20zX4drsSbKsD/0=
=G/6h
-----END PGP SIGNATURE-----



-------------------------------------------------------
This SF.Net email is sponsored by: IBM Linux Tutorials
Free Linux tutorial presented by Daniel Robbins, President and CEO of
GenToo technologies. Learn everything from fundamentals to system
administration.http://ads.osdn.com/?ad_id70&alloc_id638&op=click