[Fresco-devel] [ReFresco 01] Why CORBA is inappropriate for Fresco

I've been thinking for a while about how Fresco should change.
Finally in the last week or so, I've had some time to start writing up
the results of those thoughts in a series of notes.  I was hoping to
start posting these a few days ago so people'd have more time to read
them before the meeting, but that didn't work out... ah well, we'll
see how it goes.

Here's the first one:

ReFresco 01: Why CORBA is inappropriate for Fresco
==================================================

Okay, I know this is heresy of a sort, but I'm going to argue that
using CORBA for Fresco is a mistake.  A few points before I begin:

  -- These are not the normal arguments against CORBA; I think part of
     why it's taken so long to realize that CORBA is a bad idea for us
     is that we're so used to reflexively defending against the
     standard arguments.

     No, bloat is not a priori a bad thing; we do use most of what
     CORBA offers.  (On the other hand, I think unnecessary complexity
     _is_ a priori a bad thing, and that we can get by with rather
     less than what CORBA offers.  E.g., sure, we use the POA's policy
     setting mechanisms -- only to set one policy.  That's means it's
     unnecessary complexity, even if we use it.)

     No, we don't need our protocol to be as efficient as X's, because
     we really do send far less information down the wire.  (Though
     there are applications that will require sending lots of data
     down the wire -- the gimp reading mouse motions, for example --
     and being gratuitously slower isn't really something to be proud
     of.)

     No, we don't need to be "more asynchronous than X" just because
     some X developer once said that that was how X should be
     improved[0].  (But the need for pipelining probably will come up
     in real applications; in a comment on the wiki[1] you can see me
     not quite able to ignore this, even when I was much more
     optimistic about CORBA.)

  -- These critiques feed into each other, and some are more important
     than others.  I hope no-one tries to argue against this by
     picking out a few points from the first section and saying "well,
     but that's not _too_ bad" or "well, yes, but there is a
     workaround if we...".  It's true, a number of these points are
     not compelling on their own.  But I think some of them
     (especially in the second section) are, and I think taken
     together the preponderance of evidence is very much against
     CORBA.

  -- Note that this critique is entirely orthogonal to the debate
     about whether the scene graph should be exposed via CORBA.  Well,
     not entirely orthogonal; many of these are general problems that
     showed up first and most dramatically in the context of the scene
     graph.  But they all still apply even if the scene graph is
     pulled all the way inside the server.

  -- I should mention also what I don't dislike about CORBA.  I am not
     arguing against the basic "there are lots of objects in different
     address spaces and they send each other messages" model that
     CORBA has; that model is fundamental to Fresco's design.  Someone
     else might want to argue against it, I suppose, but at that point
     we're not even talking about Fresco anymore.  My argument is
     against a lot of the things CORBA does in realizing that model,
     that make it useless for us.

Right, enough excuses.  Let's see where the problems are.

[0] BTW, does anyone have a citation on this?  I've heard it repeated
    lots of times in more or less this form, but have no idea what the
    original context was.

[1] http://wiki.fresco.org/FrescoVsXComments

General CORBA suckage
---------------------

- Ugly generally
  - Designed by committee
  - Everything is possible in theory
  - ...which obscures that in practice, so much is impossible or
    extraordinarily difficult.

Complexity is the mind-killer.  It makes everything harder, more
confusing, more difficult to understand and predict and fix.
Simplicity is the most important goal of software engineering.  CORBA
is not simple, so any flaws are very difficult to fix.  (And I believe
I say this as the only person on the list who has actually worked on
ORB internals.)  And this complexity spreads to everything CORBA
touches; Fresco is just hard to work on, confusing, and with a huge
learning curve.  We're stagnant in large part because no-one can
figure out how to fix things, because the architecture is confusing
and CORBA makes working on stuff a slog.

- Very few real implementations
    Java: JacORB
    C++: OmniORB, possibly TAO and MICO
    Python: OmniORBpy
    C: none
    Perl: none
    Ruby: none
    TCL: none
    Smalltalk: none
    Lisp/Scheme: none
    ...: none
  - Because ugly and designed by committee, very unlikely will see
    more or better ORBs.  (Note that in all the years since
    Fresco/Berlin started using CORBA, the only real change in ORB
    support is that JacORB and MICO got the little bit better that was
    necessary to support Fresco.  Or at least I think MICO did, I'm
    not even sure.  I'm not holding my breath for more implementations
    to show up.)
  - Shortage of ORBs also means that we are very dependent on the ones
    that exist (i.e., "What if Duncan Grisby gets hit by a truck?").
  - Lots of people argue that the C++ mapping sucks, and the
    conventional answer is that this is a client-side issue, and
    should be fixed without changing the rest of CORBA.  Which is a
    fine answer, but such arguers are admitting that the C++ client
    mapping is flawed; and if you believe that, then C++ doesn't have
    any good ORBs either, leaving _only_ Java and Python that are well
    supported.

    Fresco needs to work across languages; CORBA is supposed to give
    this "for free".  We shouldn't fool ourselves any longer: this is
    a lie.  In fact, CORBA makes cross-language support near
    impossible.  Do you want to write fully compliant ORBs for every
    language worth supporting?  Do you expect other people to show up
    to do so?  CORBA is just too complicated for this to make sense.
    I wouldn't be surprised if in the end, implementing a working ORB
    turns out to be more complex then implementing a full working
    Fresco server.

- No security
  - The way CORBA works, if I can get the address for your Fresco
    server, I can do _anything_ to it.  There is no need for me to
    authenticate; in fact, it is impossible for the server to know I
    have connected at all.  And I can do anything that any client
    could do -- including manipulate the objects of normal clients in
    arbitrary ways.  There is no way to stop this without modifying
    the CORBA protocol and the CORBA API, probably in drastic ways.
  - There is a "security service" which is supposed to address this,
    but it's yet another bit of CORBA that only exists on paper.
    And for the reasons discussed above, I wouldn't hold my breath for
    it to be implemented soon.
  - I've tried and failed a few times to actually read the security
    service spec, so don't quote me on this part, but I got the
    impression that it wouldn't even fix our problem; we'd have to
    re-check credentials on every single call or something, and keep
    per-object tables of which clients were allowed to touch each
    object...

- No way to hop firewalls or control connectivity
  - Realistically, this is necessary today.  Put simply, if Fresco
    connections cannot tunnel through ssh, then we will fail.
  - But CORBA uses a many-to-many model for communications and
    generally creates and tears down connections as it pleases.
  - CORBA can not be forwarded through ssh or NATed firewalls
    or... anything, really.
  - This is a fundamental part of the protocol -- it is impossible to
    even create a smart proxy, because it would have to know how to
    munge every CORBA request so that all object identifiers were
    changed, and to do this it would have to know the full IDL of all
    communications that occur.
    - Meaning, any (otherwise compatible) changes to the server
      interface, or even to purely between-client interfaces (think
      of, e.g., an out-of-server http renderer that clients can make
      use of), would have to be explained to all proxies before they
      could be used.

Note that I'm not the only person to think these things.  Even Michi
Henning is now arguing that CORBA is a mistake, for generally these
reasons:
  http://www.zeroc.com/iceVsCorba_body.html
  http://www.zeroc.com/ieeeArticle_body.html

Particular mismatches between Fresco's needs and CORBA's model
--------------------------------------------------------------

Even if the above flaws were all fixed, CORBA would not be a good fit
for Fresco.  ICE (http://zeroc.com), for example, is a "fixed CORBA",
but I think it is still inappropriate, because of certain aspects of
its design that are intentional features:

- No connections -- alluded to above.  Right now we have to do this
  ridiculous 'pinging' thing, which is not at all robust and cannot be
  made so[2], just to emulate something TCP provides for free.

  This is something X got right.  We have clients talking to a server;
  they should just open a single TCP or Unix domain connection and
  talk.  People have spent a _lot_ of time on making this do the right
  thing.  It's tunnelable, it works through NATs (as well as anything
  works through NATs), etc.  This also means that client-to-client
  communication should all be routed _through_ the server; that
  clients can connect to the server is the only thing we can
  realistically assume, and anything else adds lots of complexity.
  Neither CORBA nor ICE provide any good way for this sort of
  forwarding to happen.  (ICE could sorta do it, maybe, I guess, but
  it's not really designed for this model.)
  
  A connection oriented protocol is also, obviously, required for
  connection-based authentication and the like.  If you want to be
  sure you know who your clients are, and that you can do things like
  clean up after them once they're gone, then you need to be able to
  count connections; with CORBA or ICE that's just impossible.

  [2] See thread starting at
     http://lists.fresco.org/pipermail/fresco-devel/2002-December/018935.html

- Basic model is of synchronous calls -- with CORBA, every network
  operation must block waiting for a response.  Firstly, this is just
  absurd; we sent something down the network, and it's going to take
  thousands or millions of cycles before we get a response; it's
  absurd that we can't get any work done during this period.  Networks
  are about sending messages, not
  here-borrow-my-stack-and-instruction-pointer.  (And incidentally,
  there's no reason why all messages should even have responses.)

  Aside from this inelegance and wastefulness, this is a source of
  serious problems; we simply cannot safely call back to clients at
  the moment, because there is no guarantee that they will reply, and
  thus no guarantee that our ORB will ever give us back our flow of
  control.

  There's an unimplemented specification (AMI) for working
  around this.  But right now it's unimplemented everywhere; even if
  someone implements does it, it won't be implemented everywhere
  (i.e., requiring this raises the bar yet higher for ORBs in new
  languages); and even if it does get added, it will mean extra
  complication for all code that wants to use it.

  Furthermore, AMI doesn't even help that much.  What you really want
  is to be able to pipeline calls -- if you know anything about the
  capability language E, think of "promises".  Even with AMI lots of
  things require round trips, because there are dependencies, where
  you have to get the result of one call before you have something to
  pass to the next.

  - This also (as a minor point) means that we have to use threads, which
    I'm not entirely convinced is a good idea.  Certainly we totally fail
    to be thread-safe right now, despite our best efforts, which suggests
    we're doing something wrong; locking is also the source of huge
    amounts of our overhead[3].

    [3] I _think_ that even when CORBA locking was eliminated with my
    shortcut patch, pthread_mutex_{lock,unlock} still showed up pretty
    high on the profiles; but I may be misremembering.

  As for reliable one way calls?  I've heard rumors that the OMG will
  eventually spit out yet another piece of paper that will solve all
  our problems.  I'm skeptical.

- Location transparency.  Frankly, location transparency is dumb.
  There are a lot of conventional arguments against location
  transparency[4], but I think they don't really apply to Fresco (at
  least once we take the scene graph out of CORBA).  They're all
  centered on the fact that local and remote really are different, so
  you can't just pretend they're the same.  In Fresco, we don't
  pretend that local and remote are the same anyway; we explicitly use
  CORBA as a communications protocol, not as a way to not think about
  networking issues (e.g., think of all the debate we have on where
  objects should properly be located).

  So my problem with location transparency isn't that location
  transparency is bad, per se (though it is).  My problem with CORBA's
  location transparency is that we don't even _want_ location
  transparency; CORBA is really just insisting on providing location
  obscurity.  There are lots of cases where it's really important to
  know where an object is really located, or to find the servant
  implementing an object, but CORBA just won't let us do it.

  Of course, I'm not arguing that CORBA's syntax for remote calls
  isn't convenient, but that's all it is: convenient.  A system could
  perfectly well provide that syntactic sugar without also thumbing
  its nose at the curious programmer and saying "nyah nyah, won't
  tell!"

  [4] See, e.g.:
        http://research.sun.com/techrep/1994/abstract-29.html
        http://www.advogato.org/person/graydon/diary.html?start=91
        http://www.erights.org/elib/concurrency/semi-transparent.html

Conclusion
----------

Fresco should not use CORBA, or even ICE.  Fresco should instead use a
networking layer that is:
  -- simple and understandable, to make reasoning, fixing, and
     implementation easy
  -- connection oriented, to allow tunneling, basic authentication,
     and connection management
  -- routable: clients should be able to exchange messages among
     themselves using the server as an intermediary; does not require
     arbitrary many-to-many communication
  -- extensible and object based: CORBA got this right, though we can
     do a better job at an on the wire format than IIOP
  -- secure from the ground up
  -- asynchronous by default, with "one way" calls that work

I realize this is a big shift, especially since I don't know of any
existing communications architectures that meet these requirements; I
am proposing creating a new protocol, which, yes, is a bit of a pain.
However, I do not see that we have any choice in the matter.  If
someone finds an existing system that meets the needs outlined above,
I will be very happy, but if not, then I genuinely believe we must
either create our own or give up entirely.

Fortunately, I think I do know how to build such a beast; later
notes will outline my design.


-- Nathaniel

-- 
"...All of this suggests that if we wished to find a modern-day model
for British and American speech of the late eighteenth century, we could
probably do no better than Yosemite Sam."