experimental ZODB prefetch support in ZEO

For years, people have been asking for a bulk load feature, especially
in ZEO and especially to aid startup. For years, I've been threatening
to add a prefetch feature to address this requirement.  I finally took
the first step towards this yesterday.

https://github.com/zopefoundation/ZEO/pull/48

This adds a prefetch method to ZEO.  Prefetch starts retrieving data
for an iterable of oids.  This makes retrieval more or less parallel,
objects are loaded all at once rather than making sequential
round-trips to the server for each object.  Applications can then go
on about their business without waiting for the loads to complete.
When an application needs data for a prefetch object, it will get it
from the ZEO cache, or, if the fetch for that object isn't complete,
it will wait for the original fetch to complete, without making a new
request.

In the coming days, I'll add a Connection prefetch option to ZODB. It
will work something like:

    conn.prefetch(ob1, ob2, ...)

where the arguments can be object ids, persistent objects (ghosts) or iterables.

In informal tests (similar to zodbshootout), prefetch provided a 60%
reduction in load time for cold loads. This was with the server
running on the same host as the clients. In a more realistic
configuration, I'd expect the benefit to be greater.  Of course, the
benefit of prefetch depend on your ability to predict what needs to be
fetched.

An idea I had that might help automate this is to add an _p_prefetch
class attribute.  If a class defines this, then when a ghost of that
class is created, prefetch would be called for it.  This would be
useful in situations where objects are very likely to be needed when
their parent/referring objects are.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.