Re: experimental ZODB prefetch support in ZEO

This has been released as ZEO 5.0.0a1 and ZODB 5.0.0a6.

Jim

On Fri, Jul 15, 2016 at 8:47 AM, Jim Fulton <[email protected]> wrote:

> For years, people have been asking for a bulk load feature, especially
> in ZEO and especially to aid startup. For years, I've been threatening
> to add a prefetch feature to address this requirement.  I finally took
> the first step towards this yesterday.
>
> https://github.com/zopefoundation/ZEO/pull/48
>
> This adds a prefetch method to ZEO.  Prefetch starts retrieving data
> for an iterable of oids.  This makes retrieval more or less parallel,
> objects are loaded all at once rather than making sequential
> round-trips to the server for each object.  Applications can then go
> on about their business without waiting for the loads to complete.
> When an application needs data for a prefetch object, it will get it
> from the ZEO cache, or, if the fetch for that object isn't complete,
> it will wait for the original fetch to complete, without making a new
> request.
>
> In the coming days, I'll add a Connection prefetch option to ZODB. It
> will work something like:
>
>     conn.prefetch(ob1, ob2, ...)
>
> where the arguments can be object ids, persistent objects (ghosts) or
> iterables.
>
> In informal tests (similar to zodbshootout), prefetch provided a 60%
> reduction in load time for cold loads. This was with the server
> running on the same host as the clients. In a more realistic
> configuration, I'd expect the benefit to be greater.  Of course, the
> benefit of prefetch depend on your ability to predict what needs to be
> fetched.
>
> An idea I had that might help automate this is to add an _p_prefetch
> class attribute.  If a class defines this, then when a ghost of that
> class is created, prefetch would be called for it.  This would be
> useful in situations where objects are very likely to be needed when
> their parent/referring objects are.
>
> Jim
>
> --
> Jim Fulton
> http://jimfulton.info
>



-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.