Re: do I need ZEO?

On Tue, Sep 27, 2016 at 4:24 PM,  <[email protected]> wrote:
> Hi.
>
> I   plan to use ZODB 5 to store information about different runs from
> long-running jobs (running the whole bunch may take a week).
> These are all launched from different threads of the same process.
>
> While these are running, I may want to access the database to check on older
> runs or to see the status of the current runs.
>
> I think that if my only use were  from multiple threads of the same process
> I could use file-based storage.  But if I want to access from another
> process, I should use ZEO, and both the programs in the first and second
> paragraph above should be clients to ZEO.
>
> Is my understanding correct?  Any tips?

Yes. RelStorage and NEO are options as well.

http://www.zodb.org/en/latest/reference/storages.html#base-storages

>
> BTW, there may be on the order of 500 threads, though at most 30 are likely
> to be active at once.  They will only write to the database at the start and
> end of their runs.  Everything is happening on one machine and the objects
> being saved are smallish.

You should open connections at the beginning and end of each run, like this:

   conn = db.open()
   try:
       for attempt in transaction.manager.attempts():
           with attempt:
               .... do stuff with the connection, conn
  finally:
       conn.close()

See:

http://transaction.readthedocs.io/en/latest/convenience.html#retries

I'm sorry that's so complicated (the magicalness of the attempts API
makes me weep), but you need to be prepared to retry transactions on
conflict, as I'm guessing it would be sad if you couldn't write to the
database after a long run..

...

> P.S. It would be good if https://pypi.python.org/pypi/ZODB/5.0.0 and similar
> pages (e.g., for 4.0) pointed to this list rather than the old one.

Thanks. Sadly there's a lot of duplicated documentation and it's hard
to keep it all up to date.

I'll fix this.

Jim

-- 
Jim Fulton
http://jimfulton.info

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.