Re: Using substanced with Zeo and keep getting database conflict errors

OK, thanks. I will ask on the substanced group but it isn't very well 
frequented.

As I said I have now written two applications with identical web 
interfaces, bar differences in the database structure, and very similar 
data uploads. The other application didn't have these commit issues. It 
seems to happen for lots of different objects but then the data on this 
site is more nested than that on the other site.

So maybe I am misunderstanding zeo then. I was previously using file 
storage and I wasn't able to have more than one process using the database. 
With zeo I can have several processes running at the same time. However, if 
I have two processes that want to write to the database then each process 
uses transactions and does not allow serial commits?

At what level is the zodb locked for editing by a transaction? Is it at the 
top level of a node? Is it from the edited object and then anything 
referenced by it or anything below it in the tree?

As the data import is a one off process then this may not be an issue. What 
would be an issue is if two users of the web interface aren't able to edit 
parts of the same tree. I can understand they couldn't both edit the same 
object but what are the limits?

Would it make a difference if I used relStorage instead of zeo?


On Tuesday, 13 September 2016 14:48:04 UTC+1, Jo Gilder wrote:
>
> Hi,
>
> I have setup an instance of substanced which is a pyramid based cms using 
> zodb. I am using zeo to connect to the zodb as described in the 
> documentation:
>
>
> https://substanced.readthedocs.io/en/latest/configuration.html?highlight=zeo
>
>
> I start zeo using zdaemon and a config file:
>
> /home/bodl-fox-svc/sites/bodl-fox-svc/bin/zdaemon -C 
> /home/bodl-fox-svc/sites/bodl-fox-svc/zdaemon.cfg start
>
>
> zdaemon.cfg looks like this.
>
> <runner>
>
>   program /home/bodl-fox-svc/sites/bodl-fox-svc/bin/runzeo  -C zeo.cfg
>
> </runner>
>
>
>
> and zeo.cfg looks like this:
>
> <zeo>
>
>   address /home/bodl-fox-svc/sites/bodl-fox-svc/data/zeosocket
>
> </zeo>
>
>
> <filestorage>
>
>   path /home/bodl-fox-svc/sites/bodl-fox-svc/data/data.fs
>
> </filestorage>
>
>
> <eventlog>
>
>   level INFO
>
>   <logfile>
>
>     path /home/bodl-fox-svc/sites/bodl-fox-svc/data/zeo.log
>
>     format %(asctime)s %(message)s
>
>   </logfile>
>
> </eventlog>
>
>
>
> the connection string in the ini file for the app is
> zodbconn.uri = 
> zeo:////home/bodl-fox-svc/sites/bodl-fox-svc/data/zeosocket?connection_cache_size=15000&blob_dir=/home/bodl-fox-svc/sites/bodl-fox-svc/data/blobs
>
> I've set this up before and it worked fine but this application gives me 
> conflict errors if I try and write to the database from two different 
> locations. 
>
> ConflictError: database conflict error (oid 0x30, class 
> BTrees.IOBTree.IOBTree, serial this txn started with 0x03b9e3ac20faca33 
> 2016-09-13 13:32:07.729605, serial currently committed 0x03b9e3acb62e9
>
> This is happening when I try and add data programatically from a bash file 
> running a python script at the same time as I add data via a web interface. 
> Any ideas?
>
> I have the following versions -
>
> zdaemon = 4.1.0
>
> ZEO = 4.1.0
>
> ZODB = 4.1.0
>
> zodburi = 2.0
>
> zope.copy = 4.0.3
>
> zope.component = 4.2.2
>
> zope.deprecation = 4.1.2
>
> zope.interface = 4.1.2
>
> ZopeSkel = 2.21.2
>
>
On Tuesday, 13 September 2016 16:56:29 UTC+1, Jim Fulton wrote:
>
> On Tue, Sep 13, 2016 at 9:48 AM, Jo Gilder <[email protected] 
> <javascript:>> wrote: 
> > Hi, 
> > 
> > I have setup an instance of substanced which is a pyramid based cms 
> using 
> > zodb. 
>
> ... 
>
> > I've set this up before and it worked fine but this application gives me 
> > conflict errors if I try and write to the database from two different 
> > locations. 
> > 
> > ConflictError: database conflict error (oid 0x30, class 
> > BTrees.IOBTree.IOBTree, serial this txn started with 0x03b9e3ac20faca33 
> > 2016-09-13 13:32:07.729605, serial currently committed 0x03b9e3acb62e9 
> > 
> > This is happening when I try and add data programatically from a bash 
> file 
> > running a python script at the same time as I add data via a web 
> interface. 
> > Any ideas? 
> > 
> > I have the following versions - 
>
> ... This is almost certainly an application issue. 
>
> In addition to ideas Jens mentioned (long transactions, accidental 
> write on read): 
>
> - Does the application, substanced, or your script retry transactions 
> when there are conflict errors?  Retrying will often mitigate 
> conflicts. 
>
> - Are the conflict errors for the same or same kind of object? The 
> errors should include object ids. From a Python prompt, you can use a 
> connection get message to load an object given an id.  See if there 
> are any patterns with conflicting objects. 
>
> I'd try asking this questions on the substanced mailing list, although 
> a lot of the people who are likely to help are probably also on this 
> one. 
>
> Jim 
>
> -- 
> Jim Fulton 
> http://jimfulton.info 
>

-- 
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
For more options, visit https://groups.google.com/d/optout.