Re: Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service
Claudius Teodorescu <[email protected]> Sat, 18 Oct 2025 12:38:07 +0300
| Newsgroups | gmane.text.xml.exist |
|---|---|
| Message-ID | <CAPTZ0VwkU--8cFGi22xxTP5nKN660X7AKQAsQoenj7XovTK8Jw@mail.gmail.com> |
--===============7998588140554465022== Content-Type: multipart/alternative; boundary="00000000000041e06406416b9c46" --00000000000041e06406416b9c46 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Hi! Would it help to have the ETL engine as an eXist-db module? Claudius Teodorescu On Sat, 18 Oct 2025 at 11:29, Willem van der Westhuizen <[email protected]> wrote: > Maybe there is a different question to ask. We are using exist and > xquery extensively with an etl engine that we wrote to read data from > couchdb and place into MySQL for reporting purposes. We are currently > using a shell script to launch 20 instances of the the etl with > different parameters to pick up a batch and process it. Would there be a > better way in exist to write the scheduler to make those calls, and then > to wait for each instance to be complete before calling the next, that > would not involve calling the same rest service muliple times from curl? > > Regards > > Willem > > On 2025/10/15 11:48, Willem van der Westhuizen wrote: > > Definitely not updating. Just running, > > > > On 2025/10/15 10:30, Juri Leino wrote: > >> Hi Willem! > >> > >> The behaviour you are describing is usually a sign for an attempt to > >> replace/update a running XQuery. If this is the case the easiest > >> solution is to make sure that the query you want to replace / change > >> is not being run at the moment. > >> > >> I agree that we should investigate if we even have to lock the XQuery > >> in the first place. But that is a separate discussion. > >> > >> Regards, > >> Juri > >> > >> On 12.10.25 13:01, Willem van der Westhuizen wrote: > >>> Hi > >>> > >>> We use existdb extensively to run xqueries on our external > >>> databases, couchdb and others. We dont use the existdb datastore for > >>> our data. > >>> > >>> We used to run on an old 3.3.3 version of existdb, and recently > >>> migrated to 6.2. Since then we have found that existdb locks the > >>> xquery that is being executed, causing deadlocks that eventually > >>> locks up the server. Is there an easy way to tell existdb to revert > >>> to the older locking behaviour that does not lock up the xquery > >>> being run? > >>> > >>> > >>> Regards > >>> > >> > >> > >> _______________________________________________ > >> Exist-open mailing list > >> [email protected] > >> https://lists.sourceforge.net/lists/listinfo/exist-open > > > > > > _______________________________________________ > > Exist-open mailing list > > [email protected] > > https://lists.sourceforge.net/lists/listinfo/exist-open > > > _______________________________________________ > Exist-open mailing list > [email protected] > https://lists.sourceforge.net/lists/listinfo/exist-open > --=20 Cu stim=C4=83, Claudius Teodorescu --00000000000041e06406416b9c46 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div>Hi!</div><div><br></div><div>Would it help to have th= e ETL engine as an eXist-db module?</div><div><br></div><div></div><div><br= ></div><div>Claudius Teodorescu</div></div><br><div class=3D"gmail_quote gm= ail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Sat, 18 Oct 2= 025 at 11:29, Willem van der Westhuizen <<a href=3D"mailto:willem@kwantu= .net">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_= quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,= 204);padding-left:1ex">Maybe there is a different question to ask. We are u= sing exist and <br> xquery extensively with an etl engine that we wrote to read data from <br> couchdb and place into MySQL for reporting purposes. We are currently <br> using a shell script to launch 20 instances of the the etl with <br> different parameters to pick up a batch and process it. Would there be a <b= r> better way in exist to write the scheduler to make those calls, and then <b= r> to wait for each instance to be complete before calling the next, that <br> would not involve calling the same rest service muliple times from curl?<br= > <br> Regards<br> <br> Willem<br> <br> On 2025/10/15 11:48, Willem van der Westhuizen wrote:<br> > Definitely not updating. Just running,<br> ><br> > On 2025/10/15 10:30, Juri Leino wrote:<br> >> Hi Willem!<br> >><br> >> The behaviour you are describing is usually a sign for an attempt = to <br> >> replace/update a running XQuery.=C2=A0 If this is the case the eas= iest <br> >> solution is to make sure that the query you want to replace / chan= ge <br> >> is not being run at the moment.<br> >><br> >> I agree that we should investigate if we even have to lock the XQu= ery <br> >> in the first place. But that is a separate discussion.<br> >><br> >> Regards,<br> >> Juri<br> >><br> >> On 12.10.25 13:01, Willem van der Westhuizen wrote:<br> >>> Hi<br> >>><br> >>> We use existdb extensively to run xqueries on our external <br= > >>> databases, couchdb and others. We dont use the existdb datasto= re for <br> >>> our data.<br> >>><br> >>> We used to run on an old 3.3.3 version of existdb, and recentl= y <br> >>> migrated to 6.2. Since then we have found that existdb locks t= he <br> >>> xquery that is being executed, causing deadlocks that eventual= ly <br> >>> locks up the server. Is there an easy way to tell existdb to r= evert <br> >>> to the older locking behaviour that does not lock up the xquer= y <br> >>> being run?<br> >>><br> >>><br> >>> Regards<br> >>><br> >><br> >><br> >> _______________________________________________<br> >> Exist-open mailing list<br> >> <a href=3D"mailto:[email protected]" target=3D"_bla= nk">[email protected]</a><br> >> <a href=3D"https://lists.sourceforge.net/lists/listinfo/exist-open= " rel=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/= listinfo/exist-open</a><br> ><br> ><br> > _______________________________________________<br> > Exist-open mailing list<br> > <a href=3D"mailto:[email protected]" target=3D"_blank">= [email protected]</a><br> > <a href=3D"https://lists.sourceforge.net/lists/listinfo/exist-open" re= l=3D"noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/list= info/exist-open</a><br> <br> <br> _______________________________________________<br> Exist-open mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">Exist= [email protected]</a><br> <a href=3D"https://lists.sourceforge.net/lists/listinfo/exist-open" rel=3D"= noreferrer" target=3D"_blank">https://lists.sourceforge.net/lists/listinfo/= exist-open</a><br> </blockquote></div><div><br clear=3D"all"></div><br><span class=3D"gmail_si= gnature_prefix">-- </span><br><div dir=3D"ltr" class=3D"gmail_signature"><d= iv dir=3D"ltr"><div><div dir=3D"ltr"><div>Cu stim=C4=83,</div>Claudius Teod= orescu</div></div></div></div> --00000000000041e06406416b9c46-- --===============7998588140554465022== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============7998588140554465022== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Exist-open mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/exist-open --===============7998588140554465022==--