Re: Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service
Willem van der Westhuizen <[email protected]> Sun, 26 Oct 2025 22:06:31 +0200
| Newsgroups | gmane.text.xml.exist |
|---|---|
| Organization | Kwantu Infomrmation Technology |
| Message-ID | <[email protected]> |
This is a multi-part message in MIME format. --===============6220640735489090911== Content-Type: multipart/alternative; boundary="------------QZrEp57EXZMLKLARqWO9zuyR" Content-Language: en-US This is a multi-part message in MIME format. --------------QZrEp57EXZMLKLARqWO9zuyR Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit Hi Dannes, No, it more the locking behaviour of later versions of exist where running many instances of the same xquery as a rest service that makes rest services external to existdb. Even though there are no updates, running multiple instances do seem to place some form of read lock on the xquery that locks the instance up. Regards Willem On 2025/10/26 21:34, Dannes Wessels wrote: > Hi Willem > >> On 18 Oct 2025, at 12:54, Willem van der Westhuizen >> <[email protected]> wrote: >> >> The etl engine is an existdb module, that is called by an xquery. One >> hypothesis is that each time that the xquery runs, it loads an >> instance of the module into memory, and thats is where the locking >> happens. No way to tell if that is indeed the case, and what the >> internal limiations are. >> >> The one interesting observation is that even in older version 3.3.3 >> if the xquery is pending up due to the remote service being called is >> stalled, then the number of instances in exists always stop >> responding at around 160 plus or minus, then the whole exist instance >> locks up. The number of database brokers, and memory etc. was never a >> limit. >> >> In the later versions of exist the deadlock may occur at much lover >> levels. To give an example, we have rewritten most of the critical >> code that locked up in javascript, and there was no issue in the >> downstream services. >> >> The explanation offered of recursive calling of modules might be a >> clue, because that is certainly the case in our code. We would prefer >> to keep the ETL in exist though, hence trying to find a solution. >> > Is it about the couch base driver I created a long time ago? > I never checked the code for newer eXist-db versions.. > > With kind regards > > Dannes > > --------------QZrEp57EXZMLKLARqWO9zuyR Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: 8bit <!DOCTYPE html> <html> <head> <meta http-equiv="Content-Type" content="text/html; charset=UTF-8"> </head> <body> <p>Hi Dannes,</p> <p>No, it more the locking behaviour of later versions of exist where running many instances of the same xquery as a rest service that makes rest services external to existdb. Even though there are no updates, running multiple instances do seem to place some form of read lock on the xquery that locks the instance up.</p> <p>Regards</p> <p>Willem<br> </p> <div class="moz-cite-prefix">On 2025/10/26 21:34, Dannes Wessels wrote:<br> </div> <blockquote type="cite" cite="mid:[email protected]"> <meta http-equiv="content-type" content="text/html; charset=UTF-8"> Hi Willem<br id="lineBreakAtBeginningOfMessage"> <div><br> <blockquote type="cite"> <div>On 18 Oct 2025, at 12:54, Willem van der Westhuizen <a class="moz-txt-link-rfc2396E" href="mailto:[email protected]"><[email protected]></a> wrote:</div> <br class="Apple-interchange-newline"> <div> <meta charset="UTF-8"> <p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">The etl engine is an existdb module, that is called by an xquery. One hypothesis is that each time that the xquery runs, it loads an instance of the module into memory, and thats is where the locking happens. No way to tell if that is indeed the case, and what the internal limiations are.</p> <p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">The one interesting observation is that even in older version 3.3.3 if the xquery is pending up due to the remote service being called is stalled, then the number of instances in exists always stop responding at around 160 plus or minus, then the whole exist instance locks up. The number of database brokers, and memory etc. was never a limit.</p> <p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">In the later versions of exist the deadlock may occur at much lover levels. To give an example, we have rewritten most of the critical code that locked up in javascript, and there was no issue in the downstream services.<span class="Apple-converted-space"> </span><br> </p> <p style="caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; letter-spacing: normal; text-align: start; text-indent: 0px; text-transform: none; white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; text-decoration: none;">The explanation offered of recursive calling of modules might be a clue, because that is certainly the case in our code. We would prefer to keep the ETL in exist though, hence trying to find a solution.</p> </div> </blockquote> Is it about the couch base driver I created a long time ago?</div> <div>I never checked the code for newer eXist-db versions..</div> <div><br> </div> <div>With kind regards</div> <div><br> </div> <div>Dannes</div> <div><br> </div> <br> </blockquote> </body> </html> --------------QZrEp57EXZMLKLARqWO9zuyR-- --===============6220640735489090911== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============6220640735489090911== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Exist-open mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/exist-open --===============6220640735489090911==--