Re: Urgent assistance with locking behaviour of existdb - or how to change the architecture of a scheduled ETL service
Dannes Wessels <[email protected]> Sun, 26 Oct 2025 20:34:06 +0100
| Newsgroups | gmane.text.xml.exist |
|---|---|
| Message-ID | <[email protected]> |
--===============8725927835110491386== Content-Type: multipart/alternative; boundary="Apple-Mail=_F72914BF-15F9-45B0-8BE1-7496B6285D97" --Apple-Mail=_F72914BF-15F9-45B0-8BE1-7496B6285D97 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=us-ascii Hi Willem > On 18 Oct 2025, at 12:54, Willem van der Westhuizen = <[email protected]> wrote: >=20 > The etl engine is an existdb module, that is called by an xquery. One = hypothesis is that each time that the xquery runs, it loads an instance = of the module into memory, and thats is where the locking happens. No = way to tell if that is indeed the case, and what the internal limiations = are. >=20 > The one interesting observation is that even in older version 3.3.3 if = the xquery is pending up due to the remote service being called is = stalled, then the number of instances in exists always stop responding = at around 160 plus or minus, then the whole exist instance locks up. The = number of database brokers, and memory etc. was never a limit. >=20 > In the later versions of exist the deadlock may occur at much lover = levels. To give an example, we have rewritten most of the critical code = that locked up in javascript, and there was no issue in the downstream = services.=20 >=20 > The explanation offered of recursive calling of modules might be a = clue, because that is certainly the case in our code. We would prefer to = keep the ETL in exist though, hence trying to find a solution. >=20 Is it about the couch base driver I created a long time ago? I never checked the code for newer eXist-db versions.. With kind regards Dannes --Apple-Mail=_F72914BF-15F9-45B0-8BE1-7496B6285D97 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=us-ascii <html aria-label=3D"message body"><head><meta http-equiv=3D"content-type" = content=3D"text/html; charset=3Dus-ascii"></head><body = style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; = line-break: after-white-space;">Hi Willem<br = id=3D"lineBreakAtBeginningOfMessage"><div><br><blockquote = type=3D"cite"><div>On 18 Oct 2025, at 12:54, Willem van der Westhuizen = <[email protected]> wrote:</div><br = class=3D"Apple-interchange-newline"><div><meta charset=3D"UTF-8"><p = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;">The etl engine = is an existdb module, that is called by an xquery. One hypothesis is = that each time that the xquery runs, it loads an instance of the module = into memory, and thats is where the locking happens. No way to tell if = that is indeed the case, and what the internal limiations are.</p><p = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;">The one = interesting observation is that even in older version 3.3.3 if the = xquery is pending up due to the remote service being called is stalled, = then the number of instances in exists always stop responding at around = 160 plus or minus, then the whole exist instance locks up. The number of = database brokers, and memory etc. was never a limit.</p><p = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;">In the later = versions of exist the deadlock may occur at much lover levels. To give = an example, we have rewritten most of the critical code that locked up = in javascript, and there was no issue in the downstream services.<span = class=3D"Apple-converted-space"> </span><br></p><p = style=3D"caret-color: rgb(0, 0, 0); font-family: Helvetica; font-size: = 12px; font-style: normal; font-variant-caps: normal; font-weight: 400; = letter-spacing: normal; text-align: start; text-indent: 0px; = text-transform: none; white-space: normal; word-spacing: 0px; = -webkit-text-stroke-width: 0px; text-decoration: none;">The explanation = offered of recursive calling of modules might be a clue, because that is = certainly the case in our code. We would prefer to keep the ETL in exist = though, hence trying to find a solution.</p></div></blockquote>Is it = about the couch base driver I created a long time ago?</div><div>I never = checked the code for newer eXist-db = versions..</div><div><br></div><div>With kind = regards</div><div><br></div><div>Dannes</div><div><br></div><br></body></h= tml>= --Apple-Mail=_F72914BF-15F9-45B0-8BE1-7496B6285D97-- --===============8725927835110491386== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline --===============8725927835110491386== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ Exist-open mailing list [email protected] https://lists.sourceforge.net/lists/listinfo/exist-open --===============8725927835110491386==--