Re: [bdbxml] Lockers that don't go away with database queries??
George Feinberg <[email protected]> Tue, 27 Sep 2005 19:17:54 -0400
| Newsgroups | gmane.comp.db.dbxml.general |
|---|---|
| Message-ID | <[email protected]> |
--Apple-Mail-5--767980537
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
charset=WINDOWS-1252;
delsp=yes;
format=flowed
> After doing additional investigation on the segfault that I =20
> mentioned yesterday, it turns out that we=92re running out of lockers =20=
> (as well as memory). We=92re using the latest Berkeley XML, with the =20=
> default of 1000 max lockers. Our database server is pretty =20
> straightforward =96 we only use transactions on write=92s and =
delete=92s, =20
> not on queries/reads. We=92re writing several different types of =20
> small XML documents into the database. The steps we go through are =20
> basically this (I=92m in the process of writing a standalone test =20
> utility that will mimic the database access portion of our software =20=
> suite):
>
>
>
> For new document A =96 query the database to ensure that a key value =20=
> in the XML is unique, then write the new document to container A =20
> (so the number of these documents in the database continues to =20
> increase)
>
> For new document B =96 simply write out the new document to container =20=
> B (again, the number of these documents increases)
>
> For document C =96 write a new version of the document and delete the =20=
> old version (2 separate operations =96 we=92re not using =20
> UpdateDocument) in container A, so at most we have 2 of these in =20
> the database at one time
>
>
>
> What we=92re seeing is the following:
>
>
>
> After roughly 200 iterations of the above steps, we have 400 =20
> current lockers. After 480 iterations of the above steps, we run =20
> out of lockers and the database PANICs. On another system where we =20
> follow roughly the same steps WITHOUT the query, we don=92t see the =20=
> same increase in lockers.
Wendy,
Are you deleting your XmlDocument objects as well as XmlValue and =20
XmlResults
objects?
One way to locate the specific operation that causes the leak is to
collect DB stats on lockers frequently, and look for unexpected =20
increases.
The code to do this looks like this:
LockStats theLockStats =3D dbEnvironment.getLockStats=20
(StatsConfig.DEFAULT);
System.out.println("num lockers:" + theLockStats.getNumLockers());
System.out.println("num locks:" + theLockStats.getNumLocks());
Where dbEnvironment is your Environment object.
> Why are we seeing a constant increate in the number of lockers? The =20=
> transactions that we create are very transient =96 create a =20
> transaction, write (or delete) the document, commit the =20
> transaction, all within the requisite try/catch block. I understand =20=
> that the regular Berkeley database does stuff under the covers, but =20=
> why are there all these lockers left lying around with no locks =20
> being held? Even if we increase the number of lockers, aren=92t we =20
> just delaying the inevitable?
In Berkeley DB, lockers are usually either a transaction or a =20
database cursor.
Since you are using, and committing transactions, they are not the =20
culprit.
I have to assume that you are performing un-transacted read operations.
Those are the ones that are consuming lockers.
In 2.1.8, using node storage, there is a locker associated with each =20
XmlDocument object
that is read outside of a transaction.
If those objects are not cleaned up (i.e. deleted), lockers will =20
accumulate.
XmlResults and XmlValue objects hold references to XmlDocument objects,
which is why they must be released also. If the document was =20
acquired in
a transaction, the transaction is the locker, not the cursor held by =20
the document.
You'll be happy to know that 2.2 does not have this problem, as its =20
documents
do not hold onto lockers (which are Berkeley DB database cursors, to be
precise).
Regards,
George
--Apple-Mail-5--767980537
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
charset=WINDOWS-1252
<HTML><BODY style=3D"word-wrap: break-word; -khtml-nbsp-mode: space; =
-khtml-line-break: after-white-space; "><DIV><BR =
class=3D"Apple-interchange-newline"><BLOCKQUOTE type=3D"cite"><SPAN =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
normal; letter-spacing: normal; line-height: normal; text-align: auto; =
-khtml-text-decorations-in-effect: none; text-indent: 0px; =
-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
white-space: normal; widows: 2; word-spacing: 0px; "><O:SMARTTAGTYPE =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"City"><O:SMARTTAGTYPE =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"place"><DIV class=3D"Section1"><P class=3D"MsoNormal"><FONT =
size=3D"2" face=3D"Arial"><SPAN style=3D"font-size:10.0pt; =
font-family:Arial; font-size: 13.3333px; "><SPAN =
class=3D"Apple-style-span" style=3D"font-family: Arial; font-size: =
13.3333px; ">After doing additional investigation on the segfault that I =
mentioned yesterday, it turns out that we=92re running out of lockers =
(as well as memory). We=92re using the latest Berkeley XML, with the =
default of 1000 max lockers. Our database server is pretty =
straightforward =96 we only use transactions on write=92s and delete=92s, =
not on queries/reads. We=92re writing several different types of small =
XML documents into the database. The steps we go through are basically =
this (I=92m in the process of writing a standalone test utility that =
will mimic the database access portion of our software =
suite):</SPAN><O:P style=3D"font-family: Arial; font-size: 13.3333px; =
"></O:P></SPAN></FONT></P><P class=3D"MsoNormal"><FONT size=3D"2" =
face=3D"Arial"><SPAN style=3D"font-size:10.0pt; font-family:Arial; =
font-size: 13.3333px; "><O:P style=3D"font-family: Arial; font-size: =
13.3333px; "><SPAN class=3D"Apple-style-span" style=3D"font-family: =
Arial; font-size: 13.3333px; ">=A0</SPAN></O:P></SPAN></FONT></P><P =
class=3D"MsoNormal"><FONT size=3D"2" face=3D"Arial"><SPAN =
style=3D"font-size:10.0pt; font-family:Arial; font-size: 13.3333px; =
"><SPAN class=3D"Apple-style-span" style=3D"font-family: Arial; =
font-size: 13.3333px; ">For new document A =96 query the database to =
ensure that a key value in the XML is unique, then write the new =
document to container A (so the number of these documents in the =
database continues to increase)</SPAN><O:P style=3D"font-family: Arial; =
font-size: 13.3333px; "></O:P></SPAN></FONT></P><P =
class=3D"MsoNormal"><FONT size=3D"2" face=3D"Arial"><SPAN =
style=3D"font-size:10.0pt; font-family:Arial; font-size: 13.3333px; =
"><SPAN class=3D"Apple-style-span" style=3D"font-family: Arial; =
font-size: 13.3333px; ">For new document B =96 simply write out the new =
document to container B (again, the number of these documents =
increases)</SPAN><O:P style=3D"font-family: Arial; font-size: 13.3333px; =
"></O:P></SPAN></FONT></P><P class=3D"MsoNormal"><FONT size=3D"2" =
face=3D"Arial"><SPAN style=3D"font-size:10.0pt; font-family:Arial; =
font-size: 13.3333px; "><SPAN class=3D"Apple-style-span" =
style=3D"font-family: Arial; font-size: 13.3333px; ">For document C =96 =
write a new version of the document and delete the old version (2 =
separate operations =96 we=92re not using UpdateDocument) in container =
A, so at most we have 2 of these in the database at one time</SPAN><O:P =
style=3D"font-family: Arial; font-size: 13.3333px; =
"></O:P></SPAN></FONT></P><P class=3D"MsoNormal"><FONT size=3D"2" =
face=3D"Arial"><SPAN style=3D"font-size:10.0pt; font-family:Arial; =
font-size: 13.3333px; "><O:P style=3D"font-family: Arial; font-size: =
13.3333px; "><SPAN class=3D"Apple-style-span" style=3D"font-family: =
Arial; font-size: 13.3333px; ">=A0</SPAN></O:P></SPAN></FONT></P><P =
class=3D"MsoNormal"><FONT size=3D"2" face=3D"Arial"><SPAN =
style=3D"font-size:10.0pt; font-family:Arial; font-size: 13.3333px; =
"><SPAN class=3D"Apple-style-span" style=3D"font-family: Arial; =
font-size: 13.3333px; ">What we=92re seeing is the following:</SPAN><O:P =
style=3D"font-family: Arial; font-size: 13.3333px; =
"></O:P></SPAN></FONT></P><P class=3D"MsoNormal"><FONT size=3D"2" =
face=3D"Arial"><SPAN style=3D"font-size:10.0pt; font-family:Arial; =
font-size: 13.3333px; "><O:P style=3D"font-family: Arial; font-size: =
13.3333px; "><SPAN class=3D"Apple-style-span" style=3D"font-family: =
Arial; font-size: 13.3333px; ">=A0</SPAN></O:P></SPAN></FONT></P><P =
class=3D"MsoNormal"><FONT size=3D"2" face=3D"Arial"><SPAN =
style=3D"font-size:10.0pt; font-family:Arial; font-size: 13.3333px; =
"><SPAN class=3D"Apple-style-span" style=3D"font-family: Arial; =
font-size: 13.3333px; ">After roughly 200 iterations of the above steps, =
we have 400 current lockers. After 480 iterations of the above steps, we =
run out of lockers and the database PANICs. On another system where we =
follow roughly the same steps WITHOUT the query, we don=92t see the same =
increase in =
lockers.</SPAN></SPAN></FONT></P></DIV></O:SMARTTAGTYPE></O:SMARTTAGTYPE><=
/SPAN></BLOCKQUOTE><DIV><BR =
class=3D"khtml-block-placeholder"></DIV>Wendy,</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>Are you deleting your =
XmlDocument objects as well as XmlValue and =
XmlResults</DIV><DIV>objects?</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>One way to locate the =
specific operation that causes the leak=A0 is to=A0</DIV><DIV>collect DB =
stats on lockers frequently, and look for unexpected =
increases.</DIV><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV>The =
code to do this looks like this:</DIV><DIV>LockStats theLockStats =3D =
dbEnvironment.getLockStats(StatsConfig.DEFAULT);</DIV><DIV>System.out.prin=
tln("num lockers:" + =
theLockStats.getNumLockers());</DIV><DIV>System.out.println("num locks:" =
+ theLockStats.getNumLocks());</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>Where dbEnvironment is your =
Environment object.</DIV><DIV><BLOCKQUOTE type=3D"cite"><SPAN =
class=3D"Apple-style-span" style=3D"border-collapse: separate; =
border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: Helvetica; =
font-size: 12px; font-style: normal; font-variant: normal; font-weight: =
normal; letter-spacing: normal; line-height: normal; text-align: auto; =
-khtml-text-decorations-in-effect: none; text-indent: 0px; =
-apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
white-space: normal; widows: 2; word-spacing: 0px; "><O:SMARTTAGTYPE =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"City"><O:SMARTTAGTYPE =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"place"><DIV class=3D"Section1"><P class=3D"MsoNormal"><FONT =
size=3D"2" face=3D"Arial"><SPAN style=3D"font-size:10.0pt; =
font-family:Arial; font-size: 13.3333px; "><SPAN =
class=3D"Apple-style-span" style=3D"font-family: Arial; font-size: =
13.3333px; ">Why are we seeing a constant increate in the number of =
lockers? The transactions that we create are very transient =96 create a =
transaction, write (or delete) the document, commit the transaction, all =
within the requisite try/catch block. I understand that the regular =
</SPAN><ST1:CITY w:st=3D"on"><ST1:PLACE w:st=3D"on"><SPAN =
class=3D"Apple-style-span" style=3D"font-family: Arial; font-size: =
13.3333px; ">Berkeley</SPAN></ST1:PLACE></ST1:CITY><SPAN =
class=3D"Apple-style-span" style=3D"font-family: Arial; font-size: =
13.3333px; "> database does stuff under the covers, but why are there =
all these lockers left lying around with no locks being held? Even if we =
increase the number of lockers, aren=92t we just delaying the =
inevitable?</SPAN></SPAN></FONT></P></DIV></O:SMARTTAGTYPE></O:SMARTTAGTYP=
E></SPAN></BLOCKQUOTE>In Berkeley DB, lockers are usually either a =
transaction or a database cursor.</DIV><DIV>Since you are using, and =
committing transactions, they are not the culprit.</DIV><DIV>I have to =
assume that you are performing un-transacted read =
operations.</DIV><DIV>Those are the ones that are consuming =
lockers.<BR><DIV><BR class=3D"khtml-block-placeholder"></DIV><DIV>In =
2.1.8, using node storage, there is a locker associated with each =
XmlDocument object</DIV><DIV>that is read outside of a =
transaction.</DIV><DIV>If those objects are not cleaned up (i.e. =
deleted), lockers will accumulate.</DIV><DIV>XmlResults and XmlValue =
objects hold references to XmlDocument objects,</DIV><DIV>which is why =
they must be released also.=A0 If the document was acquired =
in=A0</DIV><DIV>a transaction, the transaction is the locker, not the =
cursor held by the document.</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>You'll be happy to know =
that 2.2 does not have this problem, as its documents</DIV><DIV>do not =
hold onto lockers (which are Berkeley DB database cursors, to =
be</DIV><DIV>precise).</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>Regards,</DIV><DIV><BR =
class=3D"khtml-block-placeholder"></DIV><DIV>George</DIV><BLOCKQUOTE =
type=3D"cite"><SPAN class=3D"Apple-style-span" style=3D"border-collapse: =
separate; border-spacing: 0px 0px; color: rgb(0, 0, 0); font-family: =
Helvetica; font-size: 12px; font-style: normal; font-variant: normal; =
font-weight: normal; letter-spacing: normal; line-height: normal; =
text-align: auto; -khtml-text-decorations-in-effect: none; text-indent: =
0px; -apple-text-size-adjust: auto; text-transform: none; orphans: 2; =
white-space: normal; widows: 2; word-spacing: 0px; "><O:SMARTTAGTYPE =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"City"><O:SMARTTAGTYPE =
namespaceuri=3D"urn:schemas-microsoft-com:office:smarttags" =
name=3D"place"><DIV =
class=3D"Section1"></DIV></O:SMARTTAGTYPE></O:SMARTTAGTYPE></SPAN></BLOCKQ=
UOTE></DIV><FONT class=3D"Apple-style-span" face=3D"Arial" =
size=3D"4"><SPAN class=3D"Apple-style-span" style=3D"font-size: =
13.3333px;"><BR =
class=3D"khtml-block-placeholder"></SPAN></FONT></BODY></HTML>=
--Apple-Mail-5--767980537--