Re: ZODB eating CPU but RAM is free

--000000000000329fae057d38556b
Content-Type: text/plain; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

On Sun, Dec 16, 2018, 5:11 PM Vlad Oles <[email protected] wrote:

> On Saturday, December 15, 2018 at 8:30:16 AM UTC-8, Jim Fulton wrote:
>>
>>
>>
>> On Fri, Dec 14, 2018 at 9:54 PM Vlad Oles <[email protected]> wrote:
>>
>>> On Wednesday, July 19, 2017 at 9:54:05 AM UTC-7, Jim Fulton wrote:
>>>>
>>>>
>>>> One thing to understand is that ZODB doesn't limit memory usage.
>>>> Ghostifies objects to meet memory settings at certain times like
>>>> transaction boundaries and when asked to explicitly.  So a transaction=
 that
>>>> loads a lot of objects can grow ram without bound.  This is why
>>>> applications that load *lots* of objects should explicitly reduce
>>>> cache size by calling _p_jar.cacheGC() occasionally.
>>>>
>>>
>>> Does it mean that after a database is finalized (no more writing and
>>> thus transactions), there is no way to bound its RAM cache size other t=
han
>>> manually resetting cache (via cacheGC() or cacheMinimize()) after every=
 N
>>> reads?
>>>
>>
>> Define finalized.
>>
>> When a transaction is committed or a connection closed, the connection's
>> cache is reduced to it's target size. When a database is closed, the
>> connections and their memory is freed. (If there are cycles, the Python
>> garbage collection may take some time to deal with them.)
>>
>
> By "finalized" I meant the connection is still open but no more
> transactions are being commited, so the database is used only for reads.
>

Reads are executed in transactions. In fact, most transactions are read
transactions.



> I think I can confirm my initial assumption from your answer though =E2=
=80=94 the
> target cache size configuration is not going to be applied automatically =
in
> this scenario, and I'll have to free database cache manually by caling
> cacheMinimize().
>

Cache  management is applied at transaction boundaries. You can and
generally should commit periodically when reading. If you don't, you won't
see updates from other users or processes.

Historically, cacheMinimize is used for large write transactions that also
read a lot of data.

Jim







--=20
> You received this message because you are subscribed to the Google Groups
> "zodb" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to [email protected].
> For more options, visit https://groups.google.com/d/optout.
>

--=20
You received this message because you are subscribed to the Google Groups "=
zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to [email protected].
For more options, visit https://groups.google.com/d/optout.

--000000000000329fae057d38556b
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"auto"><div><br><br><div class=3D"gmail_quote"><div dir=3D"ltr">=
On Sun, Dec 16, 2018, 5:11 PM Vlad Oles &lt;<a href=3D"mailto:vladyslav.ole=
[email protected]">[email protected]</a> wrote:<br></div><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr">On Saturday, December 15, 2018 at 8:30:16 AM=
 UTC-8, Jim Fulton wrote:<blockquote class=3D"gmail_quote" style=3D"margin:=
0;margin-left:0.8ex;border-left:1px #ccc solid;padding-left:1ex"><div dir=
=3D"ltr"><br><br><div class=3D"gmail_quote"><div dir=3D"ltr">On Fri, Dec 14=
, 2018 at 9:54 PM Vlad Oles &lt;<a rel=3D"nofollow noreferrer">vladysl...@g=
mail.com</a>&gt; wrote:<br></div><blockquote class=3D"gmail_quote" style=3D=
"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-le=
ft:1ex"><div dir=3D"ltr">On Wednesday, July 19, 2017 at 9:54:05 AM UTC-7, J=
im Fulton wrote:<blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0=
px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"><div dir=
=3D"ltr"><br><div><div class=3D"gmail_quote"><div>One thing to understand i=
s that ZODB doesn&#39;t limit memory usage.=C2=A0 Ghostifies objects to mee=
t memory settings at certain times like transaction boundaries and when ask=
ed to explicitly.=C2=A0 So a transaction that loads a lot of objects can gr=
ow ram without bound.=C2=A0 This is why applications that load <b>lots</b> =
of objects should explicitly reduce cache size by calling <font face=3D"mon=
ospace, monospace">_p_jar.cacheGC()</font> occasionally. <br></div></div></=
div></div></blockquote><div><br></div><div>Does it mean that after a databa=
se is finalized (no more writing and thus transactions), there is no way to=
 bound its RAM cache size other than manually resetting cache (via cacheGC(=
) or cacheMinimize()) after every N reads?</div></div></blockquote><div><br=
></div><div>Define finalized.</div><div><br></div><div>When a transaction i=
s committed or a connection closed, the connection&#39;s cache is reduced t=
o it&#39;s target size. When a database is closed, the connections and thei=
r memory is freed. (If there are cycles, the Python garbage collection may =
take some time to deal with them.)</div></div></div></blockquote><div><br><=
/div><div>By &quot;finalized&quot; I meant the connection is still open but=
 no more transactions are being commited, so the database is used only for =
reads.</div></div></blockquote></div></div><div dir=3D"auto"><br></div><div=
 dir=3D"auto">Reads are executed in transactions. In fact, most transaction=
s are read transactions.</div><div dir=3D"auto"><br></div><div dir=3D"auto"=
><br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquote class=
=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc solid;padd=
ing-left:1ex"><div dir=3D"ltr"><div><br></div><div>I think I can confirm my=
 initial assumption from your answer though =E2=80=94 the target cache size=
 configuration is not going to be applied automatically in this scenario, a=
nd I&#39;ll have to free database cache manually by caling cacheMinimize().=
<br></div></div></blockquote></div></div><div dir=3D"auto"><br></div><div d=
ir=3D"auto">Cache=C2=A0 management is applied at transaction boundaries. Yo=
u can and generally should commit periodically when reading. If you don&#39=
;t, you won&#39;t see updates from other users or processes.=C2=A0</div><di=
v dir=3D"auto"><br></div><div dir=3D"auto">Historically, cacheMinimize is u=
sed for large write transactions that also read a lot of data.</div><div di=
r=3D"auto"><br></div><div dir=3D"auto">Jim</div><div dir=3D"auto"><br></div=
><div dir=3D"auto"><br></div><div dir=3D"auto"><br></div><div dir=3D"auto">=
<br></div><div dir=3D"auto"><br></div><div dir=3D"auto"><br></div><div dir=
=3D"auto"><br></div><div dir=3D"auto"><div class=3D"gmail_quote"><blockquot=
e class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px #ccc sol=
id;padding-left:1ex"><div dir=3D"ltr"><div></div></div>

<p></p>

-- <br>
You received this message because you are subscribed to the Google Groups &=
quot;zodb&quot; group.<br>
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:[email protected]" target=3D"_bla=
nk" rel=3D"noreferrer">[email protected]</a>.<br>
For more options, visit <a href=3D"https://groups.google.com/d/optout" targ=
et=3D"_blank" rel=3D"noreferrer">https://groups.google.com/d/optout</a>.<br=
>
</blockquote></div></div></div>

<p></p>

-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;zodb&quot; group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:[email protected]">zodb+unsubscri=
[email protected]</a>.<br />
For more options, visit <a href=3D"https://groups.google.com/d/optout">http=
s://groups.google.com/d/optout</a>.<br />

--000000000000329fae057d38556b--