Re: In memory cache in twisted

--===============5919468365019765401==
Content-Type: multipart/alternative; boundary="00000000000059fdce059380c3d1"

--00000000000059fdce059380c3d1
Content-Type: text/plain; charset="UTF-8"

Hi Maarten,
   I think you have hit the problem in the head. I do think this is
feasible as I have observed that as size of cache increases, things do get
better which might support your theory.

Is there a simple example you can add on "put a Deferred for the fetch
operation ". I am really just getting started with twisted.
Thanks for all the help.


On Thu, Sep 26, 2019 at 11:40 PM Maarten ter Huurne <[email protected]>
wrote:

> On Friday, 27 September 2019 04:38:46 CEST Waqar Khan wrote:
> > Hi,
> >   What's a good way to use a simple dictionary as a cache in twisted
> > framework?
> > Basically, I have this callback chain where I ultimately make a rest
> > call (in non-blocking way using treq) to fetch some data. But before
> > I make the call, I am using a dictionary to see if the value is
> > available or not. But, I have noticed that the event loop gets pretty
> > busy(sometimes, things get stuck and twisted server stops) as soon as
> > I add this logic.. Which is pretty much
> >
> > @defer.inlinecallbacks
> > def fetch(key):
> >       if key in cache:
> >                return cache[key]
> >       # else call back to treq to fetch value
> >        cache[key] = value
> >        return value
> >
> > This dict can grow to around 50k.. What's a good way to solve this
> > issue?
>
> If it gets stuck, then the cause for that is probably in the part of the
> code you omitted. So it would help to elaborate on how the value is
> fetched exactly.
>
> I can see two other problems with this caching mechanism though:
>
> 1. Items are never removed from the cache, so unless there is a limit to
> the number of different keys that can be used, the cache can grow
> indefinitely. You might want something like an LRU cache rather than a
> plain dictionary.
>
> https://docs.python.org/3/library/functools.html#functools.lru_cache
>
> 2. If a lot of clients are requesting the same thing, you won't see any
> benefits from caching until the first request completes. So you could
> get a pattern like this:
>
> T=0: key A requested, A is not cached, start fetch #1 of A
> T=1: key A requested, A is not cached, start fetch #2 of A
> T=2: key A requested, A is not cached, start fetch #3 of A
> T=3: key A requested, A is not cached, start fetch #4 of A
> T=4: key A requested, A is not cached, start fetch #5 of A
> T=5: fetch #1 of A completes and is added to the cache
> T=6: key A requested, A is cached, return value immediately
>
> In this example, the value for A is fetched 5 times despite the caching
> mechanism. If the fetching takes a long time compared to the rate at
> which requests are coming in, this effect gets worse at a quadratic
> rate: the total time spent fetching is the number of requests that come
> in during the fetching of the first request times the duration of the
> fetch.
>
> To avoid this, you could put a Deferred for the fetch operation in the
> cache or in a separate dictionary and if you get another request for the
> same key before the fetch completes, return that Deferred instead of
> starting another fetch.
>
> Bye,
>                 Maarten
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> [email protected]
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>

--00000000000059fdce059380c3d1
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable

<div dir=3D"ltr">Hi Maarten,<div>=C2=A0 =C2=A0I think you have hit the prob=
lem in the head. I do think this is feasible as I have observed that as siz=
e of cache increases, things do get better which might support your theory.=
</div><div><br></div><div>Is there a simple example you can add on &quot;pu=
t a Deferred for the fetch operation &quot;. I am really just getting start=
ed with twisted.</div><div>Thanks for all the help.</div><div><br></div></d=
iv><br><div class=3D"gmail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On =
Thu, Sep 26, 2019 at 11:40 PM Maarten ter Huurne &lt;<a href=3D"mailto:maar=
[email protected]">[email protected]</a>&gt; wrote:<br></div><blockqu=
ote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px=
 solid rgb(204,204,204);padding-left:1ex">On Friday, 27 September 2019 04:3=
8:46 CEST Waqar Khan wrote:<br>
&gt; Hi,<br>
&gt;=C2=A0 =C2=A0What&#39;s a good way to use a simple dictionary as a cach=
e in twisted<br>
&gt; framework?<br>
&gt; Basically, I have this callback chain where I ultimately make a rest<b=
r>
&gt; call (in non-blocking way using treq) to fetch some data. But before<b=
r>
&gt; I make the call, I am using a dictionary to see if the value is<br>
&gt; available or not. But, I have noticed that the event loop gets pretty<=
br>
&gt; busy(sometimes, things get stuck and twisted server stops) as soon as<=
br>
&gt; I add this logic.. Which is pretty much<br>
&gt; <br>
&gt; @defer.inlinecallbacks<br>
&gt; def fetch(key):<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0if key in cache:<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 return cache[ke=
y]<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0# else call back to treq to fetch value<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 cache[key] =3D value<br>
&gt;=C2=A0 =C2=A0 =C2=A0 =C2=A0 return value<br>
&gt; <br>
&gt; This dict can grow to around 50k.. What&#39;s a good way to solve this=
<br>
&gt; issue?<br>
<br>
If it gets stuck, then the cause for that is probably in the part of the <b=
r>
code you omitted. So it would help to elaborate on how the value is <br>
fetched exactly.<br>
<br>
I can see two other problems with this caching mechanism though:<br>
<br>
1. Items are never removed from the cache, so unless there is a limit to <b=
r>
the number of different keys that can be used, the cache can grow <br>
indefinitely. You might want something like an LRU cache rather than a <br>
plain dictionary.<br>
<br>
<a href=3D"https://docs.python.org/3/library/functools.html#functools.lru_c=
ache" rel=3D"noreferrer" target=3D"_blank">https://docs.python.org/3/librar=
y/functools.html#functools.lru_cache</a><br>
<br>
2. If a lot of clients are requesting the same thing, you won&#39;t see any=
 <br>
benefits from caching until the first request completes. So you could <br>
get a pattern like this:<br>
<br>
T=3D0: key A requested, A is not cached, start fetch #1 of A<br>
T=3D1: key A requested, A is not cached, start fetch #2 of A<br>
T=3D2: key A requested, A is not cached, start fetch #3 of A<br>
T=3D3: key A requested, A is not cached, start fetch #4 of A<br>
T=3D4: key A requested, A is not cached, start fetch #5 of A<br>
T=3D5: fetch #1 of A completes and is added to the cache<br>
T=3D6: key A requested, A is cached, return value immediately<br>
<br>
In this example, the value for A is fetched 5 times despite the caching=C2=
=A0 <br>
mechanism. If the fetching takes a long time compared to the rate at <br>
which requests are coming in, this effect gets worse at a quadratic <br>
rate: the total time spent fetching is the number of requests that come <br=
>
in during the fetching of the first request times the duration of the <br>
fetch.<br>
<br>
To avoid this, you could put a Deferred for the fetch operation in the <br>
cache or in a separate dictionary and if you get another request for the <b=
r>
same key before the fetch completes, return that Deferred instead of <br>
starting another fetch.<br>
<br>
Bye,<br>
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 Maarten<br>
<br>
<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Twist=
[email protected]</a><br>
<a href=3D"https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-pytho=
n" rel=3D"noreferrer" target=3D"_blank">https://twistedmatrix.com/cgi-bin/m=
ailman/listinfo/twisted-python</a><br>
</blockquote></div>

--00000000000059fdce059380c3d1--

--===============5919468365019765401==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline

X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVHdpc3RlZC1Q
eXRob24gbWFpbGluZyBsaXN0ClR3aXN0ZWQtUHl0aG9uQHR3aXN0ZWRtYXRyaXguY29tCmh0dHBz
Oi8vdHdpc3RlZG1hdHJpeC5jb20vY2dpLWJpbi9tYWlsbWFuL2xpc3RpbmZvL3R3aXN0ZWQtcHl0
aG9uCg==

--===============5919468365019765401==--