Re: sharing a dict between child processes
Waqar Khan <[email protected]> Wed, 6 Nov 2019 08:43:52 -0800
| Newsgroups | gmane.comp.python.twisted |
|---|---|
| Message-ID | <CAJuJkHPX4eiqse5R+BK_uYMRAO=osJNQ2Hmqkm3AU9T1xiiBqQ@mail.gmail.com> |
--===============0959720123133529670==
Content-Type: multipart/alternative; boundary="000000000000a54d730596b0419d"
--000000000000a54d730596b0419d
Content-Type: text/plain; charset="UTF-8"
Hi Barry,
Thanks for the response. Where can I read more about (1). It seems
like that is something I need to explore.
As we already have (2) (cache for each process).
Thanks again for your help.
On Wed, Nov 6, 2019 at 8:39 AM Scott, Barry <[email protected]>
wrote:
> On Wednesday, 6 November 2019 14:21:22 GMT Maarten ter Huurne wrote:
> > On Wednesday, 6 November 2019 07:19:56 CET Waqar Khan wrote:
> > > Hi,
> > > So, I am writing a twisted server. This server spawn multiple child
> > > processes using reactor spawnProcess that initializes a process
> > > protocol.
> > >
> > > Now, each of the childprocess receives some REST requests. Each
> > > process has a dict that acts as cache.
> > > Now, I want to share dict across processes.
> > > In general, python has SharedMemoryManager in multiprocessing module
> > > which would have helped.
> > > https://docs.python.org/3/library/multiprocessing.shared_memory.html#m
> > > ultiprocessing.managers.SharedMemoryManager.SharedMemory But since I
> > > am using twisted internal process implementation, how do I share this
> > > dict across the processes so that all the processes use this common
> > > cache?
> >
> > Keeping a dictionary in SharedMemoryManager seems far from trivial. I
> > don't think you can allocate arbitrary Python objects in the shared
> > memory and even if you could, you would run into problems when one
> > process mutates the dictionary while another is looking up something or
> > also mutating it.
> >
> > It could in theory work if you implement a custom lock-less dictionary,
> > but that would be a lot of work and hard to get right. Also having
> > shared memory mutations be synced between multiple CPU cores could
> > degrade performance, since keeping core-local CPU caches in sync is
> > expensive.
> >
> > Would it be an option to have only one process accept the REST requests,
> > check whether the result is in the cache and only distribute work to the
> > other processes if you get a cache miss? Typically the case where an
> > answer is cached is pretty fast, so perhaps you don't need multiple
> > processes to handle incoming requests.
>
> We have used a couple of ways to cache.
> 1. Use a singleton process to hold the cache and ask it, via IPC, for
> answers
> from the other process.
> 2. have a cache in each process
>
> Barry
>
>
> >
> > Bye,
> > Maarten
> >
> >
> >
> > _______________________________________________
> > Twisted-Python mailing list
> > [email protected]
> > https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
>
>
>
> _______________________________________________
> Twisted-Python mailing list
> [email protected]
> https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-python
>
--000000000000a54d730596b0419d
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi Barry,<div>=C2=A0 =C2=A0 =C2=A0 =C2=A0 Thanks for the r=
esponse. Where can I read more about (1). It seems like that is something I=
need to explore.</div><div>As we already have (2) (cache for each process)=
.</div><div>Thanks again=C2=A0for your help.</div></div><br><div class=3D"g=
mail_quote"><div dir=3D"ltr" class=3D"gmail_attr">On Wed, Nov 6, 2019 at 8:=
39 AM Scott, Barry <<a href=3D"mailto:[email protected]">barry.=
[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_quo=
te" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204=
);padding-left:1ex">On Wednesday, 6 November 2019 14:21:22 GMT Maarten ter =
Huurne wrote:<br>
> On Wednesday, 6 November 2019 07:19:56 CET Waqar Khan wrote:<br>
> > Hi,<br>
> > So, I am writing a twisted server. This server spawn multiple chi=
ld<br>
> > processes using reactor spawnProcess that initializes a process<b=
r>
> > protocol.<br>
> > <br>
> > Now, each of the childprocess receives some REST requests. Each<b=
r>
> > process has a dict that acts as cache.<br>
> > Now, I want to share dict across processes.<br>
> > In general, python has SharedMemoryManager in multiprocessing mod=
ule<br>
> > which would have helped.<br>
> > <a href=3D"https://docs.python.org/3/library/multiprocessing.shar=
ed_memory.html#m" rel=3D"noreferrer" target=3D"_blank">https://docs.python.=
org/3/library/multiprocessing.shared_memory.html#m</a><br>
> > ultiprocessing.managers.SharedMemoryManager.SharedMemory But sinc=
e I<br>
> > am using twisted internal process implementation, how do I share =
this<br>
> > dict across the processes so that all the processes use this comm=
on<br>
> > cache?<br>
> <br>
> Keeping a dictionary in SharedMemoryManager seems far from trivial. I<=
br>
> don't think you can allocate arbitrary Python objects in the share=
d<br>
> memory and even if you could, you would run into problems when one<br>
> process mutates the dictionary while another is looking up something o=
r<br>
> also mutating it.<br>
> <br>
> It could in theory work if you implement a custom lock-less dictionary=
,<br>
> but that would be a lot of work and hard to get right. Also having<br>
> shared memory mutations be synced between multiple CPU cores could<br>
> degrade performance, since keeping core-local CPU caches in sync is<br=
>
> expensive.<br>
> <br>
> Would it be an option to have only one process accept the REST request=
s,<br>
> check whether the result is in the cache and only distribute work to t=
he<br>
> other processes if you get a cache miss? Typically the case where an<b=
r>
> answer is cached is pretty fast, so perhaps you don't need multipl=
e<br>
> processes to handle incoming requests.<br>
<br>
We have used a couple of ways to cache.<br>
1. Use a singleton process to hold the cache and ask it, via IPC, for answe=
rs <br>
from the other process.<br>
2. have a cache in each process<br>
<br>
Barry<br>
<br>
<br>
> <br>
> Bye,<br>
>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0Maarten<br>
> <br>
> <br>
> <br>
> _______________________________________________<br>
> Twisted-Python mailing list<br>
> <a href=3D"mailto:[email protected]" target=3D"_blank">=
[email protected]</a><br>
> <a href=3D"https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-=
python" rel=3D"noreferrer" target=3D"_blank">https://twistedmatrix.com/cgi-=
bin/mailman/listinfo/twisted-python</a><br>
<br>
<br>
<br>
<br>
_______________________________________________<br>
Twisted-Python mailing list<br>
<a href=3D"mailto:[email protected]" target=3D"_blank">Twist=
[email protected]</a><br>
<a href=3D"https://twistedmatrix.com/cgi-bin/mailman/listinfo/twisted-pytho=
n" rel=3D"noreferrer" target=3D"_blank">https://twistedmatrix.com/cgi-bin/m=
ailman/listinfo/twisted-python</a><br>
</blockquote></div>
--000000000000a54d730596b0419d--
--===============0959720123133529670==
Content-Type: text/plain; charset="utf-8"
MIME-Version: 1.0
Content-Transfer-Encoding: base64
Content-Disposition: inline
X19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX19fX18KVHdpc3RlZC1Q
eXRob24gbWFpbGluZyBsaXN0ClR3aXN0ZWQtUHl0aG9uQHR3aXN0ZWRtYXRyaXguY29tCmh0dHBz
Oi8vdHdpc3RlZG1hdHJpeC5jb20vY2dpLWJpbi9tYWlsbWFuL2xpc3RpbmZvL3R3aXN0ZWQtcHl0
aG9uCg==
--===============0959720123133529670==--