Re: Session file locking not working properly in multithreaded environment
Ralph Heinkel <ralph.heinkel-S0/[email protected]> Fri, 10 May 2019 09:56:42 -0700 (PDT)
| Newsgroups | gmane.comp.python.cherrypy |
|---|---|
| Message-ID | <[email protected]> |
------=_Part_1043_1440267294.1557507403015
Content-Type: multipart/alternative;
boundary="----=_Part_1044_1173022219.1557507403016"
------=_Part_1044_1173022219.1557507403016
Content-Type: text/plain; charset="UTF-8"
I'm not completely convinced that this is true. Here is an example where a
second lock on the same file fails within the same thread:
Python 3.6.5 (default, Mar 31 2018, 19:45:04) [GCC] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import fcntl
>>> flags = fcntl.LOCK_EX | fcntl.LOCK_NB
>>> fp1 = open('x.lock', 'a+') # first file handle and file descriptor for
the lock file
>>> fp2 = open('x.lock', 'a+') # second file handle and descriptor for the
lock file
>>> lock1 = fcntl.flock(fp1.fileno(), flags)
>>> lock2 = fcntl.flock(fp2.fileno(), flags)
Traceback (most recent call last):
File "<stdin>", line 1, in <module>
BlockingIOError: [Errno 11] Resource temporarily unavailable
Only if lock2 was obtained from fp1.fileno() then no exception would have
been raised. As soon as a separate file descriptor (for the same file) is
used a second lock also fails.
But this is exaclty the point that makes me wonder why it then works in
cherrypy. From the log you can see that each thread uses a different file
descriptor for the lock file, so it should fail there as well... I'm still
puzzled.
On Friday, 10 May 2019 18:33:27 UTC+2, Tim Roberts wrote:
>
>
> Ah, you've stumbled across one of the classic blunders, as they say in
> Princess Bride. The Unix "flock" call only blocks simultaneous access
> between different processes. It doesn't protect multiple threads within a
> single process. A single process can call flock repeatedly on a single
> file, and all of the calls will succeed.
>
> If you want to protect a resource between threads of a single process, you
> need to use something other than flock, like a Lock from the "threading"
> module.
>
> --
> Tim Roberts, [email protected] <javascript:>
> Providenza & Boekelheide, Inc.
>
>
--
You received this message because you are subscribed to the Google Groups "cherrypy-users" group.
To unsubscribe from this group and stop receiving emails from it, send an email to cherrypy-users+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
To post to this group, send email to cherrypy-users-/JYPxA39Uh5TLH3MbocFF+G/[email protected]
Visit this group at https://groups.google.com/group/cherrypy-users.
To view this discussion on the web visit https://groups.google.com/d/msgid/cherrypy-users/82728770-ea14-4a5a-ac5b-cad92c18854c%40googlegroups.com.
For more options, visit https://groups.google.com/d/optout.
------=_Part_1044_1173022219.1557507403016
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr"><div>I'm not completely convinced that this is true. H=
ere is an example where a second lock on the same file fails within the sam=
e thread:</div><div><br></div><div><div style=3D"background-color: rgb(250,=
250, 250); border-color: rgb(187, 187, 187); border-style: solid; border-w=
idth: 1px; overflow-wrap: break-word;" class=3D"prettyprint"><code class=3D=
"prettyprint"><div class=3D"subprettyprint"><span style=3D"color: #606;" cl=
ass=3D"styled-by-prettify">Python</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"> </span><span style=3D"color: #066;" class=3D"style=
d-by-prettify">3.6</span><span style=3D"color: #660;" class=3D"styled-by-pr=
ettify">.</span><span style=3D"color: #066;" class=3D"styled-by-prettify">5=
</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> </span><s=
pan style=3D"color: #660;" class=3D"styled-by-prettify">(</span><span style=
=3D"color: #008;" class=3D"styled-by-prettify">default</span><span style=3D=
"color: #660;" class=3D"styled-by-prettify">,</span><span style=3D"color: #=
000;" class=3D"styled-by-prettify"> </span><span style=3D"color: #606;" cla=
ss=3D"styled-by-prettify">Mar</span><span style=3D"color: #000;" class=3D"s=
tyled-by-prettify"> </span><span style=3D"color: #066;" class=3D"styled-by-=
prettify">31</span><span style=3D"color: #000;" class=3D"styled-by-prettify=
"> </span><span style=3D"color: #066;" class=3D"styled-by-prettify">2018</s=
pan><span style=3D"color: #660;" class=3D"styled-by-prettify">,</span><span=
style=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D=
"color: #066;" class=3D"styled-by-prettify">19</span><span style=3D"color: =
#660;" class=3D"styled-by-prettify">:</span><span style=3D"color: #066;" cl=
ass=3D"styled-by-prettify">45</span><span style=3D"color: #660;" class=3D"s=
tyled-by-prettify">:</span><span style=3D"color: #066;" class=3D"styled-by-=
prettify">04</span><span style=3D"color: #660;" class=3D"styled-by-prettify=
">)</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> </span=
><span style=3D"color: #660;" class=3D"styled-by-prettify">[</span><span st=
yle=3D"color: #000;" class=3D"styled-by-prettify">GCC</span><span style=3D"=
color: #660;" class=3D"styled-by-prettify">]</span><span style=3D"color: #0=
00;" class=3D"styled-by-prettify"> on linux <br></span><span style=3D"color=
: #606;" class=3D"styled-by-prettify">Type</span><span style=3D"color: #000=
;" class=3D"styled-by-prettify"> </span><span style=3D"color: #080;" class=
=3D"styled-by-prettify">"help"</span><span style=3D"color: #660;"=
class=3D"styled-by-prettify">,</span><span style=3D"color: #000;" class=3D=
"styled-by-prettify"> </span><span style=3D"color: #080;" class=3D"styled-b=
y-prettify">"copyright"</span><span style=3D"color: #660;" class=
=3D"styled-by-prettify">,</span><span style=3D"color: #000;" class=3D"style=
d-by-prettify"> </span><span style=3D"color: #080;" class=3D"styled-by-pret=
tify">"credits"</span><span style=3D"color: #000;" class=3D"style=
d-by-prettify"> </span><span style=3D"color: #008;" class=3D"styled-by-pret=
tify">or</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> <=
/span><span style=3D"color: #080;" class=3D"styled-by-prettify">"licen=
se"</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> <=
/span><span style=3D"color: #008;" class=3D"styled-by-prettify">for</span><=
span style=3D"color: #000;" class=3D"styled-by-prettify"> more information<=
/span><span style=3D"color: #660;" class=3D"styled-by-prettify">.</span><sp=
an style=3D"color: #000;" class=3D"styled-by-prettify"> <br></span><span st=
yle=3D"color: #660;" class=3D"styled-by-prettify">>>></span><span =
style=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D"=
color: #008;" class=3D"styled-by-prettify">import</span><span style=3D"colo=
r: #000;" class=3D"styled-by-prettify"> fcntl <br></span><span style=3D"col=
or: #660;" class=3D"styled-by-prettify">>>></span><span style=3D"c=
olor: #000;" class=3D"styled-by-prettify"> flags </span><span style=3D"colo=
r: #660;" class=3D"styled-by-prettify">=3D</span><span style=3D"color: #000=
;" class=3D"styled-by-prettify"> fcntl</span><span style=3D"color: #660;" c=
lass=3D"styled-by-prettify">.</span><span style=3D"color: #000;" class=3D"s=
tyled-by-prettify">LOCK_EX </span><span style=3D"color: #660;" class=3D"sty=
led-by-prettify">|</span><span style=3D"color: #000;" class=3D"styled-by-pr=
ettify"> fcntl</span><span style=3D"color: #660;" class=3D"styled-by-pretti=
fy">.</span><span style=3D"color: #000;" class=3D"styled-by-prettify">LOCK_=
NB <br></span><span style=3D"color: #660;" class=3D"styled-by-prettify">>=
;>></span><span style=3D"color: #000;" class=3D"styled-by-prettify"> =
fp1 </span><span style=3D"color: #660;" class=3D"styled-by-prettify">=3D</s=
pan><span style=3D"color: #000;" class=3D"styled-by-prettify"> open</span><=
span style=3D"color: #660;" class=3D"styled-by-prettify">(</span><span styl=
e=3D"color: #080;" class=3D"styled-by-prettify">'x.lock'</span><spa=
n style=3D"color: #660;" class=3D"styled-by-prettify">,</span><span style=
=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D"color=
: #080;" class=3D"styled-by-prettify">'a+'</span><span style=3D"col=
or: #660;" class=3D"styled-by-prettify">)</span><span style=3D"color: #000;=
" class=3D"styled-by-prettify"> =C2=A0</span><span style=3D"color: #800;" c=
lass=3D"styled-by-prettify"># first file handle and file descriptor for the=
lock file</span><span style=3D"color: #000;" class=3D"styled-by-prettify">=
<br></span><span style=3D"color: #660;" class=3D"styled-by-prettify">>&g=
t;></span><span style=3D"color: #000;" class=3D"styled-by-prettify"> fp2=
</span><span style=3D"color: #660;" class=3D"styled-by-prettify">=3D</span=
><span style=3D"color: #000;" class=3D"styled-by-prettify"> open</span><spa=
n style=3D"color: #660;" class=3D"styled-by-prettify">(</span><span style=
=3D"color: #080;" class=3D"styled-by-prettify">'x.lock'</span><span=
style=3D"color: #660;" class=3D"styled-by-prettify">,</span><span style=3D=
"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D"color: #=
080;" class=3D"styled-by-prettify">'a+'</span><span style=3D"color:=
#660;" class=3D"styled-by-prettify">)</span><span style=3D"color: #000;" c=
lass=3D"styled-by-prettify"> =C2=A0</span><span style=3D"color: #800;" clas=
s=3D"styled-by-prettify"># second file handle and descriptor for the lock f=
ile</span><span style=3D"color: #000;" class=3D"styled-by-prettify"><br></s=
pan><span style=3D"color: #660;" class=3D"styled-by-prettify">>>><=
/span><span style=3D"color: #000;" class=3D"styled-by-prettify"> lock1 </sp=
an><span style=3D"color: #660;" class=3D"styled-by-prettify">=3D</span><spa=
n style=3D"color: #000;" class=3D"styled-by-prettify"> fcntl</span><span st=
yle=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D"co=
lor: #000;" class=3D"styled-by-prettify">flock</span><span style=3D"color: =
#660;" class=3D"styled-by-prettify">(</span><span style=3D"color: #000;" cl=
ass=3D"styled-by-prettify">fp1</span><span style=3D"color: #660;" class=3D"=
styled-by-prettify">.</span><span style=3D"color: #000;" class=3D"styled-by=
-prettify">fileno</span><span style=3D"color: #660;" class=3D"styled-by-pre=
ttify">(),</span><span style=3D"color: #000;" class=3D"styled-by-prettify">=
flags</span><span style=3D"color: #660;" class=3D"styled-by-prettify">)</s=
pan><span style=3D"color: #000;" class=3D"styled-by-prettify"> <br></span><=
span style=3D"color: #660;" class=3D"styled-by-prettify">>>></span=
><span style=3D"color: #000;" class=3D"styled-by-prettify"> lock2 </span><s=
pan style=3D"color: #660;" class=3D"styled-by-prettify">=3D</span><span sty=
le=3D"color: #000;" class=3D"styled-by-prettify"> fcntl</span><span style=
=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D"color=
: #000;" class=3D"styled-by-prettify">flock</span><span style=3D"color: #66=
0;" class=3D"styled-by-prettify">(</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify">fp2</span><span style=3D"color: #660;" class=3D"sty=
led-by-prettify">.</span><span style=3D"color: #000;" class=3D"styled-by-pr=
ettify">fileno</span><span style=3D"color: #660;" class=3D"styled-by-pretti=
fy">(),</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> fl=
ags</span><span style=3D"color: #660;" class=3D"styled-by-prettify">)</span=
><span style=3D"color: #000;" class=3D"styled-by-prettify"> <br></span><spa=
n style=3D"color: #606;" class=3D"styled-by-prettify">Traceback</span><span=
style=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D=
"color: #660;" class=3D"styled-by-prettify">(</span><span style=3D"color: #=
000;" class=3D"styled-by-prettify">most recent call </span><span style=3D"c=
olor: #008;" class=3D"styled-by-prettify">last</span><span style=3D"color: =
#660;" class=3D"styled-by-prettify">):</span><span style=3D"color: #000;" c=
lass=3D"styled-by-prettify"> <br>=C2=A0 </span><span style=3D"color: #606;"=
class=3D"styled-by-prettify">File</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"> </span><span style=3D"color: #080;" class=3D"style=
d-by-prettify">"<stdin>"</span><span style=3D"color: #660;"=
class=3D"styled-by-prettify">,</span><span style=3D"color: #000;" class=3D=
"styled-by-prettify"> line </span><span style=3D"color: #066;" class=3D"sty=
led-by-prettify">1</span><span style=3D"color: #660;" class=3D"styled-by-pr=
ettify">,</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> =
</span><span style=3D"color: #008;" class=3D"styled-by-prettify">in</span><=
span style=3D"color: #000;" class=3D"styled-by-prettify"> </span><span styl=
e=3D"color: #080;" class=3D"styled-by-prettify"><module></span><span =
style=3D"color: #000;" class=3D"styled-by-prettify"> <br></span><span style=
=3D"color: #606;" class=3D"styled-by-prettify">BlockingIOError</span><span =
style=3D"color: #660;" class=3D"styled-by-prettify">:</span><span style=3D"=
color: #000;" class=3D"styled-by-prettify"> </span><span style=3D"color: #6=
60;" class=3D"styled-by-prettify">[</span><span style=3D"color: #606;" clas=
s=3D"styled-by-prettify">Errno</span><span style=3D"color: #000;" class=3D"=
styled-by-prettify"> </span><span style=3D"color: #066;" class=3D"styled-by=
-prettify">11</span><span style=3D"color: #660;" class=3D"styled-by-prettif=
y">]</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> </spa=
n><span style=3D"color: #606;" class=3D"styled-by-prettify">Resource</span>=
<span style=3D"color: #000;" class=3D"styled-by-prettify"> temporarily unav=
ailable </span></div></code></div><br></div><div>Only if lock2 was obtained=
from fp1.fileno() then no exception would have been raised. As soon as a s=
eparate file descriptor (for the same file) is used a second lock also fail=
s. <br></div><div>But this is exaclty the point that makes me wonder why it=
then works in cherrypy. From the log you can see that each thread uses a d=
ifferent file descriptor for the lock file, so it should fail there as well=
...=C2=A0 I'm still puzzled.<br></div><br>On Friday, 10 May 2019 18:33:=
27 UTC+2, Tim Roberts wrote:<blockquote class=3D"gmail_quote" style=3D"mar=
gin: 0;margin-left: 0.8ex;border-left: 1px #ccc solid;padding-left: 1ex;">
=20
=20
=20
<div text=3D"#000000" bgcolor=3D"#FFFFFF"><br><p>Ah, you've stumbled =
across one of the classic blunders, as they
say in Princess Bride.=C2=A0 The Unix "flock" call only blo=
cks
simultaneous access between different processes.=C2=A0 It doesn't
protect multiple threads within a single process.=C2=A0 A single
process can call flock repeatedly on a single file, and all of the
calls will succeed.</p>
<p>If you want to protect a resource between threads of a single
process, you need to use something other than flock, like a=C2=A0 Loc=
k
from the "threading" module.<br>
</p>
<pre cols=3D"72">--=20
Tim Roberts, <a href=3D"javascript:" target=3D"_blank" gdf-obfuscated-mailt=
o=3D"RymfcqF5AQAJ" rel=3D"nofollow" onmousedown=3D"this.href=3D'javascr=
ipt:';return true;" onclick=3D"this.href=3D'javascript:';return=
true;">[email protected]</a>
Providenza & Boekelheide, Inc.
</pre>
</div>
</blockquote></div>
<p></p>
-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;cherrypy-users" group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:cherrypy-users+unsubscribe-/[email protected]">cher=
rypy-users+unsubscribe-/[email protected]</a>.<br />
To post to this group, send email to <a href=3D"mailto:cherrypy-users@googl=
egroups.com">cherrypy-users-/[email protected]</a>.<br />
Visit this group at <a href=3D"https://groups.google.com/group/cherrypy-use=
rs">https://groups.google.com/group/cherrypy-users</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msgid/cherrypy-users/82728770-ea14-4a5a-ac5b-cad92c18854c%40googlegrou=
ps.com?utm_medium=3Demail&utm_source=3Dfooter">https://groups.google.com/d/=
msgid/cherrypy-users/82728770-ea14-4a5a-ac5b-cad92c18854c%40googlegroups.co=
m</a>.<br />
For more options, visit <a href=3D"https://groups.google.com/d/optout">http=
s://groups.google.com/d/optout</a>.<br />
------=_Part_1044_1173022219.1557507403016--
------=_Part_1043_1440267294.1557507403015--