Sqlite checkpoint issue
Venugopal Thotakura <[email protected]> Tue, 30 Jun 2020 00:13:42 -0700 (PDT)
| Newsgroups | gmane.comp.web.zope.zodb |
|---|---|
| Message-ID | <[email protected]> |
------=_Part_1214_1419811628.1593501222537
Content-Type: multipart/alternative;
boundary="----=_Part_1215_2129066879.1593501222538"
------=_Part_1215_2129066879.1593501222538
Content-Type: text/plain; charset="UTF-8"
Hi,
We are using sqlite storage engine, we are facing issues with checkpointing
(WAL size going into GB.. It looks like by default checkpointing is
disabled, so we enabled auto checkpointing..
<pragmas>
wal_autocheckpoint 100
wal_checkpoint FULL
</pragmas>
Now, It looks like its trying to do checkpointing, however due to the
connections arleady open, it couldn't do that..
So to give overview of our architecture..
We connect to db from multiple processes
- 2 read process (async read with our async connection pool to limit the
connections)
- 1 write process (async write)
Also we have tried to manually run the checkpoint, for every 1min (but no
luck)
conn._storage._load_connection.cursor.execute("PRAGMA wal_checkpoint(3)")
we are getting database locked issue.. so I presume, since read connections
are already opened, its unable to complete the process.. I could confirm
when read processes are killed and write can execute it.
Traceback (most recent call last):
File "/opt/app/xxx.py", line 508, in updater
conn._storage._load_connection.cursor.execute("PRAGMA wal_checkpoint(3)")
File
"/usr/local/lib/python3.7/site-packages/relstorage/adapters/sqlite/drivers.py"
, line 113, in execute
return sqlite3.Cursor.execute(self, stmt)
sqlite3.OperationalError: database table is locked
Also I could confirm same behaviour from sqlite3 client.
/opt/app # sqlite3 main.sqlite3
SQLite version 3.28.0 2019-04-16 19:49:53
Enter ".help" for usage hints.
sqlite> PRAGMA wal_checkpoint(3);
0|1148|1148
sqlite>
We have tried closing connections, but zodb connection object close doesn't
close the underlying storage connections.. instead it put them in the
internal pool.. we could find a way to close storage connections with
public APIs.
We have tried to the drop the internal storage conenctions (_load &
_store), then it seems to be working well..
(env) venu@venu:~/workspace/xxx$ lsof main.sqlite3
COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME
python 1953 venu 8ur REG 8,2 376709120 14429800 main.sqlite3
python 1953 venu 12ur REG 8,2 376709120 14429800 main.sqlite3
Here is our async pool.. we prefork the connections (we are seeing
sometimes db.open is taking longer time to create connections, so we are
doing prefork) and use them.
def ctor(self):
tm = transaction.TransactionManager()
conn = db.open(tm)
return conn, tm
store = {
"read": [asyncio.Semaphore(value=rlimit), []],
"write": [asyncio.Semaphore(value=wlimit), []],
}
store["read"][1] = [ctor() for i in range(rlimit)]
store["write"][1] = [ctor() for i in range(wlimit)]
@asynccontextmanager
async def pooled(mode="read"):
lock = store[mode][0]
conns = store[mode][1]
await lock.acquire()
try:
conn = conns.pop(0)
if conn[0].opened is None:
logging.info("connection closed, creating new one")
conn = ctor()
yield conn
finally:
conns.append(conn)
# conn[0].close()
conn[0]._storage._load_connection.drop()
conn[0]._storage._store_connection.drop()
lock.release()
We find this not proper way, kind of hackish and also, lots of stuff is
obscure for us still..
So would like to confirm with you whether we are missing anything or is
there any API's exsits to clear storage conenctions to allow sqlite to do
checkpointing..
Thanks in advace..
- Venu
--
You received this message because you are subscribed to the Google Groups "zodb" group.
To unsubscribe from this group and stop receiving emails from it, send an email to [email protected].
To view this discussion on the web visit https://groups.google.com/d/msgid/zodb/57f8c28c-9e98-43d8-acd0-8f42609c32c2o%40googlegroups.com.
------=_Part_1215_2129066879.1593501222538
Content-Type: text/html; charset="UTF-8"
Content-Transfer-Encoding: quoted-printable
<div dir=3D"ltr">Hi,<div><br></div><div><div>We are using sqlite storage en=
gine, we are facing issues with checkpointing (WAL size going into GB.. It =
looks like by default checkpointing is disabled, so we enabled auto checkpo=
inting..</div><div><br></div><div class=3D"prettyprint" style=3D"background=
-color: rgb(250, 250, 250); border-color: rgb(187, 187, 187); border-style:=
solid; border-width: 1px; overflow-wrap: break-word;"><code class=3D"prett=
yprint"><div class=3D"subprettyprint"><span style=3D"color: #008;" class=3D=
"styled-by-prettify"><pragmas></span><span style=3D"color: #000;" cla=
ss=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0wal_autocheckpoint=
100<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0wal_checkpoint FULL<br></span><span styl=
e=3D"color: #008;" class=3D"styled-by-prettify"></pragmas></span></di=
v></code></div><div><br></div><div><br></div><div>Now, It looks like its tr=
ying to do checkpointing, however due to the connections arleady open, it c=
ouldn't do that..</div><div><br></div><div>So to give overview of our a=
rchitecture..</div><div><br></div><div>We connect to db from multiple proce=
sses</div><div>- 2 read process (async read with our async connection pool =
to limit the connections)</div><div>- 1 write process (async write)</div><d=
iv><br></div><div>Also we have tried to manually run the checkpoint,=C2=A0 =
for every 1min (but no luck)</div><div><div class=3D"prettyprint" style=3D"=
background-color: rgb(250, 250, 250); border-color: rgb(187, 187, 187); bor=
der-style: solid; border-width: 1px; overflow-wrap: break-word;"><code clas=
s=3D"prettyprint"><div class=3D"subprettyprint"><span style=3D"color: #000;=
" class=3D"styled-by-prettify">conn</span><span style=3D"color: #660;" clas=
s=3D"styled-by-prettify">.</span><span style=3D"color: #000;" class=3D"styl=
ed-by-prettify">_storage</span><span style=3D"color: #660;" class=3D"styled=
-by-prettify">.</span><span style=3D"color: #000;" class=3D"styled-by-prett=
ify">_load_connection</span><span style=3D"color: #660;" class=3D"styled-by=
-prettify">.</span><span style=3D"color: #000;" class=3D"styled-by-prettify=
">cursor</span><span style=3D"color: #660;" class=3D"styled-by-prettify">.<=
/span><span style=3D"color: #000;" class=3D"styled-by-prettify">execute</sp=
an><span style=3D"color: #660;" class=3D"styled-by-prettify">(</span><span =
style=3D"color: #080;" class=3D"styled-by-prettify">"PRAGMA wal_checkp=
oint(3)"</span><span style=3D"color: #660;" class=3D"styled-by-prettif=
y">)</span></div></code></div></div><div><br></div><div><br></div><div>we a=
re getting database locked issue.. so I presume, since read connections are=
already opened, its unable to complete the process.. I could confirm when =
read processes are killed and write can execute it.</div><div><div class=3D=
"prettyprint" style=3D"background-color: rgb(250, 250, 250); border-color: =
rgb(187, 187, 187); border-style: solid; border-width: 1px; overflow-wrap: =
break-word;"><code class=3D"prettyprint"><div class=3D"subprettyprint"><spa=
n style=3D"color: #606;" class=3D"styled-by-prettify">Traceback</span><span=
style=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D=
"color: #660;" class=3D"styled-by-prettify">(</span><span style=3D"color: #=
000;" class=3D"styled-by-prettify">most recent call </span><span style=3D"c=
olor: #008;" class=3D"styled-by-prettify">last</span><span style=3D"color: =
#660;" class=3D"styled-by-prettify">):</span><span style=3D"color: #000;" c=
lass=3D"styled-by-prettify"><br>=C2=A0</span><span style=3D"color: #606;" c=
lass=3D"styled-by-prettify">File</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"> </span><span style=3D"color: #080;" class=3D"style=
d-by-prettify">"/opt/app/xxx.py"</span><span style=3D"color: #660=
;" class=3D"styled-by-prettify">,</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"> line </span><span style=3D"color: #066;" class=3D"=
styled-by-prettify">508</span><span style=3D"color: #660;" class=3D"styled-=
by-prettify">,</span><span style=3D"color: #000;" class=3D"styled-by-pretti=
fy"> </span><span style=3D"color: #008;" class=3D"styled-by-prettify">in</s=
pan><span style=3D"color: #000;" class=3D"styled-by-prettify"> updater<br>=
=C2=A0conn</span><span style=3D"color: #660;" class=3D"styled-by-prettify">=
.</span><span style=3D"color: #000;" class=3D"styled-by-prettify">_storage<=
/span><span style=3D"color: #660;" class=3D"styled-by-prettify">.</span><sp=
an style=3D"color: #000;" class=3D"styled-by-prettify">_load_connection</sp=
an><span style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span =
style=3D"color: #000;" class=3D"styled-by-prettify">cursor</span><span styl=
e=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D"colo=
r: #000;" class=3D"styled-by-prettify">execute</span><span style=3D"color: =
#660;" class=3D"styled-by-prettify">(</span><span style=3D"color: #080;" cl=
ass=3D"styled-by-prettify">"PRAGMA wal_checkpoint(3)"</span><span=
style=3D"color: #660;" class=3D"styled-by-prettify">)</span><span style=3D=
"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0</span><span style=3D=
"color: #606;" class=3D"styled-by-prettify">File</span><span style=3D"color=
: #000;" class=3D"styled-by-prettify"> </span><span style=3D"color: #080;" =
class=3D"styled-by-prettify">"/usr/local/lib/python3.7/site-packages/r=
elstorage/adapters/sqlite/drivers.py"</span><span style=3D"color: #660=
;" class=3D"styled-by-prettify">,</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"> line </span><span style=3D"color: #066;" class=3D"=
styled-by-prettify">113</span><span style=3D"color: #660;" class=3D"styled-=
by-prettify">,</span><span style=3D"color: #000;" class=3D"styled-by-pretti=
fy"> </span><span style=3D"color: #008;" class=3D"styled-by-prettify">in</s=
pan><span style=3D"color: #000;" class=3D"styled-by-prettify"> execute<br>=
=C2=A0</span><span style=3D"color: #008;" class=3D"styled-by-prettify">retu=
rn</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> sqlite3=
</span><span style=3D"color: #660;" class=3D"styled-by-prettify">.</span><s=
pan style=3D"color: #606;" class=3D"styled-by-prettify">Cursor</span><span =
style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D"=
color: #000;" class=3D"styled-by-prettify">execute</span><span style=3D"col=
or: #660;" class=3D"styled-by-prettify">(</span><span style=3D"color: #008;=
" class=3D"styled-by-prettify">self</span><span style=3D"color: #660;" clas=
s=3D"styled-by-prettify">,</span><span style=3D"color: #000;" class=3D"styl=
ed-by-prettify"> stmt</span><span style=3D"color: #660;" class=3D"styled-by=
-prettify">)</span><span style=3D"color: #000;" class=3D"styled-by-prettify=
"><br>sqlite3</span><span style=3D"color: #660;" class=3D"styled-by-prettif=
y">.</span><span style=3D"color: #606;" class=3D"styled-by-prettify">Operat=
ionalError</span><span style=3D"color: #660;" class=3D"styled-by-prettify">=
:</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> database=
table </span><span style=3D"color: #008;" class=3D"styled-by-prettify">is<=
/span><span style=3D"color: #000;" class=3D"styled-by-prettify"> locked</sp=
an></div></code></div><div><br></div></div><div><br></div><div>Also I could=
confirm same behaviour from sqlite3 client.</div><div class=3D"prettyprint=
" style=3D"background-color: rgb(250, 250, 250); border-color: rgb(187, 187=
, 187); border-style: solid; border-width: 1px; overflow-wrap: break-word;"=
><code class=3D"prettyprint"><div class=3D"subprettyprint"><span style=3D"c=
olor: #080;" class=3D"styled-by-prettify">/opt/</span><span style=3D"color:=
#000;" class=3D"styled-by-prettify">app </span><span style=3D"color: #800;=
" class=3D"styled-by-prettify"># sqlite3 main.sqlite3</span><span style=3D"=
color: #000;" class=3D"styled-by-prettify"><br></span><span style=3D"color:=
#606;" class=3D"styled-by-prettify">SQLite</span><span style=3D"color: #00=
0;" class=3D"styled-by-prettify"> version </span><span style=3D"color: #066=
;" class=3D"styled-by-prettify">3.28</span><span style=3D"color: #660;" cla=
ss=3D"styled-by-prettify">.</span><span style=3D"color: #066;" class=3D"sty=
led-by-prettify">0</span><span style=3D"color: #000;" class=3D"styled-by-pr=
ettify"> </span><span style=3D"color: #066;" class=3D"styled-by-prettify">2=
019</span><span style=3D"color: #660;" class=3D"styled-by-prettify">-</span=
><span style=3D"color: #066;" class=3D"styled-by-prettify">04</span><span s=
tyle=3D"color: #660;" class=3D"styled-by-prettify">-</span><span style=3D"c=
olor: #066;" class=3D"styled-by-prettify">16</span><span style=3D"color: #0=
00;" class=3D"styled-by-prettify"> </span><span style=3D"color: #066;" clas=
s=3D"styled-by-prettify">19</span><span style=3D"color: #660;" class=3D"sty=
led-by-prettify">:</span><span style=3D"color: #066;" class=3D"styled-by-pr=
ettify">49</span><span style=3D"color: #660;" class=3D"styled-by-prettify">=
:</span><span style=3D"color: #066;" class=3D"styled-by-prettify">53</span>=
<span style=3D"color: #000;" class=3D"styled-by-prettify"><br></span><span =
style=3D"color: #606;" class=3D"styled-by-prettify">Enter</span><span style=
=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D"color=
: #080;" class=3D"styled-by-prettify">".help"</span><span style=
=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=3D"color=
: #008;" class=3D"styled-by-prettify">for</span><span style=3D"color: #000;=
" class=3D"styled-by-prettify"> usage hints</span><span style=3D"color: #66=
0;" class=3D"styled-by-prettify">.</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"><br>sqlite</span><span style=3D"color: #660;" class=
=3D"styled-by-prettify">></span><span style=3D"color: #000;" class=3D"st=
yled-by-prettify"> PRAGMA wal_checkpoint</span><span style=3D"color: #660;"=
class=3D"styled-by-prettify">(</span><span style=3D"color: #066;" class=3D=
"styled-by-prettify">3</span><span style=3D"color: #660;" class=3D"styled-b=
y-prettify">);</span><span style=3D"color: #000;" class=3D"styled-by-pretti=
fy"><br></span><span style=3D"color: #066;" class=3D"styled-by-prettify">0<=
/span><span style=3D"color: #660;" class=3D"styled-by-prettify">|</span><sp=
an style=3D"color: #066;" class=3D"styled-by-prettify">1148</span><span sty=
le=3D"color: #660;" class=3D"styled-by-prettify">|</span><span style=3D"col=
or: #066;" class=3D"styled-by-prettify">1148</span><span style=3D"color: #0=
00;" class=3D"styled-by-prettify"><br>sqlite</span><span style=3D"color: #6=
60;" class=3D"styled-by-prettify">></span><span style=3D"color: #000;" c=
lass=3D"styled-by-prettify"> </span></div></code></div><div><br></div><div>=
<br></div><div>We have tried closing connections, but zodb connection objec=
t close doesn't close the underlying storage connections.. instead it p=
ut them in the internal pool.. we could find a way to close storage connect=
ions with public APIs.</div><div><br></div><div>We have tried to the drop t=
he internal storage conenctions (_load & _store), then it seems to be w=
orking well..</div><div class=3D"prettyprint" style=3D"background-color: rg=
b(250, 250, 250); border-color: rgb(187, 187, 187); border-style: solid; bo=
rder-width: 1px; overflow-wrap: break-word;"><code class=3D"prettyprint"><d=
iv class=3D"subprettyprint"><span style=3D"color: #660;" class=3D"styled-by=
-prettify">(</span><span style=3D"color: #000;" class=3D"styled-by-prettify=
">env</span><span style=3D"color: #660;" class=3D"styled-by-prettify">)</sp=
an><span style=3D"color: #000;" class=3D"styled-by-prettify"> venu@venu</sp=
an><span style=3D"color: #660;" class=3D"styled-by-prettify">:~/</span><spa=
n style=3D"color: #000;" class=3D"styled-by-prettify">workspace</span><span=
style=3D"color: #660;" class=3D"styled-by-prettify">/</span><span style=3D=
"color: #000;" class=3D"styled-by-prettify">xxx$ lsof main</span><span styl=
e=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D"colo=
r: #000;" class=3D"styled-by-prettify">sqlite3<br>COMMAND PID USER FD TYPE =
DEVICE SIZE</span><span style=3D"color: #660;" class=3D"styled-by-prettify"=
>/</span><span style=3D"color: #000;" class=3D"styled-by-prettify">OFF NODE=
NAME<br>python </span><span style=3D"color: #066;" class=3D"styled-by-pret=
tify">1953</span><span style=3D"color: #000;" class=3D"styled-by-prettify">=
venu </span><span style=3D"color: #066;" class=3D"styled-by-prettify">8ur<=
/span><span style=3D"color: #000;" class=3D"styled-by-prettify"> REG </span=
><span style=3D"color: #066;" class=3D"styled-by-prettify">8</span><span st=
yle=3D"color: #660;" class=3D"styled-by-prettify">,</span><span style=3D"co=
lor: #066;" class=3D"styled-by-prettify">2</span><span style=3D"color: #000=
;" class=3D"styled-by-prettify"> </span><span style=3D"color: #066;" class=
=3D"styled-by-prettify">376709120</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"> </span><span style=3D"color: #066;" class=3D"style=
d-by-prettify">14429800</span><span style=3D"color: #000;" class=3D"styled-=
by-prettify"> main</span><span style=3D"color: #660;" class=3D"styled-by-pr=
ettify">.</span><span style=3D"color: #000;" class=3D"styled-by-prettify">s=
qlite3<br>python </span><span style=3D"color: #066;" class=3D"styled-by-pre=
ttify">1953</span><span style=3D"color: #000;" class=3D"styled-by-prettify"=
> venu </span><span style=3D"color: #066;" class=3D"styled-by-prettify">12u=
r</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> REG </sp=
an><span style=3D"color: #066;" class=3D"styled-by-prettify">8</span><span =
style=3D"color: #660;" class=3D"styled-by-prettify">,</span><span style=3D"=
color: #066;" class=3D"styled-by-prettify">2</span><span style=3D"color: #0=
00;" class=3D"styled-by-prettify"> </span><span style=3D"color: #066;" clas=
s=3D"styled-by-prettify">376709120</span><span style=3D"color: #000;" class=
=3D"styled-by-prettify"> </span><span style=3D"color: #066;" class=3D"style=
d-by-prettify">14429800</span><span style=3D"color: #000;" class=3D"styled-=
by-prettify"> main</span><span style=3D"color: #660;" class=3D"styled-by-pr=
ettify">.</span><span style=3D"color: #000;" class=3D"styled-by-prettify">s=
qlite3</span></div></code></div><div><br></div><div><br></div><div>Here is =
our async pool.. we prefork the connections (we are seeing sometimes db.ope=
n is taking longer time to create connections, so we are doing prefork) and=
use them.</div><div><br></div><div class=3D"prettyprint" style=3D"backgrou=
nd-color: rgb(250, 250, 250); border-color: rgb(187, 187, 187); border-styl=
e: solid; border-width: 1px; overflow-wrap: break-word;"><code class=3D"pre=
ttyprint"><div class=3D"subprettyprint"><span style=3D"color: #008;" class=
=3D"styled-by-prettify">def</span><span style=3D"color: #000;" class=3D"sty=
led-by-prettify"> ctor</span><span style=3D"color: #660;" class=3D"styled-b=
y-prettify">(</span><span style=3D"color: #008;" class=3D"styled-by-prettif=
y">self</span><span style=3D"color: #660;" class=3D"styled-by-prettify">):<=
/span><span style=3D"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =
=C2=A0 =C2=A0 =C2=A0 tm </span><span style=3D"color: #660;" class=3D"styled=
-by-prettify">=3D</span><span style=3D"color: #000;" class=3D"styled-by-pre=
ttify"> transaction</span><span style=3D"color: #660;" class=3D"styled-by-p=
rettify">.</span><span style=3D"color: #606;" class=3D"styled-by-prettify">=
TransactionManager</span><span style=3D"color: #660;" class=3D"styled-by-pr=
ettify">()</span><span style=3D"color: #000;" class=3D"styled-by-prettify">=
<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 conn </span><span style=3D"color: #660;" cl=
ass=3D"styled-by-prettify">=3D</span><span style=3D"color: #000;" class=3D"=
styled-by-prettify"> db</span><span style=3D"color: #660;" class=3D"styled-=
by-prettify">.</span><span style=3D"color: #000;" class=3D"styled-by-pretti=
fy">open</span><span style=3D"color: #660;" class=3D"styled-by-prettify">(<=
/span><span style=3D"color: #000;" class=3D"styled-by-prettify">tm</span><s=
pan style=3D"color: #660;" class=3D"styled-by-prettify">)</span><span style=
=3D"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=
=A0 </span><span style=3D"color: #008;" class=3D"styled-by-prettify">return=
</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> conn</spa=
n><span style=3D"color: #660;" class=3D"styled-by-prettify">,</span><span s=
tyle=3D"color: #000;" class=3D"styled-by-prettify"> tm<br><br><br>store </s=
pan><span style=3D"color: #660;" class=3D"styled-by-prettify">=3D</span><sp=
an style=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=
=3D"color: #660;" class=3D"styled-by-prettify">{</span><span style=3D"color=
: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 </span><span style=3D"color: #080;" class=3D"styled-by-prettify"=
>"read"</span><span style=3D"color: #660;" class=3D"styled-by-pre=
ttify">:</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> <=
/span><span style=3D"color: #660;" class=3D"styled-by-prettify">[</span><sp=
an style=3D"color: #000;" class=3D"styled-by-prettify">asyncio</span><span =
style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D"=
color: #606;" class=3D"styled-by-prettify">Semaphore</span><span style=3D"c=
olor: #660;" class=3D"styled-by-prettify">(</span><span style=3D"color: #00=
0;" class=3D"styled-by-prettify">value</span><span style=3D"color: #660;" c=
lass=3D"styled-by-prettify">=3D</span><span style=3D"color: #000;" class=3D=
"styled-by-prettify">rlimit</span><span style=3D"color: #660;" class=3D"sty=
led-by-prettify">),</span><span style=3D"color: #000;" class=3D"styled-by-p=
rettify"> </span><span style=3D"color: #660;" class=3D"styled-by-prettify">=
[]],</span><span style=3D"color: #000;" class=3D"styled-by-prettify"><br>=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 </span><span style=3D"color: #080=
;" class=3D"styled-by-prettify">"write"</span><span style=3D"colo=
r: #660;" class=3D"styled-by-prettify">:</span><span style=3D"color: #000;"=
class=3D"styled-by-prettify"> </span><span style=3D"color: #660;" class=3D=
"styled-by-prettify">[</span><span style=3D"color: #000;" class=3D"styled-b=
y-prettify">asyncio</span><span style=3D"color: #660;" class=3D"styled-by-p=
rettify">.</span><span style=3D"color: #606;" class=3D"styled-by-prettify">=
Semaphore</span><span style=3D"color: #660;" class=3D"styled-by-prettify">(=
</span><span style=3D"color: #000;" class=3D"styled-by-prettify">value</spa=
n><span style=3D"color: #660;" class=3D"styled-by-prettify">=3D</span><span=
style=3D"color: #000;" class=3D"styled-by-prettify">wlimit</span><span sty=
le=3D"color: #660;" class=3D"styled-by-prettify">),</span><span style=3D"co=
lor: #000;" class=3D"styled-by-prettify"> </span><span style=3D"color: #660=
;" class=3D"styled-by-prettify">[]],</span><span style=3D"color: #000;" cla=
ss=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 </span><span styl=
e=3D"color: #660;" class=3D"styled-by-prettify">}</span><span style=3D"colo=
r: #000;" class=3D"styled-by-prettify"><br>store</span><span style=3D"color=
: #660;" class=3D"styled-by-prettify">[</span><span style=3D"color: #080;" =
class=3D"styled-by-prettify">"read"</span><span style=3D"color: #=
660;" class=3D"styled-by-prettify">][</span><span style=3D"color: #066;" cl=
ass=3D"styled-by-prettify">1</span><span style=3D"color: #660;" class=3D"st=
yled-by-prettify">]</span><span style=3D"color: #000;" class=3D"styled-by-p=
rettify"> </span><span style=3D"color: #660;" class=3D"styled-by-prettify">=
=3D</span><span style=3D"color: #000;" class=3D"styled-by-prettify"> </span=
><span style=3D"color: #660;" class=3D"styled-by-prettify">[</span><span st=
yle=3D"color: #000;" class=3D"styled-by-prettify">ctor</span><span style=3D=
"color: #660;" class=3D"styled-by-prettify">()</span><span style=3D"color: =
#000;" class=3D"styled-by-prettify"> </span><span style=3D"color: #008;" cl=
ass=3D"styled-by-prettify">for</span><span style=3D"color: #000;" class=3D"=
styled-by-prettify"> i </span><span style=3D"color: #008;" class=3D"styled-=
by-prettify">in</span><span style=3D"color: #000;" class=3D"styled-by-prett=
ify"> range</span><span style=3D"color: #660;" class=3D"styled-by-prettify"=
>(</span><span style=3D"color: #000;" class=3D"styled-by-prettify">rlimit</=
span><span style=3D"color: #660;" class=3D"styled-by-prettify">)]</span><sp=
an style=3D"color: #000;" class=3D"styled-by-prettify"><br>store</span><spa=
n style=3D"color: #660;" class=3D"styled-by-prettify">[</span><span style=
=3D"color: #080;" class=3D"styled-by-prettify">"write"</span><spa=
n style=3D"color: #660;" class=3D"styled-by-prettify">][</span><span style=
=3D"color: #066;" class=3D"styled-by-prettify">1</span><span style=3D"color=
: #660;" class=3D"styled-by-prettify">]</span><span style=3D"color: #000;" =
class=3D"styled-by-prettify"> </span><span style=3D"color: #660;" class=3D"=
styled-by-prettify">=3D</span><span style=3D"color: #000;" class=3D"styled-=
by-prettify"> </span><span style=3D"color: #660;" class=3D"styled-by-pretti=
fy">[</span><span style=3D"color: #000;" class=3D"styled-by-prettify">ctor<=
/span><span style=3D"color: #660;" class=3D"styled-by-prettify">()</span><s=
pan style=3D"color: #000;" class=3D"styled-by-prettify"> </span><span style=
=3D"color: #008;" class=3D"styled-by-prettify">for</span><span style=3D"col=
or: #000;" class=3D"styled-by-prettify"> i </span><span style=3D"color: #00=
8;" class=3D"styled-by-prettify">in</span><span style=3D"color: #000;" clas=
s=3D"styled-by-prettify"> range</span><span style=3D"color: #660;" class=3D=
"styled-by-prettify">(</span><span style=3D"color: #000;" class=3D"styled-b=
y-prettify">wlimit</span><span style=3D"color: #660;" class=3D"styled-by-pr=
ettify">)]</span><span style=3D"color: #000;" class=3D"styled-by-prettify">=
<br><br><br></span><span style=3D"color: #066;" class=3D"styled-by-prettify=
">@asynccontextmanager</span><span style=3D"color: #000;" class=3D"styled-b=
y-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 async </span><span style=3D"col=
or: #008;" class=3D"styled-by-prettify">def</span><span style=3D"color: #00=
0;" class=3D"styled-by-prettify"> pooled</span><span style=3D"color: #660;"=
class=3D"styled-by-prettify">(</span><span style=3D"color: #000;" class=3D=
"styled-by-prettify">mode</span><span style=3D"color: #660;" class=3D"style=
d-by-prettify">=3D</span><span style=3D"color: #080;" class=3D"styled-by-pr=
ettify">"read"</span><span style=3D"color: #660;" class=3D"styled=
-by-prettify">):</span><span style=3D"color: #000;" class=3D"styled-by-pret=
tify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 </span><span style=3D"c=
olor: #008;" class=3D"styled-by-prettify">lock</span><span style=3D"color: =
#000;" class=3D"styled-by-prettify"> </span><span style=3D"color: #660;" cl=
ass=3D"styled-by-prettify">=3D</span><span style=3D"color: #000;" class=3D"=
styled-by-prettify"> store</span><span style=3D"color: #660;" class=3D"styl=
ed-by-prettify">[</span><span style=3D"color: #000;" class=3D"styled-by-pre=
ttify">mode</span><span style=3D"color: #660;" class=3D"styled-by-prettify"=
>][</span><span style=3D"color: #066;" class=3D"styled-by-prettify">0</span=
><span style=3D"color: #660;" class=3D"styled-by-prettify">]</span><span st=
yle=3D"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 conns </span><span style=3D"color: #660;" class=3D"sty=
led-by-prettify">=3D</span><span style=3D"color: #000;" class=3D"styled-by-=
prettify"> store</span><span style=3D"color: #660;" class=3D"styled-by-pret=
tify">[</span><span style=3D"color: #000;" class=3D"styled-by-prettify">mod=
e</span><span style=3D"color: #660;" class=3D"styled-by-prettify">][</span>=
<span style=3D"color: #066;" class=3D"styled-by-prettify">1</span><span sty=
le=3D"color: #660;" class=3D"styled-by-prettify">]</span><span style=3D"col=
or: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 await </span><span style=3D"color: #008;" class=3D"styled-by-pre=
ttify">lock</span><span style=3D"color: #660;" class=3D"styled-by-prettify"=
>.</span><span style=3D"color: #000;" class=3D"styled-by-prettify">acquire<=
/span><span style=3D"color: #660;" class=3D"styled-by-prettify">()</span><s=
pan style=3D"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 </span><span style=3D"color: #008;" class=3D"st=
yled-by-prettify">try</span><span style=3D"color: #660;" class=3D"styled-by=
-prettify">:</span><span style=3D"color: #000;" class=3D"styled-by-prettify=
"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 conn </span><=
span style=3D"color: #660;" class=3D"styled-by-prettify">=3D</span><span st=
yle=3D"color: #000;" class=3D"styled-by-prettify"> conns</span><span style=
=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D"color=
: #000;" class=3D"styled-by-prettify">pop</span><span style=3D"color: #660;=
" class=3D"styled-by-prettify">(</span><span style=3D"color: #066;" class=
=3D"styled-by-prettify">0</span><span style=3D"color: #660;" class=3D"style=
d-by-prettify">)</span><span style=3D"color: #000;" class=3D"styled-by-pret=
tify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 </span><s=
pan style=3D"color: #008;" class=3D"styled-by-prettify">if</span><span styl=
e=3D"color: #000;" class=3D"styled-by-prettify"> conn</span><span style=3D"=
color: #660;" class=3D"styled-by-prettify">[</span><span style=3D"color: #0=
66;" class=3D"styled-by-prettify">0</span><span style=3D"color: #660;" clas=
s=3D"styled-by-prettify">].</span><span style=3D"color: #000;" class=3D"sty=
led-by-prettify">opened </span><span style=3D"color: #008;" class=3D"styled=
-by-prettify">is</span><span style=3D"color: #000;" class=3D"styled-by-pret=
tify"> </span><span style=3D"color: #008;" class=3D"styled-by-prettify">Non=
e</span><span style=3D"color: #660;" class=3D"styled-by-prettify">:</span><=
span style=3D"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 logging</span><span=
style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=3D=
"color: #000;" class=3D"styled-by-prettify">info</span><span style=3D"color=
: #660;" class=3D"styled-by-prettify">(</span><span style=3D"color: #080;" =
class=3D"styled-by-prettify">"connection closed, creating new one"=
;</span><span style=3D"color: #660;" class=3D"styled-by-prettify">)</span><=
span style=3D"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 conn </span><span s=
tyle=3D"color: #660;" class=3D"styled-by-prettify">=3D</span><span style=3D=
"color: #000;" class=3D"styled-by-prettify"> ctor</span><span style=3D"colo=
r: #660;" class=3D"styled-by-prettify">()</span><span style=3D"color: #000;=
" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 </span><span style=3D"color: #008;" class=3D"styled-by-pr=
ettify">yield</span><span style=3D"color: #000;" class=3D"styled-by-prettif=
y"> conn<br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 </span><span style=3D=
"color: #008;" class=3D"styled-by-prettify">finally</span><span style=3D"co=
lor: #660;" class=3D"styled-by-prettify">:</span><span style=3D"color: #000=
;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 conns</span><span style=3D"color: #660;" class=3D"styled-=
by-prettify">.</span><span style=3D"color: #000;" class=3D"styled-by-pretti=
fy">append</span><span style=3D"color: #660;" class=3D"styled-by-prettify">=
(</span><span style=3D"color: #000;" class=3D"styled-by-prettify">conn</spa=
n><span style=3D"color: #660;" class=3D"styled-by-prettify">)</span><span s=
tyle=3D"color: #000;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0=
=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 </span><span style=3D"color: #800;" cla=
ss=3D"styled-by-prettify"># conn[0].close()</span><span style=3D"color: #00=
0;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 conn</span><span style=3D"color: #660;" class=3D"styled-b=
y-prettify">[</span><span style=3D"color: #066;" class=3D"styled-by-prettif=
y">0</span><span style=3D"color: #660;" class=3D"styled-by-prettify">].</sp=
an><span style=3D"color: #000;" class=3D"styled-by-prettify">_storage</span=
><span style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span st=
yle=3D"color: #000;" class=3D"styled-by-prettify">_load_connection</span><s=
pan style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span style=
=3D"color: #000;" class=3D"styled-by-prettify">drop</span><span style=3D"co=
lor: #660;" class=3D"styled-by-prettify">()</span><span style=3D"color: #00=
0;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=
=A0 =C2=A0 =C2=A0 conn</span><span style=3D"color: #660;" class=3D"styled-b=
y-prettify">[</span><span style=3D"color: #066;" class=3D"styled-by-prettif=
y">0</span><span style=3D"color: #660;" class=3D"styled-by-prettify">].</sp=
an><span style=3D"color: #000;" class=3D"styled-by-prettify">_storage</span=
><span style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span st=
yle=3D"color: #000;" class=3D"styled-by-prettify">_store_connection</span><=
span style=3D"color: #660;" class=3D"styled-by-prettify">.</span><span styl=
e=3D"color: #000;" class=3D"styled-by-prettify">drop</span><span style=3D"c=
olor: #660;" class=3D"styled-by-prettify">()</span><span style=3D"color: #0=
00;" class=3D"styled-by-prettify"><br>=C2=A0 =C2=A0 =C2=A0 =C2=A0 =C2=A0 =
=C2=A0 =C2=A0 =C2=A0 </span><span style=3D"color: #008;" class=3D"styled-by=
-prettify">lock</span><span style=3D"color: #660;" class=3D"styled-by-prett=
ify">.</span><span style=3D"color: #000;" class=3D"styled-by-prettify">rele=
ase</span><span style=3D"color: #660;" class=3D"styled-by-prettify">()</spa=
n><span style=3D"color: #000;" class=3D"styled-by-prettify"><br><br></span>=
</div></code></div><div><br><br></div><div>We find this not proper way, kin=
d of hackish and also, lots of stuff is obscure for us still..=C2=A0</div><=
div><br></div><div>So would like to confirm with you whether we are missing=
anything or is there any API's exsits to clear storage conenctions to =
allow sqlite to do checkpointing..</div><div><br></div><div>Thanks in advac=
e..<br><br><br>- Venu</div></div></div>
<p></p>
-- <br />
You received this message because you are subscribed to the Google Groups &=
quot;zodb" group.<br />
To unsubscribe from this group and stop receiving emails from it, send an e=
mail to <a href=3D"mailto:[email protected]">zodb+unsubscri=
[email protected]</a>.<br />
To view this discussion on the web visit <a href=3D"https://groups.google.c=
om/d/msgid/zodb/57f8c28c-9e98-43d8-acd0-8f42609c32c2o%40googlegroups.com?ut=
m_medium=3Demail&utm_source=3Dfooter">https://groups.google.com/d/msgid/zod=
b/57f8c28c-9e98-43d8-acd0-8f42609c32c2o%40googlegroups.com</a>.<br />
------=_Part_1215_2129066879.1593501222538--
------=_Part_1214_1419811628.1593501222537--