Re: [SPDK] Bug in BlobFS
"Harris, James R" <[email protected]> Thu, 11 May 2023 22:48:28 +0000
| Newsgroups | dev.linux.lists.spdk |
|---|---|
| Message-ID | <BYAPR11MB33039C915104E0D5688440DBDE749@BYAPR11MB3303.namprd11.prod.outlook.com> |
--_000_BYAPR11MB33039C915104E0D5688440DBDE749BYAPR11MB3303namp_
Content-Type: text/plain; charset="gb2312"
Content-Transfer-Encoding: base64
SGkgR3Vhbmd5YW5nLA0KDQpZZXMsIHRoaXMgZG9lcyBsb29rIGxpa2UgYSBidWcuDQoNCl9fcmVh
ZF9kb25lKCkgc2hvdWxkIGZpcnN0IGRvIGEgY2hlY2sgbGlrZSB0aGlzOg0KDQppZiAoYnNlcnJu
byAhPSAwKSB7DQogIF9fcndfZG9uZShyZXEsIGJzZXJybm8pOw0KICByZXR1cm47DQp9DQoNClRo
aXMgd2lsbCBhdCBsZWFzdCByZXBvcnQgdGhlIGVycm9yIGNvcnJlY3RseSB1cCB0aGUgc3RhY2su
DQoNCkJ1dCBpdCBhbHNvIGxvb2tzIGxpa2UgYXQgbGVhc3QgZm9yIHlvdXIgd29ya2xvYWQsIGRl
ZmF1bHQgbnVtYmVyIG9mIG1heF9jaGFubmVsX29wcyBpcyBub3Qgc3VmZmljaWVudCBmb3IgdGhl
IHVuZGVybHlpbmcgYmxvYnN0b3JlLiAgQ2FuIHlvdSBkZXNjcmliZSB5b3VyIHdvcmtsb2FkIGlu
IG1vcmUgZGV0YWlsPyAgQnkgZGVmYXVsdCwgaXQgc3VwcG9ydHMgdXAgdG8gNTEyIEkvTyBhdCBh
IHRpbWUgqEMgdGhpcyBjYW4gYmUgaW5jcmVhc2VkIGJ1dCBJoa9kIGxpa2UgdG8gdW5kZXJzdGFu
ZCB0aGUgY29uZmlndXJhdGlvbiBhbmQgd29ya2xvYWQgdGhhdCBpcyBnZW5lcmF0aW5nIG1vcmUg
SS9PIHRoYW4gdGhpcy4NCg0KTm90ZSwgeW91IGNhbiBpbmNyZWFzZSB0aGlzIG51bWJlciBieSBh
ZGRpbmcgdGhlIGZvbGxvd2luZyB0byBzcGRrX2ZzX2xvYWQoKToNCg0Kc3Bka19ic19vcHRzX2lu
aXQoJmJzX29wdHMsIHNpemVvZihic19vcHRzKSk7DQo+Pj4gYnNfb3B0cy5tYXhfY2hhbm5lbF9v
cHMgKj0gNDsgLyogQWxsb2NhdGUgNHggYXMgbWFueSBvcHMgYXMgdXN1YWwuICovDQpic19vcHRz
Lml0ZXJfY2JfZm4gPSBpdGVyX2NiOw0KDQpSZWdhcmRzLA0KDQpKaW0NCg0KDQoNCkZyb206IFNQ
REsgPHNwZGstYm91bmNlc0BsaXN0cy5saW51eGZvdW5kYXRpb24ub3JnPiBvbiBiZWhhbGYgb2Yg
uqu54tH0IDxndWFuZ3lhbmcuaGFuQGZveG1haWwuY29tPg0KRGF0ZTogVGh1cnNkYXksIE1heSAx
MSwgMjAyMyBhdCAzOjM0IFBNDQpUbzogc3BkayA8c3Bka0BsaXN0cy5saW51eGZvdW5kYXRpb24u
b3JnPg0KU3ViamVjdDogW1NQREtdIEJ1ZyBpbiBCbG9iRlMNCkhlbGxvLCB3ZSBzZWVtcyB0byBm
aW5kIGEgZmF0YWwgYnVnIGluIEJsb2JGUywgYW5kIG5lZWQgeW91ciBjb25maXJtLiBUaGFua3Mh
DQoNCkluIG15IGxhc3QgZW1haWwsIEkgc2FpZCB3ZSBlbmNvdW50ZXJlZCB3aXRoIGEgd2VpcmVk
IGJ1Zy4gUm9ja3NkYiByZXBvcnRlZCAiQ29ycnVwdGlvbjogYmxvY2sgY2hlY2tzdW0gbWlzbWF0
Y2giIHdoZW4gdXNpbmcgc3BkayBibG9iZnMgdG8gbWFuYWdlIElPLg0KDQpBZnRlciBtb250aGx5
IGRlYnVnLCB3ZSBmb3VuZCBCbG9iRlMgc2hvdWxkIGJlIGJsYW1lLiBJbiBGVU5DIF9fcmVhZF9k
b25lLCBpdCByZWdhcmQgbGVzcyBvZiB0aGUgaW5jb21pbmcgYXJnICJic2Vycm5vIiwgY2FsbGlu
ZyBfX3J3X2RvbmUocmVxLCAwKSB3aXRoIGVycm5vIDAgZGlyZWN0bHkuDQoNCkFjdHVhbGx5LCB3
ZSBmb3VuZCBGVU5DIHNwZGtfYmxvYl9pb19yZWFkIG1heSBmYWlsIGR1ZSB0byBsYWNrIG9mIHJl
c291cmNlcywgaW5jbHVkaW5nIHJ1bm5pbmcgb3V0IG9mIHJlcSBpbiBCbG9iZnMgc3luYyBkZXYg
b3IgcmVxIGluIGJzX2NoYW5uZWwuIEJsb2JmcyBzaG91bGQgcmVwb3J0ZWQgRU5PTUVNIHRvIFJv
Y2tzZGIsIGJ1dCBjYWxsaW5nIF9fcndfZG9uZShyZXEsIDApIG1lYW5zIG5vIGVycm9yLCBzbyBS
b2Nrc2RiIGNhbGN1bGF0ZSBDU1VNIG9uIGEgbm9uc2Vuc2UgYnVmLCBsZWFkaW5nIHRvICJDb3Jy
dXB0aW9uOiBibG9jayBjaGVja3N1bSBtaXNtYXRjaCIuDQoNCkJ5IHNlYXJjaGluZyBtb2RpZnkg
aGlzdG9yeSwgd2UgZm91bmQgaXQgaXMgdGhlIG9yaWdpbmFsIGNvZGUgd2hlbiBibG9iZnMgd2Fz
IGludHJvZHVjZWQsIGlzIHRoYXQgcmVhbGx5IGEgYnVnLCB1aC1odWg/DQo=
--_000_BYAPR11MB33039C915104E0D5688440DBDE749BYAPR11MB3303namp_
Content-Type: text/html; charset="gb2312"
Content-Transfer-Encoding: quoted-printable
<html xmlns:o=3D"urn:schemas-microsoft-com:office:office" xmlns:w=3D"urn:sc=
hemas-microsoft-com:office:word" xmlns:m=3D"http://schemas.microsoft.com/of=
fice/2004/12/omml" xmlns=3D"http://www.w3.org/TR/REC-html40">
<head>
<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dgb2312">
<meta name=3D"Generator" content=3D"Microsoft Word 15 (filtered medium)">
<style><!--
/* Font Definitions */
@font-face
{font-family:"MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
@font-face
{font-family:"Cambria Math";
panose-1:2 4 5 3 5 4 6 3 2 4;}
@font-face
{font-family:Calibri;
panose-1:2 15 5 2 2 2 4 3 2 4;}
@font-face
{font-family:"PingFang TC";
panose-1:2 11 4 0 0 0 0 0 0 0;}
@font-face
{font-family:"\@PingFang TC";}
@font-face
{font-family:"\@MS Gothic";
panose-1:2 11 6 9 7 2 5 8 2 4;}
/* Style Definitions */
p.MsoNormal, li.MsoNormal, div.MsoNormal
{margin:0in;
font-size:10.0pt;
font-family:"Calibri",sans-serif;}
span.EmailStyle18
{mso-style-type:personal-reply;
font-family:"Calibri",sans-serif;
color:windowtext;}
.MsoChpDefault
{mso-style-type:export-only;
font-size:10.0pt;
mso-ligatures:none;}
@page WordSection1
{size:8.5in 11.0in;
margin:1.0in 1.0in 1.0in 1.0in;}
div.WordSection1
{page:WordSection1;}
--></style>
</head>
<body lang=3D"EN-US" link=3D"#0563C1" vlink=3D"#954F72" style=3D"word-wrap:=
break-word">
<div class=3D"WordSection1">
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">Hi Guangyang,<o:p><=
/o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">Yes, this does look=
like a bug.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">__read_done() shoul=
d first do a check like this:<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">if (bserrno !=3D 0)=
{<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"> __rw_done(re=
q, bserrno);<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"> return;<o:p>=
</o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">}<o:p></o:p></span>=
</p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">This will at least =
report the error correctly up the stack.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">But it also looks l=
ike at least for your workload, default number of max_channel_ops is not su=
fficient for the underlying blobstore. Can you describe your workload=
in more detail? By default, it supports
up to 512 I/O at a time =A8C this can be increased but I=A1=AFd like to un=
derstand the configuration and workload that is generating more I/O than th=
is.<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">Note, you can incre=
ase this number by adding the following to spdk_fs_load():<o:p></o:p></span=
></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">spdk_bs_opts_init(&=
amp;bs_opts, sizeof(bs_opts));<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">>>> bs_opt=
s.max_channel_ops *<b>=3D 4; /</b>* Allocate 4x as many ops as usual. */<o:=
p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">bs_opts.iter_cb_fn =
=3D iter_cb;<o:p></o:p></span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">Regards,<o:p></o:p>=
</span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt">Jim<o:p></o:p></spa=
n></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<p class=3D"MsoNormal"><span style=3D"font-size:11.0pt"><o:p> </o:p></=
span></p>
<div style=3D"border:none;border-top:solid #B5C4DF 1.0pt;padding:3.0pt 0in =
0in 0in">
<p class=3D"MsoNormal" style=3D"mso-margin-top-alt:0in;margin-right:0in;mar=
gin-bottom:12.0pt;margin-left:.5in">
<b><span style=3D"font-size:12.0pt;color:black">From: </span></b><span styl=
e=3D"font-size:12.0pt;color:black">SPDK <[email protected]=
ion.org> on behalf of
</span><span style=3D"font-size:12.0pt;font-family:"PingFang TC",=
sans-serif;color:black">=BA=AB</span><span style=3D"font-size:12.0pt;font-f=
amily:"MS Gothic";color:black">=B9=E2=D1=F4</span><span style=3D"=
font-size:12.0pt;color:black"> <[email protected]><br>
<b>Date: </b>Thursday, May 11, 2023 at 3:34 PM<br>
<b>To: </b>spdk <[email protected]><br>
<b>Subject: </b>[SPDK] Bug in BlobFS<o:p></o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt">Hello, we seems to find a fatal bug in BlobFS, and need your confir=
m. Thanks!<o:p></o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt">In my last email, I said we encountered with a weired bug. Rocksdb =
reported "Corruption: block checksum mismatch" when using spdk bl=
obfs to manage IO. <o:p></o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt">After monthly debug, we found BlobFS should be blame. In FUNC __rea=
d_done, it regard less of the incoming arg "bserrno", calling __r=
w_done(req, 0) with errno 0 directly.<o:p></o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt">Actually, we found FUNC spdk_blob_io_read may fail due to lack of r=
esources, including running out of req in Blobfs sync dev or req in bs_chan=
nel. Blobfs should reported ENOMEM to
Rocksdb, but calling __rw_done(req, 0) means no error, so Rocksdb cal=
culate CSUM on a nonsense buf, leading to "Corruption: block checksum =
mismatch".<o:p></o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt"><o:p> </o:p></span></p>
</div>
<div>
<p class=3D"MsoNormal" style=3D"margin-left:.5in"><span style=3D"font-size:=
11.0pt">By searching modify history, we found it is the original code when =
blobfs was introduced, is that really a bug, uh-huh?<o:p></o:p></span></p>
</div>
</div>
</body>
</html>
--_000_BYAPR11MB33039C915104E0D5688440DBDE749BYAPR11MB3303namp_--