Re: [lustre-devel] RFC: Spill device for Lustre OSD
Jinshan Xiong via lustre-devel <[email protected]> Tue, 4 Nov 2025 16:28:05 -0800
| Newsgroups | org.lustre.lists.lustre-devel |
|---|---|
| Message-ID | <CAEp8vphfZW4CjgQpCQi6yDBACoTP_3n-5EAFckrOi802_Ajmjw@mail.gmail.com> |
--===============2938527279369961605== Content-Type: multipart/alternative; boundary="0000000000004b06af0642ce0698" --0000000000004b06af0642ce0698 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 4, 2025 at 4:20=E2=80=AFPM Andreas Dilger via lustre-devel < [email protected]> wrote: > On Nov 4, 2025, at 4:42 PM, Jinshan Xiong wrote: > > On Nov 4, 2025, at 15:07, Andreas Dilger <[email protected]> wrote: > >> Timothy Day <[email protected]> wrote: > >>> Overall, I think the concept is interesting. It reminds me of how > >>> Bcachefs handle multi-device support. Each device can be > >>> designated as holding metadata or data replicas. And you > >>> can control the promotion and migration between different > >>> targets (all managed by a migration daemon). But this design is > >>> too limited, IMHO. If we're going to accept the additional complexity > >>> in the OSD, the solution has to be extensible. What if I want to > >>> replicate to multiple targets? What if I want more than two tiers? > >>> What if I want to transparently migrate data from one spill device to > >>> another? We don't need this for the initial implementation, sure. > >>> But these seem like natural extensions. > > > > It=E2=80=99s possible to extend the design to have multiple spill devic= es in the > OSD; you could have two spill devices and mirror them, or raid0 to make a > larger device. I don=E2=80=99t see the design would not allow you to do t= hat. > > > >> This is essentially replicating Lustre file layouts in the end, which > >> was my original suggestion - to use FLR and/or PCC-RO foreign > >> mirror layouts for this, even if it is not directly accessible from > >> clients. That avoids reimplementing tools/formats that already > >> exist in Lustre today for relatively little benefit. > > > > One of the goals is to not have a file system-level scanner, which is > not good. Otherwise, we can just use FLR-based tiered storage. > > I don't see how the layout xattr format is related to filesystem-level > scanning? This is "just" the xattr format *STORED ON THE OST OBJECT*, > and it could be handled by direct device-level (OST) scan tools as well. > I see. I initially thought you proposed the spill device would function as a mirror and that the existing software stack would handle it. Are we going to store the spilled object's status in the layout? It sounds difficult because you will have to initiate a layout change from the OSTs. I think we still need to store such an xattr locally in the OST object. > > This would allow mirrors (FLR), multiple spill devices (multiple FLR > mirrors), concatenated devices (PFL), etc. It just avoids adding new > formats that need to be parsed, printed, etc. > > Cheers, Andreas > > > > > > _______________________________________________ > lustre-devel mailing list > [email protected] > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > --0000000000004b06af0642ce0698 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Nov 4, = 2025 at 4:20=E2=80=AFPM Andreas Dilger via lustre-devel <<a href=3D"mail= to:[email protected]">[email protected]</a>> wro= te:<br></div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px = 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">On Nov 4, 20= 25, at 4:42 PM, Jinshan Xiong wrote:<br> > On Nov 4, 2025, at 15:07, Andreas Dilger <<a href=3D"mailto:adilger= @ddn.com" target=3D"_blank">[email protected]</a>> wrote:<br> >> Timothy Day <<a href=3D"mailto:[email protected]" target=3D"_bl= ank">[email protected]</a>> wrote:<br> >>> Overall, I think the concept is interesting. It reminds me of = how<br> >>> Bcachefs handle multi-device support. Each device can be<br> >>> designated as holding metadata or data replicas. And you<br> >>> can control the promotion and migration between different<br> >>> targets (all managed by a migration daemon). But this design i= s<br> >>> too limited, IMHO. If we're going to accept the additional= complexity<br> >>> in the OSD, the solution has to be extensible. What if I want = to<br> >>> replicate to multiple targets? What if I want more than two ti= ers?<br> >>> What if I want to transparently migrate data from one spill de= vice to<br> >>> another? We don't need this for the initial implementation= , sure.<br> >>> But these seem like natural extensions.<br> > <br> > It=E2=80=99s possible to extend the design to have multiple spill devi= ces in the OSD; you could have two spill devices and mirror them, or raid0 = to make a larger device. I don=E2=80=99t see the design would not allow you= to do that.<br> > <br> >> This is essentially replicating Lustre file layouts in the end, wh= ich<br> >> was my original suggestion - to use FLR and/or PCC-RO foreign<br> >> mirror layouts for this, even if it is not directly accessible fro= m<br> >> clients.=C2=A0 That avoids reimplementing tools/formats that alrea= dy<br> >> exist in Lustre today for relatively little benefit.<br> > <br> > One of the goals is to not have a file system-level scanner, which is = not good. Otherwise, we can just use FLR-based tiered storage.<br> <br> I don't see how the layout xattr format is related to filesystem-level<= br> scanning?=C2=A0 This is "just" the xattr format *STORED ON THE OS= T OBJECT*,<br> and it could be handled by direct device-level (OST) scan tools as well.<br= ></blockquote><div><br></div><div>I see. I initially thought you proposed t= he spill device would function as a mirror and that the existing software s= tack would handle it.</div><div><br></div><div>Are we going to store the sp= illed object's status in the layout? It sounds difficult because you wi= ll have to initiate a layout change from the OSTs. I think we still need to= store such an xattr locally in the OST object.</div><div>=C2=A0</div><bloc= kquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:= 1px solid rgb(204,204,204);padding-left:1ex"> <br> This would allow mirrors (FLR), multiple spill devices (multiple FLR<br> mirrors), concatenated devices (PFL), etc.=C2=A0 It just avoids adding new<= br> formats that need to be parsed, printed, etc.<br> <br> Cheers, Andreas<br> <br> <br> <br> <br> <br> _______________________________________________<br> lustre-devel mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">lustre-d= [email protected]</a><br> <a href=3D"http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org" re= l=3D"noreferrer" target=3D"_blank">http://lists.lustre.org/listinfo.cgi/lus= tre-devel-lustre.org</a><br> </blockquote></div></div> --0000000000004b06af0642ce0698-- --===============2938527279369961605== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lustre-devel mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org --===============2938527279369961605==--