Re: [lustre-devel] RFC: Spill device for Lustre OSD
Jinshan Xiong via lustre-devel <[email protected]> Tue, 4 Nov 2025 11:02:54 -0800
| Newsgroups | org.lustre.lists.lustre-devel |
|---|---|
| Message-ID | <CAEp8vpgg=Q+10L-PSt7NF6_bcVmd5gT=Y9hWx9KR2hhOtcmYKQ@mail.gmail.com> |
--===============6056882253021907676== Content-Type: multipart/alternative; boundary="000000000000570d160642c97b63" --000000000000570d160642c97b63 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Tue, Nov 4, 2025 at 10:39=E2=80=AFAM Patrick Farrell <[email protected]> = wrote: > Right, but if this spill device is separated from the OST, how can it fai= l > over with the OST? If it *can* fail over with the OST, like eg it's just > another network attached device that the OST would access somehow*, why n= ot > either use it for the principal OST or make it a separate OST? > I'm not sure I understand the question. I don't know why failover would get in the way. It would just umount primary OSD and then spill device, and then reverse the process on the new OSS. An OST requires a fully functional OSD. The spill device would use s3fs or gcsfuse, so it won't be a fully functional OSD. > > *this is actually ignoring several questions about failover to another > machine - for example, unless you plan to have a clustered file system on > it, writing to it from a different location requires a handoff and > exclusion, etc > > -Patrick > ------------------------------ > *From:* Jinshan Xiong <[email protected]> > *Sent:* Tuesday, November 4, 2025 11:38 AM > *To:* Patrick Farrell <[email protected]> > *Cc:* Oleg Drokin via lustre-devel <[email protected]>; Oleg > Drokin <[email protected]> > *Subject:* Re: [lustre-devel] RFC: Spill device for Lustre OSD > > > > On Mon, Nov 3, 2025 at 6:47=E2=80=AFPM Patrick Farrell <[email protected]>= wrote: > > I haven=E2=80=99t seen any mention of failover yet in this conversation (= may have > missed it), but if the device is truly local, then in failed over > configurations the data is inaccessible. If it=E2=80=99s *not* local, wh= y not just > make the device part of the OST or an independent OST? > > > It won't be local. Actually, this is designed for the cloud. > > We already have tiered storage based on mirroring; however, that still > requires clients to move data and a file system level scanner to decide > which files move to the cold tier. It's cumbersome to maintain those > clients. > > > > > > > > Or even if it is local only - it could be made an OST > ------------------------------ > *From:* lustre-devel <[email protected]> on behalf of > Oleg Drokin via lustre-devel <[email protected]> > *Sent:* Monday, November 3, 2025 7:58 PM > *To:* [email protected] <[email protected]> > *Cc:* [email protected] <[email protected]> > *Subject:* Re: [lustre-devel] RFC: Spill device for Lustre OSD > > On Mon, 2025-11-03 at 16:33 -0800, Jinshan Xiong wrote: > > > > > > > > > > I am not sure it's a much better idea than the already existing > > > > > HSM > > > > > capabilities we have that would allow you to have "offline" > > > > > objects > > > > > that would be pulled back in when used, but are otherwise just > > > > > visible > > > > > in the metadata only. > > > > > The underlying capabilities are pretty rich esp. if we also > > > > > take > > > > > into > > > > > account the eventual WBC stuff. > > > > > > > > The major problem of current HSM is that it has to have dedicated > > > > clients to move data. Also, scanning the entire Lustre file > > > > system > > > > > > This (dedicated client) is an implementation detail. It could be > > > improved in many ways and the effort spent on this would bring > > > great > > > benefit to everyone? > > > > > > Almost all designs assume some upfront implementation. We (as the > > Lustre team) considered running clients on OST nodes, but cloud users > > are sensitive about their data being exposed elsewhere. > > > > Can you list a few improvements that come to mind? > > Clients between server nodes is probably one of the most obvious > choices indeed. Considering the data already resides on those nodes, I > am not sure I understand the concerns about "exposing" data that's > already on those nodes. If customers are so sensitive, we support data > encryption. > We could also do some direct server-server migration of some sort where > OSTs exchange data without bringing up real clients and doing copies > from userspace. That might be desirable for other reasons for future > functionality (e.g. various caching things people have been long > envisioning) > > > > > > > > takes very long time so it resorts to databases in order to make > > > > correct decisions about which file should be released. By the > > > > time, > > > > the two system will be out of sync. That makes it practically > > > > unusable. > > > > > > This again is an implementation detail, not even hardcoded > > > anywhere. > > > How do you plan for the OST to to know what stuff is not used > > > without > > > resorting to some database or scan? Now take this method and make > > > it > > > report "upstream" where currently HSM implementations resort to > > > databases or scans. > > > > > > > The assumption is that OST sizes are relatively small, up to 100TB. > > Also, scanning local devices in kernel spaces is much faster. So yeah > > there is no database in the way. > > I am not sure why? In general small OSTs are a relatively rare thing > because to reach large FS sizes you would need to many of them, space > balancing becomes a chore and so on. So relatively few do it for some > fringe reasons (e.g. Google) . Majority of people prefer large OSTs. > > Also nothing stops you from doing a per-OST scan (when you do have > small OSTs) and then kicking the results up to the acting agent to do > something about it (or the other way around, the hsm engine can ask > OSTs one by one (picking less busy ones or ones that have the least > free space, or some other factor). And there's absolutely no need to > wait out to query all OSTs, you can get results from one and work on > the data from it while the other OSTs are still thinking (or not, > there's absolutely no requirement to get full filesystem data before > making any decisions). > > > I guess users won't have 1PB OSTs, will they? > > There probably are already? NASA has a known 0.5P OST configuration: > > https://www.nas.nasa.gov/hecc/support/kb/lustre-progressive-file-layout-(= pfl)-with-ssd-and-hdd-pools_680.html#:~:text=3DThe%20available%20SSD%20spac= e%20in%20each%20filesystem,decimal%20(far%20right)%20labels%20of%20each%20O= ST > . > > > > Rereading your proposal, I see that this particular detail is not > > > covered and it's just assumed that "infrequently accessed data" > > > would > > > be somehow known. > > > > I should have mentioned that in the migration section. Also, we need > > to slightly update the OST read to use a local transaction to update > > an object's access time (atime) if it's older than a predefined > > threshold, for example, 10 minutes. > > This is going to be fragile in the face of varying clock times on > different clients potentially not synced with the servers. > Also in the face of -o noatime. > > But yes, I guess it's one way to get this "on the cheap", and the other > trouble I foresee is you are going to have a biased set. Only recently > touched objects (so with fresh atime), unless you plan to retain a > database and update it from such transaction flow, which certainly is > possible, but I am not sure how practical vs some sort of a scan. > > > > > > If the argument is "but OSTs know best what stuff is used" > > > > > (which I > > > > > am > > > > > not sure I buy, after all before you could use something off > > > > > OSTs > > > > > you > > > > > need to open a file I would hope) even then OSTs could just > > > > > signal > > > > > a > > > > > list of "inactive objects" that then a higher level system > > > > > would > > > > > take > > > > > care of by relocatiing somewhere more sensical and changing the > > > > > layout > > > > > to indicate those objects now live elsewhere. > > > > > > > > > > The plus here is you don't need to attach this "Wart" to every > > > > > OST > > > > > and > > > > > configure it everywhere and such, but rather have a central > > > > > location > > > > > that is centrally managed. > > > > > > > > > > > _______________________________________________ > > > lustre-devel mailing list > > > [email protected] > > > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > > > > > _______________________________________________ > lustre-devel mailing list > [email protected] > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > > --000000000000570d160642c97b63 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Tue, Nov 4, = 2025 at 10:39=E2=80=AFAM Patrick Farrell <<a href=3D"mailto:pfarrell@ddn= .com">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_q= uote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,2= 04);padding-left:1ex"><div class=3D"msg-774908302111501245"> <div dir=3D"ltr"> <div style=3D"font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Cali= bri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> Right, but if this spill device is separated from the OST, how can it fail = over with the OST?=C2=A0 If it <i>can</i>=C2=A0fail over with the OST, like eg it's just another netwo= rk attached device that the OST would access somehow*, why not either use i= t for the principal OST or make it a separate OST?</div></div></div></block= quote><div><br></div><div>I'm not sure I understand the question. I don= 't know why failover would get in the way.=C2=A0It would just umount pr= imary OSD and then spill device, and then reverse the process on the new OS= S.</div><div><br></div><div>An OST requires a fully=C2=A0functional OSD. Th= e spill device would use s3fs or gcsfuse, so it won't be a fully functi= onal OSD.</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"= margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-lef= t:1ex"><div class=3D"msg-774908302111501245"><div dir=3D"ltr"> <div style=3D"font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Cali= bri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> <br> *this is actually ignoring several questions about failover to another mach= ine - for example, unless you plan to have a clustered file system on it, w= riting to it from a different location requires a handoff and exclusion, et= c<br> <br> </div> <div style=3D"font-family:Aptos,Aptos_EmbeddedFont,Aptos_MSFontService,Cali= bri,Helvetica,sans-serif;font-size:12pt;color:rgb(0,0,0)"> -Patrick</div> <div id=3D"m_-774908302111501245appendonsend"></div> <hr style=3D"display:inline-block;width:98%"> <div id=3D"m_-774908302111501245divRplyFwdMsg" dir=3D"ltr"><font face=3D"Ca= libri, sans-serif" style=3D"font-size:11pt" color=3D"#000000"><b>From:</b> = Jinshan Xiong <<a href=3D"mailto:[email protected]" target=3D"_blank">= [email protected]</a>><br> <b>Sent:</b> Tuesday, November 4, 2025 11:38 AM<br> <b>To:</b> Patrick Farrell <<a href=3D"mailto:[email protected]" target= =3D"_blank">[email protected]</a>><br> <b>Cc:</b> Oleg Drokin via lustre-devel <<a href=3D"mailto:lustre-devel@= lists.lustre.org" target=3D"_blank">[email protected]</a>>; = Oleg Drokin <<a href=3D"mailto:[email protected]" target=3D"_blank">gr= [email protected]</a>><br> <b>Subject:</b> Re: [lustre-devel] RFC: Spill device for Lustre OSD</font> <div>=C2=A0</div> </div> <div> <div dir=3D"ltr"> <div dir=3D"ltr"><br> </div> <br> <div> <div dir=3D"ltr">On Mon, Nov 3, 2025 at 6:47=E2=80=AFPM Patrick Farrell <= ;<a href=3D"mailto:[email protected]" target=3D"_blank">[email protected]</a>= > wrote:<br> </div> <blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex"> <div> <div><span>I haven=E2=80=99t seen any mention of failover yet in this conve= rsation (may have missed it), but if the device is truly local, then in fai= led over configurations the data is inaccessible.=C2=A0 If it=E2=80=99s *no= t* local, why not just make the device part of the OST or an independent OST?</span></div> </div> </blockquote> <div><br> </div> <div>It won't be local. Actually, this is designed for the cloud.</div> <div><br> </div> <div>We already have tiered storage based on mirroring; however, that still= requires clients to move data and a file system level scanner to decide wh= ich files move to the cold tier. It's cumbersome=C2=A0to maintain those= clients.</div> <div><br> </div> <div>=C2=A0</div> <blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex"> <div> <div>=C2=A0</div> </div> </blockquote> <blockquote style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204= ,204,204);padding-left:1ex"> <div> <div><br> </div> <div><br> </div> <div>Or even if it is local only - it could be made an OST</div> <div></div> <hr style=3D"display:inline-block;width:98%"> <div id=3D"m_-774908302111501245x_m_-3839173029790542839divRplyFwdMsg" dir= =3D"ltr"><font face=3D"Calibri, sans-serif" color=3D"#000000" style=3D"font= -size:11pt"><b>From:</b> lustre-devel <<a href=3D"mailto:lustre-devel-bo= [email protected]" target=3D"_blank">[email protected]= .org</a>> on behalf of Oleg Drokin via lustre-devel <<a href=3D"mailto:lustre-dev= [email protected]" target=3D"_blank">[email protected]</a>>= ;<br> <b>Sent:</b> Monday, November 3, 2025 7:58 PM<br> <b>To:</b> <a href=3D"mailto:[email protected]" target=3D"_blank">jinshan= [email protected]</a> <<a href=3D"mailto:[email protected]" target=3D"_blan= k">[email protected]</a>><br> <b>Cc:</b> <a href=3D"mailto:[email protected]" target=3D"_blan= k">[email protected]</a> <<a href=3D"mailto:lustre-devel@lis= ts.lustre.org" target=3D"_blank">[email protected]</a>><br> <b>Subject:</b> Re: [lustre-devel] RFC: Spill device for Lustre OSD</font> <div>=C2=A0</div> </div> <div><font size=3D"2"><span style=3D"font-size:11pt"> <div>On Mon, 2025-11-03 at 16:33 -0800, Jinshan Xiong wrote:<br> > > > > <br> > > > > I am not sure it's a much better idea than the alre= ady existing<br> > > > > HSM<br> > > > > capabilities we have that would allow you to have "= ;offline"<br> > > > > objects<br> > > > > that would be pulled back in when used, but are otherwi= se just<br> > > > > visible<br> > > > > in the metadata only.<br> > > > > The underlying capabilities are pretty rich esp. if we = also<br> > > > > take<br> > > > > into<br> > > > > account the eventual WBC stuff.<br> > > > <br> > > > The major problem of current HSM is that it has to have dedi= cated<br> > > > clients to move data. Also, scanning the entire Lustre file<= br> > > > system<br> > > <br> > > This (dedicated client) is an implementation detail. It could be<= br> > > improved in many ways and the effort spent on this would bring<br= > > > great<br> > > benefit to everyone?<br> > <br> > <br> > Almost all designs assume some upfront implementation. We (as the<br> > Lustre team) considered running clients on OST nodes, but cloud users<= br> > are sensitive about their data being exposed elsewhere.<br> > <br> > Can you list a few improvements that come to mind?<br> <br> Clients between server nodes is probably one of the most obvious<br> choices indeed. Considering the data already resides on those nodes, I<br> am not sure I understand the concerns about "exposing" data that&= #39;s<br> already on those nodes. If customers are so sensitive, we support data<br> encryption.<br> We could also do some direct server-server migration of some sort where<br> OSTs exchange data without bringing up real clients and doing copies<br> from userspace. That might be desirable for other reasons for future<br> functionality (e.g. various caching things people have been long<br> envisioning)<br> =C2=A0<br> > > <br> > > > takes very long time so it resorts to databases in order to = make<br> > > > correct decisions about which file should be released. By th= e<br> > > > time,<br> > > > the two system will be out of sync. That makes it practicall= y<br> > > > unusable.<br> > > <br> > > This again is an implementation detail, not even hardcoded<br> > > anywhere.<br> > > How do you plan for the OST to to know what stuff is not used<br> > > without<br> > > resorting to some database or scan? Now take this method and make= <br> > > it<br> > > report "upstream" where currently HSM implementations r= esort to<br> > > databases or scans.<br> > > <br> > <br> > The assumption is that OST sizes are relatively small, up to 100TB.<br= > > Also, scanning local devices in kernel spaces is much faster. So yeah<= br> > there is no database in the way.<br> <br> I am not sure why? In general small OSTs are a relatively rare thing<br> because to reach large FS sizes you would need to many of them, space<br> balancing becomes a chore and so on. So relatively few do it for some<br> fringe reasons (e.g. Google) . Majority of people prefer large OSTs.<br> <br> Also nothing stops you from doing a per-OST scan (when you do have<br> small OSTs) and then kicking the results up to the acting agent to do<br> something about it (or the other way around, the hsm engine can ask<br> OSTs one by one (picking less busy ones or ones that have the least<br> free space, or some other factor). And there's absolutely no need to<br= > wait out to query all OSTs, you can get results from one and work on<br> the data from it while the other OSTs are still thinking (or not,<br> there's absolutely no requirement to get full filesystem data before<br= > making any decisions).<br> <br> > I guess users won't have 1PB OSTs, will they?<br> <br> There probably are already? NASA has a known 0.5P OST configuration:<br> <a href=3D"https://www.nas.nasa.gov/hecc/support/kb/lustre-progressive-file= -layout-(pfl)-with-ssd-and-hdd-pools_680.html#:~:text=3DThe%20available%20S= SD%20space%20in%20each%20filesystem,decimal%20(far%20right)%20labels%20of%2= 0each%20OST" target=3D"_blank">https://www.nas.nasa.gov/hecc/support/kb/lus= tre-progressive-file-layout-(pfl)-with-ssd-and-hdd-pools_680.html#:~:text= =3DThe%20available%20SSD%20space%20in%20each%20filesystem,decimal%20(far%20= right)%20labels%20of%20each%20OST</a><br> .<br> <br> > > Rereading your proposal, I see that this particular detail is not= <br> > > covered and it's just assumed that "infrequently accesse= d data"<br> > > would<br> > > be somehow known.<br> > <br> > I should have mentioned that in the migration section. Also, we need<b= r> > to slightly update the OST read to use a local transaction to update<b= r> > an object's access time (atime) if it's older than a predefine= d<br> > threshold, for example, 10 minutes.=C2=A0<br> <br> This is going to be fragile in the face of varying clock times on<br> different clients potentially not synced with the servers.<br> Also in the face of -o noatime.<br> <br> But yes, I guess it's one way to get this "on the cheap", and= the other<br> trouble I foresee is you are going to have a biased set. Only recently<br> touched objects (so with fresh atime), unless you plan to retain a<br> database and update it from such transaction flow, which certainly is<br> possible, but I am not sure how practical vs some sort of a scan.<br> <br> > > > > If the argument is "but OSTs know best what stuff = is used"<br> > > > > (which I<br> > > > > am<br> > > > > not sure I buy, after all before you could use somethin= g off<br> > > > > OSTs<br> > > > > you<br> > > > > need to open a file I would hope) even then OSTs could = just<br> > > > > signal<br> > > > > a<br> > > > > list of "inactive objects" that then a higher= level system<br> > > > > would<br> > > > > take<br> > > > > care of by relocatiing somewhere more sensical and chan= ging the<br> > > > > layout<br> > > > > to indicate those objects now live elsewhere.<br> > > > > <br> > > > > The plus here is you don't need to attach this &quo= t;Wart" to every<br> > > > > OST<br> > > > > and<br> > > > > configure it everywhere and such, but rather have a cen= tral<br> > > > > location<br> > > > > that is centrally managed.<br> > > > > <br> > > <br> > > _______________________________________________<br> > > lustre-devel mailing list<br> > > <a href=3D"mailto:[email protected]" target=3D"_blank= ">[email protected]</a><br> > > <a href=3D"http://lists.lustre.org/listinfo.cgi/lustre-devel-lust= re.org" target=3D"_blank"> http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org</a><br> > > <br> <br> _______________________________________________<br> lustre-devel mailing list<br> <a href=3D"mailto:[email protected]" target=3D"_blank">lustre-d= [email protected]</a><br> <a href=3D"http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org" ta= rget=3D"_blank">http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.or= g</a><br> </div> </span></font></div> </div> </blockquote> </div> </div> </div> </div> </div></blockquote></div></div> --000000000000570d160642c97b63-- --===============6056882253021907676== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lustre-devel mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org --===============6056882253021907676==--