Re: [lustre-devel] RFC: Spill device for Lustre OSD
Jinshan Xiong via lustre-devel <[email protected]> Tue, 4 Nov 2025 09:47:36 -0800
| Newsgroups | org.lustre.lists.lustre-devel |
|---|---|
| Message-ID | <CAEp8vpj35nY7atGPwXvjsPKdnKYfyE+w7uJ7gihXrC3_W7G15Q@mail.gmail.com> |
--===============6784438486419368419== Content-Type: multipart/alternative; boundary="0000000000000d76ba0642c86e5e" --0000000000000d76ba0642c86e5e Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable On Mon, Nov 3, 2025 at 5:58=E2=80=AFPM Oleg Drokin <[email protected]> wr= ote: > On Mon, 2025-11-03 at 16:33 -0800, Jinshan Xiong wrote: > > > > > > > > > > I am not sure it's a much better idea than the already existing > > > > > HSM > > > > > capabilities we have that would allow you to have "offline" > > > > > objects > > > > > that would be pulled back in when used, but are otherwise just > > > > > visible > > > > > in the metadata only. > > > > > The underlying capabilities are pretty rich esp. if we also > > > > > take > > > > > into > > > > > account the eventual WBC stuff. > > > > > > > > The major problem of current HSM is that it has to have dedicated > > > > clients to move data. Also, scanning the entire Lustre file > > > > system > > > > > > This (dedicated client) is an implementation detail. It could be > > > improved in many ways and the effort spent on this would bring > > > great > > > benefit to everyone? > > > > > > Almost all designs assume some upfront implementation. We (as the > > Lustre team) considered running clients on OST nodes, but cloud users > > are sensitive about their data being exposed elsewhere. > > > > Can you list a few improvements that come to mind? > > Clients between server nodes is probably one of the most obvious > choices indeed. Considering the data already resides on those nodes, I > am not sure I understand the concerns about "exposing" data that's > already on those nodes. If customers are so sensitive, we support data > encryption. > We could also do some direct server-server migration of some sort where > OSTs exchange data without bringing up real clients and doing copies > from userspace. That might be desirable for other reasons for future > functionality (e.g. various caching things people have been long > envisioning) This is going too far ;-) > > > > > > > > takes very long time so it resorts to databases in order to make > > > > correct decisions about which file should be released. By the > > > > time, > > > > the two system will be out of sync. That makes it practically > > > > unusable. > > > > > > This again is an implementation detail, not even hardcoded > > > anywhere. > > > How do you plan for the OST to to know what stuff is not used > > > without > > > resorting to some database or scan? Now take this method and make > > > it > > > report "upstream" where currently HSM implementations resort to > > > databases or scans. > > > > > > > The assumption is that OST sizes are relatively small, up to 100TB. > > Also, scanning local devices in kernel spaces is much faster. So yeah > > there is no database in the way. > > I am not sure why? In general small OSTs are a relatively rare thing > because to reach large FS sizes you would need to many of them, space > balancing becomes a chore and so on. So relatively few do it for some > fringe reasons (e.g. Google) . Majority of people prefer large OSTs. > > Also nothing stops you from doing a per-OST scan (when you do have > small OSTs) and then kicking the results up to the acting agent to do > something about it (or the other way around, the hsm engine can ask > OSTs one by one (picking less busy ones or ones that have the least > free space, or some other factor). And there's absolutely no need to > wait out to query all OSTs, you can get results from one and work on > the data from it while the other OSTs are still thinking (or not, > there's absolutely no requirement to get full filesystem data before > making any decisions). > > > I guess users won't have 1PB OSTs, will they? > > There probably are already? NASA has a known 0.5P OST configuration: > > https://www.nas.nasa.gov/hecc/support/kb/lustre-progressive-file-layout-(= pfl)-with-ssd-and-hdd-pools_680.html#:~:text=3DThe%20available%20SSD%20spac= e%20in%20each%20filesystem,decimal%20(far%20right)%20labels%20of%20each%20O= ST > . > > > > Rereading your proposal, I see that this particular detail is not > > > covered and it's just assumed that "infrequently accessed data" > > > would > > > be somehow known. > > > > I should have mentioned that in the migration section. Also, we need > > to slightly update the OST read to use a local transaction to update > > an object's access time (atime) if it's older than a predefined > > threshold, for example, 10 minutes. > > This is going to be fragile in the face of varying clock times on > different clients potentially not synced with the servers. > Also in the face of -o noatime. > It doesn't use client timestamps. Also, it won't be part of read because read doesn't initiate a transaction. It can simply use the OSS local time and start a local transaction to update the atime. > > But yes, I guess it's one way to get this "on the cheap", and the other > trouble I foresee is you are going to have a biased set. Only recently > touched objects (so with fresh atime), unless you plan to retain a > database and update it from such transaction flow, which certainly is > possible, but I am not sure how practical vs some sort of a scan. > I don't see that as an issue. It's going to update the atime in memory. And yeah if the OSS is crashed and the scanner may choose a wrong file to migrate, this should be rare and I don't think this would become a severe issue. > > > > > > If the argument is "but OSTs know best what stuff is used" > > > > > (which I > > > > > am > > > > > not sure I buy, after all before you could use something off > > > > > OSTs > > > > > you > > > > > need to open a file I would hope) even then OSTs could just > > > > > signal > > > > > a > > > > > list of "inactive objects" that then a higher level system > > > > > would > > > > > take > > > > > care of by relocatiing somewhere more sensical and changing the > > > > > layout > > > > > to indicate those objects now live elsewhere. > > > > > > > > > > The plus here is you don't need to attach this "Wart" to every > > > > > OST > > > > > and > > > > > configure it everywhere and such, but rather have a central > > > > > location > > > > > that is centrally managed. > > > > > > > > > > > _______________________________________________ > > > lustre-devel mailing list > > > [email protected] > > > http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org > > > > > --0000000000000d76ba0642c86e5e Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable <div dir=3D"ltr"><div dir=3D"ltr"><br></div><br><div class=3D"gmail_quote g= mail_quote_container"><div dir=3D"ltr" class=3D"gmail_attr">On Mon, Nov 3, = 2025 at 5:58=E2=80=AFPM Oleg Drokin <<a href=3D"mailto:[email protected]= om">[email protected]</a>> wrote:<br></div><blockquote class=3D"gmail_= quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,= 204);padding-left:1ex">On Mon, 2025-11-03 at 16:33 -0800, Jinshan Xiong wro= te:<br> > > > > <br> > > > > I am not sure it's a much better idea than the alre= ady existing<br> > > > > HSM<br> > > > > capabilities we have that would allow you to have "= ;offline"<br> > > > > objects<br> > > > > that would be pulled back in when used, but are otherwi= se just<br> > > > > visible<br> > > > > in the metadata only.<br> > > > > The underlying capabilities are pretty rich esp. if we = also<br> > > > > take<br> > > > > into<br> > > > > account the eventual WBC stuff.<br> > > > <br> > > > The major problem of current HSM is that it has to have dedi= cated<br> > > > clients to move data. Also, scanning the entire Lustre file<= br> > > > system<br> > > <br> > > This (dedicated client) is an implementation detail. It could be<= br> > > improved in many ways and the effort spent on this would bring<br= > > > great<br> > > benefit to everyone?<br> > <br> > <br> > Almost all designs assume some upfront implementation. We (as the<br> > Lustre team) considered running clients on OST nodes, but cloud users<= br> > are sensitive about their data being exposed elsewhere.<br> > <br> > Can you list a few improvements that come to mind?<br> <br> Clients between server nodes is probably one of the most obvious<br> choices indeed. Considering the data already resides on those nodes, I<br> am not sure I understand the concerns about "exposing" data that&= #39;s<br> already on those nodes. If customers are so sensitive, we support data<br> encryption.<br> We could also do some direct server-server migration of some sort where<br> OSTs exchange data without bringing up real clients and doing copies<br> from userspace. That might be desirable for other reasons for future<br> functionality (e.g. various caching things people have been long<br> envisioning)</blockquote><div><br></div><div>This is going too far ;-)</div= ><div>=C2=A0</div><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px= 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex">=C2=A0<= /blockquote><blockquote class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0= .8ex;border-left:1px solid rgb(204,204,204);padding-left:1ex"> =C2=A0<br> > > <br> > > > takes very long time so it resorts to databases in order to = make<br> > > > correct decisions about which file should be released. By th= e<br> > > > time,<br> > > > the two system will be out of sync. That makes it practicall= y<br> > > > unusable.<br> > > <br> > > This again is an implementation detail, not even hardcoded<br> > > anywhere.<br> > > How do you plan for the OST to to know what stuff is not used<br> > > without<br> > > resorting to some database or scan? Now take this method and make= <br> > > it<br> > > report "upstream" where currently HSM implementations r= esort to<br> > > databases or scans.<br> > > <br> > <br> > The assumption is that OST sizes are relatively small, up to 100TB.<br= > > Also, scanning local devices in kernel spaces is much faster. So yeah<= br> > there is no database in the way.<br> <br> I am not sure why? In general small OSTs are a relatively rare thing<br> because to reach large FS sizes you would need to many of them, space<br> balancing becomes a chore and so on. So relatively few do it for some<br> fringe reasons (e.g. Google) . Majority of people prefer large OSTs.<br> <br> Also nothing stops you from doing a per-OST scan (when you do have<br> small OSTs) and then kicking the results up to the acting agent to do<br> something about it (or the other way around, the hsm engine can ask<br> OSTs one by one (picking less busy ones or ones that have the least<br> free space, or some other factor). And there's absolutely no need to<br= > wait out to query all OSTs, you can get results from one and work on<br> the data from it while the other OSTs are still thinking (or not,<br> there's absolutely no requirement to get full filesystem data before<br= > making any decisions).<br> <br> > I guess users won't have 1PB OSTs, will they?<br> <br> There probably are already? NASA has a known 0.5P OST configuration:<br> <a href=3D"https://www.nas.nasa.gov/hecc/support/kb/lustre-progressive-file= -layout-(pfl)-with-ssd-and-hdd-pools_680.html#:~:text=3DThe%20available%20S= SD%20space%20in%20each%20filesystem,decimal%20(far%20right)%20labels%20of%2= 0each%20OST" rel=3D"noreferrer" target=3D"_blank">https://www.nas.nasa.gov/= hecc/support/kb/lustre-progressive-file-layout-(pfl)-with-ssd-and-hdd-pools= _680.html#:~:text=3DThe%20available%20SSD%20space%20in%20each%20filesystem,= decimal%20(far%20right)%20labels%20of%20each%20OST</a><br> .<br> <br> > > Rereading your proposal, I see that this particular detail is not= <br> > > covered and it's just assumed that "infrequently accesse= d data"<br> > > would<br> > > be somehow known.<br> > <br> > I should have mentioned that in the migration section. Also, we need<b= r> > to slightly update the OST read to use a local transaction to update<b= r> > an object's access time (atime) if it's older than a predefine= d<br> > threshold, for example, 10 minutes.=C2=A0<br> <br> This is going to be fragile in the face of varying clock times on<br> different clients potentially not synced with the servers.<br> Also in the face of -o noatime.<br></blockquote><div><br></div><div>It does= n't use client timestamps. Also, it won't be part of read because r= ead doesn't initiate a transaction.</div><div><br></div><div>It=C2=A0ca= n simply use the OSS local time and start a local transaction to update the= atime.=C2=A0</div><div>=C2=A0</div><blockquote class=3D"gmail_quote" style= =3D"margin:0px 0px 0px 0.8ex;border-left:1px solid rgb(204,204,204);padding= -left:1ex"> <br> But yes, I guess it's one way to get this "on the cheap", and= the other<br> trouble I foresee is you are going to have a biased set. Only recently<br> touched objects (so with fresh atime), unless you plan to retain a<br> database and update it from such transaction flow, which certainly is<br> possible, but I am not sure how practical vs some sort of a scan.<br></bloc= kquote><div><br></div><div>I don't see that as an issue. It's going= to update the atime in memory. And yeah if the OSS is crashed and the scan= ner may choose a wrong file to migrate, this should be rare and I don't= think this would become a severe issue.</div><div>=C2=A0</div><blockquote = class=3D"gmail_quote" style=3D"margin:0px 0px 0px 0.8ex;border-left:1px sol= id rgb(204,204,204);padding-left:1ex"> <br> > > > > If the argument is "but OSTs know best what stuff = is used"<br> > > > > (which I<br> > > > > am<br> > > > > not sure I buy, after all before you could use somethin= g off<br> > > > > OSTs<br> > > > > you<br> > > > > need to open a file I would hope) even then OSTs could = just<br> > > > > signal<br> > > > > a<br> > > > > list of "inactive objects" that then a higher= level system<br> > > > > would<br> > > > > take<br> > > > > care of by relocatiing somewhere more sensical and chan= ging the<br> > > > > layout<br> > > > > to indicate those objects now live elsewhere.<br> > > > > <br> > > > > The plus here is you don't need to attach this &quo= t;Wart" to every<br> > > > > OST<br> > > > > and<br> > > > > configure it everywhere and such, but rather have a cen= tral<br> > > > > location<br> > > > > that is centrally managed.<br> > > > > <br> > > <br> > > _______________________________________________<br> > > lustre-devel mailing list<br> > > <a href=3D"mailto:[email protected]" target=3D"_blank= ">[email protected]</a><br> > > <a href=3D"http://lists.lustre.org/listinfo.cgi/lustre-devel-lust= re.org" rel=3D"noreferrer" target=3D"_blank">http://lists.lustre.org/listin= fo.cgi/lustre-devel-lustre.org</a><br> > > <br> <br> </blockquote></div></div> --0000000000000d76ba0642c86e5e-- --===============6784438486419368419== Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Content-Disposition: inline _______________________________________________ lustre-devel mailing list [email protected] http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org --===============6784438486419368419==--