Re: [lustre-devel] RFC: Spill device for Lustre OSD

--===============7808215221663793929==
Content-Type: multipart/alternative;
	boundary="Apple-Mail=_44A77994-14E6-4B71-8FCB-73DDA2890107"


--Apple-Mail=_44A77994-14E6-4B71-8FCB-73DDA2890107
Content-Transfer-Encoding: quoted-printable
Content-Type: text/plain;
	charset=utf-8



> On Nov 4, 2025, at 15:07, Andreas Dilger <[email protected]> wrote:
>=20
> On Nov 4, 2025, at 14:11, Day, Timothy <[email protected]> wrote:
>>=20
>>> VFS OSD won=E2=80=99t give us everything to make it an OSD. =
Transaction is one of the issue as Oleg mentioned.
>>=20
>> We'll have to implement a lot of complexity in kernel space for
>> the design you're suggesting. At some point, implementing
>> a transaction log and delegating the rest to user space might
>> be the more maintainable option? Doing that via VFS/Fuse is
>> one option. Something similar to ublk for OSD could be another
>> option.
>=20
> If the writes to the spill device are well managed, it may be possible
> to do it without transaction support, so long as they are not exposed
> directly for writing to the clients.  Otherwise they essentially need
> to be OSDs that expose transactions and recovery semantics.
>=20
>> I'm not saying this is preferrable, but is this something you've
>> considered?
>>=20
>>> Extended attributes would be another thing since not all file =
systems would support it.
>>=20
>> I think it's fine not accept filesystems that don't support EA.
>> Or have the OSD advertise that it doesn't support EA.
>=20
> I don't think any storage system we care about today does not support
> EAs or tags or similar metadata that can be used for this.
>=20
>>>>> deliver better performance than block-level cache with dmcache, =
where
>>>>> recovery time is lengthy if the cache size is huge.
>>>>=20
>>>> This seems speculative. Could you elaborate more on why you think
>>>> this is the case?
>>>=20
>>> DMcache doesn=E2=80=99t persist the bitmap used to indicate which =
blocks are holding dirty data, so an ungraceful shutdown will lead to a =
scanning of the entire cache in order to determine which blocks are > =
dirty.
>>=20
>> This is a failing of DMcache rather than an indication that this
>> problem can't be solved on the block layer.
>=20
> That was my original discussion with Jinshan as well.  There was a
> similar issue with mdraid, and they ended up with persistent bitmaps
> in a flash device.  It should be possible to manage this with logs
> or bitmaps stored in the fast OSD device instead of the spill device.
>=20
>> Overall, I think the concept is interesting. It reminds me of how
>> Bcachefs handle multi-device support. Each device can be
>> designated as holding metadata or data replicas. And you
>> can control the promotion and migration between different
>> targets (all managed by a migration daemon). But this design is
>> too limited, IMHO. If we're going to accept the additional complexity
>> in the OSD, the solution has to be extensible. What if I want to
>> replicate to multiple targets? What if I want more than two tiers?
>> What if I want to transparently migrate data from one spill device to
>> another? We don't need this for the initial implementation, sure.
>> But these seem like natural extensions.

It=E2=80=99s possible to extend the design to have multiple spill =
devices in the OSD; you could have two spill devices and mirror them, or =
raid0 to make a larger device. I don=E2=80=99t see the design would not =
allow you to do that.=20

>=20
> This is essentially replicating Lustre file layouts in the end, which
> was my original suggestion - to use FLR and/or PCC-RO foreign
> mirror layouts for this, even if it is not directly accessible from
> clients.  That avoids reimplementing tools/formats that already
> exist in Lustre today for relatively little benefit.


One of the goals is to not have a file system-level scanner, which is =
not good. Otherwise, we can just use FLR-based tiered storage.

>=20
>> I think we need to use some kind of common API for the
>> different devices. Even if the spill device doesn't support atomic
>> transactions, I don't see why we couldn't still use the common OSD
>> API and implement the migration daemon as a stacking driver on-top
>> of that. The spill device OSD driver could be made to advertise that =
it
>> doesn't support atomic transactions and may not support EA. But we
>> get the added benefit of being able to use existing OSDs with this
>> feature, pretty much for free.
>=20
> Also my argument.  Using the VFS directly is constraining (lack of
> transactions), but of backends that _can_ be a full OSD (or already
> have an OSD like ldiskfs, ZFS, memfs) it is a drop-in replacement.
>=20
> There is already an OSD API to query the functionality of the backing
> storage, so it should be straight forward to add "transaction", =
"xattr",
> and other supported features to that.
>=20
> If we can implement a no-transaction osd-vfs, that would expose a
> lot of flexibility for other reasons as well.  Possibly the osd-vfs =
could
> implement a journal or other logging layer internally to make up for
> lack of transactions, whether initially or at a later stage?

What would be the benefit of having a limited OSD in the stack? I =
don=E2=80=99t have a strong opinion of not doing it, but I just didn=E2=80=
=99t see any benefits of doing it.

>=20
> Cheers, Andreas
> =E2=80=94
> Andreas Dilger
> Lustre Principal Architect
> Whamcloud/DDN
>=20
>=20
>=20
>=20


--Apple-Mail=_44A77994-14E6-4B71-8FCB-73DDA2890107
Content-Transfer-Encoding: quoted-printable
Content-Type: text/html;
	charset=utf-8

<html aria-label=3D"message body"><head><meta http-equiv=3D"content-type" =
content=3D"text/html; charset=3Dutf-8"></head><body =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;"><br =
id=3D"lineBreakAtBeginningOfMessage"><div><br><blockquote =
type=3D"cite"><div>On Nov 4, 2025, at 15:07, Andreas Dilger =
&lt;[email protected]&gt; wrote:</div><br =
class=3D"Apple-interchange-newline"><div>

<meta http-equiv=3D"Content-Type" content=3D"text/html; charset=3Dutf-8">

<div style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;">
On Nov 4, 2025, at 14:11, Day, Timothy &lt;[email protected]&gt; =
wrote:<br>
<div>
<blockquote type=3D"cite"><br class=3D"Apple-interchange-newline">
<div>
<div>
<blockquote type=3D"cite">VFS OSD won=E2=80=99t give us everything to =
make it an OSD. Transaction is one of the issue as Oleg mentioned.<br>
</blockquote>
<br>
We'll have to implement a lot of complexity in kernel space for<br>
the design you're suggesting. At some point, implementing<br>
a transaction log and delegating the rest to user space might<br>
be the more maintainable option? Doing that via VFS/Fuse is<br>
one option. Something similar to ublk for OSD could be another<br>
option.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
If the writes to the spill device are well managed, it may be =
possible</div>
<div>to do it without transaction support, so long as they are not =
exposed</div>
<div>directly for writing to the clients. &nbsp;Otherwise they =
essentially need</div>
<div>to be OSDs that expose transactions and recovery semantics.</div>
<div><br>
<blockquote type=3D"cite">
<div>
<div>I'm not saying this is preferrable, but is this something =
you've<br>
considered?<br>
<br>
<blockquote type=3D"cite">Extended attributes would be another thing =
since not all file systems would support it.<br>
</blockquote>
<br>
I think it's fine not accept filesystems that don't support EA.</div>
</div>
</blockquote>
<blockquote type=3D"cite">
<div>
<div>Or have the&nbsp;OSD advertise that it doesn't support EA.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
I don't think any storage system we care about today does not =
support</div>
<div>EAs or tags or similar metadata that can be used for this.</div>
<div><br>
<blockquote type=3D"cite">
<div>
<div>
<blockquote type=3D"cite">
<blockquote type=3D"cite">
<blockquote type=3D"cite">deliver better performance than block-level =
cache with dmcache, where<br>
recovery time is lengthy if the cache size is huge.<br>
</blockquote>
<br>
This seems speculative. Could you elaborate more on why you think<br>
this is the case?<br>
</blockquote>
<br>
DMcache doesn=E2=80=99t persist the bitmap used to indicate which blocks =
are holding dirty data, so an ungraceful shutdown will lead to a =
scanning of the entire cache in order to determine which blocks are &gt; =
dirty.<br>
</blockquote>
<br>
This is a failing of DMcache rather than an indication that this<br>
problem can't be solved on the block layer.<br>
</div>
</div>
</blockquote>
<div><br>
</div>
That was my original discussion with Jinshan as well. &nbsp;There was =
a</div>
<div>similar issue with mdraid, and they ended up with persistent =
bitmaps</div>
<div>in a flash device. &nbsp;It should be possible to manage this with =
logs</div>
<div>or bitmaps stored in the fast OSD device instead of the spill =
device.</div>
<div><br>
<blockquote type=3D"cite">
<div>
<div>Overall, I think the concept is interesting. It reminds me of =
how<br>
Bcachefs handle multi-device support. Each device can be<br>
designated as holding metadata or data replicas. And you<br>
can control the promotion and migration between different<br>
targets (all managed by a migration daemon). But this design is<br>
too limited, IMHO. If we're going to accept the additional =
complexity<br>
in the OSD, the solution has to be extensible. What if I want to<br>
replicate to multiple targets? What if I want more than two tiers?<br>
What if I want to transparently migrate data from one spill device =
to<br>
another? We don't need this for the initial implementation, sure.<br>
But these seem like natural =
extensions.<br></div></div></blockquote></div></div></div></blockquote><di=
v><br></div><div>It=E2=80=99s possible to extend the design to have =
multiple spill devices in the OSD; you could have two spill devices and =
mirror them, or raid0 to make a larger device. I don=E2=80=99t see the =
design would not allow you to do that.&nbsp;</div><br><blockquote =
type=3D"cite"><div><div style=3D"overflow-wrap: break-word; =
-webkit-nbsp-mode: space; line-break: =
after-white-space;"><div><blockquote type=3D"cite"><div><div>
</div>
</div>
</blockquote>
<div><br>
</div>
This is essentially replicating Lustre file layouts in the end, =
which</div>
<div>was my original suggestion - to use FLR and/or PCC-RO foreign</div>
<div>mirror layouts for this, even if it is not directly accessible =
from</div>
<div>clients. &nbsp;That avoids reimplementing tools/formats that =
already</div>
<div>exist in Lustre today for relatively little =
benefit.</div></div></div></blockquote><div><br></div><div><br></div><div>=
One of the goals is to not have a file system-level scanner, which is =
not good. Otherwise, we can just use FLR-based tiered =
storage.</div><br><blockquote type=3D"cite"><div><div =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;">
<div><br>
<blockquote type=3D"cite">
<div>
<div>I think we need to use some kind of common API for the<br>
different devices. Even if the spill device doesn't support atomic<br>
transactions, I don't see why we couldn't still use the common OSD<br>
API and implement the migration daemon as a stacking driver on-top<br>
of that. The spill device OSD driver could be made to advertise that =
it<br>
doesn't support atomic transactions and may not support EA. But we<br>
get the added benefit of being able to use existing OSDs with this<br>
feature, pretty much for free.<br>
</div>
</div>
</blockquote>
<br>
</div>
<div>Also my argument. &nbsp;Using the VFS directly is constraining =
(lack of</div>
<div>transactions), but of backends that _can_ be a full OSD (or =
already</div>
<div>have an OSD like ldiskfs, ZFS, memfs) it is a drop-in =
replacement.</div>
<div><br>
</div>
<div>There is already an OSD API to query the functionality of the =
backing</div>
<div>storage, so it should be straight forward to add "transaction", =
"xattr",</div>
<div>and other supported features to =
that.</div></div></div></blockquote><blockquote type=3D"cite"><div><div =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;">
<div><br>
</div>
<div>If we can implement a no-transaction osd-vfs, that would expose =
a</div>
<div>lot of flexibility for other reasons as well. &nbsp;Possibly the =
osd-vfs could</div>
<div>implement a journal or other logging layer internally to make up =
for</div>
<div>lack of transactions, whether initially or at a later =
stage?</div></div></div></blockquote><div><br></div><div>What would be =
the benefit of having a limited OSD in the stack? I don=E2=80=99t have a =
strong opinion of not doing it, but I just didn=E2=80=99t see any =
benefits of doing it.</div><br><blockquote type=3D"cite"><div><div =
style=3D"overflow-wrap: break-word; -webkit-nbsp-mode: space; =
line-break: after-white-space;">
<br>
<div>
<div dir=3D"auto" style=3D"caret-color: rgb(0, 0, 0); letter-spacing: =
normal; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: =
space; line-break: after-white-space;">
<div dir=3D"auto" style=3D"caret-color: rgb(0, 0, 0); letter-spacing: =
normal; text-align: start; text-indent: 0px; text-transform: none; =
white-space: normal; word-spacing: 0px; -webkit-text-stroke-width: 0px; =
text-decoration: none; overflow-wrap: break-word; -webkit-nbsp-mode: =
space; line-break: after-white-space;">
<div>Cheers, Andreas</div>
<div>=E2=80=94</div>
<div>Andreas Dilger</div>
<div>Lustre Principal Architect</div>
<div>Whamcloud/DDN</div>
</div>
<br class=3D"Apple-interchange-newline">
</div>
<br class=3D"Apple-interchange-newline">
<br class=3D"Apple-interchange-newline">
</div>
<br>
</div>

</div></blockquote></div><br></body></html>=

--Apple-Mail=_44A77994-14E6-4B71-8FCB-73DDA2890107--

--===============7808215221663793929==
Content-Type: text/plain; charset="us-ascii"
MIME-Version: 1.0
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

_______________________________________________
lustre-devel mailing list
[email protected]
http://lists.lustre.org/listinfo.cgi/lustre-devel-lustre.org

--===============7808215221663793929==--