Re: [PATCH v2] ceph: fix denial of service issue in ceph_update_snap_trace()

On Mon, Jul 6, 2026 at 11:32=E2=80=AFPM Viacheslav Dubeyko <[email protected]=
om> wrote:
>
> From: Viacheslav Dubeyko <[email protected]>
>
> A WARN_ON() fires inside ceph_update_snap_trace() when the client
> receives a malformed snap trace from the MDS. The kernel additionally
> logs:
>
> ceph_update_snap_trace do remount to continue after corrupted snaptrace i=
s fixed
>
> indicating the client cannot recover the snap state and forces
> the operator to remount the filesystem. Triggered with the call chain
> reaching the warn from ceph_con_process_message() -> mds_dispatch()
> -> ceph_update_snap_trace(). Impact: denial of service of the affected
> mount until remount; the client refuses further snap-related operations
> after the warn, so any open file in a snap-realm becomes unusable.
>
> [  230.026879] WARNING: fs/ceph/snap.c:926 at ceph_update_snap_trace+0x30=
8/0x3300, CPU#1: kworker/1:1/58
> [  230.028125] Modules linked in:
> [  230.028427] CPU: 1 UID: 0 PID: 58 Comm: kworker/1:1 Not tainted 6.19.0=
-g44331bd6a610-dirty #9 PREEMPT(lazy)
> [  230.029089] Hardware name: QEMU Standard PC (i440FX + PIIX, 1996), BIO=
S rel-1.14.0-0-g155821a1990b-prebuilt.qemu.org 04/01/2014
> [  230.029815] Workqueue: ceph-msgr ceph_con_workfn
> [  230.030177] RIP: 0010:ceph_update_snap_trace+0x31c/0x3300
> [  230.030550] Code: fa 48 c1 ea 03 80 3c 02 00 0f 85 23 26 00 00 48 8d 3=
d 28 d2 3a 05 48 8b 73 28 49 89 e9 4d 89 e8 4c 89 f1 48 c7 c2 40 40 8e 9e <=
67> 48 0f b9 3a e8 3a 7c db fe 48 b8 00 00 00 00 00 fc ff df 4d 8d
> [  230.031708] RSP: 0018:ffffc900003f7728 EFLAGS: 00010246
> [  230.032084] RAX: dffffc0000000000 RBX: ffff88812282d000 RCX: ffffffff9=
e8e3b40
> [  230.032533] RDX: ffffffff9e8e4040 RSI: 0000000000016f97 RDI: ffffffffa=
087e7d0
> [  230.033008] RBP: ffffffff9e8e3c00 R08: ffffffff9e8e3b40 R09: ffffffff9=
e8e3c00
> [  230.033460] R10: 00000000fffffffb R11: ffff888102928040 R12: ffff88811=
031a3e0
> [  230.033937] R13: ffffffff9e8e3b40 R14: ffffffff9e8e3b40 R15: 1ffff9200=
007eefd
> [  230.034401] FS:  0000000000000000(0000) GS:ffff888254a05000(0000) knlG=
S:0000000000000000
> [  230.034941] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [  230.035315] CR2: 00007f7efa7be740 CR3: 000000012435c000 CR4: 000000000=
0750ef0
> [  230.035803] PKRU: 55555554
> [  230.036001] Call Trace:
> [  230.036175]  <TASK>
> [  230.036342]  ? mds_dispatch+0x1ceb/0x6f60
> [  230.036645]  ? __pfx___might_resched+0x10/0x10
> [  230.037010]  ? iget5_locked+0x44/0xb0
> [  230.037316]  ? __pfx_ceph_update_snap_trace+0x10/0x10
> [  230.037670]  ? __pfx_down_write+0x10/0x10
> [  230.038005]  mds_dispatch+0x1dd1/0x6f60
> [  230.038306]  ? ceph_con_process_message+0x1ab/0x270
> [  230.039024]  ? lock_release+0xc7/0x270
> [  230.039321]  ? __pfx_mds_dispatch+0x10/0x10
> [  230.039622]  ? __local_bh_enable_ip+0xa1/0x110
> [  230.039992]  ceph_con_process_message+0x1f4/0x270
> --
> [  321.986495] libceph: failed to decode MOSDOpReply for tid 9: -22
> [  330.121991] ------------[ cut here ]------------
>
> The error path in ceph_update_snap_trace() is reached whenever snap trace
> decoding fails. The two entry points are:
> (1) bad =E2=80=94 a ceph_decode_need() macro fails when the encoded snap =
trace does not
> contain enough bytes for the declared number of snaps or prior-parent sna=
ps.
> (2) fail =E2=80=94 reached directly from -ENOMEM returns (ceph_create_sna=
p_realm(),
> dup_array(), adjust_snap_realm_parent()).
>
> This patch fixes the issue by changing WARN(1, ...) on
> pr_warn_ratelimited_client().
>
> Fixes: a68e564adcaa ("ceph: blocklist the kclient when receiving corrupte=
d snap trace")
> Signed-off-by: Viacheslav Dubeyko <[email protected]>
> cc: Alex Markuze <[email protected]>
> cc: Ilya Dryomov <[email protected]>
> cc: Patrick Donnelly <[email protected]>
> cc: Ceph Development <[email protected]>
> ---
>  fs/ceph/snap.c | 10 +++++-----
>  1 file changed, 5 insertions(+), 5 deletions(-)
>
> diff --git a/fs/ceph/snap.c b/fs/ceph/snap.c
> index 9b79a5eaca93..e5d8d59ffc17 100644
> --- a/fs/ceph/snap.c
> +++ b/fs/ceph/snap.c
> @@ -923,11 +923,11 @@ int ceph_update_snap_trace(struct ceph_mds_client *=
mdsc,
>                 pr_err_client(cl, "failed to blocklist %s: %d\n",
>                               ceph_pr_addr(&client->msgr.inst.addr), ret)=
;
>
> -       WARN(1, "[client.%lld] %s %s%sdo remount to continue%s",
> -            client->monc.auth->global_id, __func__,
> -            ret ? "" : ceph_pr_addr(&client->msgr.inst.addr),
> -            ret ? "" : " was blocklisted, ",
> -            err =3D=3D -EIO ? " after corrupted snaptrace is fixed" : ""=
);
> +       pr_warn_ratelimited_client(cl, "%s %s%sdo remount to continue%s",
> +                           __func__,
> +                           ret ? "" : ceph_pr_addr(&client->msgr.inst.ad=
dr),
> +                           ret ? "" : " was blocklisted, ",
> +                           err =3D=3D -EIO ? " after corrupted snaptrace=
 is fixed" : "");
>
>         return err;
>  }
> --
> 2.54.0
>

Hi Slava,

I'm confused by the patch title and description.  The title is

> ceph: fix denial of service issue in ceph_update_snap_trace()

and in the description that is expanded to

> Impact: denial of service of the affected mount until remount; the
> client refuses further snap-related operations after the warn, so any
> open file in a snap-realm becomes unusable.

The description ends with

> This patch fixes the issue by changing WARN(1, ...) on
> pr_warn_ratelimited_client().

but WARN -> pr_warn_ratelimited_client change that doesn't magically
make the blocklisting that precedes the warning go away.  All (not just
snap-related as the description suggests) further operations would
still be refused one way or the other, the MDS session would be killed,
etc.  What exactly is the denial of service that is getting fixed here?

Thanks,

                Ilya