Re: [PATCH RFC] nbd: fix hung task and integer overflow in timeout configuration
Uladzislau Zhauniarovich <[email protected]> Thu, 23 Jul 2026 12:12:34 +0200
| Newsgroups | dev.linux.lists.syzbot |
|---|---|
| Message-ID | <[email protected]> |
The v1 patch correctly identifies that wait_for_reconnect() blocks in
TASK_UNINTERRUPTIBLE for a user-controlled NBD_ATTR_DEAD_CONN_TIMEOUT,
and it is right to split the netlink parsing into a local u64 timeout
and to guard the jiffies conversion against overflow. Keep the local
variable, the overflow-safe conversion at both nbd_genl_connect() and
nbd_genl_reconfigure(), and the Reported-by/Closes tags.
However, the fix is incomplete: changing the sleep state of
wait_for_reconnect() (to TASK_IDLE via wait_event_idle_timeout()) does
not stop the hung-task panic, and the MAX_SCHEDULE_TIMEOUT clamp does
not bound the actual hang.
nbd allocates its tag set with BLK_MQ_F_BLOCKING (nbd_dev_add()), so
nbd_queue_rq() is not guaranteed to run in the submitter's context. In
the common case the block layer dispatches it from a kworker
(blk_mq_run_work_fn), while the submitting task blocks in
folio_wait_bit_common() (io_schedule, plain TASK_UNINTERRUPTIBLE)
waiting for the bio to complete during the partition scan (bdev_open ->
bdev_disk_changed -> do_read_cache_folio). That submitter wait is caught
by the base hung-task detector regardless of what state
wait_for_reconnect() sleeps in — the "blocked in I/O wait" message is
printed unconditionally in check_hung_task() (t->in_iowait ? " in I/O
wait" : "") and is not gated on CONFIG_DETECT_HUNG_TASK_BLOCKER.
wait_event_idle_timeout() only hides the wait_for_reconnect() frame (the
"inline dispatch" manifestation from the sample report); it cannot hide
the submitter's I/O-completion wait, so the panic still fires.
The clamp is also not a bound. min_t(u64, timeout, U32_MAX) then
min_t(u64, timeout * HZ, MAX_SCHEDULE_TIMEOUT) only prevents the u64 *
HZ overflow; the resulting wait is still ~136 years, i.e. effectively
unbounded. Because the submitter's I/O-completion wait cannot be made
interruptible, the only way to guarantee the request fails before the
detector threshold is to bound the reconnect wait itself.
This was confirmed on a KASAN build (hung-task timeout lowered to 20s)
with a reproducer that configures a single-connection nbd device with a
huge dead_conn_timeout, kills the socket, and reads /dev/nbd0:
unpatched, huge timeout: PANIC.
v1 (idle + MAX_SCHEDULE_TIMEOUT clamp), huge timeout: STILL PANIC
(nbd_synth blocked in I/O wait, folio_wait_bit_common, Kernel panic -
hung_task).
wait_event_killable_timeout + MAX_SCHEDULE_TIMEOUT clamp, huge timeout:
STILL PANIC (identical).
unpatched, dead_conn_timeout = 8s: CLEAN — the read returns -EIO after
8s, no hung task.
Required corrections:
Bound dead_conn_timeout to a real maximum that keeps the effective
reconnect wait below the hung-task detection threshold, not merely below
MAX_SCHEDULE_TIMEOUT. The clamp must cap the wait to a finite, small
value (on the order of the default hung_task_timeout_secs, e.g. a few
tens of seconds) so that a request against a dead connection fails with
-EIO in bounded time. Keep the overflow-safe u64 timeout conversion, but
the ceiling must bound the hang, not just the arithmetic. The exact
ceiling is a semantics decision for the nbd maintainers; state it
explicitly in the commit message.
Replace wait_event_idle_timeout() with wait_event_killable_timeout()
rather than TASK_IDLE. Both are skipped by the hung-task detector, but
TASK_KILLABLE additionally lets SIGKILL/the OOM killer reap a submitter
that does run wait_for_reconnect() inline. Capture the return value in a
long and treat ret <= 0 (timeout, or -ERESTARTSYS from a fatal signal)
as "no reconnect" and fail the request. Non-fatal signals must still not
end the wait, preserving the intent of commit ff57dc94faec ("nbd: wait
uninterruptible for the dead timeout").
Wake the reconnect waitqueue when the connection is torn down.
sock_shutdown() sets NBD_RT_DISCONNECTED but does not wake
config->conn_wait, so a concurrent waiter only observes the flag after
its own timeout. Add wake_up(&config->conn_wait) in sock_shutdown()
after the sockets are marked dead.
Do not rely on the sleep-state change alone to close the report. Note in
the commit message that the BLK_MQ_F_BLOCKING async-dispatch path leaves
the submitter in an uninterruptible I/O-completion wait, and that
bounding the timeout (correction 1) is what actually prevents the panic;
the killable wait (correction 2) is reapability hardening, not the
primary fix.
Use Fixes: 560bc4b39952 ("nbd: handle dead connections"), which
introduced both wait_for_reconnect() and the unbounded
dead_conn_timeout. Reference ff57dc94faec in the body when explaining
why the wait must stay signal-safe against non-fatal signals, but it is
not the root of the unbounded-wait DoS.
On 15/07/2026 17:32, syzbot wrote:
> An unprivileged user can configure an NBD device with an arbitrarily large
> dead connection timeout via the NBD_ATTR_DEAD_CONN_TIMEOUT netlink
> attribute. When a request is queued and the connection is dead,
> nbd_handle_cmd() calls wait_for_reconnect(), which blocks the current
> thread using wait_event_timeout(). Because this wait is in the
> TASK_UNINTERRUPTIBLE state, a timeout larger than 120 seconds triggers the
> hung task detector, leading to a kernel panic:
>
> INFO: task kworker/1:1H:667 blocked for more than 143 seconds.
> Workqueue: kblockd blk_mq_run_work_fn
> Call Trace:
> <TASK>
> __schedule+0x17e7/0x5630
> schedule+0x164/0x2b0
> schedule_timeout+0x152/0x2c0
> wait_for_reconnect drivers/block/nbd.c:1107 [inline]
> nbd_handle_cmd drivers/block/nbd.c:1149 [inline]
> nbd_queue_rq+0x797/0xfb0 drivers/block/nbd.c:1207
> blk_mq_dispatch_rq_list+0x499/0x1990
> __blk_mq_sched_dispatch_requests+0xd36/0x1580
> blk_mq_sched_dispatch_requests+0xd7/0x190
> blk_mq_run_work_fn+0x16c/0x300
> process_one_work kernel/workqueue.c:3322 [inline]
> process_scheduled_works+0xa8e/0x14e0
> worker_thread+0x92d/0xe10
> kthread+0x388/0x470
> ret_from_fork+0x514/0xb70
> ret_from_fork_asm+0x1a/0x30
> </TASK>
>
> Historically, wait_for_reconnect() used wait_event_interruptible_timeout(),
> but commit ff57dc94faec ("nbd: wait uninterruptible for the dead timeout")
> changed it to wait_event_timeout() to prevent signals from prematurely
> interrupting the wait. However, using TASK_UNINTERRUPTIBLE for arbitrarily
> long waits is an anti-pattern that trips the hung task detector.
>
> Fix this by replacing wait_event_timeout() with wait_event_idle_timeout().
> This puts the task into the TASK_IDLE state (TASK_UNINTERRUPTIBLE |
> TASK_NOLOAD), which ignores signals as intended but is explicitly ignored
> by the hung task detector.
>
> Additionally, add defense-in-depth by capping the parsed timeout to U32_MAX
> and the resulting jiffies to MAX_SCHEDULE_TIMEOUT. This prevents integer
> overflow when the u64 timeout is implicitly cast to a signed long by
> schedule_timeout(), which could otherwise result in a negative timeout and
> cause the wait to return immediately.
>
> Fixes: ff57dc94faec ("nbd: wait uninterruptible for the dead timeout")
> Assisted-by: Gemini:gemini-3.1-pro-preview syzbot
> Reported-by: [email protected]
> Closes: https://syzkaller.appspot.com/bug?extid=30c16035531e3248dcbc
> Link: https://syzkaller.appspot.com/ai_job?id=f07c3611-0a61-4d67-9f4d-033b1cb82e1e
> To: "Jens Axboe" <[email protected]>
> To: "Josef Bacik" <[email protected]>
> To: <[email protected]>
> To: <[email protected]>
> To: "Josef Bacik" <[email protected]>
> Cc: <[email protected]>
>
> ---
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index 8f10762e9..874a916f8 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -1104,11 +1104,11 @@ static int wait_for_reconnect(struct nbd_device *nbd)
> if (!config->dead_conn_timeout)
> return 0;
>
> - if (!wait_event_timeout(config->conn_wait,
> - test_bit(NBD_RT_DISCONNECTED,
> - &config->runtime_flags) ||
> - atomic_read(&config->live_connections) > 0,
> - config->dead_conn_timeout))
> + if (!wait_event_idle_timeout(
> + config->conn_wait,
> + test_bit(NBD_RT_DISCONNECTED, &config->runtime_flags) ||
> + atomic_read(&config->live_connections) > 0,
> + config->dead_conn_timeout))
> return 0;
>
> return !test_bit(NBD_RT_DISCONNECTED, &config->runtime_flags);
> @@ -2194,9 +2194,11 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
> nbd_set_cmd_timeout(nbd,
> nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT]));
> if (info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]) {
> - config->dead_conn_timeout =
> + u64 timeout =
> nla_get_u64(info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]);
> - config->dead_conn_timeout *= HZ;
> + timeout = min_t(u64, timeout, U32_MAX);
> + config->dead_conn_timeout =
> + min_t(u64, timeout * HZ, MAX_SCHEDULE_TIMEOUT);
> }
> if (info->attrs[NBD_ATTR_SERVER_FLAGS])
> config->flags =
> @@ -2415,9 +2417,11 @@ static int nbd_genl_reconfigure(struct sk_buff *skb, struct genl_info *info)
> nbd_set_cmd_timeout(nbd,
> nla_get_u64(info->attrs[NBD_ATTR_TIMEOUT]));
> if (info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]) {
> - config->dead_conn_timeout =
> + u64 timeout =
> nla_get_u64(info->attrs[NBD_ATTR_DEAD_CONN_TIMEOUT]);
> - config->dead_conn_timeout *= HZ;
> + timeout = min_t(u64, timeout, U32_MAX);
> + config->dead_conn_timeout =
> + min_t(u64, timeout * HZ, MAX_SCHEDULE_TIMEOUT);
> }
> if (info->attrs[NBD_ATTR_CLIENT_FLAGS]) {
> u64 flags = nla_get_u64(info->attrs[NBD_ATTR_CLIENT_FLAGS]);
>
>
> base-commit: a13c140cc289c0b7b3770bce5b3ad42ab35074aa