Re: [PATCH RFC] nbd: fix I/O hang on dead socket and console spam
Kusaram Devineni <[email protected]> Wed, 15 Jul 2026 17:34:34 +0530
| Newsgroups | dev.linux.lists.syzbot |
|---|---|
| Message-ID | <[email protected]> |
On 09-07-2026 06:11 pm, syzbot wrote:
> When an NBD device is configured without NBD_CFLAG_DISCONNECT_ON_CLOSE and
> with a timeout of 0, a closed socket can lead to a permanent I/O hang. If
> the connection is closed, the NBD recv_work thread marks the socket as
> dead. However, if an I/O request is sent, it will eventually time out. In
> nbd_xmit_timeout(), if the configured timeout is 0, the code currently only
> checks if the socket has been replaced. It fails to check if the socket is
> dead. As a result, the request timer is unconditionally reset and the
> request stays in-flight forever, causing tasks like udevd to hang
> indefinitely in TASK_UNINTERRUPTIBLE and triggering the hung task detector:
>
the case here is an in-flight request sent while the socket was usable,
followed by that same socket becoming dead before a reply arrives. state
the sequence precisely to avoid ambiguity. i.e. request sent and marked
in flight, socket becomes dead, no reply can arrive, and the
zero-timeout handler keeps resetting the timer because the cookie still
matches.
> INFO: task udevd:5877 blocked in I/O wait for more than 143 seconds.
> task:udevd state:D
> Call Trace:
> <TASK>
> __schedule+0x17e7/0x5630
> schedule+0x164/0x2b0
> io_schedule+0x7f/0xd0
> folio_wait_bit_common+0x836/0xbc0
> do_read_cache_folio+0x1ac/0x590
> read_part_sector+0xb6/0x2b0
> adfspart_check_POWERTEC+0x9a/0x7a0
> bdev_disk_changed+0x851/0x17a0
> blkdev_get_whole+0x372/0x510
> bdev_open+0x324/0xd70
> blkdev_open+0x461/0x600
> do_dentry_open+0x816/0x1380
> vfs_open+0x3b/0x340
> path_openat+0x2e44/0x3830
> do_file_open+0x23e/0x4a0
> do_sys_openat2+0x115/0x200
> __x64_sys_openat+0x138/0x170
> do_syscall_64+0x15f/0x560
> entry_SYSCALL_64_after_hwframe+0x77/0x7f
> </TASK>
>
> Additionally, repeated attempts to connect to an already-in-use NBD device
> can cause console spam because the "nbd%d already in use" error message in
> nbd_genl_connect() is not rate-limited. This can delay console_unlock() and
> trigger NMI backtraces.
keep the logging hunk, but present/highlight it as a secondary issue
observed.
>
> Fix the I/O hang by checking nsock->dead in addition to the cookie check in
> nbd_xmit_timeout(). If the socket is dead, the command is requeued. When
> nbd_handle_cmd() subsequently processes it, it will evaluate the dead
> socket and properly fail the I/O request, gracefully terminating the hung
> read operation.
the phrase "properly fail the I/O request" implies that failure is the
inevitable result of requeue and hides the other existing
nbd_handle_cmd() outcomes: selecting a live fallback or waiting for
reconnection. it also does not explain why failure is expected in this
report. describe requeue as returning the request to the existing
fallback/reconnect/failure policy. if immediate failure is mentioned,
qualify it as the outcome for the reported configuration rather than a
universal one.
>
> Fix the console spam by changing the pr_err() in nbd_genl_connect() to
> pr_err_ratelimited().
>
> Fixes: 2c272542baee ("nbd: requeue command if the soecket is changed")
> Assisted-by: Gemini:gemini-3.1-pro-preview Gemini:gemini-3-flash-preview syzbot
> Reported-by: [email protected]
> Closes: https://syzkaller.appspot.com/bug?extid=82de77d3f217960f087d
> Link: https://syzkaller.appspot.com/ai_job?id=13cee482-fa85-4a3d-bcc1-e2cc9ae8e909
> To: "Jens Axboe" <[email protected]>
> To: "Josef Bacik" <[email protected]>
> To: <[email protected]>
> To: <[email protected]>
> Cc: <[email protected]>
>
> ---
> diff --git a/drivers/block/nbd.c b/drivers/block/nbd.c
> index 8f10762e9..eedb1c870 100644
> --- a/drivers/block/nbd.c
> +++ b/drivers/block/nbd.c
> @@ -523,7 +523,7 @@ static enum blk_eh_timer_return nbd_xmit_timeout(struct request *req)
> blk_rq_bytes(req), (req->timeout / HZ) * cmd->retries);
>
> mutex_lock(&nsock->tx_lock);
> - if (cmd->cookie != nsock->cookie) {
> + if (cmd->cookie != nsock->cookie || nsock->dead) {
> nbd_requeue_cmd(cmd);
> mutex_unlock(&nsock->tx_lock);
> mutex_unlock(&cmd->lock);
> @@ -2172,7 +2172,7 @@ static int nbd_genl_connect(struct sk_buff *skb, struct genl_info *info)
> nbd_put(nbd);
> if (index == -1)
> goto again;
> - pr_err("nbd%d already in use\n", index);
> + pr_err_ratelimited("nbd%d already in use\n", index);
> return -EBUSY;
> }
>
>
>
> base-commit: 8cdeaa50eae8dad34885515f62559ee83e7e8dda
-kusaram