Re: [PATCH v3] libceph: reset OSD session when keepalive2 acks stop arriving

Hello,
Pinging about this patch. Is there any additional feedback or changes needed?
We have been successfully testing this patch since it was posted.
Thanks,
--chris

On Mon, Jul 20, 2026 at 5:43 PM Chris J Arges <[email protected]> wrote:
>
> handle_timeout() in osd_client.c sends CEPH_MSGR2_TAG_KEEPALIVE2 frames
> to OSDs with stalled requests, but libceph never verifies if the ACKs
> actually return.
>
> Consequently, if an OSD messenger queue wedges while the underlying
> TCP socket remains ESTABLISHED, the client will block indefinitely in
> ceph_osdc_wait_request(), causing tasks to hang in D state.
>
> Fix this by introducing a watchdog in handle_timeout() that checks
> ceph_con_keepalive_expired() against a new CEPH_OSD_PING_TIMEOUT (60s).
> On expiry, reset the sparse-read state, call reopen_osd(), and kick
> outstanding requests when the session is reopened.
>
> Because OSD keepalives are only sent to OSDs with stalled requests,
> last_keepalive_ack can be stale on an otherwise healthy connection that
> has simply been idle. Track the start of each slow/probing episode per
> OSD and require the episode to last CEPH_OSD_PING_TIMEOUT before checking
> for an expired keepalive ack, so the watchdog only fires after we have
> been actively pinging.
>
> Additionally, seed last_keepalive_ack to the current time in
> ceph_con_open() to prevent the watchdog from firing spuriously on fresh
> connections for both OSD and monitor clients.
>
> Fixes: 8b9558aab853 ("libceph: use keepalive2 to verify the mon session is alive")
> Link: https://tracker.ceph.com/issues/76202
> Co-developed-by: Andrew DeMaria <[email protected]>
> Signed-off-by: Andrew DeMaria <[email protected]>
> Signed-off-by: Chris J Arges <[email protected]>
> Reviewed-by: Viacheslav Dubeyko <[email protected]>
> ---
> This patch fixes an issue where the kernel rbd client and an OSD have an
> ESTABLISHED TCP connection, but keepalive2 ACKs stop returning from the
> OSD. When that happens, outstanding OSD requests can remain held by the
> client and callers can hang in D state. We were able to mitgiate this
> issue by using ss -K to kill the affected OSD TCP connection which
> reopened the OSD session.
>
> We were able to reproduce this issue in production a few times, and
> synthetically by dropping OSD to rbd application frames while allowing
> TCP ACKs through.
>
> https://tracker.ceph.com/issues/76202 describes the same situation.
>
> The following patch addresses this by creating a watchdog that resets the
> OSD session if this situation is detected.
> ---
> Changes in v3:
> - Rebase and collected Reviewed-By tag
> - Link to v2: https://patch.msgid.link/[email protected]
>
> Changes in v2:
> - add static function to check if osd keepalive timed out
> - Link to v1: https://patch.msgid.link/[email protected]
>
> To: Ilya Dryomov <[email protected]>
> To: Alex Markuze <[email protected]>
> To: Viacheslav Dubeyko <[email protected]>
> Cc: [email protected]
> Cc: [email protected]
> ---
>  include/linux/ceph/libceph.h    |  1 +
>  include/linux/ceph/osd_client.h |  1 +
>  net/ceph/messenger.c            |  3 +++
>  net/ceph/osd_client.c           | 29 ++++++++++++++++++++++++++++-
>  4 files changed, 33 insertions(+), 1 deletion(-)
>
> diff --git a/include/linux/ceph/libceph.h b/include/linux/ceph/libceph.h
> index 63e0e2aa1ce9..1a117c5f1964 100644
> --- a/include/linux/ceph/libceph.h
> +++ b/include/linux/ceph/libceph.h
> @@ -74,6 +74,7 @@ struct ceph_options {
>   */
>  #define CEPH_MOUNT_TIMEOUT_DEFAULT     msecs_to_jiffies(60 * 1000)
>  #define CEPH_OSD_KEEPALIVE_DEFAULT     msecs_to_jiffies(5 * 1000)
> +#define CEPH_OSD_PING_TIMEOUT          msecs_to_jiffies(60 * 1000)
>  #define CEPH_OSD_IDLE_TTL_DEFAULT      msecs_to_jiffies(60 * 1000)
>  #define CEPH_OSD_REQUEST_TIMEOUT_DEFAULT 0  /* no timeout */
>  #define CEPH_READ_FROM_REPLICA_DEFAULT 0  /* read from primary */
> diff --git a/include/linux/ceph/osd_client.h b/include/linux/ceph/osd_client.h
> index 50b14a5661c7..52eb76e9d62a 100644
> --- a/include/linux/ceph/osd_client.h
> +++ b/include/linux/ceph/osd_client.h
> @@ -94,6 +94,7 @@ struct ceph_osd {
>         struct ceph_auth_handshake o_auth;
>         unsigned long lru_ttl;
>         struct list_head o_keepalive_item;
> +       unsigned long o_keepalive_stamp;
>         struct mutex lock;
>         struct ceph_sparse_read o_sparse_read;
>  };
> diff --git a/net/ceph/messenger.c b/net/ceph/messenger.c
> index 34b3097b4c7b..f7776d83d506 100644
> --- a/net/ceph/messenger.c
> +++ b/net/ceph/messenger.c
> @@ -610,6 +610,9 @@ void ceph_con_open(struct ceph_connection *con,
>
>         memcpy(&con->peer_addr, addr, sizeof(*addr));
>         con->delay = 0;      /* reset backoff memory */
> +
> +       ktime_get_real_ts64(&con->last_keepalive_ack);
> +
>         mutex_unlock(&con->mutex);
>         queue_con(con);
>  }
> diff --git a/net/ceph/osd_client.c b/net/ceph/osd_client.c
> index 2ff00070c181..c4e1c0534801 100644
> --- a/net/ceph/osd_client.c
> +++ b/net/ceph/osd_client.c
> @@ -53,6 +53,7 @@ static void link_linger(struct ceph_osd *osd,
>  static void unlink_linger(struct ceph_osd *osd,
>                           struct ceph_osd_linger_request *lreq);
>  static void clear_backoffs(struct ceph_osd *osd);
> +static void kick_osd_requests(struct ceph_osd *osd);
>
>  #if 1
>  static inline bool rwsem_is_wrlocked(struct rw_semaphore *sem)
> @@ -3420,6 +3421,15 @@ static int linger_notify_finish_wait(struct ceph_osd_linger_request *lreq,
>         return left;
>  }
>
> +static bool osd_keepalive_timed_out(struct ceph_osd *osd)
> +{
> +       if (!time_after_eq(jiffies,
> +                          osd->o_keepalive_stamp + CEPH_OSD_PING_TIMEOUT))
> +               return false;
> +
> +       return ceph_con_keepalive_expired(&osd->o_con, CEPH_OSD_PING_TIMEOUT);
> +}
> +
>  /*
>   * Timeout callback, called every N seconds.  When 1 or more OSD
>   * requests has been active for more than N seconds, we send a keepalive
> @@ -3480,8 +3490,11 @@ static void handle_timeout(struct work_struct *work)
>                         mutex_unlock(&lreq->lock);
>                 }
>
> -               if (found)
> +               if (found) {
>                         list_move_tail(&osd->o_keepalive_item, &slow_osds);
> +               } else {
> +                       osd->o_keepalive_stamp = 0;
> +               }
>         }
>
>         if (opts->osd_request_timeout) {
> @@ -3507,6 +3520,20 @@ static void handle_timeout(struct work_struct *work)
>                                                         struct ceph_osd,
>                                                         o_keepalive_item);
>                 list_del_init(&osd->o_keepalive_item);
> +
> +               /* Record start of ping timeout from the first slow tick. */
> +               if (!osd->o_keepalive_stamp) {
> +                       osd->o_keepalive_stamp = jiffies;
> +               } else if (osd_keepalive_timed_out(osd)) {
> +                       pr_warn_ratelimited("osd%d not responding to keepalives, resetting session\n",
> +                                           osd->o_osd);
> +                       osd->o_sparse_op_idx = -1;
> +                       ceph_init_sparse_read(&osd->o_sparse_read);
> +                       if (!reopen_osd(osd))
> +                               kick_osd_requests(osd);
> +                       continue;
> +               }
> +
>                 ceph_con_keepalive(&osd->o_con);
>         }
>
>
> ---
> base-commit: b95f03f04d475aa6719d15a636ddf32222d55657
> change-id: 20260707-fix-rbd-keepalives-664fe9266c35
>
> Best regards,
> --
> Chris J Arges <[email protected]>
>