Re: [bug report] libceph: Oversized front segment in CEPH_MSG_OSD_OPREPLY not in all cases correctly handled

cc: [email protected]

On Mon, 2026-08-17 at 12:21 +0200, Raphael Zimmer wrote:
> Hi,
> 
> I discovered a problem with the handling of CEPH_MSG_OSD_OPREPLY
> messages with oversized front segments in get_reply()
> (net/ceph/osd_client.c), which can be used to trigger a BUG()
> assertion.
> 
> get_reply() is used to get the already allocated ceph_msg for
> receiving
> the reply to an osd request. In this function, there is a part that
> handles incoming messages with a front segment that is larger than
> the
> preallocated one. In this part, a new message with a larger front
> segment is allocated and swapped in for the reply message. This does
> not
> consider that the message may have a data segment. If the now
> replaced
> ceph_msg had data items, the newly allocated one doesn't contain them
> anymore. This may lead to the subsequent
> 	if (!srlen && data_len > req->r_reply->data_length)
> statement being true and skipping this message even though a new
> front
> segment was allocated to allow for handling it.
> 
> Additionally, this behavior can trigger a
> 	BUG_ON(msg->num_data_items >= msg->max_data_items)
> assertion in ceph_msg_data_add() (net/ceph/messenger.c). This happens
> when a reply message for a request that initially contained data
> items
> in its preallocated reply message is received that doesn't contain a
> data segment, but an oversized front segment. This could happen,
> e.g.,
> for a CEPH_OSD_OP_READ. In this case, the message is not skipped, but
> processed. If the message is otherwise corrupted, or an error occurs
> during processing, this is treated as a connection fault and
> con_fault_finish() is called. If the connection to the osd can be
> reopened, pending requests will get resubmitted, which results in
> resending the request. During this, ceph_msg_data_add() will
> eventually
> be called, where the assertion triggers because max_data_items still
> has
> its initial value of zero after reallocating the reply message.
> 
> This further allows a malicious peer to intentionally trigger this
> assertion by sending a message to the client.
> 
> The problem could be fixed by also adding the needed data items to
> the
> new message. However, for a request that has data items in both its
> request and reply message, the normal way of creating the message
> wouldn't work, as setup_request_data() is not meant to be called when
> either of the messages already has its data items. Therefore, this
> would
> likely require manual creation of the data items for the new message.
> With this in mind, it seems simpler to not allocate a new message at
> all, but only allocate a new front segment buffer and replace the one
> in
> the preallocated message with it.
> 
> I can submit a patch following this approach. However, I would like
> to
> clarify first if there was any reason behind this implementation.
> Should
> a larger front segment intentionally only be accepted for reply
> messages
> that don't contain data items, and does it even happen under normal
> operation? In case it never actually occurs as long as no error
> occurs
> or a message is corrupted, an osd request reply message with an
> oversized front segment may be discarded right away without
> reallocating
> the buffer.
> 
> Best regards,
> Raphael