[virtio-dev] Re: [PATCH 2/5] virtio: introduce SUSPEND bit in device status

On 9/18/2023 10:56 AM, Zhu, Lingshan wrote:
>
>
> On 9/15/2023 7:10 PM, Michael S. Tsirkin wrote:
>> On Fri, Sep 15, 2023 at 10:57:33AM +0800, Zhu, Lingshan wrote:
>>>
>>> On 9/14/2023 7:34 PM, Michael S. Tsirkin wrote:
>>>> On Wed, Sep 06, 2023 at 04:16:34PM +0800, Zhu Lingshan wrote:
>>>>> This patch introduces a new status bit in the device status: SUSPEND.
>>>>>
>>>>> This SUSPEND bit can be used by the driver to suspend a device,
>>>>> in order to stabilize the device states and virtqueue states.
>>>>>
>>>>> Its main use case is live migration.
>>>>>
>>>>> Signed-off-by: Zhu Lingshan <[email protected]>
>>>>> Signed-off-by: Jason Wang <[email protected]>
>>>>> Signed-off-by: Eugenio Pérez <[email protected]>
>>>>> ---
>>>>>    content.tex | 31 +++++++++++++++++++++++++++++++
>>>>>    1 file changed, 31 insertions(+)
>>>>>
>>>>> diff --git a/content.tex b/content.tex
>>>>> index 0e492cd..0fab537 100644
>>>>> --- a/content.tex
>>>>> +++ b/content.tex
>>>>> @@ -47,6 +47,9 @@ \section{\field{Device Status} 
>>>>> Field}\label{sec:Basic Facilities of a Virtio Dev
>>>>>    \item[DRIVER_OK (4)] Indicates that the driver is set up and 
>>>>> ready to
>>>>>      drive the device.
>>>>> +\item[SUSPEND (16)] When VIRTIO_F_SUSPEND is negotiated, 
>>>>> indicates that the
>>>>> +  device has been suspended by the driver.
>>>>> +
>>>>>    \item[DEVICE_NEEDS_RESET (64)] Indicates that the device has 
>>>>> experienced
>>>>>      an error from which it can't recover.
>>>>>    \end{description}
>>>>> @@ -73,6 +76,10 @@ \section{\field{Device Status} 
>>>>> Field}\label{sec:Basic Facilities of a Virtio Dev
>>>>>    recover by issuing a reset.
>>>>>    \end{note}
>>>>> +The driver SHOULD NOT set SUSPEND if FEATURES_OK is not set.
>>>>> +
>>>>> +When setting SUSPEND, the driver MUST re-read \field{device 
>>>>> status} to ensure the SUSPEND bit is set.
>>>>> +
>>>>>    \devicenormative{\subsection}{Device Status Field}{Basic 
>>>>> Facilities of a Virtio Device / Device Status Field}
>>>>>    The device MUST NOT consume buffers or send any used buffer
>>>>> @@ -82,6 +89,26 @@ \section{\field{Device Status} 
>>>>> Field}\label{sec:Basic Facilities of a Virtio Dev
>>>>>    that a reset is needed.  If DRIVER_OK is set, after it sets 
>>>>> DEVICE_NEEDS_RESET, the device
>>>>>    MUST send a device configuration change notification to the 
>>>>> driver.
>>>>> +The device MUST ignore SUSPEND if FEATURES_OK is not set.
>>>>> +
>>>>> +The device MUST ignore SUSPEND if VIRTIO_F_SUSPEND is not 
>>>>> negotiated.
>>>> why? let's just forbid driver from setting it.
>>> OK
>>>>> +
>>>>> +The device SHOULD allow settings to \field{device status} even 
>>>>> when SUSPEND is set.
>>>>> +
>>>>> +If VIRTIO_F_SUSPEND is negotiated and SUSPEND is set, the device 
>>>>> SHOULD clear SUSPEND
>>>>> +and resumes operation upon DRIVER_OK.
>>>>> +
>>>> sorry what?
>>> In case of a failed or cancelled Live Migration, the device needs to 
>>> resume
>>> operation.
>>> However the spec forbids the driver to clear a device status bit, so
>>> re-writing
>>> DRIVER_OK is expected to clear SUSPEND and the device resume operation.
>> No, DRIVER_OK is already set. Setting a bit that is already set should
>> not have side effects. In fact auto-clearing suspend is problematic too.
> The spec says: Set the DRIVER_OK status bit. At this point the device 
> is “live”.
>
> So semantically DRIVER_OK can bring the device to live even from SUSPEND.
>
> In the implementation, the device can check whether SUSPEND is set, then
> decide what to do. Just don't ignore DRIVER_OK if it is already
> set, and the driver should not clear a device status bit.
>
>>
>>
>>>>> +If VIRTIO_F_SUSPEND is negotiated, when the driver sets SUSPEND,
>>>>> +the device SHOULD perform the following actions before presenting 
>>>>> SUSPEND bit in the \field{device status}:
>>>>> +
>>>>> +\begin{itemize}
>>>>> +\item Stop consuming buffers of any virtqueues and mark all 
>>>>> finished descritors as used.
>>>>> +\item Wait until all descriptors that being processed to finish 
>>>>> and mark them as used.
>>>>> +\item Flush all used buffer and send used buffer notifications to 
>>>>> the driver.
>>>> flush how?
>>> This is device-type-specific, and we will include tracking inflight
>>> descriptors(buffers) in V2.
>>>>> +\item Record Virtqueue State of each enabled virtqueue, see 
>>>>> section \ref{sec:Virtqueues / Virtqueue State}
>>>> record where?
>>> This is transport specific, for PCI, patch 5 introduces two new 
>>> fields for
>>> avail and used state
>> they clearly can't store state for all vqs, these are just two 16 bit 
>> fields.
> vq states filed can work with queue_select like other vq fields.
> I will document this in the comment.
>>
>>>>> +\item Pause its operation except \field{device status} and 
>>>>> preserve configurations in its Device Configuration Space, see 
>>>>> \ref{sec:Basic Facilities of a Virtio Device / Device 
>>>>> Configuration Space}
>>>> pause in what sense? completely? this does not seem realistic.
>>>> e.g. pci express link has to stay active or device will die.
>>> only pause virtio, I will rephrase the sentence as "pause its virtio
>>> operation".
>> that is vague too. for example what happens to link state of
>> a networking device?
> Then how about we say: pause operation in both data-path and 
> control-path?
>
> Or do you have any suggestion?
>>
>>> Others like PCI link in the example is out of the spec and we don't 
>>> need
>>> to migrate them.
>>>>
>>>> also, presumably here it is except a bunch of other fields.
>>>> e.g. what about queue select and all related queue fields?
>>> For now they are forbidden.
>>>
>>> As SiWei suggested, we will introduce a new feature bit to control 
>>> whether
>>> allowing resetting a VQ after SUSPEND. We can use more feature bits if
>>> there are requirements to perform anything after SUSPEND. But for now
>>> they are forbidden.
>> I don't know how this means, but whatever. you need to make
>> all this explicit though.
> a new feature bit: VIRTIO_F_RING_SUSPEND_RESET. If this feature
> bit has been negotiated then the device allow reset a vq after SUSPEND.
Hi Michael,

Rethink of this, as you suggested before, In V2, I will forbid resetting
VQs after SUSPEND.

Thanks
>>
>>>>> +\end{itemize}
>>>>> +
>>>>>    \section{Feature Bits}\label{sec:Basic Facilities of a Virtio 
>>>>> Device / Feature Bits}
>>>>>    Each virtio device offers all the features it understands.  During
>>>>> @@ -937,6 +964,10 @@ \chapter{Reserved Feature 
>>>>> Bits}\label{sec:Reserved Feature Bits}
>>>>>        \ref{devicenormative:Basic Facilities of a Virtio Device / 
>>>>> Feature Bits} for
>>>>>        handling features reserved for future use.
>>>>> +  \item[VIRTIO_F_SUSPEND(42)] This feature indicates that the 
>>>>> driver can
>>>>> +   SUSPEND the device.
>>>>> +   See \ref{sec:Basic Facilities of a Virtio Device / Device 
>>>>> Status Field}.
>>>>> +
>>>>>    \end{description}
>>>>>    \drivernormative{\section}{Reserved Feature Bits}{Reserved 
>>>>> Feature Bits}
>>>>> -- 
>>>>> 2.35.3
>


---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]