Re: [PATCH v10 1/1] virtio-video: Add virtio video device specification
Alexander Gordeev <[email protected]> Wed, 25 Feb 2026 15:21:30 +0100
| Newsgroups | dev.linux.lists.virtio-comment,org.kernel.vger.linux-media |
|---|---|
| Message-ID | <[email protected]> |
On 15/02/2026 16:11, Michael S. Tsirkin wrote: > On Sun, Feb 15, 2026 at 10:49:22AM +0000, Parav Pandit wrote: >> >>> From: Alexander Gordeev <[email protected]> >>> Sent: 13 February 2026 02:23 PM >>> >>> From: Alexander Gordeev <[email protected]> >>> >>> Add the specification of the video decoder and encoder device, which >>> can be used to provide host-accelerated video operations to the guest. >>> >>> Signed-off-by: Alexander Gordeev <[email protected]> >>> --- >>> conformance.tex | 12 +- >>> content.tex | 1 + >>> device-types/video/description.tex | 1592 +++++++++++++++++++++ >>> device-types/video/device-conformance.tex | 22 + >>> device-types/video/driver-conformance.tex | 20 + >>> introduction.tex | 21 + >>> 6 files changed, 1664 insertions(+), 4 deletions(-) >>> create mode 100644 device-types/video/description.tex >>> create mode 100644 device-types/video/device-conformance.tex >>> create mode 100644 device-types/video/driver-conformance.tex >>> >>> diff --git a/conformance.tex b/conformance.tex >>> index 9af31e2..f34e600 100644 >>> --- a/conformance.tex >>> +++ b/conformance.tex >>> @@ -37,8 +37,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \ref{sec:Conformance / Driver Conformance / PMEM Driver Conformance}, >>> \ref{sec:Conformance / Driver Conformance / CAN Driver Conformance}, >>> \ref{sec:Conformance / Driver Conformance / SPI Controller Driver Conformance}, >>> -\ref{sec:Conformance / Driver Conformance / Media Driver Conformance} or >>> -\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance}. >>> +\ref{sec:Conformance / Driver Conformance / Media Driver Conformance}, >>> +\ref{sec:Conformance / Driver Conformance / RTC Driver Conformance} or >>> +\ref{sec:Conformance / Driver Conformance / Video Driver Conformance}. >>> >>> \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. >>> \end{itemize} >>> @@ -68,8 +69,9 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \ref{sec:Conformance / Device Conformance / PMEM Device Conformance}, >>> \ref{sec:Conformance / Device Conformance / CAN Device Conformance}, >>> \ref{sec:Conformance / Device Conformance / SPI Controller Device Conformance}, >>> -\ref{sec:Conformance / Device Conformance / Media Device Conformance} or >>> -\ref{sec:Conformance / Device Conformance / RTC Device Conformance}. >>> +\ref{sec:Conformance / Device Conformance / Media Device Conformance}, >>> +\ref{sec:Conformance / Device Conformance / RTC Device Conformance} or >>> +\ref{sec:Conformance / Device Conformance / Video Device Conformance}. >>> >>> \item Clause \ref{sec:Conformance / Legacy Interface: Transitional Device and Transitional Driver Conformance}. >>> \end{itemize} >>> @@ -170,6 +172,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \input{device-types/spi/driver-conformance.tex} >>> \input{device-types/media/driver-conformance.tex} >>> \input{device-types/rtc/driver-conformance.tex} >>> +\input{device-types/video/driver-conformance.tex} >>> >>> \conformance{\section}{Device Conformance}\label{sec:Conformance / Device Conformance} >>> >>> @@ -264,6 +267,7 @@ \section{Conformance Targets}\label{sec:Conformance / Conformance Targets} >>> \input{device-types/spi/device-conformance.tex} >>> \input{device-types/media/device-conformance.tex} >>> \input{device-types/rtc/device-conformance.tex} >>> +\input{device-types/video/device-conformance.tex} >>> >>> \conformance{\section}{Legacy Interface: Transitional Device and Transitional Driver Conformance}\label{sec:Conformance / Legacy >>> Interface: Transitional Device and Transitional Driver Conformance} >>> A conformant implementation MUST be either transitional or >>> diff --git a/content.tex b/content.tex >>> index 5de811f..0c13f68 100644 >>> --- a/content.tex >>> +++ b/content.tex >>> @@ -835,6 +835,7 @@ \chapter{Device Types}\label{sec:Device Types} >>> \input{device-types/spi/description.tex} >>> \input{device-types/media/description.tex} >>> \input{device-types/rtc/description.tex} >>> +\input{device-types/video/description.tex} >>> >>> \chapter{Reserved Feature Bits}\label{sec:Reserved Feature Bits} >>> >>> diff --git a/device-types/video/description.tex b/device-types/video/description.tex >>> new file mode 100644 >>> index 0000000..8945e26 >>> --- /dev/null >>> +++ b/device-types/video/description.tex >>> @@ -0,0 +1,1592 @@ >>> +\section{Video Device} >>> +\label{sec:Device Types / Video Device} >>> + >>> +The virtio video device provides support for host-accelerated video encoding >>> +and decoding. >>> + >>> +\subsection{Device ID} >>> +\label{sec:Device Types / Video Device / Device ID} >>> + >>> +50 >>> + >>> +\subsection{Virtqueues} >>> +\label{sec:Device Types / Video Device / Virtqueues} >>> + >>> +\begin{description} >>> + \item[0] >>> + commandq - driver commands >>> + \item[1] >>> + eventq - device async responses to commands and standalone device events >>> +\end{description} >>> + >>> +\subsection{Feature bits} >>> +\label{sec:Device Types / Video Device / Feature bits} >>> + >>> +\begin{description} >>> + \item[VIRTIO_VIDEO_F_ENCODER (0)] >>> + The device can encode video. >>> + \item[VIRTIO_VIDEO_F_DECODER (1)] >>> + The device can decode video. >>> + % Use-case: the device can support both encoding and decoding, so having both >>> + % here can save resources. >>> + \item[VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES (2)] >>> + Guest pages can be used as the backing memory of resources. >>> + \item[VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG (3)] >>> + The device can use non-contiguous guest memory as the backing memory of >>> + resources. Only meaningful if VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES is also >>> + set. >>> + \item[VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT (4)] >>> + Objects exported by another virtio device can be used as the backing memory >>> + of resources. >>> + \item[VIRTIO_VIDEO_F_V4L2_COMPATIBLE_LAST_BUFFER (5)] >>> + The device releases an extra empty output buffer after a drain or DPC so that >>> + the driver can send a buffer with V4L2_BUF_FLAG_LAST set in the V4L2 way. >> This does not seem relavant to the video device. >> It should not be attached to any V4L2 implementation. >> Can you please craft it differently? > > My understanding is that this is a V4L2 device just like virtio input makes > the linux input layer into a device. Well, the device isn't really V4L2 and doesn't even use V4L2, but indeed the V4L2 driver is our main target for now. There is already virtio-media for exposing V4L2 devices from the host. There are other drivers to come though, so we're going to have setups with no V4L2 at all. >>> +\end{description} >>> + >>> +\devicenormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >>> + >>> +The device MUST set at least one of VIRTIO_VIDEO_F_ENCODER or VIRTIO_VIDEO_F_DECODER. >>> + >>> +The device MUST set at least one of VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES or >>> +VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT, since the absence of both bits would >>> +mean that no memory can be used at all for resources. >>> + >>> +The device MUST NOT set VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG unless it also sets >>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES. >>> + >>> +\drivernormative{\subsubsection}{Feature bits}{Device Types / Video Device / Feature bits} >>> + >>> +The driver MUST negotiate at least one of the VIRTIO_VIDEO_F_ENCODER and >>> +VIRTIO_VIDEO_F_DECODER features. >>> + >>> +The driver MUST negotiate at least one of the >>> +VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES and VIRTIO_VIDEO_F_RESOURCE_VIRTIO_OBJECT >>> +features. >>> + >> You wouldn't need virtio_object feature bit. It should be covered using the capability. > > To be frank I'm pretty split on this. Capabilities make sense > if there are envisioned to be lots of these. > I'll let the contributor decide if that is the case. I'd rather not change this. We won't have many flags I think. >>> +If VIRTIO_VIDEO_F_RESOURCE_GUEST_PAGES has been negotiated, but not >>> +VIRTIO_VIDEO_F_RESOURCE_NON_CONTIG, the driver MUST use physically >>> +contiguous memory for all the buffers it allocates. >>> + >>> +\subsection{Device configuration layout} >>> +\label{sec:Device Types / Video Device / Device configuration layout} >>> + >>> +The video device configuration space uses the following layout: >>> + >>> +\begin{lstlisting} >>> +struct virtio_video_config { >>> + le32 max_streams; >>> + le32 caps_length; >>> +}; >>> +\end{lstlisting} >>> + >>> +\begin{description} >>> + \item[\field{max_streams}] >>> + is the maximum number of concurrent streams the device supports. >>> + \item[\field{caps_length}] >>> + is the minimum length in bytes that a device-writable buffer must have >>> + in order to receive the response to VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS, see >>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Device Commands / QUERY CAPS}. >>> +\end{description} >>> + >>> +\devicenormative{\subsubsection}{Device configuration layout}{Device Types / Video Device / Device configuration layout} >>> + >>> +\field{max_streams} MUST be positive. >>> + >>> +\field{caps_length} MUST be set to the response size of >>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS. >>> + >> This plumbing can be easily done using 'Device and driver capabilities' located in the 'Group administration commands'. >> Please rework the patch to use the existing basic facility. >> This can also possibly eliminate plumbing device specific command q. >> More below. > > admin commands are kind of heavyweight, they are designed for control > path. Question is, is this a data path or a control path thing? > If data path it is not appropriate, if control path it is. > > >>> +\subsection{Device Initialization} >>> +\label{sec:Device Types / Video Device / Device Initialization} >>> +\begin{enumerate} >>> + \item >>> + The driver reads the feature bits and negotiates the features it needs. >>> + \item >>> + The driver sets up the commandq and the eventq. >>> + \item >>> + The driver reads the \field{caps_length} field of the configuration >>> + space, prepares a buffer of at least that size and sends the buffer on the >>> + commandq with the VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS command. >>> + \item >>> + The device sends a response over commandq to >>> + VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS via used descriptors provided with the >>> + command. >>> + \item >>> + The driver receives the response from the device, and parses its capabilities. >>> +\end{enumerate} >>> + >>> +\subsection{Device Operation} >>> +\label{sec:Device Types / Video Device / Device Operation} >>> + >>> +The device supports opening and operating a number of parallel streams up to >>> +\field{max_streams}. Each stream has three internal device queues: mainqX, >>> +inputqX and outputqX, where X is the stream id. Each stream command has a >>> +field that dispatches the command to the specific internal queue. >>> + >> If these queues are internal to the device it does not need exposure in the spec here. If they have a meaning to the driver, >> Than keyword 'internal' should be dropped and rephased. > > I think the implication is that within each queue commands > are consumed in order? > > This isn't terribly clear. Noted, I'll make sure to improve the description in the next draft. >>> +% Use-case: there might be different real-time requirements for different >>> +% streams, so more virtqueues can be added in the future if necessary. >>> +% The internal queues don't change, the data formats don't change, only the >>> +% mapping of streams/internal queues to particular virtqueues changes. >>> + >>> +The mainqX is used to open a stream with VIRTIO_VIDEO_CMD_STREAM_OPEN, >>> +close a stream with VIRTIO_VIDEO_CMD_STREAM_CLOSE, reset inputqX or outputqX >>> +using VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET, set some of the stream parameters out >>> +of band with high priority with VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, unblock >>> +the outputqX with VIRTIO_VIDEO_CMD_STREAM_UNBLOCK if it gets blocked for >>> +format negotiation. >>> + >>> +The inputqX and outputqX are used to queue input or output resources using >>> +VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE. Additionally inputqX is used to set input and >>> +output parameters using VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, to complete >>> +processing of all queued input resources and make the resulting output >>> +resources available to the driver using VIRTIO_VIDEO_CMD_STREAM_DRAIN. >>> + >>> +All the stream commands start async operations, return the results >>> +using async responses over eventq. >>> +The eventq is used by the device to send the device's async responses to >>> +stream commands and the device's standalone events. >>> + >>> +% This way eventq becomes the single source of truth about the device's state, >>> +% the driver doesn't have to tediously synchronize commandq's and eventq's >>> +% used queues the way it was necessary in the past (similarly to V4L2's DQBUF >>> +% and DQEVENT). One more benefit is that commandq is processed fast and >>> +% strictly in order, so commandq descriptors exhaustion should never happen in >>> +% practice. >>> + >> Video device should be implementable without any V4L2 binding/description etc. >> >>> +Parameters allow the driver to configure the stream including setting up the >>> +resources. Available parameters depend on the device type, see >>> +\ref{sec:Device Types / Video Device / Device capabilities and parameters}. >>> + >>> +A resource is a set of memory buffers that contain a unit of data that >>> +the device can process or produce. Most resources have only one buffer, >>> +raw frames using a multi-planar format can have several. >>> +Input resources are filled by the driver with compressed (coded) video data >>> +for a decoder and raw frames for an encoder, output resources are filled by >>> +the device as the result of processing the input resources with decoded raw >>> +frames for a decoder and compressed (encoded) data for an encoder. >>> +Resources from inputqX and outputqX are consumed independently, not in pairs. >>> +One input resource can result in zero to many produced output resources. >>> +A decoder device dequeues the output decoded frames in presentation order. >>> +An encoder device dequeues the output decoded frames in decoding order. >>> +The driver can reuse a queued resource after receiving a corresponding async >>> +response. Dequeued output resources can still be used by the device as >>> +reference frames, so the driver can't write to them. >>> + >>> +% TODO: maybe send the second RESOURCE_QUEUE async response, when the dequeued >>> +% output resource is not used by the device anymore and therefore becomes >>> +% writeable? >>> + >>> +The device can detect standalone stream-related events: errors and dynamic >>> +parameters changes that require intervention from the driver (e.g. >>> +reallocating backing memory of output resources to fit the new parameters). >>> +The events are signalled on the eventq, see >>> +\ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >>> + >>> +\devicenormative{\subsubsection}{Device Operation}{Device Types / Video Device / Device Operation} >>> + >>> +The device MUST set to zero all unused, disabled or padding bits in its >>> +responses. >>> + >>> +\subsubsection{Device Operation: Command Virtqueue} >>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Command Virtqueue} >>> + >>> +This section lists the commands that can be sent by the driver to commandq. >>> + >>> +Different structures are used for each command and response. A command >>> +structure starts with the requested command code, defined as follows: >>> + >>> +\begin{lstlisting} >>> +/* Device */ >>> +#define VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS 0x100 >>> + >>> +/* Stream */ >>> +#define VIRTIO_VIDEO_CMD_STREAM_OPEN 0x200 >>> +#define VIRTIO_VIDEO_CMD_STREAM_CLOSE 0x201 >> You can craft the stream using a existing basic facility of resource object. >> Where each stream is just a resource object, that be queried or modified. >> >>> +#define VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS 0x202 >>> +#define VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS 0x203 >>> +#define VIRTIO_VIDEO_CMD_STREAM_UNBLOCK 0x204 >>> +#define VIRTIO_VIDEO_CMD_STREAM_DRAIN 0x205 >>> +#define VIRTIO_VIDEO_CMD_STREAM_QUEUE_RESET 0x206 >>> +#define VIRTIO_VIDEO_CMD_STREAM_RESOURCE_QUEUE 0x207 >>> +\end{lstlisting} >>> + >>> +Stream commands start with a header: >>> + >>> +\begin{lstlisting} >>> +#define VIRTIO_VIDEO_QUEUE_TYPE_MAIN 0 >>> +#define VIRTIO_VIDEO_QUEUE_TYPE_INPUT 1 >>> +#define VIRTIO_VIDEO_QUEUE_TYPE_OUTPUT 2 >>> + >>> +struct virtio_video_stream_cmd_header { >>> + le32 type; /* One of VIRTIO_VIDEO_CMD_STREAM_* */ >>> + le32 stream_id; >>> + le32 queue_type; /* One of VIRTIO_VIDEO_QUEUE_TYPE_* */ >>> + le32 async_response_cookie; >>> +}; >>> +\end{lstlisting} >>> + >>> +\begin{description} >>> + \item[\field{async_response_cookie}] >>> + is an async response cookie provided by the driver, that allows >>> + to relate an async response to the previously submitted command. >>> +\end{description} >>> + >>> +\subsubsection{Device Operation: Event Virtqueue} >>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: Event Virtqueue} >>> + >>> +The eventq is used by the device to send async responses to commands queued >>> +by the driver on commandq and standalone events. Stream errors and dynamic >>> +parameters changes are caused by changes in the device's state, not by >>> +commands, still they are delivered as responses to implicit >>> +VIRTIO_VIDEO_CMD_STREAM_CLOSE and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS, >>> +respectively. >>> + >>> +Events start with a header: >>> + >>> +\begin{lstlisting} >>> +#define VIRTIO_VIDEO_EVENT_FLAG_ERROR (1 << 0) >>> +#define VIRTIO_VIDEO_EVENT_FLAG_STANDALONE (1 << 1) >>> +#define VIRTIO_VIDEO_EVENT_FLAG_CANCELED (1 << 2) >>> +#define VIRTIO_VIDEO_EVENT_FLAG_BLOCKED (1 << 3) >>> + >>> +struct virtio_video_event_header { >>> + le32 event_type; /* VIRTIO_VIDEO_CMD_STREAM_* */ >>> + le32 stream_id; >>> + le32 async_response_cookie; >>> + le32 event_flags; /* Bitmask of VIRTIO_VIDEO_EVENT_FLAG_* */ >>> +}; >>> +\end{lstlisting} >>> + >>> +\begin{description} >>> + \item[\field{event_type}] >>> + is the type of the event. >>> + \item[\field{stream_id}] >>> + is the ID of a valid stream. >>> + \item[\field{async_response_cookie}] >>> + is an async response cookie provided by the driver, that allows >>> + to relate the event to a previously submitted command. >>> + \item[\field{event_flags}] >>> + is a bitmask of VIRTIO_VIDEO_EVENT_FLAG_* flags >>> + >>> + \begin{description} >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_ERROR] >>> + is set if the command finished with an error due to an >>> + invalid argument or for other reasons. >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_STANDALONE] >>> + is set for standalone events, see >>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Standalone Events}. >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_CANCELED] >>> + is set if the command has been canceled by another >>> + command, that has higher priority. Doesn't make sense >>> + for standalone events. >>> + \item[VIRTIO_VIDEO_EVENT_FLAG_BLOCKED] >>> + is set if the command triggered a block on the >>> + outputqX to allow output format negotiation. >>> + When the negotiation is finished the block has to be >>> + removed using VIRTIO_VIDEO_CMD_STREAM_UNBLOCK >>> + command, see >>> + \ref{sec:Device Types / Video Device / Device Operation / Device Operation: Stream commands / UNBLOCK}. >>> + \end{description} >>> +\end{description} >>> + >>> +The particular data structure representing the event is selected according to >>> +the \field{event_type}. >>> + >>> +\drivernormative{\paragraph}{Device Operation: Event Virtqueue}{Device Types / Video Device / Device Operation / Device Operation: >>> Event Virtqueue} >>> + >>> +The driver MUST at any time have at least one descriptor with a used >>> +buffer large enough to contain a \field{struct virtio_video_event} >>> +queued on the eventq. >>> + >>> +The driver MUST NOT put device-readable descriptors into the eventq. >>> + >>> +The driver MUST account for the fact that the async responses to commands >>> +might come out-of-order (i.e. after other commands sent to the device), >>> +and that some of them can be cancelled. >>> + >>> +The driver SHOULD wait for an async response of command A, that caused >>> +cancellation of command B, before queueing the command B again. >>> + >>> +\subsubsection{Device Operation: TLV format} >>> +\label{sec:Device Types / Video Device / Device Operation / Device Operation: TLV format} >>> + >>> +VIRTIO_VIDEO_CMD_DEVICE_QUERY_CAPS and VIRTIO_VIDEO_CMD_STREAM_SET_PARAMS/ >>> +VIRTIO_VIDEO_CMD_STREAM_GET_PARAMS >>> +commands represent device capabilities and corresponding device parameters >>> +in the form of TLV (Type-Length-Value): >>> + >>> +\begin{lstlisting} >>> +struct virtio_video_tlv { >>> + le32 type; >>> + le32 length; >>> + u8 value[length]; >>> +}; >>> +\end{lstlisting} > > will cause all kind of padding mischief if length is not a multiple of > 4. Yes, true. It is indeed mentioned further: + \item[\field{length}] + specifies the \field{value} size in bytes aligned to 4 bytes.