| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
.. c:namespace:: V4L |
| 3 |
|
| 4 |
.. _stateless_decoder: |
| 5 |
|
| 6 |
************************************************** |
| 7 |
Memory-to-memory Stateless Video Decoder Interface |
| 8 |
************************************************** |
| 9 |
|
| 10 |
A stateless decoder is a decoder that works without retaining any kind of state |
| 11 |
between processed frames. This means that each frame is decoded independently |
| 12 |
of any previous and future frames, and that the client is responsible for |
| 13 |
maintaining the decoding state and providing it to the decoder with each |
| 14 |
decoding request. This is in contrast to the stateful video decoder interface, |
| 15 |
where the hardware and driver maintain the decoding state and all the client |
| 16 |
has to do is to provide the raw encoded stream and dequeue decoded frames in |
| 17 |
display order. |
| 18 |
|
| 19 |
This section describes how user-space ("the client") is expected to communicate |
| 20 |
with stateless decoders in order to successfully decode an encoded stream. |
| 21 |
Compared to stateful codecs, the decoder/client sequence is simpler, but the |
| 22 |
cost of this simplicity is extra complexity in the client which is responsible |
| 23 |
for maintaining a consistent decoding state. |
| 24 |
|
| 25 |
Stateless decoders make use of the :ref:`media-request-api`. A stateless |
| 26 |
decoder must expose the ``V4L2_BUF_CAP_SUPPORTS_REQUESTS`` capability on its |
| 27 |
``OUTPUT`` queue when :c:func:`VIDIOC_REQBUFS` or :c:func:`VIDIOC_CREATE_BUFS` |
| 28 |
are invoked. |
| 29 |
|
| 30 |
Depending on the encoded formats supported by the decoder, a single decoded |
| 31 |
frame may be the result of several decode requests (for instance, H.264 streams |
| 32 |
with multiple slices per frame). Decoders that support such formats must also |
| 33 |
expose the ``V4L2_BUF_CAP_SUPPORTS_M2M_HOLD_CAPTURE_BUF`` capability on their |
| 34 |
``OUTPUT`` queue. |
| 35 |
|
| 36 |
Querying capabilities |
| 37 |
===================== |
| 38 |
|
| 39 |
1. To enumerate the set of coded formats supported by the decoder, the client |
| 40 |
calls :c:func:`VIDIOC_ENUM_FMT` on the ``OUTPUT`` queue. |
| 41 |
|
| 42 |
* The driver must always return the full set of supported ``OUTPUT`` formats, |
| 43 |
irrespective of the format currently set on the ``CAPTURE`` queue. |
| 44 |
|
| 45 |
* Simultaneously, the driver must restrain the set of values returned by |
| 46 |
codec-specific capability controls (such as H.264 profiles) to the set |
| 47 |
actually supported by the hardware. |
| 48 |
|
| 49 |
2. To enumerate the set of supported raw formats, the client calls |
| 50 |
:c:func:`VIDIOC_ENUM_FMT` on the ``CAPTURE`` queue. |
| 51 |
|
| 52 |
* The driver must return only the formats supported for the format currently |
| 53 |
active on the ``OUTPUT`` queue. |
| 54 |
|
| 55 |
* Depending on the currently set ``OUTPUT`` format, the set of supported raw |
| 56 |
formats may depend on the value of some codec-dependent controls. |
| 57 |
The client is responsible for making sure that these controls are set |
| 58 |
before querying the ``CAPTURE`` queue. Failure to do so will result in the |
| 59 |
default values for these controls being used, and a returned set of formats |
| 60 |
that may not be usable for the media the client is trying to decode. |
| 61 |
|
| 62 |
3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported |
| 63 |
resolutions for a given format, passing desired pixel format in |
| 64 |
:c:type:`v4l2_frmsizeenum`'s ``pixel_format``. |
| 65 |
|
| 66 |
4. Supported profiles and levels for the current ``OUTPUT`` format, if |
| 67 |
applicable, may be queried using their respective controls via |
| 68 |
:c:func:`VIDIOC_QUERYCTRL`. |
| 69 |
|
| 70 |
Initialization |
| 71 |
============== |
| 72 |
|
| 73 |
1. Set the coded format on the ``OUTPUT`` queue via :c:func:`VIDIOC_S_FMT`. |
| 74 |
|
| 75 |
* **Required fields:** |
| 76 |
|
| 77 |
``type`` |
| 78 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 79 |
|
| 80 |
``pixelformat`` |
| 81 |
a coded pixel format. |
| 82 |
|
| 83 |
``width``, ``height`` |
| 84 |
coded width and height parsed from the stream. |
| 85 |
|
| 86 |
other fields |
| 87 |
follow standard semantics. |
| 88 |
|
| 89 |
.. note:: |
| 90 |
|
| 91 |
Changing the ``OUTPUT`` format may change the currently set ``CAPTURE`` |
| 92 |
format. The driver will derive a new ``CAPTURE`` format from the |
| 93 |
``OUTPUT`` format being set, including resolution, colorimetry |
| 94 |
parameters, etc. If the client needs a specific ``CAPTURE`` format, |
| 95 |
it must adjust it afterwards. |
| 96 |
|
| 97 |
2. Call :c:func:`VIDIOC_S_EXT_CTRLS` to set all the controls (parsed headers, |
| 98 |
etc.) required by the ``OUTPUT`` format to enumerate the ``CAPTURE`` formats. |
| 99 |
|
| 100 |
3. Call :c:func:`VIDIOC_G_FMT` for ``CAPTURE`` queue to get the format for the |
| 101 |
destination buffers parsed/decoded from the bytestream. |
| 102 |
|
| 103 |
* **Required fields:** |
| 104 |
|
| 105 |
``type`` |
| 106 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 107 |
|
| 108 |
* **Returned fields:** |
| 109 |
|
| 110 |
``width``, ``height`` |
| 111 |
frame buffer resolution for the decoded frames. |
| 112 |
|
| 113 |
``pixelformat`` |
| 114 |
pixel format for decoded frames. |
| 115 |
|
| 116 |
``num_planes`` (for _MPLANE ``type`` only) |
| 117 |
number of planes for pixelformat. |
| 118 |
|
| 119 |
``sizeimage``, ``bytesperline`` |
| 120 |
as per standard semantics; matching frame buffer format. |
| 121 |
|
| 122 |
.. note:: |
| 123 |
|
| 124 |
The value of ``pixelformat`` may be any pixel format supported for the |
| 125 |
``OUTPUT`` format, based on the hardware capabilities. It is suggested |
| 126 |
that the driver chooses the preferred/optimal format for the current |
| 127 |
configuration. For example, a YUV format may be preferred over an RGB |
| 128 |
format, if an additional conversion step would be required for RGB. |
| 129 |
|
| 130 |
4. *[optional]* Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on |
| 131 |
the ``CAPTURE`` queue. The client may use this ioctl to discover which |
| 132 |
alternative raw formats are supported for the current ``OUTPUT`` format and |
| 133 |
select one of them via :c:func:`VIDIOC_S_FMT`. |
| 134 |
|
| 135 |
.. note:: |
| 136 |
|
| 137 |
The driver will return only formats supported for the currently selected |
| 138 |
``OUTPUT`` format and currently set controls, even if more formats may be |
| 139 |
supported by the decoder in general. |
| 140 |
|
| 141 |
For example, a decoder may support YUV and RGB formats for |
| 142 |
resolutions 1920x1088 and lower, but only YUV for higher resolutions (due |
| 143 |
to hardware limitations). After setting a resolution of 1920x1088 or lower |
| 144 |
as the ``OUTPUT`` format, :c:func:`VIDIOC_ENUM_FMT` may return a set of |
| 145 |
YUV and RGB pixel formats, but after setting a resolution higher than |
| 146 |
1920x1088, the driver will not return RGB pixel formats, since they are |
| 147 |
unsupported for this resolution. |
| 148 |
|
| 149 |
5. *[optional]* Choose a different ``CAPTURE`` format than suggested via |
| 150 |
:c:func:`VIDIOC_S_FMT` on ``CAPTURE`` queue. It is possible for the client to |
| 151 |
choose a different format than selected/suggested by the driver in |
| 152 |
:c:func:`VIDIOC_G_FMT`. |
| 153 |
|
| 154 |
* **Required fields:** |
| 155 |
|
| 156 |
``type`` |
| 157 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 158 |
|
| 159 |
``pixelformat`` |
| 160 |
a raw pixel format. |
| 161 |
|
| 162 |
``width``, ``height`` |
| 163 |
frame buffer resolution of the decoded stream; typically unchanged from |
| 164 |
what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different |
| 165 |
if the hardware supports composition and/or scaling. |
| 166 |
|
| 167 |
After performing this step, the client must perform step 3 again in order |
| 168 |
to obtain up-to-date information about the buffers size and layout. |
| 169 |
|
| 170 |
6. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on |
| 171 |
``OUTPUT`` queue. |
| 172 |
|
| 173 |
* **Required fields:** |
| 174 |
|
| 175 |
``count`` |
| 176 |
requested number of buffers to allocate; greater than zero. |
| 177 |
|
| 178 |
``type`` |
| 179 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 180 |
|
| 181 |
``memory`` |
| 182 |
follows standard semantics. |
| 183 |
|
| 184 |
* **Returned fields:** |
| 185 |
|
| 186 |
``count`` |
| 187 |
actual number of buffers allocated. |
| 188 |
|
| 189 |
* If required, the driver will adjust ``count`` to be equal or bigger to the |
| 190 |
minimum of required number of ``OUTPUT`` buffers for the given format and |
| 191 |
requested count. The client must check this value after the ioctl returns |
| 192 |
to get the actual number of buffers allocated. |
| 193 |
|
| 194 |
7. Allocate destination (raw format) buffers via :c:func:`VIDIOC_REQBUFS` on the |
| 195 |
``CAPTURE`` queue. |
| 196 |
|
| 197 |
* **Required fields:** |
| 198 |
|
| 199 |
``count`` |
| 200 |
requested number of buffers to allocate; greater than zero. The client |
| 201 |
is responsible for deducing the minimum number of buffers required |
| 202 |
for the stream to be properly decoded (taking e.g. reference frames |
| 203 |
into account) and pass an equal or bigger number. |
| 204 |
|
| 205 |
``type`` |
| 206 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 207 |
|
| 208 |
``memory`` |
| 209 |
follows standard semantics. ``V4L2_MEMORY_USERPTR`` is not supported |
| 210 |
for ``CAPTURE`` buffers. |
| 211 |
|
| 212 |
* **Returned fields:** |
| 213 |
|
| 214 |
``count`` |
| 215 |
adjusted to allocated number of buffers, in case the codec requires |
| 216 |
more buffers than requested. |
| 217 |
|
| 218 |
* The driver must adjust count to the minimum of required number of |
| 219 |
``CAPTURE`` buffers for the current format, stream configuration and |
| 220 |
requested count. The client must check this value after the ioctl |
| 221 |
returns to get the number of buffers allocated. |
| 222 |
|
| 223 |
8. Allocate requests (likely one per ``OUTPUT`` buffer) via |
| 224 |
:c:func:`MEDIA_IOC_REQUEST_ALLOC` on the media device. |
| 225 |
|
| 226 |
9. Start streaming on both ``OUTPUT`` and ``CAPTURE`` queues via |
| 227 |
:c:func:`VIDIOC_STREAMON`. |
| 228 |
|
| 229 |
Decoding |
| 230 |
======== |
| 231 |
|
| 232 |
For each frame, the client is responsible for submitting at least one request to |
| 233 |
which the following is attached: |
| 234 |
|
| 235 |
* The amount of encoded data expected by the codec for its current |
| 236 |
configuration, as a buffer submitted to the ``OUTPUT`` queue. Typically, this |
| 237 |
corresponds to one frame worth of encoded data, but some formats may allow (or |
| 238 |
require) different amounts per unit. |
| 239 |
* All the metadata needed to decode the submitted encoded data, in the form of |
| 240 |
controls relevant to the format being decoded. |
| 241 |
|
| 242 |
The amount of data and contents of the source ``OUTPUT`` buffer, as well as the |
| 243 |
controls that must be set on the request, depend on the active coded pixel |
| 244 |
format and might be affected by codec-specific extended controls, as stated in |
| 245 |
documentation of each format. |
| 246 |
|
| 247 |
If there is a possibility that the decoded frame will require one or more |
| 248 |
decode requests after the current one in order to be produced, then the client |
| 249 |
must set the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag on the ``OUTPUT`` |
| 250 |
buffer. This will result in the (potentially partially) decoded ``CAPTURE`` |
| 251 |
buffer not being made available for dequeueing, and reused for the next decode |
| 252 |
request if the timestamp of the next ``OUTPUT`` buffer has not changed. |
| 253 |
|
| 254 |
A typical frame would thus be decoded using the following sequence: |
| 255 |
|
| 256 |
1. Queue an ``OUTPUT`` buffer containing one unit of encoded bytestream data for |
| 257 |
the decoding request, using :c:func:`VIDIOC_QBUF`. |
| 258 |
|
| 259 |
* **Required fields:** |
| 260 |
|
| 261 |
``index`` |
| 262 |
index of the buffer being queued. |
| 263 |
|
| 264 |
``type`` |
| 265 |
type of the buffer. |
| 266 |
|
| 267 |
``bytesused`` |
| 268 |
number of bytes taken by the encoded data frame in the buffer. |
| 269 |
|
| 270 |
``flags`` |
| 271 |
the ``V4L2_BUF_FLAG_REQUEST_FD`` flag must be set. Additionally, if |
| 272 |
we are not sure that the current decode request is the last one needed |
| 273 |
to produce a fully decoded frame, then |
| 274 |
``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` must also be set. |
| 275 |
|
| 276 |
``request_fd`` |
| 277 |
must be set to the file descriptor of the decoding request. |
| 278 |
|
| 279 |
``timestamp`` |
| 280 |
must be set to a unique value per frame. This value will be propagated |
| 281 |
into the decoded frame's buffer and can also be used to use this frame |
| 282 |
as the reference of another. If using multiple decode requests per |
| 283 |
frame, then the timestamps of all the ``OUTPUT`` buffers for a given |
| 284 |
frame must be identical. If the timestamp changes, then the currently |
| 285 |
held ``CAPTURE`` buffer will be made available for dequeuing and the |
| 286 |
current request will work on a new ``CAPTURE`` buffer. |
| 287 |
|
| 288 |
2. Set the codec-specific controls for the decoding request, using |
| 289 |
:c:func:`VIDIOC_S_EXT_CTRLS`. |
| 290 |
|
| 291 |
* **Required fields:** |
| 292 |
|
| 293 |
``which`` |
| 294 |
must be ``V4L2_CTRL_WHICH_REQUEST_VAL``. |
| 295 |
|
| 296 |
``request_fd`` |
| 297 |
must be set to the file descriptor of the decoding request. |
| 298 |
|
| 299 |
other fields |
| 300 |
other fields are set as usual when setting controls. The ``controls`` |
| 301 |
array must contain all the codec-specific controls required to decode |
| 302 |
a frame. |
| 303 |
|
| 304 |
.. note:: |
| 305 |
|
| 306 |
It is possible to specify the controls in different invocations of |
| 307 |
:c:func:`VIDIOC_S_EXT_CTRLS`, or to overwrite a previously set control, as |
| 308 |
long as ``request_fd`` and ``which`` are properly set. The controls state |
| 309 |
at the moment of request submission is the one that will be considered. |
| 310 |
|
| 311 |
.. note:: |
| 312 |
|
| 313 |
The order in which steps 1 and 2 take place is interchangeable. |
| 314 |
|
| 315 |
3. Submit the request by invoking :c:func:`MEDIA_REQUEST_IOC_QUEUE` on the |
| 316 |
request FD. |
| 317 |
|
| 318 |
If the request is submitted without an ``OUTPUT`` buffer, or if some of the |
| 319 |
required controls are missing from the request, then |
| 320 |
:c:func:`MEDIA_REQUEST_IOC_QUEUE` will return ``-ENOENT``. If more than one |
| 321 |
``OUTPUT`` buffer is queued, then it will return ``-EINVAL``. |
| 322 |
:c:func:`MEDIA_REQUEST_IOC_QUEUE` returning non-zero means that no |
| 323 |
``CAPTURE`` buffer will be produced for this request. |
| 324 |
|
| 325 |
``CAPTURE`` buffers must not be part of the request, and are queued |
| 326 |
independently. They are returned in decode order (i.e. the same order as coded |
| 327 |
frames were submitted to the ``OUTPUT`` queue). |
| 328 |
|
| 329 |
Runtime decoding errors are signaled by the dequeued ``CAPTURE`` buffers |
| 330 |
carrying the ``V4L2_BUF_FLAG_ERROR`` flag. If a decoded reference frame has an |
| 331 |
error, then all following decoded frames that refer to it also have the |
| 332 |
``V4L2_BUF_FLAG_ERROR`` flag set, although the decoder will still try to |
| 333 |
produce (likely corrupted) frames. |
| 334 |
|
| 335 |
Buffer management while decoding |
| 336 |
================================ |
| 337 |
Contrary to stateful decoders, a stateless decoder does not perform any kind of |
| 338 |
buffer management: it only guarantees that dequeued ``CAPTURE`` buffers can be |
| 339 |
used by the client for as long as they are not queued again. "Used" here |
| 340 |
encompasses using the buffer for compositing or display. |
| 341 |
|
| 342 |
A dequeued capture buffer can also be used as the reference frame of another |
| 343 |
buffer. |
| 344 |
|
| 345 |
A frame is specified as reference by converting its timestamp into nanoseconds, |
| 346 |
and storing it into the relevant member of a codec-dependent control structure. |
| 347 |
The :c:func:`v4l2_timeval_to_ns` function must be used to perform that |
| 348 |
conversion. The timestamp of a frame can be used to reference it as soon as all |
| 349 |
its units of encoded data are successfully submitted to the ``OUTPUT`` queue. |
| 350 |
|
| 351 |
A decoded buffer containing a reference frame must not be reused as a decoding |
| 352 |
target until all the frames referencing it have been decoded. The safest way to |
| 353 |
achieve this is to refrain from queueing a reference buffer until all the |
| 354 |
decoded frames referencing it have been dequeued. However, if the driver can |
| 355 |
guarantee that buffers queued to the ``CAPTURE`` queue are processed in queued |
| 356 |
order, then user-space can take advantage of this guarantee and queue a |
| 357 |
reference buffer when the following conditions are met: |
| 358 |
|
| 359 |
1. All the requests for frames affected by the reference frame have been |
| 360 |
queued, and |
| 361 |
|
| 362 |
2. A sufficient number of ``CAPTURE`` buffers to cover all the decoded |
| 363 |
referencing frames have been queued. |
| 364 |
|
| 365 |
When queuing a decoding request, the driver will increase the reference count of |
| 366 |
all the resources associated with reference frames. This means that the client |
| 367 |
can e.g. close the DMABUF file descriptors of reference frame buffers if it |
| 368 |
won't need them afterwards. |
| 369 |
|
| 370 |
Seeking |
| 371 |
======= |
| 372 |
In order to seek, the client just needs to submit requests using input buffers |
| 373 |
corresponding to the new stream position. It must however be aware that |
| 374 |
resolution may have changed and follow the dynamic resolution change sequence in |
| 375 |
that case. Also depending on the codec used, picture parameters (e.g. SPS/PPS |
| 376 |
for H.264) may have changed and the client is responsible for making sure that a |
| 377 |
valid state is sent to the decoder. |
| 378 |
|
| 379 |
The client is then free to ignore any returned ``CAPTURE`` buffer that comes |
| 380 |
from the pre-seek position. |
| 381 |
|
| 382 |
Pausing |
| 383 |
======= |
| 384 |
|
| 385 |
In order to pause, the client can just cease queuing buffers onto the ``OUTPUT`` |
| 386 |
queue. Without source bytestream data, there is no data to process and the codec |
| 387 |
will remain idle. |
| 388 |
|
| 389 |
Dynamic resolution change |
| 390 |
========================= |
| 391 |
|
| 392 |
If the client detects a resolution change in the stream, it will need to perform |
| 393 |
the initialization sequence again with the new resolution: |
| 394 |
|
| 395 |
1. If the last submitted request resulted in a ``CAPTURE`` buffer being |
| 396 |
held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the |
| 397 |
last frame is not available on the ``CAPTURE`` queue. In this case, a |
| 398 |
``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver |
| 399 |
dequeue the held ``CAPTURE`` buffer. |
| 400 |
|
| 401 |
2. Wait until all submitted requests have completed and dequeue the |
| 402 |
corresponding output buffers. |
| 403 |
|
| 404 |
3. Call :c:func:`VIDIOC_STREAMOFF` on both the ``OUTPUT`` and ``CAPTURE`` |
| 405 |
queues. |
| 406 |
|
| 407 |
4. Free all ``CAPTURE`` buffers by calling :c:func:`VIDIOC_REQBUFS` on the |
| 408 |
``CAPTURE`` queue with a buffer count of zero. |
| 409 |
|
| 410 |
5. Perform the initialization sequence again (minus the allocation of |
| 411 |
``OUTPUT`` buffers), with the new resolution set on the ``OUTPUT`` queue. |
| 412 |
Note that due to resolution constraints, a different format may need to be |
| 413 |
picked on the ``CAPTURE`` queue. |
| 414 |
|
| 415 |
Drain |
| 416 |
===== |
| 417 |
|
| 418 |
If the last submitted request resulted in a ``CAPTURE`` buffer being |
| 419 |
held by the use of the ``V4L2_BUF_FLAG_M2M_HOLD_CAPTURE_BUF`` flag, then the |
| 420 |
last frame is not available on the ``CAPTURE`` queue. In this case, a |
| 421 |
``V4L2_DEC_CMD_FLUSH`` command shall be sent. This will make the driver |
| 422 |
dequeue the held ``CAPTURE`` buffer. |
| 423 |
|
| 424 |
After that, in order to drain the stream on a stateless decoder, the client |
| 425 |
just needs to wait until all the submitted requests are completed. |