| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
.. c:namespace:: V4L |
| 3 |
|
| 4 |
.. _decoder: |
| 5 |
|
| 6 |
************************************************* |
| 7 |
Memory-to-Memory Stateful Video Decoder Interface |
| 8 |
************************************************* |
| 9 |
|
| 10 |
A stateful video decoder takes complete chunks of the bytestream (e.g. Annex-B |
| 11 |
H.264/HEVC stream, raw VP8/9 stream) and decodes them into raw video frames in |
| 12 |
display order. The decoder is expected not to require any additional information |
| 13 |
from the client to process these buffers. |
| 14 |
|
| 15 |
Performing software parsing, processing etc. of the stream in the driver in |
| 16 |
order to support this interface is strongly discouraged. In case such |
| 17 |
operations are needed, use of the Stateless Video Decoder Interface (in |
| 18 |
development) is strongly advised. |
| 19 |
|
| 20 |
Conventions and Notations Used in This Document |
| 21 |
=============================================== |
| 22 |
|
| 23 |
1. The general V4L2 API rules apply if not specified in this document |
| 24 |
otherwise. |
| 25 |
|
| 26 |
2. The meaning of words "must", "may", "should", etc. is as per `RFC |
| 27 |
2119 <https://tools.ietf.org/html/rfc2119>`_. |
| 28 |
|
| 29 |
3. All steps not marked "optional" are required. |
| 30 |
|
| 31 |
4. :c:func:`VIDIOC_G_EXT_CTRLS` and :c:func:`VIDIOC_S_EXT_CTRLS` may be used |
| 32 |
interchangeably with :c:func:`VIDIOC_G_CTRL` and :c:func:`VIDIOC_S_CTRL`, |
| 33 |
unless specified otherwise. |
| 34 |
|
| 35 |
5. Single-planar API (see :ref:`planar-apis`) and applicable structures may be |
| 36 |
used interchangeably with multi-planar API, unless specified otherwise, |
| 37 |
depending on decoder capabilities and following the general V4L2 guidelines. |
| 38 |
|
| 39 |
6. i = [a..b]: sequence of integers from a to b, inclusive, i.e. i = |
| 40 |
[0..2]: i = 0, 1, 2. |
| 41 |
|
| 42 |
7. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE`` |
| 43 |
queue containing data that resulted from processing buffer A. |
| 44 |
|
| 45 |
.. _decoder-glossary: |
| 46 |
|
| 47 |
Glossary |
| 48 |
======== |
| 49 |
|
| 50 |
CAPTURE |
| 51 |
the destination buffer queue; for decoders, the queue of buffers containing |
| 52 |
decoded frames; for encoders, the queue of buffers containing an encoded |
| 53 |
bytestream; ``V4L2_BUF_TYPE_VIDEO_CAPTURE`` or |
| 54 |
``V4L2_BUF_TYPE_VIDEO_CAPTURE_MPLANE``; data is captured from the hardware |
| 55 |
into ``CAPTURE`` buffers. |
| 56 |
|
| 57 |
client |
| 58 |
the application communicating with the decoder or encoder implementing |
| 59 |
this interface. |
| 60 |
|
| 61 |
coded format |
| 62 |
encoded/compressed video bytestream format (e.g. H.264, VP8, etc.); see |
| 63 |
also: raw format. |
| 64 |
|
| 65 |
coded height |
| 66 |
height for given coded resolution. |
| 67 |
|
| 68 |
coded resolution |
| 69 |
stream resolution in pixels aligned to codec and hardware requirements; |
| 70 |
typically visible resolution rounded up to full macroblocks; |
| 71 |
see also: visible resolution. |
| 72 |
|
| 73 |
coded width |
| 74 |
width for given coded resolution. |
| 75 |
|
| 76 |
coding tree unit |
| 77 |
processing unit of the HEVC codec (corresponds to macroblock units in |
| 78 |
H.264, VP8, VP9), |
| 79 |
can use block structures of up to 64×64 pixels. |
| 80 |
Good at sub-partitioning the picture into variable sized structures. |
| 81 |
|
| 82 |
decode order |
| 83 |
the order in which frames are decoded; may differ from display order if the |
| 84 |
coded format includes a feature of frame reordering; for decoders, |
| 85 |
``OUTPUT`` buffers must be queued by the client in decode order; for |
| 86 |
encoders ``CAPTURE`` buffers must be returned by the encoder in decode order. |
| 87 |
|
| 88 |
destination |
| 89 |
data resulting from the decode process; see ``CAPTURE``. |
| 90 |
|
| 91 |
display order |
| 92 |
the order in which frames must be displayed; for encoders, ``OUTPUT`` |
| 93 |
buffers must be queued by the client in display order; for decoders, |
| 94 |
``CAPTURE`` buffers must be returned by the decoder in display order. |
| 95 |
|
| 96 |
DPB |
| 97 |
Decoded Picture Buffer; an H.264/HEVC term for a buffer that stores a decoded |
| 98 |
raw frame available for reference in further decoding steps. |
| 99 |
|
| 100 |
EOS |
| 101 |
end of stream. |
| 102 |
|
| 103 |
IDR |
| 104 |
Instantaneous Decoder Refresh; a type of a keyframe in an H.264/HEVC-encoded |
| 105 |
stream, which clears the list of earlier reference frames (DPBs). |
| 106 |
|
| 107 |
keyframe |
| 108 |
an encoded frame that does not reference frames decoded earlier, i.e. |
| 109 |
can be decoded fully on its own. |
| 110 |
|
| 111 |
macroblock |
| 112 |
a processing unit in image and video compression formats based on linear |
| 113 |
block transforms (e.g. H.264, VP8, VP9); codec-specific, but for most of |
| 114 |
popular codecs the size is 16x16 samples (pixels). The HEVC codec uses a |
| 115 |
slightly more flexible processing unit called coding tree unit (CTU). |
| 116 |
|
| 117 |
OUTPUT |
| 118 |
the source buffer queue; for decoders, the queue of buffers containing |
| 119 |
an encoded bytestream; for encoders, the queue of buffers containing raw |
| 120 |
frames; ``V4L2_BUF_TYPE_VIDEO_OUTPUT`` or |
| 121 |
``V4L2_BUF_TYPE_VIDEO_OUTPUT_MPLANE``; the hardware is fed with data |
| 122 |
from ``OUTPUT`` buffers. |
| 123 |
|
| 124 |
PPS |
| 125 |
Picture Parameter Set; a type of metadata entity in an H.264/HEVC bytestream. |
| 126 |
|
| 127 |
raw format |
| 128 |
uncompressed format containing raw pixel data (e.g. YUV, RGB formats). |
| 129 |
|
| 130 |
resume point |
| 131 |
a point in the bytestream from which decoding may start/continue, without |
| 132 |
any previous state/data present, e.g.: a keyframe (VP8/VP9) or |
| 133 |
SPS/PPS/IDR sequence (H.264/HEVC); a resume point is required to start decode |
| 134 |
of a new stream, or to resume decoding after a seek. |
| 135 |
|
| 136 |
source |
| 137 |
data fed to the decoder or encoder; see ``OUTPUT``. |
| 138 |
|
| 139 |
source height |
| 140 |
height in pixels for given source resolution; relevant to encoders only. |
| 141 |
|
| 142 |
source resolution |
| 143 |
resolution in pixels of source frames being source to the encoder and |
| 144 |
subject to further cropping to the bounds of visible resolution; relevant to |
| 145 |
encoders only. |
| 146 |
|
| 147 |
source width |
| 148 |
width in pixels for given source resolution; relevant to encoders only. |
| 149 |
|
| 150 |
SPS |
| 151 |
Sequence Parameter Set; a type of metadata entity in an H.264/HEVC bytestream. |
| 152 |
|
| 153 |
stream metadata |
| 154 |
additional (non-visual) information contained inside encoded bytestream; |
| 155 |
for example: coded resolution, visible resolution, codec profile. |
| 156 |
|
| 157 |
visible height |
| 158 |
height for given visible resolution; display height. |
| 159 |
|
| 160 |
visible resolution |
| 161 |
stream resolution of the visible picture, in pixels, to be used for |
| 162 |
display purposes; must be smaller or equal to coded resolution; |
| 163 |
display resolution. |
| 164 |
|
| 165 |
visible width |
| 166 |
width for given visible resolution; display width. |
| 167 |
|
| 168 |
State Machine |
| 169 |
============= |
| 170 |
|
| 171 |
.. kernel-render:: DOT |
| 172 |
:alt: DOT digraph of decoder state machine |
| 173 |
:caption: Decoder State Machine |
| 174 |
|
| 175 |
digraph decoder_state_machine { |
| 176 |
node [shape = doublecircle, label="Decoding"] Decoding; |
| 177 |
|
| 178 |
node [shape = circle, label="Initialization"] Initialization; |
| 179 |
node [shape = circle, label="Capture\nsetup"] CaptureSetup; |
| 180 |
node [shape = circle, label="Dynamic\nResolution\nChange"] ResChange; |
| 181 |
node [shape = circle, label="Stopped"] Stopped; |
| 182 |
node [shape = circle, label="Drain"] Drain; |
| 183 |
node [shape = circle, label="Seek"] Seek; |
| 184 |
node [shape = circle, label="End of Stream"] EoS; |
| 185 |
|
| 186 |
node [shape = point]; qi |
| 187 |
qi -> Initialization [ label = "open()" ]; |
| 188 |
|
| 189 |
Initialization -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ]; |
| 190 |
|
| 191 |
CaptureSetup -> Stopped [ label = "CAPTURE\nbuffers\nready" ]; |
| 192 |
|
| 193 |
Decoding -> ResChange [ label = "Stream\nresolution\nchange" ]; |
| 194 |
Decoding -> Drain [ label = "V4L2_DEC_CMD_STOP" ]; |
| 195 |
Decoding -> EoS [ label = "EoS mark\nin the stream" ]; |
| 196 |
Decoding -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; |
| 197 |
Decoding -> Stopped [ label = "VIDIOC_STREAMOFF(CAPTURE)" ]; |
| 198 |
Decoding -> Decoding; |
| 199 |
|
| 200 |
ResChange -> CaptureSetup [ label = "CAPTURE\nformat\nestablished" ]; |
| 201 |
ResChange -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; |
| 202 |
|
| 203 |
EoS -> Drain [ label = "Implicit\ndrain" ]; |
| 204 |
|
| 205 |
Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(CAPTURE)" ]; |
| 206 |
Drain -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; |
| 207 |
|
| 208 |
Seek -> Decoding [ label = "VIDIOC_STREAMON(OUTPUT)" ]; |
| 209 |
Seek -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ]; |
| 210 |
|
| 211 |
Stopped -> Decoding [ label = "V4L2_DEC_CMD_START\nor\nVIDIOC_STREAMON(CAPTURE)" ]; |
| 212 |
Stopped -> Seek [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; |
| 213 |
} |
| 214 |
|
| 215 |
Querying Capabilities |
| 216 |
===================== |
| 217 |
|
| 218 |
1. To enumerate the set of coded formats supported by the decoder, the |
| 219 |
client may call :c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``. |
| 220 |
|
| 221 |
* The full set of supported formats will be returned, regardless of the |
| 222 |
format set on ``CAPTURE``. |
| 223 |
* Check the flags field of :c:type:`v4l2_fmtdesc` for more information |
| 224 |
about the decoder's capabilities with respect to each coded format. |
| 225 |
In particular whether or not the decoder has a full-fledged bytestream |
| 226 |
parser and if the decoder supports dynamic resolution changes. |
| 227 |
|
| 228 |
2. To enumerate the set of supported raw formats, the client may call |
| 229 |
:c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``. |
| 230 |
|
| 231 |
* Only the formats supported for the format currently active on ``OUTPUT`` |
| 232 |
will be returned. |
| 233 |
|
| 234 |
* In order to enumerate raw formats supported by a given coded format, |
| 235 |
the client must first set that coded format on ``OUTPUT`` and then |
| 236 |
enumerate formats on ``CAPTURE``. |
| 237 |
|
| 238 |
3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported |
| 239 |
resolutions for a given format, passing desired pixel format in |
| 240 |
:c:type:`v4l2_frmsizeenum` ``pixel_format``. |
| 241 |
|
| 242 |
* Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a coded pixel |
| 243 |
format will include all possible coded resolutions supported by the |
| 244 |
decoder for given coded pixel format. |
| 245 |
|
| 246 |
* Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a raw pixel format |
| 247 |
will include all possible frame buffer resolutions supported by the |
| 248 |
decoder for given raw pixel format and the coded format currently set on |
| 249 |
``OUTPUT``. |
| 250 |
|
| 251 |
4. Supported profiles and levels for the coded format currently set on |
| 252 |
``OUTPUT``, if applicable, may be queried using their respective controls |
| 253 |
via :c:func:`VIDIOC_QUERYCTRL`. |
| 254 |
|
| 255 |
Initialization |
| 256 |
============== |
| 257 |
|
| 258 |
1. Set the coded format on ``OUTPUT`` via :c:func:`VIDIOC_S_FMT`. |
| 259 |
|
| 260 |
* **Required fields:** |
| 261 |
|
| 262 |
``type`` |
| 263 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 264 |
|
| 265 |
``pixelformat`` |
| 266 |
a coded pixel format. |
| 267 |
|
| 268 |
``width``, ``height`` |
| 269 |
coded resolution of the stream; required only if it cannot be parsed |
| 270 |
from the stream for the given coded format; otherwise the decoder will |
| 271 |
use this resolution as a placeholder resolution that will likely change |
| 272 |
as soon as it can parse the actual coded resolution from the stream. |
| 273 |
|
| 274 |
``sizeimage`` |
| 275 |
desired size of ``OUTPUT`` buffers; the decoder may adjust it to |
| 276 |
match hardware requirements. |
| 277 |
|
| 278 |
other fields |
| 279 |
follow standard semantics. |
| 280 |
|
| 281 |
* **Returned fields:** |
| 282 |
|
| 283 |
``sizeimage`` |
| 284 |
adjusted size of ``OUTPUT`` buffers. |
| 285 |
|
| 286 |
* The ``CAPTURE`` format will be updated with an appropriate frame buffer |
| 287 |
resolution instantly based on the width and height returned by |
| 288 |
:c:func:`VIDIOC_S_FMT`. |
| 289 |
However, for coded formats that include stream resolution information, |
| 290 |
after the decoder is done parsing the information from the stream, it will |
| 291 |
update the ``CAPTURE`` format with new values and signal a source change |
| 292 |
event, regardless of whether they match the values set by the client or |
| 293 |
not. |
| 294 |
|
| 295 |
.. important:: |
| 296 |
|
| 297 |
Changing the ``OUTPUT`` format may change the currently set ``CAPTURE`` |
| 298 |
format. How the new ``CAPTURE`` format is determined is up to the decoder |
| 299 |
and the client must ensure it matches its needs afterwards. |
| 300 |
|
| 301 |
2. Allocate source (bytestream) buffers via :c:func:`VIDIOC_REQBUFS` on |
| 302 |
``OUTPUT``. |
| 303 |
|
| 304 |
* **Required fields:** |
| 305 |
|
| 306 |
``count`` |
| 307 |
requested number of buffers to allocate; greater than zero. |
| 308 |
|
| 309 |
``type`` |
| 310 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 311 |
|
| 312 |
``memory`` |
| 313 |
follows standard semantics. |
| 314 |
|
| 315 |
* **Returned fields:** |
| 316 |
|
| 317 |
``count`` |
| 318 |
the actual number of buffers allocated. |
| 319 |
|
| 320 |
.. warning:: |
| 321 |
|
| 322 |
The actual number of allocated buffers may differ from the ``count`` |
| 323 |
given. The client must check the updated value of ``count`` after the |
| 324 |
call returns. |
| 325 |
|
| 326 |
Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``OUTPUT`` queue can be |
| 327 |
used to have more control over buffer allocation. |
| 328 |
|
| 329 |
* **Required fields:** |
| 330 |
|
| 331 |
``count`` |
| 332 |
requested number of buffers to allocate; greater than zero. |
| 333 |
|
| 334 |
``type`` |
| 335 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 336 |
|
| 337 |
``memory`` |
| 338 |
follows standard semantics. |
| 339 |
|
| 340 |
``format`` |
| 341 |
follows standard semantics. |
| 342 |
|
| 343 |
* **Returned fields:** |
| 344 |
|
| 345 |
``count`` |
| 346 |
adjusted to the number of allocated buffers. |
| 347 |
|
| 348 |
.. warning:: |
| 349 |
|
| 350 |
The actual number of allocated buffers may differ from the ``count`` |
| 351 |
given. The client must check the updated value of ``count`` after the |
| 352 |
call returns. |
| 353 |
|
| 354 |
3. Start streaming on the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`. |
| 355 |
|
| 356 |
4. **This step only applies to coded formats that contain resolution information |
| 357 |
in the stream.** Continue queuing/dequeuing bytestream buffers to/from the |
| 358 |
``OUTPUT`` queue via :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`. The |
| 359 |
buffers will be processed and returned to the client in order, until |
| 360 |
required metadata to configure the ``CAPTURE`` queue are found. This is |
| 361 |
indicated by the decoder sending a ``V4L2_EVENT_SOURCE_CHANGE`` event with |
| 362 |
``changes`` set to ``V4L2_EVENT_SRC_CH_RESOLUTION``. |
| 363 |
|
| 364 |
* It is not an error if the first buffer does not contain enough data for |
| 365 |
this to occur. Processing of the buffers will continue as long as more |
| 366 |
data is needed. |
| 367 |
|
| 368 |
* If data in a buffer that triggers the event is required to decode the |
| 369 |
first frame, it will not be returned to the client, until the |
| 370 |
initialization sequence completes and the frame is decoded. |
| 371 |
|
| 372 |
* If the client has not set the coded resolution of the stream on its own, |
| 373 |
calling :c:func:`VIDIOC_G_FMT`, :c:func:`VIDIOC_S_FMT`, |
| 374 |
:c:func:`VIDIOC_TRY_FMT` or :c:func:`VIDIOC_REQBUFS` on the ``CAPTURE`` |
| 375 |
queue will not return the real values for the stream until a |
| 376 |
``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to |
| 377 |
``V4L2_EVENT_SRC_CH_RESOLUTION`` is signaled. |
| 378 |
|
| 379 |
.. important:: |
| 380 |
|
| 381 |
Any client query issued after the decoder queues the event will return |
| 382 |
values applying to the just parsed stream, including queue formats, |
| 383 |
selection rectangles and controls. |
| 384 |
|
| 385 |
.. note:: |
| 386 |
|
| 387 |
A client capable of acquiring stream parameters from the bytestream on |
| 388 |
its own may attempt to set the width and height of the ``OUTPUT`` format |
| 389 |
to non-zero values matching the coded size of the stream, skip this step |
| 390 |
and continue with the `Capture Setup` sequence. However, it must not |
| 391 |
rely on any driver queries regarding stream parameters, such as |
| 392 |
selection rectangles and controls, since the decoder has not parsed them |
| 393 |
from the stream yet. If the values configured by the client do not match |
| 394 |
those parsed by the decoder, a `Dynamic Resolution Change` will be |
| 395 |
triggered to reconfigure them. |
| 396 |
|
| 397 |
.. note:: |
| 398 |
|
| 399 |
No decoded frames are produced during this phase. |
| 400 |
|
| 401 |
5. Continue with the `Capture Setup` sequence. |
| 402 |
|
| 403 |
Capture Setup |
| 404 |
============= |
| 405 |
|
| 406 |
1. Call :c:func:`VIDIOC_G_FMT` on the ``CAPTURE`` queue to get format for the |
| 407 |
destination buffers parsed/decoded from the bytestream. |
| 408 |
|
| 409 |
* **Required fields:** |
| 410 |
|
| 411 |
``type`` |
| 412 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 413 |
|
| 414 |
* **Returned fields:** |
| 415 |
|
| 416 |
``width``, ``height`` |
| 417 |
frame buffer resolution for the decoded frames. |
| 418 |
|
| 419 |
``pixelformat`` |
| 420 |
pixel format for decoded frames. |
| 421 |
|
| 422 |
``num_planes`` (for _MPLANE ``type`` only) |
| 423 |
number of planes for pixelformat. |
| 424 |
|
| 425 |
``sizeimage``, ``bytesperline`` |
| 426 |
as per standard semantics; matching frame buffer format. |
| 427 |
|
| 428 |
.. note:: |
| 429 |
|
| 430 |
The value of ``pixelformat`` may be any pixel format supported by the |
| 431 |
decoder for the current stream. The decoder should choose a |
| 432 |
preferred/optimal format for the default configuration. For example, a |
| 433 |
YUV format may be preferred over an RGB format if an additional |
| 434 |
conversion step would be required for the latter. |
| 435 |
|
| 436 |
2. **Optional.** Acquire the visible resolution via |
| 437 |
:c:func:`VIDIOC_G_SELECTION`. |
| 438 |
|
| 439 |
* **Required fields:** |
| 440 |
|
| 441 |
``type`` |
| 442 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 443 |
|
| 444 |
``target`` |
| 445 |
set to ``V4L2_SEL_TGT_COMPOSE``. |
| 446 |
|
| 447 |
* **Returned fields:** |
| 448 |
|
| 449 |
``r.left``, ``r.top``, ``r.width``, ``r.height`` |
| 450 |
the visible rectangle; it must fit within the frame buffer resolution |
| 451 |
returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``. |
| 452 |
|
| 453 |
* The following selection targets are supported on ``CAPTURE``: |
| 454 |
|
| 455 |
``V4L2_SEL_TGT_CROP_BOUNDS`` |
| 456 |
corresponds to the coded resolution of the stream. |
| 457 |
|
| 458 |
``V4L2_SEL_TGT_CROP_DEFAULT`` |
| 459 |
the rectangle covering the part of the ``CAPTURE`` buffer that |
| 460 |
contains meaningful picture data (visible area); width and height |
| 461 |
will be equal to the visible resolution of the stream. |
| 462 |
|
| 463 |
``V4L2_SEL_TGT_CROP`` |
| 464 |
the rectangle within the coded resolution to be output to |
| 465 |
``CAPTURE``; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``; read-only on |
| 466 |
hardware without additional compose/scaling capabilities. |
| 467 |
|
| 468 |
``V4L2_SEL_TGT_COMPOSE_BOUNDS`` |
| 469 |
the maximum rectangle within a ``CAPTURE`` buffer, which the cropped |
| 470 |
frame can be composed into; equal to ``V4L2_SEL_TGT_CROP`` if the |
| 471 |
hardware does not support compose/scaling. |
| 472 |
|
| 473 |
``V4L2_SEL_TGT_COMPOSE_DEFAULT`` |
| 474 |
equal to ``V4L2_SEL_TGT_CROP``. |
| 475 |
|
| 476 |
``V4L2_SEL_TGT_COMPOSE`` |
| 477 |
the rectangle inside a ``CAPTURE`` buffer into which the cropped |
| 478 |
frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``; |
| 479 |
read-only on hardware without additional compose/scaling capabilities. |
| 480 |
|
| 481 |
``V4L2_SEL_TGT_COMPOSE_PADDED`` |
| 482 |
the rectangle inside a ``CAPTURE`` buffer which is overwritten by the |
| 483 |
hardware; equal to ``V4L2_SEL_TGT_COMPOSE`` if the hardware does not |
| 484 |
write padding pixels. |
| 485 |
|
| 486 |
.. warning:: |
| 487 |
|
| 488 |
The values are guaranteed to be meaningful only after the decoder |
| 489 |
successfully parses the stream metadata. The client must not rely on the |
| 490 |
query before that happens. |
| 491 |
|
| 492 |
3. **Optional.** Enumerate ``CAPTURE`` formats via :c:func:`VIDIOC_ENUM_FMT` on |
| 493 |
the ``CAPTURE`` queue. Once the stream information is parsed and known, the |
| 494 |
client may use this ioctl to discover which raw formats are supported for |
| 495 |
given stream and select one of them via :c:func:`VIDIOC_S_FMT`. |
| 496 |
|
| 497 |
.. important:: |
| 498 |
|
| 499 |
The decoder will return only formats supported for the currently |
| 500 |
established coded format, as per the ``OUTPUT`` format and/or stream |
| 501 |
metadata parsed in this initialization sequence, even if more formats |
| 502 |
may be supported by the decoder in general. In other words, the set |
| 503 |
returned will be a subset of the initial query mentioned in the |
| 504 |
`Querying Capabilities` section. |
| 505 |
|
| 506 |
For example, a decoder may support YUV and RGB formats for resolutions |
| 507 |
1920x1088 and lower, but only YUV for higher resolutions (due to |
| 508 |
hardware limitations). After parsing a resolution of 1920x1088 or lower, |
| 509 |
:c:func:`VIDIOC_ENUM_FMT` may return a set of YUV and RGB pixel formats, |
| 510 |
but after parsing resolution higher than 1920x1088, the decoder will not |
| 511 |
return RGB, unsupported for this resolution. |
| 512 |
|
| 513 |
However, subsequent resolution change event triggered after |
| 514 |
discovering a resolution change within the same stream may switch |
| 515 |
the stream into a lower resolution and :c:func:`VIDIOC_ENUM_FMT` |
| 516 |
would return RGB formats again in that case. |
| 517 |
|
| 518 |
4. **Optional.** Set the ``CAPTURE`` format via :c:func:`VIDIOC_S_FMT` on the |
| 519 |
``CAPTURE`` queue. The client may choose a different format than |
| 520 |
selected/suggested by the decoder in :c:func:`VIDIOC_G_FMT`. |
| 521 |
|
| 522 |
* **Required fields:** |
| 523 |
|
| 524 |
``type`` |
| 525 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 526 |
|
| 527 |
``pixelformat`` |
| 528 |
a raw pixel format. |
| 529 |
|
| 530 |
``width``, ``height`` |
| 531 |
frame buffer resolution of the decoded stream; typically unchanged from |
| 532 |
what was returned with :c:func:`VIDIOC_G_FMT`, but it may be different |
| 533 |
if the hardware supports composition and/or scaling. |
| 534 |
|
| 535 |
* Setting the ``CAPTURE`` format will reset the compose selection rectangles |
| 536 |
to their default values, based on the new resolution, as described in the |
| 537 |
previous step. |
| 538 |
|
| 539 |
5. **Optional.** Set the compose rectangle via :c:func:`VIDIOC_S_SELECTION` on |
| 540 |
the ``CAPTURE`` queue if it is desired and if the decoder has compose and/or |
| 541 |
scaling capabilities. |
| 542 |
|
| 543 |
* **Required fields:** |
| 544 |
|
| 545 |
``type`` |
| 546 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 547 |
|
| 548 |
``target`` |
| 549 |
set to ``V4L2_SEL_TGT_COMPOSE``. |
| 550 |
|
| 551 |
``r.left``, ``r.top``, ``r.width``, ``r.height`` |
| 552 |
the rectangle inside a ``CAPTURE`` buffer into which the cropped |
| 553 |
frame is written; defaults to ``V4L2_SEL_TGT_COMPOSE_DEFAULT``; |
| 554 |
read-only on hardware without additional compose/scaling capabilities. |
| 555 |
|
| 556 |
* **Returned fields:** |
| 557 |
|
| 558 |
``r.left``, ``r.top``, ``r.width``, ``r.height`` |
| 559 |
the visible rectangle; it must fit within the frame buffer resolution |
| 560 |
returned by :c:func:`VIDIOC_G_FMT` on ``CAPTURE``. |
| 561 |
|
| 562 |
.. warning:: |
| 563 |
|
| 564 |
The decoder may adjust the compose rectangle to the nearest |
| 565 |
supported one to meet codec and hardware requirements. The client needs |
| 566 |
to check the adjusted rectangle returned by :c:func:`VIDIOC_S_SELECTION`. |
| 567 |
|
| 568 |
6. If all the following conditions are met, the client may resume the decoding |
| 569 |
instantly: |
| 570 |
|
| 571 |
* ``sizeimage`` of the new format (determined in previous steps) is less |
| 572 |
than or equal to the size of currently allocated buffers, |
| 573 |
|
| 574 |
* the number of buffers currently allocated is greater than or equal to the |
| 575 |
minimum number of buffers acquired in previous steps. To fulfill this |
| 576 |
requirement, the client may use :c:func:`VIDIOC_CREATE_BUFS` to add new |
| 577 |
buffers. |
| 578 |
|
| 579 |
In that case, the remaining steps do not apply and the client may resume |
| 580 |
the decoding by one of the following actions: |
| 581 |
|
| 582 |
* if the ``CAPTURE`` queue is streaming, call :c:func:`VIDIOC_DECODER_CMD` |
| 583 |
with the ``V4L2_DEC_CMD_START`` command, |
| 584 |
|
| 585 |
* if the ``CAPTURE`` queue is not streaming, call :c:func:`VIDIOC_STREAMON` |
| 586 |
on the ``CAPTURE`` queue. |
| 587 |
|
| 588 |
However, if the client intends to change the buffer set, to lower |
| 589 |
memory usage or for any other reasons, it may be achieved by following |
| 590 |
the steps below. |
| 591 |
|
| 592 |
7. **If the** ``CAPTURE`` **queue is streaming,** keep queuing and dequeuing |
| 593 |
buffers on the ``CAPTURE`` queue until a buffer marked with the |
| 594 |
``V4L2_BUF_FLAG_LAST`` flag is dequeued. |
| 595 |
|
| 596 |
8. **If the** ``CAPTURE`` **queue is streaming,** call :c:func:`VIDIOC_STREAMOFF` |
| 597 |
on the ``CAPTURE`` queue to stop streaming. |
| 598 |
|
| 599 |
.. warning:: |
| 600 |
|
| 601 |
The ``OUTPUT`` queue must remain streaming. Calling |
| 602 |
:c:func:`VIDIOC_STREAMOFF` on it would abort the sequence and trigger a |
| 603 |
seek. |
| 604 |
|
| 605 |
9. **If the** ``CAPTURE`` **queue has buffers allocated,** free the ``CAPTURE`` |
| 606 |
buffers using :c:func:`VIDIOC_REQBUFS`. |
| 607 |
|
| 608 |
* **Required fields:** |
| 609 |
|
| 610 |
``count`` |
| 611 |
set to 0. |
| 612 |
|
| 613 |
``type`` |
| 614 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 615 |
|
| 616 |
``memory`` |
| 617 |
follows standard semantics. |
| 618 |
|
| 619 |
10. Allocate ``CAPTURE`` buffers via :c:func:`VIDIOC_REQBUFS` on the |
| 620 |
``CAPTURE`` queue. |
| 621 |
|
| 622 |
* **Required fields:** |
| 623 |
|
| 624 |
``count`` |
| 625 |
requested number of buffers to allocate; greater than zero. |
| 626 |
|
| 627 |
``type`` |
| 628 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 629 |
|
| 630 |
``memory`` |
| 631 |
follows standard semantics. |
| 632 |
|
| 633 |
* **Returned fields:** |
| 634 |
|
| 635 |
``count`` |
| 636 |
actual number of buffers allocated. |
| 637 |
|
| 638 |
.. warning:: |
| 639 |
|
| 640 |
The actual number of allocated buffers may differ from the ``count`` |
| 641 |
given. The client must check the updated value of ``count`` after the |
| 642 |
call returns. |
| 643 |
|
| 644 |
.. note:: |
| 645 |
|
| 646 |
To allocate more than the minimum number of buffers (for pipeline |
| 647 |
depth), the client may query the ``V4L2_CID_MIN_BUFFERS_FOR_CAPTURE`` |
| 648 |
control to get the minimum number of buffers required, and pass the |
| 649 |
obtained value plus the number of additional buffers needed in the |
| 650 |
``count`` field to :c:func:`VIDIOC_REQBUFS`. |
| 651 |
|
| 652 |
Alternatively, :c:func:`VIDIOC_CREATE_BUFS` on the ``CAPTURE`` queue can be |
| 653 |
used to have more control over buffer allocation. For example, by |
| 654 |
allocating buffers larger than the current ``CAPTURE`` format, future |
| 655 |
resolution changes can be accommodated. |
| 656 |
|
| 657 |
* **Required fields:** |
| 658 |
|
| 659 |
``count`` |
| 660 |
requested number of buffers to allocate; greater than zero. |
| 661 |
|
| 662 |
``type`` |
| 663 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 664 |
|
| 665 |
``memory`` |
| 666 |
follows standard semantics. |
| 667 |
|
| 668 |
``format`` |
| 669 |
a format representing the maximum framebuffer resolution to be |
| 670 |
accommodated by newly allocated buffers. |
| 671 |
|
| 672 |
* **Returned fields:** |
| 673 |
|
| 674 |
``count`` |
| 675 |
adjusted to the number of allocated buffers. |
| 676 |
|
| 677 |
.. warning:: |
| 678 |
|
| 679 |
The actual number of allocated buffers may differ from the ``count`` |
| 680 |
given. The client must check the updated value of ``count`` after the |
| 681 |
call returns. |
| 682 |
|
| 683 |
.. note:: |
| 684 |
|
| 685 |
To allocate buffers for a format different than parsed from the stream |
| 686 |
metadata, the client must proceed as follows, before the metadata |
| 687 |
parsing is initiated: |
| 688 |
|
| 689 |
* set width and height of the ``OUTPUT`` format to desired coded resolution to |
| 690 |
let the decoder configure the ``CAPTURE`` format appropriately, |
| 691 |
|
| 692 |
* query the ``CAPTURE`` format using :c:func:`VIDIOC_G_FMT` and save it |
| 693 |
until this step. |
| 694 |
|
| 695 |
The format obtained in the query may be then used with |
| 696 |
:c:func:`VIDIOC_CREATE_BUFS` in this step to allocate the buffers. |
| 697 |
|
| 698 |
11. Call :c:func:`VIDIOC_STREAMON` on the ``CAPTURE`` queue to start decoding |
| 699 |
frames. |
| 700 |
|
| 701 |
Decoding |
| 702 |
======== |
| 703 |
|
| 704 |
This state is reached after the `Capture Setup` sequence finishes successfully. |
| 705 |
In this state, the client queues and dequeues buffers to both queues via |
| 706 |
:c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`, following the standard |
| 707 |
semantics. |
| 708 |
|
| 709 |
The content of the source ``OUTPUT`` buffers depends on the active coded pixel |
| 710 |
format and may be affected by codec-specific extended controls, as stated in |
| 711 |
the documentation of each format. |
| 712 |
|
| 713 |
Both queues operate independently, following the standard behavior of V4L2 |
| 714 |
buffer queues and memory-to-memory devices. In addition, the order of decoded |
| 715 |
frames dequeued from the ``CAPTURE`` queue may differ from the order of queuing |
| 716 |
coded frames to the ``OUTPUT`` queue, due to properties of the selected coded |
| 717 |
format, e.g. frame reordering. |
| 718 |
|
| 719 |
The client must not assume any direct relationship between ``CAPTURE`` |
| 720 |
and ``OUTPUT`` buffers and any specific timing of buffers becoming |
| 721 |
available to dequeue. Specifically: |
| 722 |
|
| 723 |
* a buffer queued to ``OUTPUT`` may result in no buffers being produced |
| 724 |
on ``CAPTURE`` (e.g. if it does not contain encoded data, or if only |
| 725 |
metadata syntax structures are present in it), |
| 726 |
|
| 727 |
* a buffer queued to ``OUTPUT`` may result in more than one buffer produced |
| 728 |
on ``CAPTURE`` (if the encoded data contained more than one frame, or if |
| 729 |
returning a decoded frame allowed the decoder to return a frame that |
| 730 |
preceded it in decode, but succeeded it in the display order), |
| 731 |
|
| 732 |
* a buffer queued to ``OUTPUT`` may result in a buffer being produced on |
| 733 |
``CAPTURE`` later into decode process, and/or after processing further |
| 734 |
``OUTPUT`` buffers, or be returned out of order, e.g. if display |
| 735 |
reordering is used, |
| 736 |
|
| 737 |
* buffers may become available on the ``CAPTURE`` queue without additional |
| 738 |
buffers queued to ``OUTPUT`` (e.g. during drain or ``EOS``), because of the |
| 739 |
``OUTPUT`` buffers queued in the past whose decoding results are only |
| 740 |
available at later time, due to specifics of the decoding process. |
| 741 |
|
| 742 |
.. note:: |
| 743 |
|
| 744 |
To allow matching decoded ``CAPTURE`` buffers with ``OUTPUT`` buffers they |
| 745 |
originated from, the client can set the ``timestamp`` field of the |
| 746 |
:c:type:`v4l2_buffer` struct when queuing an ``OUTPUT`` buffer. The |
| 747 |
``CAPTURE`` buffer(s), which resulted from decoding that ``OUTPUT`` buffer |
| 748 |
will have their ``timestamp`` field set to the same value when dequeued. |
| 749 |
|
| 750 |
In addition to the straightforward case of one ``OUTPUT`` buffer producing |
| 751 |
one ``CAPTURE`` buffer, the following cases are defined: |
| 752 |
|
| 753 |
* one ``OUTPUT`` buffer generates multiple ``CAPTURE`` buffers: the same |
| 754 |
``OUTPUT`` timestamp will be copied to multiple ``CAPTURE`` buffers. |
| 755 |
|
| 756 |
* multiple ``OUTPUT`` buffers generate one ``CAPTURE`` buffer: timestamp of |
| 757 |
the ``OUTPUT`` buffer queued first will be copied. |
| 758 |
|
| 759 |
* the decoding order differs from the display order (i.e. the ``CAPTURE`` |
| 760 |
buffers are out-of-order compared to the ``OUTPUT`` buffers): ``CAPTURE`` |
| 761 |
timestamps will not retain the order of ``OUTPUT`` timestamps. |
| 762 |
|
| 763 |
.. note:: |
| 764 |
|
| 765 |
The backing memory of ``CAPTURE`` buffers that are used as reference frames |
| 766 |
by the stream may be read by the hardware even after they are dequeued. |
| 767 |
Consequently, the client should avoid writing into this memory while the |
| 768 |
``CAPTURE`` queue is streaming. Failure to observe this may result in |
| 769 |
corruption of decoded frames. |
| 770 |
|
| 771 |
Similarly, when using a memory type other than ``V4L2_MEMORY_MMAP``, the |
| 772 |
client should make sure that each ``CAPTURE`` buffer is always queued with |
| 773 |
the same backing memory for as long as the ``CAPTURE`` queue is streaming. |
| 774 |
The reason for this is that V4L2 buffer indices can be used by drivers to |
| 775 |
identify frames. Thus, if the backing memory of a reference frame is |
| 776 |
submitted under a different buffer ID, the driver may misidentify it and |
| 777 |
decode a new frame into it while it is still in use, resulting in corruption |
| 778 |
of the following frames. |
| 779 |
|
| 780 |
During the decoding, the decoder may initiate one of the special sequences, as |
| 781 |
listed below. The sequences will result in the decoder returning all the |
| 782 |
``CAPTURE`` buffers that originated from all the ``OUTPUT`` buffers processed |
| 783 |
before the sequence started. Last of the buffers will have the |
| 784 |
``V4L2_BUF_FLAG_LAST`` flag set. To determine the sequence to follow, the client |
| 785 |
must check if there is any pending event and: |
| 786 |
|
| 787 |
* if a ``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to |
| 788 |
``V4L2_EVENT_SRC_CH_RESOLUTION`` is pending, the `Dynamic Resolution |
| 789 |
Change` sequence needs to be followed, |
| 790 |
|
| 791 |
* if a ``V4L2_EVENT_EOS`` event is pending, the `End of Stream` sequence needs |
| 792 |
to be followed. |
| 793 |
|
| 794 |
Some of the sequences can be intermixed with each other and need to be handled |
| 795 |
as they happen. The exact operation is documented for each sequence. |
| 796 |
|
| 797 |
Should a decoding error occur, it will be reported to the client with the level |
| 798 |
of details depending on the decoder capabilities. Specifically: |
| 799 |
|
| 800 |
* the CAPTURE buffer that contains the results of the failed decode operation |
| 801 |
will be returned with the V4L2_BUF_FLAG_ERROR flag set, |
| 802 |
|
| 803 |
* if the decoder is able to precisely report the OUTPUT buffer that triggered |
| 804 |
the error, such buffer will be returned with the V4L2_BUF_FLAG_ERROR flag |
| 805 |
set. |
| 806 |
|
| 807 |
In case of a fatal failure that does not allow the decoding to continue, any |
| 808 |
further operations on corresponding decoder file handle will return the -EIO |
| 809 |
error code. The client may close the file handle and open a new one, or |
| 810 |
alternatively reinitialize the instance by stopping streaming on both queues, |
| 811 |
releasing all buffers and performing the Initialization sequence again. |
| 812 |
|
| 813 |
Seek |
| 814 |
==== |
| 815 |
|
| 816 |
Seek is controlled by the ``OUTPUT`` queue, as it is the source of coded data. |
| 817 |
The seek does not require any specific operation on the ``CAPTURE`` queue, but |
| 818 |
it may be affected as per normal decoder operation. |
| 819 |
|
| 820 |
1. Stop the ``OUTPUT`` queue to begin the seek sequence via |
| 821 |
:c:func:`VIDIOC_STREAMOFF`. |
| 822 |
|
| 823 |
* **Required fields:** |
| 824 |
|
| 825 |
``type`` |
| 826 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 827 |
|
| 828 |
* The decoder will drop all the pending ``OUTPUT`` buffers and they must be |
| 829 |
treated as returned to the client (following standard semantics). |
| 830 |
|
| 831 |
2. Restart the ``OUTPUT`` queue via :c:func:`VIDIOC_STREAMON`. |
| 832 |
|
| 833 |
* **Required fields:** |
| 834 |
|
| 835 |
``type`` |
| 836 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 837 |
|
| 838 |
* The decoder will start accepting new source bytestream buffers after the |
| 839 |
call returns. |
| 840 |
|
| 841 |
3. Start queuing buffers containing coded data after the seek to the ``OUTPUT`` |
| 842 |
queue until a suitable resume point is found. |
| 843 |
|
| 844 |
.. note:: |
| 845 |
|
| 846 |
There is no requirement to begin queuing coded data starting exactly |
| 847 |
from a resume point (e.g. SPS or a keyframe). Any queued ``OUTPUT`` |
| 848 |
buffers will be processed and returned to the client until a suitable |
| 849 |
resume point is found. While looking for a resume point, the decoder |
| 850 |
should not produce any decoded frames into ``CAPTURE`` buffers. |
| 851 |
|
| 852 |
Some hardware is known to mishandle seeks to a non-resume point. Such an |
| 853 |
operation may result in an unspecified number of corrupted decoded frames |
| 854 |
being made available on the ``CAPTURE`` queue. Drivers must ensure that |
| 855 |
no fatal decoding errors or crashes occur, and implement any necessary |
| 856 |
handling and workarounds for hardware issues related to seek operations. |
| 857 |
|
| 858 |
.. warning:: |
| 859 |
|
| 860 |
In case of the H.264/HEVC codec, the client must take care not to seek |
| 861 |
over a change of SPS/PPS. Even though the target frame could be a |
| 862 |
keyframe, the stale SPS/PPS inside decoder state would lead to undefined |
| 863 |
results when decoding. Although the decoder must handle that case without |
| 864 |
a crash or a fatal decode error, the client must not expect a sensible |
| 865 |
decode output. |
| 866 |
|
| 867 |
If the hardware can detect such corrupted decoded frames, then |
| 868 |
corresponding buffers will be returned to the client with the |
| 869 |
V4L2_BUF_FLAG_ERROR set. See the `Decoding` section for further |
| 870 |
description of decode error reporting. |
| 871 |
|
| 872 |
4. After a resume point is found, the decoder will start returning ``CAPTURE`` |
| 873 |
buffers containing decoded frames. |
| 874 |
|
| 875 |
.. important:: |
| 876 |
|
| 877 |
A seek may result in the `Dynamic Resolution Change` sequence being |
| 878 |
initiated, due to the seek target having decoding parameters different from |
| 879 |
the part of the stream decoded before the seek. The sequence must be handled |
| 880 |
as per normal decoder operation. |
| 881 |
|
| 882 |
.. warning:: |
| 883 |
|
| 884 |
It is not specified when the ``CAPTURE`` queue starts producing buffers |
| 885 |
containing decoded data from the ``OUTPUT`` buffers queued after the seek, |
| 886 |
as it operates independently from the ``OUTPUT`` queue. |
| 887 |
|
| 888 |
The decoder may return a number of remaining ``CAPTURE`` buffers containing |
| 889 |
decoded frames originating from the ``OUTPUT`` buffers queued before the |
| 890 |
seek sequence is performed. |
| 891 |
|
| 892 |
The ``VIDIOC_STREAMOFF`` operation discards any remaining queued |
| 893 |
``OUTPUT`` buffers, which means that not all of the ``OUTPUT`` buffers |
| 894 |
queued before the seek sequence may have matching ``CAPTURE`` buffers |
| 895 |
produced. For example, given the sequence of operations on the |
| 896 |
``OUTPUT`` queue: |
| 897 |
|
| 898 |
QBUF(A), QBUF(B), STREAMOFF(), STREAMON(), QBUF(G), QBUF(H), |
| 899 |
|
| 900 |
any of the following results on the ``CAPTURE`` queue is allowed: |
| 901 |
|
| 902 |
{A', B', G', H'}, {A', G', H'}, {G', H'}. |
| 903 |
|
| 904 |
To determine the CAPTURE buffer containing the first decoded frame after the |
| 905 |
seek, the client may observe the timestamps to match the CAPTURE and OUTPUT |
| 906 |
buffers or use V4L2_DEC_CMD_STOP and V4L2_DEC_CMD_START to drain the |
| 907 |
decoder. |
| 908 |
|
| 909 |
.. note:: |
| 910 |
|
| 911 |
To achieve instantaneous seek, the client may restart streaming on the |
| 912 |
``CAPTURE`` queue too to discard decoded, but not yet dequeued buffers. |
| 913 |
|
| 914 |
Dynamic Resolution Change |
| 915 |
========================= |
| 916 |
|
| 917 |
Streams that include resolution metadata in the bytestream may require switching |
| 918 |
to a different resolution during the decoding. |
| 919 |
|
| 920 |
.. note:: |
| 921 |
|
| 922 |
Not all decoders can detect resolution changes. Those that do set the |
| 923 |
``V4L2_FMT_FLAG_DYN_RESOLUTION`` flag for the coded format when |
| 924 |
:c:func:`VIDIOC_ENUM_FMT` is called. |
| 925 |
|
| 926 |
The sequence starts when the decoder detects a coded frame with one or more of |
| 927 |
the following parameters different from those previously established (and |
| 928 |
reflected by corresponding queries): |
| 929 |
|
| 930 |
* coded resolution (``OUTPUT`` width and height), |
| 931 |
|
| 932 |
* visible resolution (selection rectangles), |
| 933 |
|
| 934 |
* the minimum number of buffers needed for decoding, |
| 935 |
|
| 936 |
* bit-depth of the bitstream has been changed, |
| 937 |
|
| 938 |
* colorspace of the bitstream has been changed, but it doesn't require |
| 939 |
buffer reallocation. |
| 940 |
|
| 941 |
Whenever that happens, the decoder must proceed as follows: |
| 942 |
|
| 943 |
1. After encountering a resolution change in the stream, the decoder sends a |
| 944 |
``V4L2_EVENT_SOURCE_CHANGE`` event with ``changes`` set to |
| 945 |
``V4L2_EVENT_SRC_CH_RESOLUTION``. |
| 946 |
|
| 947 |
.. important:: |
| 948 |
|
| 949 |
Any client query issued after the decoder queues the event will return |
| 950 |
values applying to the stream after the resolution change, including |
| 951 |
queue formats, selection rectangles and controls. |
| 952 |
|
| 953 |
2. The decoder will then process and decode all remaining buffers from before |
| 954 |
the resolution change point. |
| 955 |
|
| 956 |
* The last buffer from before the change must be marked with the |
| 957 |
``V4L2_BUF_FLAG_LAST`` flag, similarly to the `Drain` sequence above. |
| 958 |
|
| 959 |
.. warning:: |
| 960 |
|
| 961 |
The last buffer may be empty (with :c:type:`v4l2_buffer` ``bytesused`` |
| 962 |
= 0) and in that case it must be ignored by the client, as it does not |
| 963 |
contain a decoded frame. |
| 964 |
|
| 965 |
.. note:: |
| 966 |
|
| 967 |
Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer marked |
| 968 |
with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from |
| 969 |
:c:func:`VIDIOC_DQBUF`. |
| 970 |
|
| 971 |
The client must continue the sequence as described below to continue the |
| 972 |
decoding process. |
| 973 |
|
| 974 |
1. Dequeue the source change event. |
| 975 |
|
| 976 |
.. important:: |
| 977 |
|
| 978 |
A source change triggers an implicit decoder drain, similar to the |
| 979 |
explicit `Drain` sequence. The decoder is stopped after it completes. |
| 980 |
The decoding process must be resumed with either a pair of calls to |
| 981 |
:c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the |
| 982 |
``CAPTURE`` queue, or a call to :c:func:`VIDIOC_DECODER_CMD` with the |
| 983 |
``V4L2_DEC_CMD_START`` command. |
| 984 |
|
| 985 |
2. Continue with the `Capture Setup` sequence. |
| 986 |
|
| 987 |
.. note:: |
| 988 |
|
| 989 |
During the resolution change sequence, the ``OUTPUT`` queue must remain |
| 990 |
streaming. Calling :c:func:`VIDIOC_STREAMOFF` on the ``OUTPUT`` queue would |
| 991 |
abort the sequence and initiate a seek. |
| 992 |
|
| 993 |
In principle, the ``OUTPUT`` queue operates separately from the ``CAPTURE`` |
| 994 |
queue and this remains true for the duration of the entire resolution change |
| 995 |
sequence as well. |
| 996 |
|
| 997 |
The client should, for best performance and simplicity, keep queuing/dequeuing |
| 998 |
buffers to/from the ``OUTPUT`` queue even while processing this sequence. |
| 999 |
|
| 1000 |
Drain |
| 1001 |
===== |
| 1002 |
|
| 1003 |
To ensure that all queued ``OUTPUT`` buffers have been processed and related |
| 1004 |
``CAPTURE`` buffers are given to the client, the client must follow the drain |
| 1005 |
sequence described below. After the drain sequence ends, the client has |
| 1006 |
received all decoded frames for all ``OUTPUT`` buffers queued before the |
| 1007 |
sequence was started. |
| 1008 |
|
| 1009 |
1. Begin drain by issuing :c:func:`VIDIOC_DECODER_CMD`. |
| 1010 |
|
| 1011 |
* **Required fields:** |
| 1012 |
|
| 1013 |
``cmd`` |
| 1014 |
set to ``V4L2_DEC_CMD_STOP``. |
| 1015 |
|
| 1016 |
``flags`` |
| 1017 |
set to 0. |
| 1018 |
|
| 1019 |
``pts`` |
| 1020 |
set to 0. |
| 1021 |
|
| 1022 |
.. warning:: |
| 1023 |
|
| 1024 |
The sequence can be only initiated if both ``OUTPUT`` and ``CAPTURE`` |
| 1025 |
queues are streaming. For compatibility reasons, the call to |
| 1026 |
:c:func:`VIDIOC_DECODER_CMD` will not fail even if any of the queues is |
| 1027 |
not streaming, but at the same time it will not initiate the `Drain` |
| 1028 |
sequence and so the steps described below would not be applicable. |
| 1029 |
|
| 1030 |
2. Any ``OUTPUT`` buffers queued by the client before the |
| 1031 |
:c:func:`VIDIOC_DECODER_CMD` was issued will be processed and decoded as |
| 1032 |
normal. The client must continue to handle both queues independently, |
| 1033 |
similarly to normal decode operation. This includes: |
| 1034 |
|
| 1035 |
* handling any operations triggered as a result of processing those buffers, |
| 1036 |
such as the `Dynamic Resolution Change` sequence, before continuing with |
| 1037 |
the drain sequence, |
| 1038 |
|
| 1039 |
* queuing and dequeuing ``CAPTURE`` buffers, until a buffer marked with the |
| 1040 |
``V4L2_BUF_FLAG_LAST`` flag is dequeued, |
| 1041 |
|
| 1042 |
.. warning:: |
| 1043 |
|
| 1044 |
The last buffer may be empty (with :c:type:`v4l2_buffer` |
| 1045 |
``bytesused`` = 0) and in that case it must be ignored by the client, |
| 1046 |
as it does not contain a decoded frame. |
| 1047 |
|
| 1048 |
.. note:: |
| 1049 |
|
| 1050 |
Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer |
| 1051 |
marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from |
| 1052 |
:c:func:`VIDIOC_DQBUF`. |
| 1053 |
|
| 1054 |
* dequeuing processed ``OUTPUT`` buffers, until all the buffers queued |
| 1055 |
before the ``V4L2_DEC_CMD_STOP`` command are dequeued, |
| 1056 |
|
| 1057 |
* dequeuing the ``V4L2_EVENT_EOS`` event, if the client subscribed to it. |
| 1058 |
|
| 1059 |
.. note:: |
| 1060 |
|
| 1061 |
For backwards compatibility, the decoder will signal a ``V4L2_EVENT_EOS`` |
| 1062 |
event when the last frame has been decoded and all frames are ready to be |
| 1063 |
dequeued. It is a deprecated behavior and the client must not rely on it. |
| 1064 |
The ``V4L2_BUF_FLAG_LAST`` buffer flag should be used instead. |
| 1065 |
|
| 1066 |
3. Once all the ``OUTPUT`` buffers queued before the ``V4L2_DEC_CMD_STOP`` call |
| 1067 |
are dequeued and the last ``CAPTURE`` buffer is dequeued, the decoder is |
| 1068 |
stopped and it will accept, but not process, any newly queued ``OUTPUT`` |
| 1069 |
buffers until the client issues any of the following operations: |
| 1070 |
|
| 1071 |
* ``V4L2_DEC_CMD_START`` - the decoder will not be reset and will resume |
| 1072 |
operation normally, with all the state from before the drain, |
| 1073 |
|
| 1074 |
* a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the |
| 1075 |
``CAPTURE`` queue - the decoder will resume the operation normally, |
| 1076 |
however any ``CAPTURE`` buffers still in the queue will be returned to the |
| 1077 |
client, |
| 1078 |
|
| 1079 |
* a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the |
| 1080 |
``OUTPUT`` queue - any pending source buffers will be returned to the |
| 1081 |
client and the `Seek` sequence will be triggered. |
| 1082 |
|
| 1083 |
.. note:: |
| 1084 |
|
| 1085 |
Once the drain sequence is initiated, the client needs to drive it to |
| 1086 |
completion, as described by the steps above, unless it aborts the process by |
| 1087 |
issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE`` |
| 1088 |
queues. The client is not allowed to issue ``V4L2_DEC_CMD_START`` or |
| 1089 |
``V4L2_DEC_CMD_STOP`` again while the drain sequence is in progress and they |
| 1090 |
will fail with -EBUSY error code if attempted. |
| 1091 |
|
| 1092 |
Although not mandatory, the availability of decoder commands may be queried |
| 1093 |
using :c:func:`VIDIOC_TRY_DECODER_CMD`. |
| 1094 |
|
| 1095 |
End of Stream |
| 1096 |
============= |
| 1097 |
|
| 1098 |
If the decoder encounters an end of stream marking in the stream, the decoder |
| 1099 |
will initiate the `Drain` sequence, which the client must handle as described |
| 1100 |
above, skipping the initial :c:func:`VIDIOC_DECODER_CMD`. |
| 1101 |
|
| 1102 |
Commit Points |
| 1103 |
============= |
| 1104 |
|
| 1105 |
Setting formats and allocating buffers trigger changes in the behavior of the |
| 1106 |
decoder. |
| 1107 |
|
| 1108 |
1. Setting the format on the ``OUTPUT`` queue may change the set of formats |
| 1109 |
supported/advertised on the ``CAPTURE`` queue. In particular, it also means |
| 1110 |
that the ``CAPTURE`` format may be reset and the client must not rely on the |
| 1111 |
previously set format being preserved. |
| 1112 |
|
| 1113 |
2. Enumerating formats on the ``CAPTURE`` queue always returns only formats |
| 1114 |
supported for the current ``OUTPUT`` format. |
| 1115 |
|
| 1116 |
3. Setting the format on the ``CAPTURE`` queue does not change the list of |
| 1117 |
formats available on the ``OUTPUT`` queue. An attempt to set a ``CAPTURE`` |
| 1118 |
format that is not supported for the currently selected ``OUTPUT`` format |
| 1119 |
will result in the decoder adjusting the requested ``CAPTURE`` format to a |
| 1120 |
supported one. |
| 1121 |
|
| 1122 |
4. Enumerating formats on the ``OUTPUT`` queue always returns the full set of |
| 1123 |
supported coded formats, irrespectively of the current ``CAPTURE`` format. |
| 1124 |
|
| 1125 |
5. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues, |
| 1126 |
the client must not change the format on the ``OUTPUT`` queue. Drivers will |
| 1127 |
return the -EBUSY error code for any such format change attempt. |
| 1128 |
|
| 1129 |
To summarize, setting formats and allocation must always start with the |
| 1130 |
``OUTPUT`` queue and the ``OUTPUT`` queue is the master that governs the |
| 1131 |
set of supported formats for the ``CAPTURE`` queue. |