| 1 |
.. SPDX-License-Identifier: GPL-2.0 OR GFDL-1.1-no-invariants-or-later |
| 2 |
.. c:namespace:: V4L |
| 3 |
|
| 4 |
.. _encoder: |
| 5 |
|
| 6 |
************************************************* |
| 7 |
Memory-to-Memory Stateful Video Encoder Interface |
| 8 |
************************************************* |
| 9 |
|
| 10 |
A stateful video encoder takes raw video frames in display order and encodes |
| 11 |
them into a bytestream. It generates complete chunks of the bytestream, including |
| 12 |
all metadata, headers, etc. The resulting bytestream does not require any |
| 13 |
further post-processing by the client. |
| 14 |
|
| 15 |
Performing software stream processing, header generation etc. in the driver |
| 16 |
in order to support this interface is strongly discouraged. In case such |
| 17 |
operations are needed, use of the Stateless Video Encoder Interface (in |
| 18 |
development) is strongly advised. |
| 19 |
|
| 20 |
Conventions and Notations Used in This Document |
| 21 |
=============================================== |
| 22 |
|
| 23 |
1. The general V4L2 API rules apply if not specified in this document |
| 24 |
otherwise. |
| 25 |
|
| 26 |
2. The meaning of words "must", "may", "should", etc. is as per `RFC |
| 27 |
2119 <https://tools.ietf.org/html/rfc2119>`_. |
| 28 |
|
| 29 |
3. All steps not marked "optional" are required. |
| 30 |
|
| 31 |
4. :c:func:`VIDIOC_G_EXT_CTRLS` and :c:func:`VIDIOC_S_EXT_CTRLS` may be used |
| 32 |
interchangeably with :c:func:`VIDIOC_G_CTRL` and :c:func:`VIDIOC_S_CTRL`, |
| 33 |
unless specified otherwise. |
| 34 |
|
| 35 |
5. Single-planar API (see :ref:`planar-apis`) and applicable structures may be |
| 36 |
used interchangeably with multi-planar API, unless specified otherwise, |
| 37 |
depending on encoder capabilities and following the general V4L2 guidelines. |
| 38 |
|
| 39 |
6. i = [a..b]: sequence of integers from a to b, inclusive, i.e. i = |
| 40 |
[0..2]: i = 0, 1, 2. |
| 41 |
|
| 42 |
7. Given an ``OUTPUT`` buffer A, then A' represents a buffer on the ``CAPTURE`` |
| 43 |
queue containing data that resulted from processing buffer A. |
| 44 |
|
| 45 |
Glossary |
| 46 |
======== |
| 47 |
|
| 48 |
Refer to :ref:`decoder-glossary`. |
| 49 |
|
| 50 |
State Machine |
| 51 |
============= |
| 52 |
|
| 53 |
.. kernel-render:: DOT |
| 54 |
:alt: DOT digraph of encoder state machine |
| 55 |
:caption: Encoder State Machine |
| 56 |
|
| 57 |
digraph encoder_state_machine { |
| 58 |
node [shape = doublecircle, label="Encoding"] Encoding; |
| 59 |
|
| 60 |
node [shape = circle, label="Initialization"] Initialization; |
| 61 |
node [shape = circle, label="Stopped"] Stopped; |
| 62 |
node [shape = circle, label="Drain"] Drain; |
| 63 |
node [shape = circle, label="Reset"] Reset; |
| 64 |
|
| 65 |
node [shape = point]; qi |
| 66 |
qi -> Initialization [ label = "open()" ]; |
| 67 |
|
| 68 |
Initialization -> Encoding [ label = "Both queues streaming" ]; |
| 69 |
|
| 70 |
Encoding -> Drain [ label = "V4L2_ENC_CMD_STOP" ]; |
| 71 |
Encoding -> Reset [ label = "VIDIOC_STREAMOFF(CAPTURE)" ]; |
| 72 |
Encoding -> Stopped [ label = "VIDIOC_STREAMOFF(OUTPUT)" ]; |
| 73 |
Encoding -> Encoding; |
| 74 |
|
| 75 |
Drain -> Stopped [ label = "All CAPTURE\nbuffers dequeued\nor\nVIDIOC_STREAMOFF(OUTPUT)" ]; |
| 76 |
Drain -> Reset [ label = "VIDIOC_STREAMOFF(CAPTURE)" ]; |
| 77 |
|
| 78 |
Reset -> Encoding [ label = "VIDIOC_STREAMON(CAPTURE)" ]; |
| 79 |
Reset -> Initialization [ label = "VIDIOC_REQBUFS(OUTPUT, 0)" ]; |
| 80 |
|
| 81 |
Stopped -> Encoding [ label = "V4L2_ENC_CMD_START\nor\nVIDIOC_STREAMON(OUTPUT)" ]; |
| 82 |
Stopped -> Reset [ label = "VIDIOC_STREAMOFF(CAPTURE)" ]; |
| 83 |
} |
| 84 |
|
| 85 |
Querying Capabilities |
| 86 |
===================== |
| 87 |
|
| 88 |
1. To enumerate the set of coded formats supported by the encoder, the |
| 89 |
client may call :c:func:`VIDIOC_ENUM_FMT` on ``CAPTURE``. |
| 90 |
|
| 91 |
* The full set of supported formats will be returned, regardless of the |
| 92 |
format set on ``OUTPUT``. |
| 93 |
|
| 94 |
2. To enumerate the set of supported raw formats, the client may call |
| 95 |
:c:func:`VIDIOC_ENUM_FMT` on ``OUTPUT``. |
| 96 |
|
| 97 |
* Only the formats supported for the format currently active on ``CAPTURE`` |
| 98 |
will be returned. |
| 99 |
|
| 100 |
* In order to enumerate raw formats supported by a given coded format, |
| 101 |
the client must first set that coded format on ``CAPTURE`` and then |
| 102 |
enumerate the formats on ``OUTPUT``. |
| 103 |
|
| 104 |
3. The client may use :c:func:`VIDIOC_ENUM_FRAMESIZES` to detect supported |
| 105 |
resolutions for a given format, passing the desired pixel format in |
| 106 |
:c:type:`v4l2_frmsizeenum` ``pixel_format``. |
| 107 |
|
| 108 |
* Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a coded pixel |
| 109 |
format will include all possible coded resolutions supported by the |
| 110 |
encoder for the given coded pixel format. |
| 111 |
|
| 112 |
* Values returned by :c:func:`VIDIOC_ENUM_FRAMESIZES` for a raw pixel format |
| 113 |
will include all possible frame buffer resolutions supported by the |
| 114 |
encoder for the given raw pixel format and coded format currently set on |
| 115 |
``CAPTURE``. |
| 116 |
|
| 117 |
4. The client may use :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` to detect supported |
| 118 |
frame intervals for a given format and resolution, passing the desired pixel |
| 119 |
format in :c:type:`v4l2_frmivalenum` ``pixel_format`` and the resolution |
| 120 |
in :c:type:`v4l2_frmivalenum` ``width`` and :c:type:`v4l2_frmivalenum` |
| 121 |
``height``. |
| 122 |
|
| 123 |
* Values returned by :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` for a coded pixel |
| 124 |
format and coded resolution will include all possible frame intervals |
| 125 |
supported by the encoder for the given coded pixel format and resolution. |
| 126 |
|
| 127 |
* Values returned by :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` for a raw pixel |
| 128 |
format and resolution will include all possible frame intervals supported |
| 129 |
by the encoder for the given raw pixel format and resolution and for the |
| 130 |
coded format, coded resolution and coded frame interval currently set on |
| 131 |
``CAPTURE``. |
| 132 |
|
| 133 |
* Support for :c:func:`VIDIOC_ENUM_FRAMEINTERVALS` is optional. If it is |
| 134 |
not implemented, then there are no special restrictions other than the |
| 135 |
limits of the codec itself. |
| 136 |
|
| 137 |
5. Supported profiles and levels for the coded format currently set on |
| 138 |
``CAPTURE``, if applicable, may be queried using their respective controls |
| 139 |
via :c:func:`VIDIOC_QUERYCTRL`. |
| 140 |
|
| 141 |
6. Any additional encoder capabilities may be discovered by querying |
| 142 |
their respective controls. |
| 143 |
|
| 144 |
Initialization |
| 145 |
============== |
| 146 |
|
| 147 |
1. Set the coded format on the ``CAPTURE`` queue via :c:func:`VIDIOC_S_FMT`. |
| 148 |
|
| 149 |
* **Required fields:** |
| 150 |
|
| 151 |
``type`` |
| 152 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 153 |
|
| 154 |
``pixelformat`` |
| 155 |
the coded format to be produced. |
| 156 |
|
| 157 |
``sizeimage`` |
| 158 |
desired size of ``CAPTURE`` buffers; the encoder may adjust it to |
| 159 |
match hardware requirements. |
| 160 |
|
| 161 |
``width``, ``height`` |
| 162 |
ignored (read-only). |
| 163 |
|
| 164 |
other fields |
| 165 |
follow standard semantics. |
| 166 |
|
| 167 |
* **Returned fields:** |
| 168 |
|
| 169 |
``sizeimage`` |
| 170 |
adjusted size of ``CAPTURE`` buffers. |
| 171 |
|
| 172 |
``width``, ``height`` |
| 173 |
the coded size selected by the encoder based on current state, e.g. |
| 174 |
``OUTPUT`` format, selection rectangles, etc. (read-only). |
| 175 |
|
| 176 |
.. important:: |
| 177 |
|
| 178 |
Changing the ``CAPTURE`` format may change the currently set ``OUTPUT`` |
| 179 |
format. How the new ``OUTPUT`` format is determined is up to the encoder |
| 180 |
and the client must ensure it matches its needs afterwards. |
| 181 |
|
| 182 |
2. **Optional.** Enumerate supported ``OUTPUT`` formats (raw formats for |
| 183 |
source) for the selected coded format via :c:func:`VIDIOC_ENUM_FMT`. |
| 184 |
|
| 185 |
* **Required fields:** |
| 186 |
|
| 187 |
``type`` |
| 188 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 189 |
|
| 190 |
other fields |
| 191 |
follow standard semantics. |
| 192 |
|
| 193 |
* **Returned fields:** |
| 194 |
|
| 195 |
``pixelformat`` |
| 196 |
raw format supported for the coded format currently selected on |
| 197 |
the ``CAPTURE`` queue. |
| 198 |
|
| 199 |
other fields |
| 200 |
follow standard semantics. |
| 201 |
|
| 202 |
3. Set the raw source format on the ``OUTPUT`` queue via |
| 203 |
:c:func:`VIDIOC_S_FMT`. |
| 204 |
|
| 205 |
* **Required fields:** |
| 206 |
|
| 207 |
``type`` |
| 208 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 209 |
|
| 210 |
``pixelformat`` |
| 211 |
raw format of the source. |
| 212 |
|
| 213 |
``width``, ``height`` |
| 214 |
source resolution. |
| 215 |
|
| 216 |
other fields |
| 217 |
follow standard semantics. |
| 218 |
|
| 219 |
* **Returned fields:** |
| 220 |
|
| 221 |
``width``, ``height`` |
| 222 |
may be adjusted to match encoder minimums, maximums and alignment |
| 223 |
requirements, as required by the currently selected formats, as |
| 224 |
reported by :c:func:`VIDIOC_ENUM_FRAMESIZES`. |
| 225 |
|
| 226 |
other fields |
| 227 |
follow standard semantics. |
| 228 |
|
| 229 |
* Setting the ``OUTPUT`` format will reset the selection rectangles to their |
| 230 |
default values, based on the new resolution, as described in the next |
| 231 |
step. |
| 232 |
|
| 233 |
4. Set the raw frame interval on the ``OUTPUT`` queue via |
| 234 |
:c:func:`VIDIOC_S_PARM`. This also sets the coded frame interval on the |
| 235 |
``CAPTURE`` queue to the same value. |
| 236 |
|
| 237 |
* **Required fields:** |
| 238 |
|
| 239 |
``type`` |
| 240 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 241 |
|
| 242 |
``parm.output`` |
| 243 |
set all fields except ``parm.output.timeperframe`` to 0. |
| 244 |
|
| 245 |
``parm.output.timeperframe`` |
| 246 |
the desired frame interval; the encoder may adjust it to |
| 247 |
match hardware requirements. |
| 248 |
|
| 249 |
* **Returned fields:** |
| 250 |
|
| 251 |
``parm.output.timeperframe`` |
| 252 |
the adjusted frame interval. |
| 253 |
|
| 254 |
.. important:: |
| 255 |
|
| 256 |
Changing the ``OUTPUT`` frame interval *also* sets the framerate that |
| 257 |
the encoder uses to encode the video. So setting the frame interval |
| 258 |
to 1/24 (or 24 frames per second) will produce a coded video stream |
| 259 |
that can be played back at that speed. The frame interval for the |
| 260 |
``OUTPUT`` queue is just a hint, the application may provide raw |
| 261 |
frames at a different rate. It can be used by the driver to help |
| 262 |
schedule multiple encoders running in parallel. |
| 263 |
|
| 264 |
In the next step the ``CAPTURE`` frame interval can optionally be |
| 265 |
changed to a different value. This is useful for off-line encoding |
| 266 |
were the coded frame interval can be different from the rate at |
| 267 |
which raw frames are supplied. |
| 268 |
|
| 269 |
.. important:: |
| 270 |
|
| 271 |
``timeperframe`` deals with *frames*, not fields. So for interlaced |
| 272 |
formats this is the time per two fields, since a frame consists of |
| 273 |
a top and a bottom field. |
| 274 |
|
| 275 |
.. note:: |
| 276 |
|
| 277 |
It is due to historical reasons that changing the ``OUTPUT`` frame |
| 278 |
interval also changes the coded frame interval on the ``CAPTURE`` |
| 279 |
queue. Ideally these would be independent settings, but that would |
| 280 |
break the existing API. |
| 281 |
|
| 282 |
5. **Optional** Set the coded frame interval on the ``CAPTURE`` queue via |
| 283 |
:c:func:`VIDIOC_S_PARM`. This is only necessary if the coded frame |
| 284 |
interval is different from the raw frame interval, which is typically |
| 285 |
the case for off-line encoding. Support for this feature is signalled |
| 286 |
by the :ref:`V4L2_FMT_FLAG_ENC_CAP_FRAME_INTERVAL <fmtdesc-flags>` format flag. |
| 287 |
|
| 288 |
* **Required fields:** |
| 289 |
|
| 290 |
``type`` |
| 291 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``CAPTURE``. |
| 292 |
|
| 293 |
``parm.capture`` |
| 294 |
set all fields except ``parm.capture.timeperframe`` to 0. |
| 295 |
|
| 296 |
``parm.capture.timeperframe`` |
| 297 |
the desired coded frame interval; the encoder may adjust it to |
| 298 |
match hardware requirements. |
| 299 |
|
| 300 |
* **Returned fields:** |
| 301 |
|
| 302 |
``parm.capture.timeperframe`` |
| 303 |
the adjusted frame interval. |
| 304 |
|
| 305 |
.. important:: |
| 306 |
|
| 307 |
Changing the ``CAPTURE`` frame interval sets the framerate for the |
| 308 |
coded video. It does *not* set the rate at which buffers arrive on the |
| 309 |
``CAPTURE`` queue, that depends on how fast the encoder is and how |
| 310 |
fast raw frames are queued on the ``OUTPUT`` queue. |
| 311 |
|
| 312 |
.. important:: |
| 313 |
|
| 314 |
``timeperframe`` deals with *frames*, not fields. So for interlaced |
| 315 |
formats this is the time per two fields, since a frame consists of |
| 316 |
a top and a bottom field. |
| 317 |
|
| 318 |
.. note:: |
| 319 |
|
| 320 |
Not all drivers support this functionality, in that case just set |
| 321 |
the desired coded frame interval for the ``OUTPUT`` queue. |
| 322 |
|
| 323 |
However, drivers that can schedule multiple encoders based on the |
| 324 |
``OUTPUT`` frame interval must support this optional feature. |
| 325 |
|
| 326 |
6. **Optional.** Set the visible resolution for the stream metadata via |
| 327 |
:c:func:`VIDIOC_S_SELECTION` on the ``OUTPUT`` queue if it is desired |
| 328 |
to be different than the full OUTPUT resolution. |
| 329 |
|
| 330 |
* **Required fields:** |
| 331 |
|
| 332 |
``type`` |
| 333 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 334 |
|
| 335 |
``target`` |
| 336 |
set to ``V4L2_SEL_TGT_CROP``. |
| 337 |
|
| 338 |
``r.left``, ``r.top``, ``r.width``, ``r.height`` |
| 339 |
visible rectangle; this must fit within the `V4L2_SEL_TGT_CROP_BOUNDS` |
| 340 |
rectangle and may be subject to adjustment to match codec and |
| 341 |
hardware constraints. |
| 342 |
|
| 343 |
* **Returned fields:** |
| 344 |
|
| 345 |
``r.left``, ``r.top``, ``r.width``, ``r.height`` |
| 346 |
visible rectangle adjusted by the encoder. |
| 347 |
|
| 348 |
* The following selection targets are supported on ``OUTPUT``: |
| 349 |
|
| 350 |
``V4L2_SEL_TGT_CROP_BOUNDS`` |
| 351 |
equal to the full source frame, matching the active ``OUTPUT`` |
| 352 |
format. |
| 353 |
|
| 354 |
``V4L2_SEL_TGT_CROP_DEFAULT`` |
| 355 |
equal to ``V4L2_SEL_TGT_CROP_BOUNDS``. |
| 356 |
|
| 357 |
``V4L2_SEL_TGT_CROP`` |
| 358 |
rectangle within the source buffer to be encoded into the |
| 359 |
``CAPTURE`` stream; defaults to ``V4L2_SEL_TGT_CROP_DEFAULT``. |
| 360 |
|
| 361 |
.. note:: |
| 362 |
|
| 363 |
A common use case for this selection target is encoding a source |
| 364 |
video with a resolution that is not a multiple of a macroblock, |
| 365 |
e.g. the common 1920x1080 resolution may require the source |
| 366 |
buffers to be aligned to 1920x1088 for codecs with 16x16 macroblock |
| 367 |
size. To avoid encoding the padding, the client needs to explicitly |
| 368 |
configure this selection target to 1920x1080. |
| 369 |
|
| 370 |
.. warning:: |
| 371 |
|
| 372 |
The encoder may adjust the crop/compose rectangles to the nearest |
| 373 |
supported ones to meet codec and hardware requirements. The client needs |
| 374 |
to check the adjusted rectangle returned by :c:func:`VIDIOC_S_SELECTION`. |
| 375 |
|
| 376 |
7. Allocate buffers for both ``OUTPUT`` and ``CAPTURE`` via |
| 377 |
:c:func:`VIDIOC_REQBUFS`. This may be performed in any order. |
| 378 |
|
| 379 |
* **Required fields:** |
| 380 |
|
| 381 |
``count`` |
| 382 |
requested number of buffers to allocate; greater than zero. |
| 383 |
|
| 384 |
``type`` |
| 385 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT`` or |
| 386 |
``CAPTURE``. |
| 387 |
|
| 388 |
other fields |
| 389 |
follow standard semantics. |
| 390 |
|
| 391 |
* **Returned fields:** |
| 392 |
|
| 393 |
``count`` |
| 394 |
actual number of buffers allocated. |
| 395 |
|
| 396 |
.. warning:: |
| 397 |
|
| 398 |
The actual number of allocated buffers may differ from the ``count`` |
| 399 |
given. The client must check the updated value of ``count`` after the |
| 400 |
call returns. |
| 401 |
|
| 402 |
.. note:: |
| 403 |
|
| 404 |
To allocate more than the minimum number of OUTPUT buffers (for pipeline |
| 405 |
depth), the client may query the ``V4L2_CID_MIN_BUFFERS_FOR_OUTPUT`` |
| 406 |
control to get the minimum number of buffers required, and pass the |
| 407 |
obtained value plus the number of additional buffers needed in the |
| 408 |
``count`` field to :c:func:`VIDIOC_REQBUFS`. |
| 409 |
|
| 410 |
Alternatively, :c:func:`VIDIOC_CREATE_BUFS` can be used to have more |
| 411 |
control over buffer allocation. |
| 412 |
|
| 413 |
* **Required fields:** |
| 414 |
|
| 415 |
``count`` |
| 416 |
requested number of buffers to allocate; greater than zero. |
| 417 |
|
| 418 |
``type`` |
| 419 |
a ``V4L2_BUF_TYPE_*`` enum appropriate for ``OUTPUT``. |
| 420 |
|
| 421 |
other fields |
| 422 |
follow standard semantics. |
| 423 |
|
| 424 |
* **Returned fields:** |
| 425 |
|
| 426 |
``count`` |
| 427 |
adjusted to the number of allocated buffers. |
| 428 |
|
| 429 |
8. Begin streaming on both ``OUTPUT`` and ``CAPTURE`` queues via |
| 430 |
:c:func:`VIDIOC_STREAMON`. This may be performed in any order. The actual |
| 431 |
encoding process starts when both queues start streaming. |
| 432 |
|
| 433 |
.. note:: |
| 434 |
|
| 435 |
If the client stops the ``CAPTURE`` queue during the encode process and then |
| 436 |
restarts it again, the encoder will begin generating a stream independent |
| 437 |
from the stream generated before the stop. The exact constraints depend |
| 438 |
on the coded format, but may include the following implications: |
| 439 |
|
| 440 |
* encoded frames produced after the restart must not reference any |
| 441 |
frames produced before the stop, e.g. no long term references for |
| 442 |
H.264/HEVC, |
| 443 |
|
| 444 |
* any headers that must be included in a standalone stream must be |
| 445 |
produced again, e.g. SPS and PPS for H.264/HEVC. |
| 446 |
|
| 447 |
Encoding |
| 448 |
======== |
| 449 |
|
| 450 |
This state is reached after the `Initialization` sequence finishes |
| 451 |
successfully. In this state, the client queues and dequeues buffers to both |
| 452 |
queues via :c:func:`VIDIOC_QBUF` and :c:func:`VIDIOC_DQBUF`, following the |
| 453 |
standard semantics. |
| 454 |
|
| 455 |
The content of encoded ``CAPTURE`` buffers depends on the active coded pixel |
| 456 |
format and may be affected by codec-specific extended controls, as stated |
| 457 |
in the documentation of each format. |
| 458 |
|
| 459 |
Both queues operate independently, following standard behavior of V4L2 buffer |
| 460 |
queues and memory-to-memory devices. In addition, the order of encoded frames |
| 461 |
dequeued from the ``CAPTURE`` queue may differ from the order of queuing raw |
| 462 |
frames to the ``OUTPUT`` queue, due to properties of the selected coded format, |
| 463 |
e.g. frame reordering. |
| 464 |
|
| 465 |
The client must not assume any direct relationship between ``CAPTURE`` and |
| 466 |
``OUTPUT`` buffers and any specific timing of buffers becoming |
| 467 |
available to dequeue. Specifically: |
| 468 |
|
| 469 |
* a buffer queued to ``OUTPUT`` may result in more than one buffer produced on |
| 470 |
``CAPTURE`` (for example, if returning an encoded frame allowed the encoder |
| 471 |
to return a frame that preceded it in display, but succeeded it in the decode |
| 472 |
order; however, there may be other reasons for this as well), |
| 473 |
|
| 474 |
* a buffer queued to ``OUTPUT`` may result in a buffer being produced on |
| 475 |
``CAPTURE`` later into encode process, and/or after processing further |
| 476 |
``OUTPUT`` buffers, or be returned out of order, e.g. if display |
| 477 |
reordering is used, |
| 478 |
|
| 479 |
* buffers may become available on the ``CAPTURE`` queue without additional |
| 480 |
buffers queued to ``OUTPUT`` (e.g. during drain or ``EOS``), because of the |
| 481 |
``OUTPUT`` buffers queued in the past whose encoding results are only |
| 482 |
available at later time, due to specifics of the encoding process, |
| 483 |
|
| 484 |
* buffers queued to ``OUTPUT`` may not become available to dequeue instantly |
| 485 |
after being encoded into a corresponding ``CAPTURE`` buffer, e.g. if the |
| 486 |
encoder needs to use the frame as a reference for encoding further frames. |
| 487 |
|
| 488 |
.. note:: |
| 489 |
|
| 490 |
To allow matching encoded ``CAPTURE`` buffers with ``OUTPUT`` buffers they |
| 491 |
originated from, the client can set the ``timestamp`` field of the |
| 492 |
:c:type:`v4l2_buffer` struct when queuing an ``OUTPUT`` buffer. The |
| 493 |
``CAPTURE`` buffer(s), which resulted from encoding that ``OUTPUT`` buffer |
| 494 |
will have their ``timestamp`` field set to the same value when dequeued. |
| 495 |
|
| 496 |
In addition to the straightforward case of one ``OUTPUT`` buffer producing |
| 497 |
one ``CAPTURE`` buffer, the following cases are defined: |
| 498 |
|
| 499 |
* one ``OUTPUT`` buffer generates multiple ``CAPTURE`` buffers: the same |
| 500 |
``OUTPUT`` timestamp will be copied to multiple ``CAPTURE`` buffers, |
| 501 |
|
| 502 |
* the encoding order differs from the presentation order (i.e. the |
| 503 |
``CAPTURE`` buffers are out-of-order compared to the ``OUTPUT`` buffers): |
| 504 |
``CAPTURE`` timestamps will not retain the order of ``OUTPUT`` timestamps. |
| 505 |
|
| 506 |
.. note:: |
| 507 |
|
| 508 |
To let the client distinguish between frame types (keyframes, intermediate |
| 509 |
frames; the exact list of types depends on the coded format), the |
| 510 |
``CAPTURE`` buffers will have corresponding flag bits set in their |
| 511 |
:c:type:`v4l2_buffer` struct when dequeued. See the documentation of |
| 512 |
:c:type:`v4l2_buffer` and each coded pixel format for exact list of flags |
| 513 |
and their meanings. |
| 514 |
|
| 515 |
Should an encoding error occur, it will be reported to the client with the level |
| 516 |
of details depending on the encoder capabilities. Specifically: |
| 517 |
|
| 518 |
* the ``CAPTURE`` buffer (if any) that contains the results of the failed encode |
| 519 |
operation will be returned with the ``V4L2_BUF_FLAG_ERROR`` flag set, |
| 520 |
|
| 521 |
* if the encoder is able to precisely report the ``OUTPUT`` buffer(s) that triggered |
| 522 |
the error, such buffer(s) will be returned with the ``V4L2_BUF_FLAG_ERROR`` flag |
| 523 |
set. |
| 524 |
|
| 525 |
.. note:: |
| 526 |
|
| 527 |
If a ``CAPTURE`` buffer is too small then it is just returned with the |
| 528 |
``V4L2_BUF_FLAG_ERROR`` flag set. More work is needed to detect that this |
| 529 |
error occurred because the buffer was too small, and to provide support to |
| 530 |
free existing buffers that were too small. |
| 531 |
|
| 532 |
In case of a fatal failure that does not allow the encoding to continue, any |
| 533 |
further operations on corresponding encoder file handle will return the -EIO |
| 534 |
error code. The client may close the file handle and open a new one, or |
| 535 |
alternatively reinitialize the instance by stopping streaming on both queues, |
| 536 |
releasing all buffers and performing the Initialization sequence again. |
| 537 |
|
| 538 |
Encoding Parameter Changes |
| 539 |
========================== |
| 540 |
|
| 541 |
The client is allowed to use :c:func:`VIDIOC_S_CTRL` to change encoder |
| 542 |
parameters at any time. The availability of parameters is encoder-specific |
| 543 |
and the client must query the encoder to find the set of available controls. |
| 544 |
|
| 545 |
The ability to change each parameter during encoding is encoder-specific, as |
| 546 |
per the standard semantics of the V4L2 control interface. The client may |
| 547 |
attempt to set a control during encoding and if the operation fails with the |
| 548 |
-EBUSY error code, the ``CAPTURE`` queue needs to be stopped for the |
| 549 |
configuration change to be allowed. To do this, it may follow the `Drain` |
| 550 |
sequence to avoid losing the already queued/encoded frames. |
| 551 |
|
| 552 |
The timing of parameter updates is encoder-specific, as per the standard |
| 553 |
semantics of the V4L2 control interface. If the client needs to apply the |
| 554 |
parameters exactly at specific frame, using the Request API |
| 555 |
(:ref:`media-request-api`) should be considered, if supported by the encoder. |
| 556 |
|
| 557 |
Drain |
| 558 |
===== |
| 559 |
|
| 560 |
To ensure that all the queued ``OUTPUT`` buffers have been processed and the |
| 561 |
related ``CAPTURE`` buffers are given to the client, the client must follow the |
| 562 |
drain sequence described below. After the drain sequence ends, the client has |
| 563 |
received all encoded frames for all ``OUTPUT`` buffers queued before the |
| 564 |
sequence was started. |
| 565 |
|
| 566 |
1. Begin the drain sequence by issuing :c:func:`VIDIOC_ENCODER_CMD`. |
| 567 |
|
| 568 |
* **Required fields:** |
| 569 |
|
| 570 |
``cmd`` |
| 571 |
set to ``V4L2_ENC_CMD_STOP``. |
| 572 |
|
| 573 |
``flags`` |
| 574 |
set to 0. |
| 575 |
|
| 576 |
``pts`` |
| 577 |
set to 0. |
| 578 |
|
| 579 |
.. warning:: |
| 580 |
|
| 581 |
The sequence can be only initiated if both ``OUTPUT`` and ``CAPTURE`` |
| 582 |
queues are streaming. For compatibility reasons, the call to |
| 583 |
:c:func:`VIDIOC_ENCODER_CMD` will not fail even if any of the queues is |
| 584 |
not streaming, but at the same time it will not initiate the `Drain` |
| 585 |
sequence and so the steps described below would not be applicable. |
| 586 |
|
| 587 |
2. Any ``OUTPUT`` buffers queued by the client before the |
| 588 |
:c:func:`VIDIOC_ENCODER_CMD` was issued will be processed and encoded as |
| 589 |
normal. The client must continue to handle both queues independently, |
| 590 |
similarly to normal encode operation. This includes: |
| 591 |
|
| 592 |
* queuing and dequeuing ``CAPTURE`` buffers, until a buffer marked with the |
| 593 |
``V4L2_BUF_FLAG_LAST`` flag is dequeued, |
| 594 |
|
| 595 |
.. warning:: |
| 596 |
|
| 597 |
The last buffer may be empty (with :c:type:`v4l2_buffer` |
| 598 |
``bytesused`` = 0) and in that case it must be ignored by the client, |
| 599 |
as it does not contain an encoded frame. |
| 600 |
|
| 601 |
.. note:: |
| 602 |
|
| 603 |
Any attempt to dequeue more ``CAPTURE`` buffers beyond the buffer |
| 604 |
marked with ``V4L2_BUF_FLAG_LAST`` will result in a -EPIPE error from |
| 605 |
:c:func:`VIDIOC_DQBUF`. |
| 606 |
|
| 607 |
* dequeuing processed ``OUTPUT`` buffers, until all the buffers queued |
| 608 |
before the ``V4L2_ENC_CMD_STOP`` command are dequeued, |
| 609 |
|
| 610 |
* dequeuing the ``V4L2_EVENT_EOS`` event, if the client subscribes to it. |
| 611 |
|
| 612 |
.. note:: |
| 613 |
|
| 614 |
For backwards compatibility, the encoder will signal a ``V4L2_EVENT_EOS`` |
| 615 |
event when the last frame has been encoded and all frames are ready to be |
| 616 |
dequeued. It is deprecated behavior and the client must not rely on it. |
| 617 |
The ``V4L2_BUF_FLAG_LAST`` buffer flag should be used instead. |
| 618 |
|
| 619 |
3. Once all ``OUTPUT`` buffers queued before the ``V4L2_ENC_CMD_STOP`` call are |
| 620 |
dequeued and the last ``CAPTURE`` buffer is dequeued, the encoder is stopped |
| 621 |
and it will accept, but not process any newly queued ``OUTPUT`` buffers |
| 622 |
until the client issues any of the following operations: |
| 623 |
|
| 624 |
* ``V4L2_ENC_CMD_START`` - the encoder will not be reset and will resume |
| 625 |
operation normally, with all the state from before the drain, |
| 626 |
|
| 627 |
* a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the |
| 628 |
``CAPTURE`` queue - the encoder will be reset (see the `Reset` sequence) |
| 629 |
and then resume encoding, |
| 630 |
|
| 631 |
* a pair of :c:func:`VIDIOC_STREAMOFF` and :c:func:`VIDIOC_STREAMON` on the |
| 632 |
``OUTPUT`` queue - the encoder will resume operation normally, however any |
| 633 |
source frames queued to the ``OUTPUT`` queue between ``V4L2_ENC_CMD_STOP`` |
| 634 |
and :c:func:`VIDIOC_STREAMOFF` will be discarded. |
| 635 |
|
| 636 |
.. note:: |
| 637 |
|
| 638 |
Once the drain sequence is initiated, the client needs to drive it to |
| 639 |
completion, as described by the steps above, unless it aborts the process by |
| 640 |
issuing :c:func:`VIDIOC_STREAMOFF` on any of the ``OUTPUT`` or ``CAPTURE`` |
| 641 |
queues. The client is not allowed to issue ``V4L2_ENC_CMD_START`` or |
| 642 |
``V4L2_ENC_CMD_STOP`` again while the drain sequence is in progress and they |
| 643 |
will fail with -EBUSY error code if attempted. |
| 644 |
|
| 645 |
For reference, handling of various corner cases is described below: |
| 646 |
|
| 647 |
* In case of no buffer in the ``OUTPUT`` queue at the time the |
| 648 |
``V4L2_ENC_CMD_STOP`` command was issued, the drain sequence completes |
| 649 |
immediately and the encoder returns an empty ``CAPTURE`` buffer with the |
| 650 |
``V4L2_BUF_FLAG_LAST`` flag set. |
| 651 |
|
| 652 |
* In case of no buffer in the ``CAPTURE`` queue at the time the drain |
| 653 |
sequence completes, the next time the client queues a ``CAPTURE`` buffer |
| 654 |
it is returned at once as an empty buffer with the ``V4L2_BUF_FLAG_LAST`` |
| 655 |
flag set. |
| 656 |
|
| 657 |
* If :c:func:`VIDIOC_STREAMOFF` is called on the ``CAPTURE`` queue in the |
| 658 |
middle of the drain sequence, the drain sequence is canceled and all |
| 659 |
``CAPTURE`` buffers are implicitly returned to the client. |
| 660 |
|
| 661 |
* If :c:func:`VIDIOC_STREAMOFF` is called on the ``OUTPUT`` queue in the |
| 662 |
middle of the drain sequence, the drain sequence completes immediately and |
| 663 |
next ``CAPTURE`` buffer will be returned empty with the |
| 664 |
``V4L2_BUF_FLAG_LAST`` flag set. |
| 665 |
|
| 666 |
Although not mandatory, the availability of encoder commands may be queried |
| 667 |
using :c:func:`VIDIOC_TRY_ENCODER_CMD`. |
| 668 |
|
| 669 |
Reset |
| 670 |
===== |
| 671 |
|
| 672 |
The client may want to request the encoder to reinitialize the encoding, so |
| 673 |
that the following stream data becomes independent from the stream data |
| 674 |
generated before. Depending on the coded format, that may imply that: |
| 675 |
|
| 676 |
* encoded frames produced after the restart must not reference any frames |
| 677 |
produced before the stop, e.g. no long term references for H.264/HEVC, |
| 678 |
|
| 679 |
* any headers that must be included in a standalone stream must be produced |
| 680 |
again, e.g. SPS and PPS for H.264/HEVC. |
| 681 |
|
| 682 |
This can be achieved by performing the reset sequence. |
| 683 |
|
| 684 |
1. Perform the `Drain` sequence to ensure all the in-flight encoding finishes |
| 685 |
and respective buffers are dequeued. |
| 686 |
|
| 687 |
2. Stop streaming on the ``CAPTURE`` queue via :c:func:`VIDIOC_STREAMOFF`. This |
| 688 |
will return all currently queued ``CAPTURE`` buffers to the client, without |
| 689 |
valid frame data. |
| 690 |
|
| 691 |
3. Start streaming on the ``CAPTURE`` queue via :c:func:`VIDIOC_STREAMON` and |
| 692 |
continue with regular encoding sequence. The encoded frames produced into |
| 693 |
``CAPTURE`` buffers from now on will contain a standalone stream that can be |
| 694 |
decoded without the need for frames encoded before the reset sequence, |
| 695 |
starting at the first ``OUTPUT`` buffer queued after issuing the |
| 696 |
`V4L2_ENC_CMD_STOP` of the `Drain` sequence. |
| 697 |
|
| 698 |
This sequence may be also used to change encoding parameters for encoders |
| 699 |
without the ability to change the parameters on the fly. |
| 700 |
|
| 701 |
Commit Points |
| 702 |
============= |
| 703 |
|
| 704 |
Setting formats and allocating buffers triggers changes in the behavior of the |
| 705 |
encoder. |
| 706 |
|
| 707 |
1. Setting the format on the ``CAPTURE`` queue may change the set of formats |
| 708 |
supported/advertised on the ``OUTPUT`` queue. In particular, it also means |
| 709 |
that the ``OUTPUT`` format may be reset and the client must not rely on the |
| 710 |
previously set format being preserved. |
| 711 |
|
| 712 |
2. Enumerating formats on the ``OUTPUT`` queue always returns only formats |
| 713 |
supported for the current ``CAPTURE`` format. |
| 714 |
|
| 715 |
3. Setting the format on the ``OUTPUT`` queue does not change the list of |
| 716 |
formats available on the ``CAPTURE`` queue. An attempt to set the ``OUTPUT`` |
| 717 |
format that is not supported for the currently selected ``CAPTURE`` format |
| 718 |
will result in the encoder adjusting the requested ``OUTPUT`` format to a |
| 719 |
supported one. |
| 720 |
|
| 721 |
4. Enumerating formats on the ``CAPTURE`` queue always returns the full set of |
| 722 |
supported coded formats, irrespective of the current ``OUTPUT`` format. |
| 723 |
|
| 724 |
5. While buffers are allocated on any of the ``OUTPUT`` or ``CAPTURE`` queues, |
| 725 |
the client must not change the format on the ``CAPTURE`` queue. Drivers will |
| 726 |
return the -EBUSY error code for any such format change attempt. |
| 727 |
|
| 728 |
To summarize, setting formats and allocation must always start with the |
| 729 |
``CAPTURE`` queue and the ``CAPTURE`` queue is the master that governs the |
| 730 |
set of supported formats for the ``OUTPUT`` queue. |