| 1 |
.. SPDX-License-Identifier: GFDL-1.1-no-invariants-or-later |
| 2 |
.. c:namespace:: V4L |
| 3 |
|
| 4 |
.. _subdev: |
| 5 |
|
| 6 |
******************** |
| 7 |
Sub-device Interface |
| 8 |
******************** |
| 9 |
|
| 10 |
The complex nature of V4L2 devices, where hardware is often made of |
| 11 |
several integrated circuits that need to interact with each other in a |
| 12 |
controlled way, leads to complex V4L2 drivers. The drivers usually |
| 13 |
reflect the hardware model in software, and model the different hardware |
| 14 |
components as software blocks called sub-devices. |
| 15 |
|
| 16 |
V4L2 sub-devices are usually kernel-only objects. If the V4L2 driver |
| 17 |
implements the media device API, they will automatically inherit from |
| 18 |
media entities. Applications will be able to enumerate the sub-devices |
| 19 |
and discover the hardware topology using the media entities, pads and |
| 20 |
links enumeration API. |
| 21 |
|
| 22 |
In addition to make sub-devices discoverable, drivers can also choose to |
| 23 |
make them directly configurable by applications. When both the |
| 24 |
sub-device driver and the V4L2 device driver support this, sub-devices |
| 25 |
will feature a character device node on which ioctls can be called to |
| 26 |
|
| 27 |
- query, read and write sub-devices controls |
| 28 |
|
| 29 |
- subscribe and unsubscribe to events and retrieve them |
| 30 |
|
| 31 |
- negotiate image formats on individual pads |
| 32 |
|
| 33 |
- inspect and modify internal data routing between pads of the same entity |
| 34 |
|
| 35 |
Sub-device character device nodes, conventionally named |
| 36 |
``/dev/v4l-subdev*``, use major number 81. |
| 37 |
|
| 38 |
Drivers may opt to limit the sub-device character devices to only expose |
| 39 |
operations that do not modify the device state. In such a case the sub-devices |
| 40 |
are referred to as ``read-only`` in the rest of this documentation, and the |
| 41 |
related restrictions are documented in individual ioctls. |
| 42 |
|
| 43 |
|
| 44 |
Controls |
| 45 |
======== |
| 46 |
|
| 47 |
Most V4L2 controls are implemented by sub-device hardware. Drivers |
| 48 |
usually merge all controls and expose them through video device nodes. |
| 49 |
Applications can control all sub-devices through a single interface. |
| 50 |
|
| 51 |
Complex devices sometimes implement the same control in different pieces |
| 52 |
of hardware. This situation is common in embedded platforms, where both |
| 53 |
sensors and image processing hardware implement identical functions, |
| 54 |
such as contrast adjustment, white balance or faulty pixels correction. |
| 55 |
As the V4L2 controls API doesn't support several identical controls in a |
| 56 |
single device, all but one of the identical controls are hidden. |
| 57 |
|
| 58 |
Applications can access those hidden controls through the sub-device |
| 59 |
node with the V4L2 control API described in :ref:`control`. The ioctls |
| 60 |
behave identically as when issued on V4L2 device nodes, with the |
| 61 |
exception that they deal only with controls implemented in the |
| 62 |
sub-device. |
| 63 |
|
| 64 |
Depending on the driver, those controls might also be exposed through |
| 65 |
one (or several) V4L2 device nodes. |
| 66 |
|
| 67 |
|
| 68 |
Events |
| 69 |
====== |
| 70 |
|
| 71 |
V4L2 sub-devices can notify applications of events as described in |
| 72 |
:ref:`event`. The API behaves identically as when used on V4L2 device |
| 73 |
nodes, with the exception that it only deals with events generated by |
| 74 |
the sub-device. Depending on the driver, those events might also be |
| 75 |
reported on one (or several) V4L2 device nodes. |
| 76 |
|
| 77 |
|
| 78 |
.. _pad-level-formats: |
| 79 |
|
| 80 |
Pad-level Formats |
| 81 |
================= |
| 82 |
|
| 83 |
.. warning:: |
| 84 |
|
| 85 |
Pad-level formats are only applicable to very complex devices that |
| 86 |
need to expose low-level format configuration to user space. Generic |
| 87 |
V4L2 applications do *not* need to use the API described in this |
| 88 |
section. |
| 89 |
|
| 90 |
.. note:: |
| 91 |
|
| 92 |
For the purpose of this section, the term *format* means the |
| 93 |
combination of media bus data format, frame width and frame height. |
| 94 |
|
| 95 |
Image formats are typically negotiated on video capture and output |
| 96 |
devices using the format and |
| 97 |
:ref:`selection <VIDIOC_SUBDEV_G_SELECTION>` ioctls. The driver is |
| 98 |
responsible for configuring every block in the video pipeline according |
| 99 |
to the requested format at the pipeline input and/or output. |
| 100 |
|
| 101 |
For complex devices, such as often found in embedded systems, identical |
| 102 |
image sizes at the output of a pipeline can be achieved using different |
| 103 |
hardware configurations. One such example is shown on |
| 104 |
:ref:`pipeline-scaling`, where image scaling can be performed on both |
| 105 |
the video sensor and the host image processing hardware. |
| 106 |
|
| 107 |
|
| 108 |
.. _pipeline-scaling: |
| 109 |
|
| 110 |
.. kernel-figure:: pipeline.dot |
| 111 |
:alt: pipeline.dot |
| 112 |
:align: center |
| 113 |
|
| 114 |
Image Format Negotiation on Pipelines |
| 115 |
|
| 116 |
High quality and high speed pipeline configuration |
| 117 |
|
| 118 |
|
| 119 |
|
| 120 |
The sensor scaler is usually of less quality than the host scaler, but |
| 121 |
scaling on the sensor is required to achieve higher frame rates. |
| 122 |
Depending on the use case (quality vs. speed), the pipeline must be |
| 123 |
configured differently. Applications need to configure the formats at |
| 124 |
every point in the pipeline explicitly. |
| 125 |
|
| 126 |
Drivers that implement the :ref:`media API <media-controller-intro>` |
| 127 |
can expose pad-level image format configuration to applications. When |
| 128 |
they do, applications can use the |
| 129 |
:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and |
| 130 |
:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls. to |
| 131 |
negotiate formats on a per-pad basis. |
| 132 |
|
| 133 |
Applications are responsible for configuring coherent parameters on the |
| 134 |
whole pipeline and making sure that connected pads have compatible |
| 135 |
formats. The pipeline is checked for formats mismatch at |
| 136 |
:ref:`VIDIOC_STREAMON <VIDIOC_STREAMON>` time, and an ``EPIPE`` error |
| 137 |
code is then returned if the configuration is invalid. |
| 138 |
|
| 139 |
Pad-level image format configuration support can be tested by calling |
| 140 |
the :ref:`VIDIOC_SUBDEV_G_FMT` ioctl on pad |
| 141 |
0. If the driver returns an ``EINVAL`` error code pad-level format |
| 142 |
configuration is not supported by the sub-device. |
| 143 |
|
| 144 |
|
| 145 |
Format Negotiation |
| 146 |
------------------ |
| 147 |
|
| 148 |
Acceptable formats on pads can (and usually do) depend on a number of |
| 149 |
external parameters, such as formats on other pads, active links, or |
| 150 |
even controls. Finding a combination of formats on all pads in a video |
| 151 |
pipeline, acceptable to both application and driver, can't rely on |
| 152 |
formats enumeration only. A format negotiation mechanism is required. |
| 153 |
|
| 154 |
Central to the format negotiation mechanism are the get/set format |
| 155 |
operations. When called with the ``which`` argument set to |
| 156 |
:ref:`V4L2_SUBDEV_FORMAT_TRY <VIDIOC_SUBDEV_G_FMT>`, the |
| 157 |
:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` and |
| 158 |
:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls operate on |
| 159 |
a set of formats parameters that are not connected to the hardware |
| 160 |
configuration. Modifying those 'try' formats leaves the device state |
| 161 |
untouched (this applies to both the software state stored in the driver |
| 162 |
and the hardware state stored in the device itself). |
| 163 |
|
| 164 |
While not kept as part of the device state, try formats are stored in |
| 165 |
the sub-device file handles. A |
| 166 |
:ref:`VIDIOC_SUBDEV_G_FMT <VIDIOC_SUBDEV_G_FMT>` call will return |
| 167 |
the last try format set *on the same sub-device file handle*. Several |
| 168 |
applications querying the same sub-device at the same time will thus not |
| 169 |
interact with each other. |
| 170 |
|
| 171 |
To find out whether a particular format is supported by the device, |
| 172 |
applications use the |
| 173 |
:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctl. Drivers |
| 174 |
verify and, if needed, change the requested ``format`` based on device |
| 175 |
requirements and return the possibly modified value. Applications can |
| 176 |
then choose to try a different format or accept the returned value and |
| 177 |
continue. |
| 178 |
|
| 179 |
Formats returned by the driver during a negotiation iteration are |
| 180 |
guaranteed to be supported by the device. In particular, drivers |
| 181 |
guarantee that a returned format will not be further changed if passed |
| 182 |
to an :ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` call as-is |
| 183 |
(as long as external parameters, such as formats on other pads or links' |
| 184 |
configuration are not changed). |
| 185 |
|
| 186 |
Drivers automatically propagate formats inside sub-devices. When a try |
| 187 |
or active format is set on a pad, corresponding formats on other pads of |
| 188 |
the same sub-device can be modified by the driver. Drivers are free to |
| 189 |
modify formats as required by the device. However, they should comply |
| 190 |
with the following rules when possible: |
| 191 |
|
| 192 |
- Formats should be propagated from sink pads to source pads. Modifying |
| 193 |
a format on a source pad should not modify the format on any sink |
| 194 |
pad. |
| 195 |
|
| 196 |
- Sub-devices that scale frames using variable scaling factors should |
| 197 |
reset the scale factors to default values when sink pads formats are |
| 198 |
modified. If the 1:1 scaling ratio is supported, this means that |
| 199 |
source pads formats should be reset to the sink pads formats. |
| 200 |
|
| 201 |
Formats are not propagated across links, as that would involve |
| 202 |
propagating them from one sub-device file handle to another. |
| 203 |
Applications must then take care to configure both ends of every link |
| 204 |
explicitly with compatible formats. Identical formats on the two ends of |
| 205 |
a link are guaranteed to be compatible. Drivers are free to accept |
| 206 |
different formats matching device requirements as being compatible. |
| 207 |
|
| 208 |
:ref:`sample-pipeline-config` shows a sample configuration sequence |
| 209 |
for the pipeline described in :ref:`pipeline-scaling` (table columns |
| 210 |
list entity names and pad numbers). |
| 211 |
|
| 212 |
|
| 213 |
.. raw:: latex |
| 214 |
|
| 215 |
\begingroup |
| 216 |
\scriptsize |
| 217 |
\setlength{\tabcolsep}{2pt} |
| 218 |
|
| 219 |
.. tabularcolumns:: |p{2.0cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}|p{2.1cm}| |
| 220 |
|
| 221 |
.. _sample-pipeline-config: |
| 222 |
|
| 223 |
.. flat-table:: Sample Pipeline Configuration |
| 224 |
:header-rows: 1 |
| 225 |
:stub-columns: 0 |
| 226 |
:widths: 5 5 5 5 5 5 5 |
| 227 |
|
| 228 |
* - |
| 229 |
- Sensor/0 |
| 230 |
|
| 231 |
format |
| 232 |
- Frontend/0 |
| 233 |
|
| 234 |
format |
| 235 |
- Frontend/1 |
| 236 |
|
| 237 |
format |
| 238 |
- Scaler/0 |
| 239 |
|
| 240 |
format |
| 241 |
- Scaler/0 |
| 242 |
|
| 243 |
compose selection rectangle |
| 244 |
- Scaler/1 |
| 245 |
|
| 246 |
format |
| 247 |
* - Initial state |
| 248 |
- 2048x1536 |
| 249 |
|
| 250 |
SGRBG8_1X8 |
| 251 |
- (default) |
| 252 |
- (default) |
| 253 |
- (default) |
| 254 |
- (default) |
| 255 |
- (default) |
| 256 |
* - Configure frontend sink format |
| 257 |
- 2048x1536 |
| 258 |
|
| 259 |
SGRBG8_1X8 |
| 260 |
- *2048x1536* |
| 261 |
|
| 262 |
*SGRBG8_1X8* |
| 263 |
- *2046x1534* |
| 264 |
|
| 265 |
*SGRBG8_1X8* |
| 266 |
- (default) |
| 267 |
- (default) |
| 268 |
- (default) |
| 269 |
* - Configure scaler sink format |
| 270 |
- 2048x1536 |
| 271 |
|
| 272 |
SGRBG8_1X8 |
| 273 |
- 2048x1536 |
| 274 |
|
| 275 |
SGRBG8_1X8 |
| 276 |
- 2046x1534 |
| 277 |
|
| 278 |
SGRBG8_1X8 |
| 279 |
- *2046x1534* |
| 280 |
|
| 281 |
*SGRBG8_1X8* |
| 282 |
- *0,0/2046x1534* |
| 283 |
- *2046x1534* |
| 284 |
|
| 285 |
*SGRBG8_1X8* |
| 286 |
* - Configure scaler sink compose selection |
| 287 |
- 2048x1536 |
| 288 |
|
| 289 |
SGRBG8_1X8 |
| 290 |
- 2048x1536 |
| 291 |
|
| 292 |
SGRBG8_1X8 |
| 293 |
- 2046x1534 |
| 294 |
|
| 295 |
SGRBG8_1X8 |
| 296 |
- 2046x1534 |
| 297 |
|
| 298 |
SGRBG8_1X8 |
| 299 |
- *0,0/1280x960* |
| 300 |
- *1280x960* |
| 301 |
|
| 302 |
*SGRBG8_1X8* |
| 303 |
|
| 304 |
.. raw:: latex |
| 305 |
|
| 306 |
\endgroup |
| 307 |
|
| 308 |
1. Initial state. The sensor source pad format is set to its native 3MP |
| 309 |
size and V4L2_MBUS_FMT_SGRBG8_1X8 media bus code. Formats on the |
| 310 |
host frontend and scaler sink and source pads have the default |
| 311 |
values, as well as the compose rectangle on the scaler's sink pad. |
| 312 |
|
| 313 |
2. The application configures the frontend sink pad format's size to |
| 314 |
2048x1536 and its media bus code to V4L2_MBUS_FMT_SGRBG_1X8. The |
| 315 |
driver propagates the format to the frontend source pad. |
| 316 |
|
| 317 |
3. The application configures the scaler sink pad format's size to |
| 318 |
2046x1534 and the media bus code to V4L2_MBUS_FMT_SGRBG_1X8 to |
| 319 |
match the frontend source size and media bus code. The media bus code |
| 320 |
on the sink pad is set to V4L2_MBUS_FMT_SGRBG_1X8. The driver |
| 321 |
propagates the size to the compose selection rectangle on the |
| 322 |
scaler's sink pad, and the format to the scaler source pad. |
| 323 |
|
| 324 |
4. The application configures the size of the compose selection |
| 325 |
rectangle of the scaler's sink pad 1280x960. The driver propagates |
| 326 |
the size to the scaler's source pad format. |
| 327 |
|
| 328 |
When satisfied with the try results, applications can set the active |
| 329 |
formats by setting the ``which`` argument to |
| 330 |
``V4L2_SUBDEV_FORMAT_ACTIVE``. Active formats are changed exactly as try |
| 331 |
formats by drivers. To avoid modifying the hardware state during format |
| 332 |
negotiation, applications should negotiate try formats first and then |
| 333 |
modify the active settings using the try formats returned during the |
| 334 |
last negotiation iteration. This guarantees that the active format will |
| 335 |
be applied as-is by the driver without being modified. |
| 336 |
|
| 337 |
|
| 338 |
.. _v4l2-subdev-selections: |
| 339 |
|
| 340 |
Selections: cropping, scaling and composition |
| 341 |
--------------------------------------------- |
| 342 |
|
| 343 |
Many sub-devices support cropping frames on their input or output pads |
| 344 |
(or possible even on both). Cropping is used to select the area of |
| 345 |
interest in an image, typically on an image sensor or a video decoder. |
| 346 |
It can also be used as part of digital zoom implementations to select |
| 347 |
the area of the image that will be scaled up. |
| 348 |
|
| 349 |
Crop settings are defined by a crop rectangle and represented in a |
| 350 |
struct :c:type:`v4l2_rect` by the coordinates of the top |
| 351 |
left corner and the rectangle size. Both the coordinates and sizes are |
| 352 |
expressed in pixels. |
| 353 |
|
| 354 |
As for pad formats, drivers store try and active rectangles for the |
| 355 |
selection targets :ref:`v4l2-selections-common`. |
| 356 |
|
| 357 |
On sink pads, cropping is applied relative to the current pad format. |
| 358 |
The pad format represents the image size as received by the sub-device |
| 359 |
from the previous block in the pipeline, and the crop rectangle |
| 360 |
represents the sub-image that will be transmitted further inside the |
| 361 |
sub-device for processing. |
| 362 |
|
| 363 |
The scaling operation changes the size of the image by scaling it to new |
| 364 |
dimensions. The scaling ratio isn't specified explicitly, but is implied |
| 365 |
from the original and scaled image sizes. Both sizes are represented by |
| 366 |
struct :c:type:`v4l2_rect`. |
| 367 |
|
| 368 |
Scaling support is optional. When supported by a subdev, the crop |
| 369 |
rectangle on the subdev's sink pad is scaled to the size configured |
| 370 |
using the |
| 371 |
:ref:`VIDIOC_SUBDEV_S_SELECTION <VIDIOC_SUBDEV_G_SELECTION>` IOCTL |
| 372 |
using ``V4L2_SEL_TGT_COMPOSE`` selection target on the same pad. If the |
| 373 |
subdev supports scaling but not composing, the top and left values are |
| 374 |
not used and must always be set to zero. |
| 375 |
|
| 376 |
On source pads, cropping is similar to sink pads, with the exception |
| 377 |
that the source size from which the cropping is performed, is the |
| 378 |
COMPOSE rectangle on the sink pad. In both sink and source pads, the |
| 379 |
crop rectangle must be entirely contained inside the source image size |
| 380 |
for the crop operation. |
| 381 |
|
| 382 |
The drivers should always use the closest possible rectangle the user |
| 383 |
requests on all selection targets, unless specifically told otherwise. |
| 384 |
``V4L2_SEL_FLAG_GE`` and ``V4L2_SEL_FLAG_LE`` flags may be used to round |
| 385 |
the image size either up or down. :ref:`v4l2-selection-flags` |
| 386 |
|
| 387 |
|
| 388 |
Types of selection targets |
| 389 |
-------------------------- |
| 390 |
|
| 391 |
|
| 392 |
Actual targets |
| 393 |
^^^^^^^^^^^^^^ |
| 394 |
|
| 395 |
Actual targets (without a postfix) reflect the actual hardware |
| 396 |
configuration at any point of time. There is a BOUNDS target |
| 397 |
corresponding to every actual target. |
| 398 |
|
| 399 |
|
| 400 |
BOUNDS targets |
| 401 |
^^^^^^^^^^^^^^ |
| 402 |
|
| 403 |
BOUNDS targets is the smallest rectangle that contains all valid actual |
| 404 |
rectangles. It may not be possible to set the actual rectangle as large |
| 405 |
as the BOUNDS rectangle, however. This may be because e.g. a sensor's |
| 406 |
pixel array is not rectangular but cross-shaped or round. The maximum |
| 407 |
size may also be smaller than the BOUNDS rectangle. |
| 408 |
|
| 409 |
|
| 410 |
.. _format-propagation: |
| 411 |
|
| 412 |
Order of configuration and format propagation |
| 413 |
--------------------------------------------- |
| 414 |
|
| 415 |
Inside subdevs, the order of image processing steps will always be from |
| 416 |
the sink pad towards the source pad. This is also reflected in the order |
| 417 |
in which the configuration must be performed by the user: the changes |
| 418 |
made will be propagated to any subsequent stages. If this behaviour is |
| 419 |
not desired, the user must set ``V4L2_SEL_FLAG_KEEP_CONFIG`` flag. This |
| 420 |
flag causes no propagation of the changes are allowed in any |
| 421 |
circumstances. This may also cause the accessed rectangle to be adjusted |
| 422 |
by the driver, depending on the properties of the underlying hardware. |
| 423 |
|
| 424 |
The coordinates to a step always refer to the actual size of the |
| 425 |
previous step. The exception to this rule is the sink compose |
| 426 |
rectangle, which refers to the sink compose bounds rectangle --- if it |
| 427 |
is supported by the hardware. |
| 428 |
|
| 429 |
1. Sink pad format. The user configures the sink pad format. This format |
| 430 |
defines the parameters of the image the entity receives through the |
| 431 |
pad for further processing. |
| 432 |
|
| 433 |
2. Sink pad actual crop selection. The sink pad crop defines the crop |
| 434 |
performed to the sink pad format. |
| 435 |
|
| 436 |
3. Sink pad actual compose selection. The size of the sink pad compose |
| 437 |
rectangle defines the scaling ratio compared to the size of the sink |
| 438 |
pad crop rectangle. The location of the compose rectangle specifies |
| 439 |
the location of the actual sink compose rectangle in the sink compose |
| 440 |
bounds rectangle. |
| 441 |
|
| 442 |
4. Source pad actual crop selection. Crop on the source pad defines crop |
| 443 |
performed to the image in the sink compose bounds rectangle. |
| 444 |
|
| 445 |
5. Source pad format. The source pad format defines the output pixel |
| 446 |
format of the subdev, as well as the other parameters with the |
| 447 |
exception of the image width and height. Width and height are defined |
| 448 |
by the size of the source pad actual crop selection. |
| 449 |
|
| 450 |
Accessing any of the above rectangles not supported by the subdev will |
| 451 |
return ``EINVAL``. Any rectangle referring to a previous unsupported |
| 452 |
rectangle coordinates will instead refer to the previous supported |
| 453 |
rectangle. For example, if sink crop is not supported, the compose |
| 454 |
selection will refer to the sink pad format dimensions instead. |
| 455 |
|
| 456 |
|
| 457 |
.. _subdev-image-processing-crop: |
| 458 |
|
| 459 |
.. kernel-figure:: subdev-image-processing-crop.svg |
| 460 |
:alt: subdev-image-processing-crop.svg |
| 461 |
:align: center |
| 462 |
|
| 463 |
Image processing in subdevs: simple crop example |
| 464 |
|
| 465 |
In the above example, the subdev supports cropping on its sink pad. To |
| 466 |
configure it, the user sets the media bus format on the subdev's sink |
| 467 |
pad. Now the actual crop rectangle can be set on the sink pad --- the |
| 468 |
location and size of this rectangle reflect the location and size of a |
| 469 |
rectangle to be cropped from the sink format. The size of the sink crop |
| 470 |
rectangle will also be the size of the format of the subdev's source |
| 471 |
pad. |
| 472 |
|
| 473 |
|
| 474 |
.. _subdev-image-processing-scaling-multi-source: |
| 475 |
|
| 476 |
.. kernel-figure:: subdev-image-processing-scaling-multi-source.svg |
| 477 |
:alt: subdev-image-processing-scaling-multi-source.svg |
| 478 |
:align: center |
| 479 |
|
| 480 |
Image processing in subdevs: scaling with multiple sources |
| 481 |
|
| 482 |
In this example, the subdev is capable of first cropping, then scaling |
| 483 |
and finally cropping for two source pads individually from the resulting |
| 484 |
scaled image. The location of the scaled image in the cropped image is |
| 485 |
ignored in sink compose target. Both of the locations of the source crop |
| 486 |
rectangles refer to the sink scaling rectangle, independently cropping |
| 487 |
an area at location specified by the source crop rectangle from it. |
| 488 |
|
| 489 |
|
| 490 |
.. _subdev-image-processing-full: |
| 491 |
|
| 492 |
.. kernel-figure:: subdev-image-processing-full.svg |
| 493 |
:alt: subdev-image-processing-full.svg |
| 494 |
:align: center |
| 495 |
|
| 496 |
Image processing in subdevs: scaling and composition with multiple sinks and sources |
| 497 |
|
| 498 |
The subdev driver supports two sink pads and two source pads. The images |
| 499 |
from both of the sink pads are individually cropped, then scaled and |
| 500 |
further composed on the composition bounds rectangle. From that, two |
| 501 |
independent streams are cropped and sent out of the subdev from the |
| 502 |
source pads. |
| 503 |
|
| 504 |
|
| 505 |
.. toctree:: |
| 506 |
:maxdepth: 1 |
| 507 |
|
| 508 |
subdev-formats |
| 509 |
|
| 510 |
.. _subdev-routing: |
| 511 |
|
| 512 |
Streams, multiplexed media pads and internal routing |
| 513 |
==================================================== |
| 514 |
|
| 515 |
Simple V4L2 sub-devices do not support multiple, unrelated video streams, |
| 516 |
and only a single stream can pass through a media link and a media pad. |
| 517 |
Thus each pad contains a format and selection configuration for that |
| 518 |
single stream. A subdev can do stream processing and split a stream into |
| 519 |
two or compose two streams into one, but the inputs and outputs for the |
| 520 |
subdev are still a single stream per pad. |
| 521 |
|
| 522 |
Some hardware, e.g. MIPI CSI-2, support multiplexed streams, that is, multiple |
| 523 |
data streams are transmitted on the same bus, which is represented by a media |
| 524 |
link connecting a transmitter source pad with a sink pad on the receiver. For |
| 525 |
example, a camera sensor can produce two distinct streams, a pixel stream and a |
| 526 |
metadata stream, which are transmitted on the multiplexed data bus, represented |
| 527 |
by a media link which connects the single sensor's source pad with the receiver |
| 528 |
sink pad. The stream-aware receiver will de-multiplex the streams received on |
| 529 |
the its sink pad and allows to route them individually to one of its source |
| 530 |
pads. |
| 531 |
|
| 532 |
Subdevice drivers that support multiplexed streams are compatible with |
| 533 |
non-multiplexed subdev drivers. However, if the driver at the sink end of a link |
| 534 |
does not support streams, then only stream 0 of source end may be captured. |
| 535 |
There may be additional limitations specific to the sink device. |
| 536 |
|
| 537 |
Understanding streams |
| 538 |
--------------------- |
| 539 |
|
| 540 |
A stream is a stream of content (e.g. pixel data or metadata) flowing through |
| 541 |
the media pipeline from a source (e.g. a sensor) towards the final sink (e.g. a |
| 542 |
receiver and demultiplexer in a SoC). Each media link carries all the enabled |
| 543 |
streams from one end of the link to the other, and sub-devices have routing |
| 544 |
tables which describe how the incoming streams from sink pads are routed to the |
| 545 |
source pads. |
| 546 |
|
| 547 |
A stream ID is a media pad-local identifier for a stream. Streams IDs of |
| 548 |
the same stream must be equal on both ends of a link. In other words, |
| 549 |
a particular stream ID must exist on both sides of a media |
| 550 |
link, but another stream ID can be used for the same stream at the other side |
| 551 |
of the sub-device. |
| 552 |
|
| 553 |
A stream at a specific point in the media pipeline is identified by the |
| 554 |
sub-device and a (pad, stream) pair. For sub-devices that do not support |
| 555 |
multiplexed streams the 'stream' field is always 0. |
| 556 |
|
| 557 |
Interaction between routes, streams, formats and selections |
| 558 |
----------------------------------------------------------- |
| 559 |
|
| 560 |
The addition of streams to the V4L2 sub-device interface moves the sub-device |
| 561 |
formats and selections from pads to (pad, stream) pairs. Besides the |
| 562 |
usual pad, also the stream ID needs to be provided for setting formats and |
| 563 |
selections. The order of configuring formats and selections along a stream is |
| 564 |
the same as without streams (see :ref:`format-propagation`). |
| 565 |
|
| 566 |
Instead of the sub-device wide merging of streams from all sink pads |
| 567 |
towards all source pads, data flows for each route are separate from each |
| 568 |
other. Any number of routes from streams on sink pads towards streams on |
| 569 |
source pads is allowed, to the extent supported by drivers. For every |
| 570 |
stream on a source pad, however, only a single route is allowed. |
| 571 |
|
| 572 |
Any configurations of a stream within a pad, such as format or selections, |
| 573 |
are independent of similar configurations on other streams. This is |
| 574 |
subject to change in the future. |
| 575 |
|
| 576 |
Device types and routing setup |
| 577 |
------------------------------ |
| 578 |
|
| 579 |
Different kinds of sub-devices have differing behaviour for route activation, |
| 580 |
depending on the hardware. In all cases, however, only routes that have the |
| 581 |
``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag set are active. |
| 582 |
|
| 583 |
Devices generating the streams may allow enabling and disabling some of the |
| 584 |
routes or have a fixed routing configuration. If the routes can be disabled, not |
| 585 |
declaring the routes (or declaring them without ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` |
| 586 |
flag set) in ``VIDIOC_SUBDEV_S_ROUTING`` will disable the routes. |
| 587 |
``VIDIOC_SUBDEV_S_ROUTING`` will still return such routes back to the user in |
| 588 |
the routes array, with the ``V4L2_SUBDEV_ROUTE_FL_ACTIVE`` flag unset. |
| 589 |
|
| 590 |
Devices transporting the streams almost always have more configurability with |
| 591 |
respect to routing. Typically any route between the sub-device's sink and source |
| 592 |
pads is possible, and multiple routes (usually up to certain limited number) may |
| 593 |
be active simultaneously. For such devices, no routes are created by the driver |
| 594 |
and user-created routes are fully replaced when ``VIDIOC_SUBDEV_S_ROUTING`` is |
| 595 |
called on the sub-device. Such newly created routes have the device's default |
| 596 |
configuration for format and selection rectangles. |
| 597 |
|
| 598 |
Configuring streams |
| 599 |
------------------- |
| 600 |
|
| 601 |
The configuration of the streams is done individually for each sub-device and |
| 602 |
the validity of the streams between sub-devices is validated when the pipeline |
| 603 |
is started. |
| 604 |
|
| 605 |
There are three steps in configuring the streams: |
| 606 |
|
| 607 |
1. Set up links. Connect the pads between sub-devices using the |
| 608 |
:ref:`Media Controller API <media_controller>` |
| 609 |
|
| 610 |
2. Streams. Streams are declared and their routing is configured by setting the |
| 611 |
routing table for the sub-device using :ref:`VIDIOC_SUBDEV_S_ROUTING |
| 612 |
<VIDIOC_SUBDEV_G_ROUTING>` ioctl. Note that setting the routing table will |
| 613 |
reset formats and selections in the sub-device to default values. |
| 614 |
|
| 615 |
3. Configure formats and selections. Formats and selections of each stream are |
| 616 |
configured separately as documented for plain sub-devices in |
| 617 |
:ref:`format-propagation`. The stream ID is set to the same stream ID |
| 618 |
associated with either sink or source pads of routes configured using the |
| 619 |
:ref:`VIDIOC_SUBDEV_S_ROUTING <VIDIOC_SUBDEV_G_ROUTING>` ioctl. |
| 620 |
|
| 621 |
Multiplexed streams setup example |
| 622 |
--------------------------------- |
| 623 |
|
| 624 |
A simple example of a multiplexed stream setup might be as follows: |
| 625 |
|
| 626 |
- Two identical sensors (Sensor A and Sensor B). Each sensor has a single source |
| 627 |
pad (pad 0) which carries a pixel data stream. |
| 628 |
|
| 629 |
- Multiplexer bridge (Bridge). The bridge has two sink pads, connected to the |
| 630 |
sensors (pads 0, 1), and one source pad (pad 2), which outputs two streams. |
| 631 |
|
| 632 |
- Receiver in the SoC (Receiver). The receiver has a single sink pad (pad 0), |
| 633 |
connected to the bridge, and two source pads (pads 1-2), going to the DMA |
| 634 |
engine. The receiver demultiplexes the incoming streams to the source pads. |
| 635 |
|
| 636 |
- DMA Engines in the SoC (DMA Engine), one for each stream. Each DMA engine is |
| 637 |
connected to a single source pad in the receiver. |
| 638 |
|
| 639 |
The sensors, the bridge and the receiver are modeled as V4L2 sub-devices, |
| 640 |
exposed to userspace via /dev/v4l-subdevX device nodes. The DMA engines are |
| 641 |
modeled as V4L2 devices, exposed to userspace via /dev/videoX nodes. |
| 642 |
|
| 643 |
To configure this pipeline, the userspace must take the following steps: |
| 644 |
|
| 645 |
1. Set up media links between entities: connect the sensors to the bridge, |
| 646 |
bridge to the receiver, and the receiver to the DMA engines. This step does |
| 647 |
not differ from normal non-multiplexed media controller setup. |
| 648 |
|
| 649 |
2. Configure routing |
| 650 |
|
| 651 |
.. flat-table:: Bridge routing table |
| 652 |
:header-rows: 1 |
| 653 |
|
| 654 |
* - Sink Pad/Stream |
| 655 |
- Source Pad/Stream |
| 656 |
- Routing Flags |
| 657 |
- Comments |
| 658 |
* - 0/0 |
| 659 |
- 2/0 |
| 660 |
- V4L2_SUBDEV_ROUTE_FL_ACTIVE |
| 661 |
- Pixel data stream from Sensor A |
| 662 |
* - 1/0 |
| 663 |
- 2/1 |
| 664 |
- V4L2_SUBDEV_ROUTE_FL_ACTIVE |
| 665 |
- Pixel data stream from Sensor B |
| 666 |
|
| 667 |
.. flat-table:: Receiver routing table |
| 668 |
:header-rows: 1 |
| 669 |
|
| 670 |
* - Sink Pad/Stream |
| 671 |
- Source Pad/Stream |
| 672 |
- Routing Flags |
| 673 |
- Comments |
| 674 |
* - 0/0 |
| 675 |
- 1/0 |
| 676 |
- V4L2_SUBDEV_ROUTE_FL_ACTIVE |
| 677 |
- Pixel data stream from Sensor A |
| 678 |
* - 0/1 |
| 679 |
- 2/0 |
| 680 |
- V4L2_SUBDEV_ROUTE_FL_ACTIVE |
| 681 |
- Pixel data stream from Sensor B |
| 682 |
|
| 683 |
3. Configure formats and selections |
| 684 |
|
| 685 |
After configuring routing, the next step is configuring the formats and |
| 686 |
selections for the streams. This is similar to performing this step without |
| 687 |
streams, with just one exception: the ``stream`` field needs to be assigned |
| 688 |
to the value of the stream ID. |
| 689 |
|
| 690 |
A common way to accomplish this is to start from the sensors and propagate |
| 691 |
the configurations along the stream towards the receiver, using |
| 692 |
:ref:`VIDIOC_SUBDEV_S_FMT <VIDIOC_SUBDEV_G_FMT>` ioctls to configure each |
| 693 |
stream endpoint in each sub-device. |