| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
Media Controller devices |
| 4 |
------------------------ |
| 5 |
|
| 6 |
Media Controller |
| 7 |
~~~~~~~~~~~~~~~~ |
| 8 |
|
| 9 |
The media controller userspace API is documented in |
| 10 |
:ref:`the Media Controller uAPI book <media_controller>`. This document focus |
| 11 |
on the kernel-side implementation of the media framework. |
| 12 |
|
| 13 |
Abstract media device model |
| 14 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 15 |
|
| 16 |
Discovering a device internal topology, and configuring it at runtime, is one |
| 17 |
of the goals of the media framework. To achieve this, hardware devices are |
| 18 |
modelled as an oriented graph of building blocks called entities connected |
| 19 |
through pads. |
| 20 |
|
| 21 |
An entity is a basic media hardware building block. It can correspond to |
| 22 |
a large variety of logical blocks such as physical hardware devices |
| 23 |
(CMOS sensor for instance), logical hardware devices (a building block |
| 24 |
in a System-on-Chip image processing pipeline), DMA channels or physical |
| 25 |
connectors. |
| 26 |
|
| 27 |
A pad is a connection endpoint through which an entity can interact with |
| 28 |
other entities. Data (not restricted to video) produced by an entity |
| 29 |
flows from the entity's output to one or more entity inputs. Pads should |
| 30 |
not be confused with physical pins at chip boundaries. |
| 31 |
|
| 32 |
A link is a point-to-point oriented connection between two pads, either |
| 33 |
on the same entity or on different entities. Data flows from a source |
| 34 |
pad to a sink pad. |
| 35 |
|
| 36 |
Media device |
| 37 |
^^^^^^^^^^^^ |
| 38 |
|
| 39 |
A media device is represented by a struct media_device |
| 40 |
instance, defined in ``include/media/media-device.h``. |
| 41 |
Allocation of the structure is handled by the media device driver, usually by |
| 42 |
embedding the :c:type:`media_device` instance in a larger driver-specific |
| 43 |
structure. |
| 44 |
|
| 45 |
Drivers initialise media device instances by calling |
| 46 |
:c:func:`media_device_init()`. After initialising a media device instance, it is |
| 47 |
registered by calling :c:func:`__media_device_register()` via the macro |
| 48 |
``media_device_register()`` and unregistered by calling |
| 49 |
:c:func:`media_device_unregister()`. An initialised media device must be |
| 50 |
eventually cleaned up by calling :c:func:`media_device_cleanup()`. |
| 51 |
|
| 52 |
Note that it is not allowed to unregister a media device instance that was not |
| 53 |
previously registered, or clean up a media device instance that was not |
| 54 |
previously initialised. |
| 55 |
|
| 56 |
Entities |
| 57 |
^^^^^^^^ |
| 58 |
|
| 59 |
Entities are represented by a struct media_entity |
| 60 |
instance, defined in ``include/media/media-entity.h``. The structure is usually |
| 61 |
embedded into a higher-level structure, such as |
| 62 |
:c:type:`v4l2_subdev` or :c:type:`video_device` |
| 63 |
instances, although drivers can allocate entities directly. |
| 64 |
|
| 65 |
Drivers initialize entity pads by calling |
| 66 |
:c:func:`media_entity_pads_init()`. |
| 67 |
|
| 68 |
Drivers register entities with a media device by calling |
| 69 |
:c:func:`media_device_register_entity()` |
| 70 |
and unregistered by calling |
| 71 |
:c:func:`media_device_unregister_entity()`. |
| 72 |
|
| 73 |
Interfaces |
| 74 |
^^^^^^^^^^ |
| 75 |
|
| 76 |
Interfaces are represented by a |
| 77 |
struct media_interface instance, defined in |
| 78 |
``include/media/media-entity.h``. Currently, only one type of interface is |
| 79 |
defined: a device node. Such interfaces are represented by a |
| 80 |
struct media_intf_devnode. |
| 81 |
|
| 82 |
Drivers initialize and create device node interfaces by calling |
| 83 |
:c:func:`media_devnode_create()` |
| 84 |
and remove them by calling: |
| 85 |
:c:func:`media_devnode_remove()`. |
| 86 |
|
| 87 |
Pads |
| 88 |
^^^^ |
| 89 |
Pads are represented by a struct media_pad instance, |
| 90 |
defined in ``include/media/media-entity.h``. Each entity stores its pads in |
| 91 |
a pads array managed by the entity driver. Drivers usually embed the array in |
| 92 |
a driver-specific structure. |
| 93 |
|
| 94 |
Pads are identified by their entity and their 0-based index in the pads |
| 95 |
array. |
| 96 |
|
| 97 |
Both information are stored in the struct media_pad, |
| 98 |
making the struct media_pad pointer the canonical way |
| 99 |
to store and pass link references. |
| 100 |
|
| 101 |
Pads have flags that describe the pad capabilities and state. |
| 102 |
|
| 103 |
``MEDIA_PAD_FL_SINK`` indicates that the pad supports sinking data. |
| 104 |
``MEDIA_PAD_FL_SOURCE`` indicates that the pad supports sourcing data. |
| 105 |
|
| 106 |
.. note:: |
| 107 |
|
| 108 |
One and only one of ``MEDIA_PAD_FL_SINK`` or ``MEDIA_PAD_FL_SOURCE`` must |
| 109 |
be set for each pad. |
| 110 |
|
| 111 |
Links |
| 112 |
^^^^^ |
| 113 |
|
| 114 |
Links are represented by a struct media_link instance, |
| 115 |
defined in ``include/media/media-entity.h``. There are two types of links: |
| 116 |
|
| 117 |
**1. pad to pad links**: |
| 118 |
|
| 119 |
Associate two entities via their PADs. Each entity has a list that points |
| 120 |
to all links originating at or targeting any of its pads. |
| 121 |
A given link is thus stored twice, once in the source entity and once in |
| 122 |
the target entity. |
| 123 |
|
| 124 |
Drivers create pad to pad links by calling: |
| 125 |
:c:func:`media_create_pad_link()` and remove with |
| 126 |
:c:func:`media_entity_remove_links()`. |
| 127 |
|
| 128 |
**2. interface to entity links**: |
| 129 |
|
| 130 |
Associate one interface to a Link. |
| 131 |
|
| 132 |
Drivers create interface to entity links by calling: |
| 133 |
:c:func:`media_create_intf_link()` and remove with |
| 134 |
:c:func:`media_remove_intf_links()`. |
| 135 |
|
| 136 |
.. note:: |
| 137 |
|
| 138 |
Links can only be created after having both ends already created. |
| 139 |
|
| 140 |
Links have flags that describe the link capabilities and state. The |
| 141 |
valid values are described at :c:func:`media_create_pad_link()` and |
| 142 |
:c:func:`media_create_intf_link()`. |
| 143 |
|
| 144 |
Graph traversal |
| 145 |
^^^^^^^^^^^^^^^ |
| 146 |
|
| 147 |
The media framework provides APIs to traverse media graphs, locating connected |
| 148 |
entities and links. |
| 149 |
|
| 150 |
To iterate over all entities belonging to a media device, drivers can use |
| 151 |
the media_device_for_each_entity macro, defined in |
| 152 |
``include/media/media-device.h``. |
| 153 |
|
| 154 |
.. code-block:: c |
| 155 |
|
| 156 |
struct media_entity *entity; |
| 157 |
|
| 158 |
media_device_for_each_entity(entity, mdev) { |
| 159 |
// entity will point to each entity in turn |
| 160 |
... |
| 161 |
} |
| 162 |
|
| 163 |
Helper functions can be used to find a link between two given pads, or a pad |
| 164 |
connected to another pad through an enabled link |
| 165 |
(:c:func:`media_entity_find_link()`, :c:func:`media_pad_remote_pad_first()`, |
| 166 |
:c:func:`media_entity_remote_source_pad_unique()` and |
| 167 |
:c:func:`media_pad_remote_pad_unique()`). |
| 168 |
|
| 169 |
Use count and power handling |
| 170 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 171 |
|
| 172 |
Due to the wide differences between drivers regarding power management |
| 173 |
needs, the media controller does not implement power management. However, |
| 174 |
the struct media_entity includes a ``use_count`` |
| 175 |
field that media drivers |
| 176 |
can use to track the number of users of every entity for power management |
| 177 |
needs. |
| 178 |
|
| 179 |
The :c:type:`media_entity<media_entity>`.\ ``use_count`` field is owned by |
| 180 |
media drivers and must not be |
| 181 |
touched by entity drivers. Access to the field must be protected by the |
| 182 |
:c:type:`media_device`.\ ``graph_mutex`` lock. |
| 183 |
|
| 184 |
Links setup |
| 185 |
^^^^^^^^^^^ |
| 186 |
|
| 187 |
Link properties can be modified at runtime by calling |
| 188 |
:c:func:`media_entity_setup_link()`. |
| 189 |
|
| 190 |
Pipelines and media streams |
| 191 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 192 |
|
| 193 |
A media stream is a stream of pixels or metadata originating from one or more |
| 194 |
source devices (such as a sensors) and flowing through media entity pads |
| 195 |
towards the final sinks. The stream can be modified on the route by the |
| 196 |
devices (e.g. scaling or pixel format conversions), or it can be split into |
| 197 |
multiple branches, or multiple branches can be merged. |
| 198 |
|
| 199 |
A media pipeline is a set of media streams which are interdependent. This |
| 200 |
interdependency can be caused by the hardware (e.g. configuration of a second |
| 201 |
stream cannot be changed if the first stream has been enabled) or by the driver |
| 202 |
due to the software design. Most commonly a media pipeline consists of a single |
| 203 |
stream which does not branch. |
| 204 |
|
| 205 |
When starting streaming, drivers must notify all entities in the pipeline to |
| 206 |
prevent link states from being modified during streaming by calling |
| 207 |
:c:func:`media_pipeline_start()`. |
| 208 |
|
| 209 |
The function will mark all the pads which are part of the pipeline as streaming. |
| 210 |
|
| 211 |
The struct media_pipeline instance pointed to by the pipe argument will be |
| 212 |
stored in every pad in the pipeline. Drivers should embed the struct |
| 213 |
media_pipeline in higher-level pipeline structures and can then access the |
| 214 |
pipeline through the struct media_pad pipe field. |
| 215 |
|
| 216 |
Calls to :c:func:`media_pipeline_start()` can be nested. |
| 217 |
The pipeline pointer must be identical for all nested calls to the function. |
| 218 |
|
| 219 |
:c:func:`media_pipeline_start()` may return an error. In that case, |
| 220 |
it will clean up any of the changes it did by itself. |
| 221 |
|
| 222 |
When stopping the stream, drivers must notify the entities with |
| 223 |
:c:func:`media_pipeline_stop()`. |
| 224 |
|
| 225 |
If multiple calls to :c:func:`media_pipeline_start()` have been |
| 226 |
made the same number of :c:func:`media_pipeline_stop()` calls |
| 227 |
are required to stop streaming. |
| 228 |
The :c:type:`media_entity`.\ ``pipe`` field is reset to ``NULL`` on the last |
| 229 |
nested stop call. |
| 230 |
|
| 231 |
Link configuration will fail with ``-EBUSY`` by default if either end of the |
| 232 |
link is a streaming entity. Links that can be modified while streaming must |
| 233 |
be marked with the ``MEDIA_LNK_FL_DYNAMIC`` flag. |
| 234 |
|
| 235 |
If other operations need to be disallowed on streaming entities (such as |
| 236 |
changing entities configuration parameters) drivers can explicitly check the |
| 237 |
media_entity stream_count field to find out if an entity is streaming. This |
| 238 |
operation must be done with the media_device graph_mutex held. |
| 239 |
|
| 240 |
Link validation |
| 241 |
^^^^^^^^^^^^^^^ |
| 242 |
|
| 243 |
Link validation is performed by :c:func:`media_pipeline_start()` |
| 244 |
for any entity which has sink pads in the pipeline. The |
| 245 |
:c:type:`media_entity`.\ ``link_validate()`` callback is used for that |
| 246 |
purpose. In ``link_validate()`` callback, entity driver should check |
| 247 |
that the properties of the source pad of the connected entity and its own |
| 248 |
sink pad match. It is up to the type of the entity (and in the end, the |
| 249 |
properties of the hardware) what matching actually means. |
| 250 |
|
| 251 |
Subsystems should facilitate link validation by providing subsystem specific |
| 252 |
helper functions to provide easy access for commonly needed information, and |
| 253 |
in the end provide a way to use driver-specific callbacks. |
| 254 |
|
| 255 |
Pipeline traversal |
| 256 |
^^^^^^^^^^^^^^^^^^ |
| 257 |
|
| 258 |
Once a pipeline has been constructed with :c:func:`media_pipeline_start()`, |
| 259 |
drivers can iterate over entities or pads in the pipeline with the |
| 260 |
:c:macro:´media_pipeline_for_each_entity` and |
| 261 |
:c:macro:´media_pipeline_for_each_pad` macros. Iterating over pads is |
| 262 |
straightforward: |
| 263 |
|
| 264 |
.. code-block:: c |
| 265 |
|
| 266 |
media_pipeline_pad_iter iter; |
| 267 |
struct media_pad *pad; |
| 268 |
|
| 269 |
media_pipeline_for_each_pad(pipe, &iter, pad) { |
| 270 |
/* 'pad' will point to each pad in turn */ |
| 271 |
... |
| 272 |
} |
| 273 |
|
| 274 |
To iterate over entities, the iterator needs to be initialized and cleaned up |
| 275 |
as an additional steps: |
| 276 |
|
| 277 |
.. code-block:: c |
| 278 |
|
| 279 |
media_pipeline_entity_iter iter; |
| 280 |
struct media_entity *entity; |
| 281 |
int ret; |
| 282 |
|
| 283 |
ret = media_pipeline_entity_iter_init(pipe, &iter); |
| 284 |
if (ret) |
| 285 |
...; |
| 286 |
|
| 287 |
media_pipeline_for_each_entity(pipe, &iter, entity) { |
| 288 |
/* 'entity' will point to each entity in turn */ |
| 289 |
... |
| 290 |
} |
| 291 |
|
| 292 |
media_pipeline_entity_iter_cleanup(&iter); |
| 293 |
|
| 294 |
Media Controller Device Allocator API |
| 295 |
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ |
| 296 |
|
| 297 |
When the media device belongs to more than one driver, the shared media |
| 298 |
device is allocated with the shared struct device as the key for look ups. |
| 299 |
|
| 300 |
The shared media device should stay in registered state until the last |
| 301 |
driver unregisters it. In addition, the media device should be released when |
| 302 |
all the references are released. Each driver gets a reference to the media |
| 303 |
device during probe, when it allocates the media device. If media device is |
| 304 |
already allocated, the allocate API bumps up the refcount and returns the |
| 305 |
existing media device. The driver puts the reference back in its disconnect |
| 306 |
routine when it calls :c:func:`media_device_delete()`. |
| 307 |
|
| 308 |
The media device is unregistered and cleaned up from the kref put handler to |
| 309 |
ensure that the media device stays in registered state until the last driver |
| 310 |
unregisters the media device. |
| 311 |
|
| 312 |
**Driver Usage** |
| 313 |
|
| 314 |
Drivers should use the appropriate media-core routines to manage the shared |
| 315 |
media device life-time handling the two states: |
| 316 |
1. allocate -> register -> delete |
| 317 |
2. get reference to already registered device -> delete |
| 318 |
|
| 319 |
call :c:func:`media_device_delete()` routine to make sure the shared media |
| 320 |
device delete is handled correctly. |
| 321 |
|
| 322 |
**driver probe:** |
| 323 |
Call :c:func:`media_device_usb_allocate()` to allocate or get a reference |
| 324 |
Call :c:func:`media_device_register()`, if media devnode isn't registered |
| 325 |
|
| 326 |
**driver disconnect:** |
| 327 |
Call :c:func:`media_device_delete()` to free the media_device. Freeing is |
| 328 |
handled by the kref put handler. |
| 329 |
|
| 330 |
API Definitions |
| 331 |
^^^^^^^^^^^^^^^ |
| 332 |
|
| 333 |
.. kernel-doc:: include/media/media-device.h |
| 334 |
|
| 335 |
.. kernel-doc:: include/media/media-devnode.h |
| 336 |
|
| 337 |
.. kernel-doc:: include/media/media-entity.h |
| 338 |
|
| 339 |
.. kernel-doc:: include/media/media-request.h |
| 340 |
|
| 341 |
.. kernel-doc:: include/media/media-dev-allocator.h |