| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
========================================== |
| 4 |
ARM Mali-C55 Image Signal Processor driver |
| 5 |
========================================== |
| 6 |
|
| 7 |
Introduction |
| 8 |
============ |
| 9 |
|
| 10 |
This file documents the driver for ARM's Mali-C55 Image Signal Processor. The |
| 11 |
driver is located under drivers/media/platform/arm/mali-c55. |
| 12 |
|
| 13 |
The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from |
| 14 |
sensors through either a parallel interface or a memory bus before processing it |
| 15 |
and outputting it through an internal DMA engine. Two output pipelines are |
| 16 |
possible (though one may not be fitted, depending on the implementation). These |
| 17 |
are referred to as "Full resolution" and "Downscale", but the naming is historic |
| 18 |
and both pipes are capable of cropping/scaling operations. The full resolution |
| 19 |
pipe is also capable of outputting RAW data, bypassing much of the ISP's |
| 20 |
processing. The downscale pipe cannot output RAW data. An integrated test |
| 21 |
pattern generator can be used to drive the ISP and produce image data in the |
| 22 |
absence of a connected camera sensor. The driver module is named mali_c55, and |
| 23 |
is enabled through the CONFIG_VIDEO_MALI_C55 config option. |
| 24 |
|
| 25 |
The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and |
| 26 |
expects camera sensors connected to the ISP to have V4L2 subdevice interfaces. |
| 27 |
|
| 28 |
Mali-C55 ISP hardware |
| 29 |
===================== |
| 30 |
|
| 31 |
A high level functional view of the Mali-C55 ISP is presented below. The ISP |
| 32 |
takes input from either a live source or through a DMA engine for memory input, |
| 33 |
depending on the SoC integration.:: |
| 34 |
|
| 35 |
+---------+ +----------+ +--------+ |
| 36 |
| Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA | |
| 37 |
+---------+ +----------+ |\ Output +--->| Writer | |
| 38 |
| | \ | +--------+ |
| 39 |
| | \ +----------+ +------+---> Streaming I/O |
| 40 |
+------------+ +------->| | | | | |
| 41 |
| | | |-->| Mali-C55 |--+ |
| 42 |
| DMA Reader |--------------->| | | ISP | | |
| 43 |
| | | / | | | +---> Streaming I/O |
| 44 |
+------------+ | / +----------+ | | |
| 45 |
|/ +------+ |
| 46 |
| +--------+ |
| 47 |
+--->| DMA | |
| 48 |
"Downscaled" | Writer | |
| 49 |
Output +--------+ |
| 50 |
|
| 51 |
Media Controller Topology |
| 52 |
========================= |
| 53 |
|
| 54 |
An example of the ISP's topology (as implemented in a system with an IMX415 |
| 55 |
camera sensor and generic CSI-2 receiver) is below: |
| 56 |
|
| 57 |
|
| 58 |
.. kernel-figure:: mali-c55-graph.dot |
| 59 |
:alt: mali-c55-graph.dot |
| 60 |
:align: center |
| 61 |
|
| 62 |
The driver has 4 V4L2 subdevices: |
| 63 |
|
| 64 |
- `mali_c55 isp`: Responsible for configuring input crop and color space |
| 65 |
conversion |
| 66 |
- `mali_c55 tpg`: The test pattern generator, emulating a camera sensor. |
| 67 |
- `mali_c55 resizer fr`: The Full-Resolution pipe resizer |
| 68 |
- `mali_c55 resizer ds`: The Downscale pipe resizer |
| 69 |
|
| 70 |
The driver has 3 V4L2 video devices: |
| 71 |
|
| 72 |
- `mali-c55 fr`: The full-resolution pipe's capture device |
| 73 |
- `mali-c55 ds`: The downscale pipe's capture device |
| 74 |
- `mali-c55 3a stats`: The 3A statistics capture device |
| 75 |
|
| 76 |
Frame sequences are synchronised across to two capture devices, meaning if one |
| 77 |
pipe is started later than the other the sequence numbers returned in its |
| 78 |
buffers will match those of the other pipe rather than starting from zero. |
| 79 |
|
| 80 |
Idiosyncrasies |
| 81 |
-------------- |
| 82 |
|
| 83 |
**mali-c55 isp** |
| 84 |
The `mali-c55 isp` subdevice has a single sink pad to which all sources of data |
| 85 |
should be connected. The active source is selected by enabling the appropriate |
| 86 |
media link and disabling all others. The ISP has two source pads, reflecting the |
| 87 |
different paths through which it can internally route data. Tap points within |
| 88 |
the ISP allow users to divert data to avoid processing by some or all of the |
| 89 |
hardware's processing steps. The diagram below is intended only to highlight how |
| 90 |
the bypassing works and is not a true reflection of those processing steps; for |
| 91 |
a high-level functional block diagram see ARM's developer page for the |
| 92 |
ISP [3]_:: |
| 93 |
|
| 94 |
+--------------------------------------------------------------+ |
| 95 |
| Possible Internal ISP Data Routes | |
| 96 |
| +------------+ +----------+ +------------+ | |
| 97 |
+---+ | | | | | Colour | +---+ |
| 98 |
| 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 | |
| 99 |
+---+ | | | | | | Conversion | +---+ |
| 100 |
| | +------------+ +----------+ +------------+ | |
| 101 |
| | +---+ |
| 102 |
| +---------------------------------------------------| 2 | |
| 103 |
| +---+ |
| 104 |
| | |
| 105 |
+--------------------------------------------------------------+ |
| 106 |
|
| 107 |
|
| 108 |
.. flat-table:: |
| 109 |
:header-rows: 1 |
| 110 |
|
| 111 |
* - Pad |
| 112 |
- Direction |
| 113 |
- Purpose |
| 114 |
|
| 115 |
* - 0 |
| 116 |
- sink |
| 117 |
- Data input, connected to the TPG and camera sensors |
| 118 |
|
| 119 |
* - 1 |
| 120 |
- source |
| 121 |
- RGB/YUV data, connected to the FR and DS V4L2 subdevices |
| 122 |
|
| 123 |
* - 2 |
| 124 |
- source |
| 125 |
- RAW bayer data, connected to the FR V4L2 subdevices |
| 126 |
|
| 127 |
The ISP is limited to both input and output resolutions between 640x480 and |
| 128 |
8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt() |
| 129 |
operations. |
| 130 |
|
| 131 |
**mali-c55 resizer fr** |
| 132 |
The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different |
| 133 |
insertion points in the hardware (either RAW or demosaiced data): |
| 134 |
|
| 135 |
.. flat-table:: |
| 136 |
:header-rows: 1 |
| 137 |
|
| 138 |
* - Pad |
| 139 |
- Direction |
| 140 |
- Purpose |
| 141 |
|
| 142 |
* - 0 |
| 143 |
- sink |
| 144 |
- Data input connected to the ISP's demosaiced stream. |
| 145 |
|
| 146 |
* - 1 |
| 147 |
- source |
| 148 |
- Data output connected to the capture video device |
| 149 |
|
| 150 |
* - 2 |
| 151 |
- sink |
| 152 |
- Data input connected to the ISP's raw data stream |
| 153 |
|
| 154 |
The data source in use is selected through the routing API; two routes each of a |
| 155 |
single stream are available: |
| 156 |
|
| 157 |
.. flat-table:: |
| 158 |
:header-rows: 1 |
| 159 |
|
| 160 |
* - Sink Pad |
| 161 |
- Source Pad |
| 162 |
- Purpose |
| 163 |
|
| 164 |
* - 0 |
| 165 |
- 1 |
| 166 |
- Demosaiced data route |
| 167 |
|
| 168 |
* - 2 |
| 169 |
- 1 |
| 170 |
- Raw data route |
| 171 |
|
| 172 |
|
| 173 |
If the demosaiced route is active then the FR pipe is only capable of output |
| 174 |
in RGB/YUV formats. If the raw route is active then the output reflects the |
| 175 |
input (which may be either Bayer or RGB/YUV data). |
| 176 |
|
| 177 |
Using the driver to capture video |
| 178 |
================================= |
| 179 |
|
| 180 |
Using the media controller APIs we can configure the input source and ISP to |
| 181 |
capture images in a variety of formats. In the examples below, configuring the |
| 182 |
media graph is done with the v4l-utils [1]_ package's media-ctl utility. |
| 183 |
Capturing the images is done with yavta [2]_. |
| 184 |
|
| 185 |
Configuring the input source |
| 186 |
---------------------------- |
| 187 |
|
| 188 |
The first step is to set the input source that we wish by enabling the correct |
| 189 |
media link. Using the example topology above, we can select the TPG as follows: |
| 190 |
|
| 191 |
.. code-block:: none |
| 192 |
|
| 193 |
media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]" |
| 194 |
media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]" |
| 195 |
|
| 196 |
Configuring which video devices will stream data |
| 197 |
------------------------------------------------ |
| 198 |
|
| 199 |
The driver will wait for all video devices to have their VIDIOC_STREAMON ioctl |
| 200 |
called before it tells the sensor to start streaming. To facilitate this we need |
| 201 |
to enable links to the video devices that we want to use. In the example below |
| 202 |
we enable the links to both of the image capture video devices |
| 203 |
|
| 204 |
.. code-block:: none |
| 205 |
|
| 206 |
media-ctl -l "'mali-c55 resizer fr':1->'mali-c55 fr':0[1]" |
| 207 |
media-ctl -l "'mali-c55 resizer ds':1->'mali-c55 ds':0[1]" |
| 208 |
|
| 209 |
Capturing bayer data from the source and processing to RGB/YUV |
| 210 |
-------------------------------------------------------------- |
| 211 |
|
| 212 |
To capture 1920x1080 bayer data from the source and push it through the ISP's |
| 213 |
full processing pipeline, we configure the data formats appropriately on the |
| 214 |
source, ISP and resizer subdevices and set the FR resizer's routing to select |
| 215 |
processed data. The media bus format on the resizer's source pad will be either |
| 216 |
RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or |
| 217 |
YUV. The ISP's debayering block outputs RGB data natively, setting the source |
| 218 |
pad format to YUV10_1X30 enables the colour space conversion block. |
| 219 |
|
| 220 |
In this example we target RGB565 output, so select RGB121212_1X36 as the resizer |
| 221 |
source pad's format: |
| 222 |
|
| 223 |
.. code-block:: none |
| 224 |
|
| 225 |
# Set formats on the TPG and ISP |
| 226 |
media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB20_1X20/1920x1080]" |
| 227 |
media-ctl -V "'mali-c55 isp':0[fmt:SRGGB20_1X20/1920x1080]" |
| 228 |
media-ctl -V "'mali-c55 isp':1[fmt:SRGGB20_1X20/1920x1080]" |
| 229 |
|
| 230 |
# Set routing on the FR resizer |
| 231 |
media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]" |
| 232 |
|
| 233 |
# Set format on the resizer, must be done AFTER the routing. |
| 234 |
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]" |
| 235 |
|
| 236 |
The downscale output can also be used to stream data at the same time. In this |
| 237 |
case since only processed data can be captured through the downscale output no |
| 238 |
routing need be set: |
| 239 |
|
| 240 |
.. code-block:: none |
| 241 |
|
| 242 |
# Set format on the resizer |
| 243 |
media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]" |
| 244 |
|
| 245 |
Following which images can be captured from both the FR and DS output's video |
| 246 |
devices (simultaneously, if desired): |
| 247 |
|
| 248 |
.. code-block:: none |
| 249 |
|
| 250 |
yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 |
| 251 |
yavta -f RGB565 -s 1920x1080 -c10 /dev/video1 |
| 252 |
|
| 253 |
Cropping the image |
| 254 |
~~~~~~~~~~~~~~~~~~ |
| 255 |
|
| 256 |
Both the full resolution and downscale pipes can crop to a minimum resolution of |
| 257 |
640x480. To crop the image simply configure the resizer's sink pad's crop and |
| 258 |
compose rectangles and set the format on the video device: |
| 259 |
|
| 260 |
.. code-block:: none |
| 261 |
|
| 262 |
media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]" |
| 263 |
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" |
| 264 |
yavta -f RGB565 -s 640x480 -c10 /dev/video0 |
| 265 |
|
| 266 |
Downscaling the image |
| 267 |
~~~~~~~~~~~~~~~~~~~~~ |
| 268 |
|
| 269 |
Both the full resolution and downscale pipes can downscale the image by up to 8x |
| 270 |
provided the minimum 640x480 output resolution is adhered to. For the best image |
| 271 |
result the scaling ratio for each direction should be the same. To configure |
| 272 |
scaling we use the compose rectangle on the resizer's sink pad: |
| 273 |
|
| 274 |
.. code-block:: none |
| 275 |
|
| 276 |
media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]" |
| 277 |
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]" |
| 278 |
yavta -f RGB565 -s 640x480 -c10 /dev/video0 |
| 279 |
|
| 280 |
Capturing images in YUV formats |
| 281 |
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ |
| 282 |
|
| 283 |
If we need to output YUV data rather than RGB the color space conversion block |
| 284 |
needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the |
| 285 |
resizer's source pad. We can then configure a capture format like NV12 (here in |
| 286 |
its multi-planar variant) |
| 287 |
|
| 288 |
.. code-block:: none |
| 289 |
|
| 290 |
media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]" |
| 291 |
yavta -f NV12M -s 1920x1080 -c10 /dev/video0 |
| 292 |
|
| 293 |
Capturing RGB data from the source and processing it with the resizers |
| 294 |
---------------------------------------------------------------------- |
| 295 |
|
| 296 |
The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this |
| 297 |
case although none of the image quality blocks would be used it can still |
| 298 |
crop/scale the data in the usual way. For this reason RGB data input to the ISP |
| 299 |
still goes through the ISP subdevice's pad 1 to the resizer. |
| 300 |
|
| 301 |
To achieve this, the ISP's sink pad's format is set to |
| 302 |
MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to |
| 303 |
work with the ISP. Converting the camera sensor's output to that format is the |
| 304 |
responsibility of external hardware. |
| 305 |
|
| 306 |
In this example we ask the test pattern generator to give us RGB data instead of |
| 307 |
bayer. |
| 308 |
|
| 309 |
.. code-block:: none |
| 310 |
|
| 311 |
media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]" |
| 312 |
media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" |
| 313 |
|
| 314 |
Cropping or scaling the data can be done in exactly the same way as outlined |
| 315 |
earlier. |
| 316 |
|
| 317 |
Capturing raw data from the source and outputting it unmodified |
| 318 |
----------------------------------------------------------------- |
| 319 |
|
| 320 |
The ISP can additionally capture raw data from the source and output it on the |
| 321 |
full resolution pipe only, completely unmodified. In this case the downscale |
| 322 |
pipe can still process the data normally and be used at the same time. |
| 323 |
|
| 324 |
To configure raw bypass the FR resizer's subdevice's routing table needs to be |
| 325 |
configured, followed by formats in the appropriate places: |
| 326 |
|
| 327 |
.. code-block:: none |
| 328 |
|
| 329 |
media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]" |
| 330 |
media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]" |
| 331 |
media-ctl -V "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080]" |
| 332 |
media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]" |
| 333 |
|
| 334 |
# Set format on the video device and stream |
| 335 |
yavta -f RGB565 -s 1920x1080 -c10 /dev/video0 |
| 336 |
|
| 337 |
.. _mali-c55-3a-stats: |
| 338 |
|
| 339 |
Capturing ISP Statistics |
| 340 |
======================== |
| 341 |
|
| 342 |
The ISP is capable of producing statistics for consumption by image processing |
| 343 |
algorithms running in userspace. These statistics can be captured by queueing |
| 344 |
buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only |
| 345 |
the :ref:`V4L2_META_FMT_MALI_C55_STATS <v4l2-meta-fmt-mali-c55-stats>` |
| 346 |
format is supported, so no format-setting need be done: |
| 347 |
|
| 348 |
.. code-block:: none |
| 349 |
|
| 350 |
# We assume the media graph has been configured to support RGB565 capture |
| 351 |
# from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics |
| 352 |
# V4L2 device is at /dev/video3 |
| 353 |
|
| 354 |
yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \ |
| 355 |
yavta -c10 -F /dev/video3 |
| 356 |
|
| 357 |
The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`, |
| 358 |
but broadly statistics are generated to support three image processing |
| 359 |
algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus). |
| 360 |
These stats can be drawn from various places in the Mali C55 ISP pipeline, known |
| 361 |
as "tap points". This high-level block diagram is intended to explain where in |
| 362 |
the processing flow the statistics can be drawn from:: |
| 363 |
|
| 364 |
+--> AEXP-2 +----> AEXP-1 +--> AF-0 |
| 365 |
| +----> AF-1 | |
| 366 |
| | | |
| 367 |
+---------+ | +--------------+ | +--------------+ | |
| 368 |
| Input +-+-->+ Digital Gain +---+-->+ Black Level +---+---+ |
| 369 |
+---------+ +--------------+ +--------------+ | |
| 370 |
+-----------------------------------------------------------------+ |
| 371 |
| |
| 372 |
| +--------------+ +---------+ +----------------+ |
| 373 |
+-->| Sinter Noise +-+ White +--+--->| Lens Shading +--+---------------+ |
| 374 |
| Reduction | | Balance | | | | | | |
| 375 |
+--------------+ +---------+ | +----------------+ | | |
| 376 |
+---> AEXP-0 (A) +--> AEXP-0 (B) | |
| 377 |
+--------------------------------------------------------------------------+ |
| 378 |
| |
| 379 |
| +----------------+ +--------------+ +----------------+ |
| 380 |
+-->| Tone mapping +-+--->| Demosaicing +->+ Purple Fringe +-+-----------+ |
| 381 |
| | | +--------------+ | Correction | | | |
| 382 |
+----------------+ +-> AEXP-IRIDIX +----------------+ +---> AWB-0 | |
| 383 |
+----------------------------------------------------------------------------+ |
| 384 |
| +-------------+ +-------------+ |
| 385 |
+------------------->| Colour +---+--->| Output | |
| 386 |
| Correction | | | Pipelines | |
| 387 |
+-------------+ | +-------------+ |
| 388 |
+--> AWB-1 |
| 389 |
|
| 390 |
By default all statistics are drawn from the 0th tap point for each algorithm; |
| 391 |
I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF |
| 392 |
statistics from AF-0. This is configurable for AEXP and AWB statsistics through |
| 393 |
programming the ISP's parameters. |
| 394 |
|
| 395 |
.. _mali-c55-3a-params: |
| 396 |
|
| 397 |
Programming ISP Parameters |
| 398 |
========================== |
| 399 |
|
| 400 |
The ISP can be programmed with various parameters from userspace to apply to the |
| 401 |
hardware before and during video stream. This allows userspace to dynamically |
| 402 |
change values such as black level, white balance and lens shading gains and so |
| 403 |
on. |
| 404 |
|
| 405 |
The buffer format and how to populate it are described by the |
| 406 |
:ref:`V4L2_META_FMT_MALI_C55_PARAMS <v4l2-meta-fmt-mali-c55-params>` format, |
| 407 |
which should be set as the data format for the `mali-c55 3a params` video node. |
| 408 |
|
| 409 |
References |
| 410 |
========== |
| 411 |
.. [1] https://git.linuxtv.org/v4l-utils.git/ |
| 412 |
.. [2] https://git.ideasonboard.org/yavta.git |
| 413 |
.. [3] https://developer.arm.com/Processors/Mali-C55 |