Debian Codegraph @

Documentation/admin-guide/media/mali-c55.rst

raw
1 .. SPDX-License-Identifier: GPL-2.0
2
3 ==========================================
4 ARM Mali-C55 Image Signal Processor driver
5 ==========================================
6
7 Introduction
8 ============
9
10 This file documents the driver for ARM's Mali-C55 Image Signal Processor. The
11 driver is located under drivers/media/platform/arm/mali-c55.
12
13 The Mali-C55 ISP receives data in either raw Bayer format or RGB/YUV format from
14 sensors through either a parallel interface or a memory bus before processing it
15 and outputting it through an internal DMA engine. Two output pipelines are
16 possible (though one may not be fitted, depending on the implementation). These
17 are referred to as "Full resolution" and "Downscale", but the naming is historic
18 and both pipes are capable of cropping/scaling operations. The full resolution
19 pipe is also capable of outputting RAW data, bypassing much of the ISP's
20 processing. The downscale pipe cannot output RAW data. An integrated test
21 pattern generator can be used to drive the ISP and produce image data in the
22 absence of a connected camera sensor. The driver module is named mali_c55, and
23 is enabled through the CONFIG_VIDEO_MALI_C55 config option.
24
25 The driver implements V4L2, Media Controller and V4L2 Subdevice interfaces and
26 expects camera sensors connected to the ISP to have V4L2 subdevice interfaces.
27
28 Mali-C55 ISP hardware
29 =====================
30
31 A high level functional view of the Mali-C55 ISP is presented below. The ISP
32 takes input from either a live source or through a DMA engine for memory input,
33 depending on the SoC integration.::
34
35 +---------+ +----------+ +--------+
36 | Sensor |--->| CSI-2 Rx | "Full Resolution" | DMA |
37 +---------+ +----------+ |\ Output +--->| Writer |
38 | | \ | +--------+
39 | | \ +----------+ +------+---> Streaming I/O
40 +------------+ +------->| | | | |
41 | | | |-->| Mali-C55 |--+
42 | DMA Reader |--------------->| | | ISP | |
43 | | | / | | | +---> Streaming I/O
44 +------------+ | / +----------+ | |
45 |/ +------+
46 | +--------+
47 +--->| DMA |
48 "Downscaled" | Writer |
49 Output +--------+
50
51 Media Controller Topology
52 =========================
53
54 An example of the ISP's topology (as implemented in a system with an IMX415
55 camera sensor and generic CSI-2 receiver) is below:
56
57
58 .. kernel-figure:: mali-c55-graph.dot
59 :alt: mali-c55-graph.dot
60 :align: center
61
62 The driver has 4 V4L2 subdevices:
63
64 - `mali_c55 isp`: Responsible for configuring input crop and color space
65 conversion
66 - `mali_c55 tpg`: The test pattern generator, emulating a camera sensor.
67 - `mali_c55 resizer fr`: The Full-Resolution pipe resizer
68 - `mali_c55 resizer ds`: The Downscale pipe resizer
69
70 The driver has 3 V4L2 video devices:
71
72 - `mali-c55 fr`: The full-resolution pipe's capture device
73 - `mali-c55 ds`: The downscale pipe's capture device
74 - `mali-c55 3a stats`: The 3A statistics capture device
75
76 Frame sequences are synchronised across to two capture devices, meaning if one
77 pipe is started later than the other the sequence numbers returned in its
78 buffers will match those of the other pipe rather than starting from zero.
79
80 Idiosyncrasies
81 --------------
82
83 **mali-c55 isp**
84 The `mali-c55 isp` subdevice has a single sink pad to which all sources of data
85 should be connected. The active source is selected by enabling the appropriate
86 media link and disabling all others. The ISP has two source pads, reflecting the
87 different paths through which it can internally route data. Tap points within
88 the ISP allow users to divert data to avoid processing by some or all of the
89 hardware's processing steps. The diagram below is intended only to highlight how
90 the bypassing works and is not a true reflection of those processing steps; for
91 a high-level functional block diagram see ARM's developer page for the
92 ISP [3]_::
93
94 +--------------------------------------------------------------+
95 | Possible Internal ISP Data Routes |
96 | +------------+ +----------+ +------------+ |
97 +---+ | | | | | Colour | +---+
98 | 0 |--+-->| Processing |->| Demosaic |->| Space |--->| 1 |
99 +---+ | | | | | | Conversion | +---+
100 | | +------------+ +----------+ +------------+ |
101 | | +---+
102 | +---------------------------------------------------| 2 |
103 | +---+
104 | |
105 +--------------------------------------------------------------+
106
107
108 .. flat-table::
109 :header-rows: 1
110
111 * - Pad
112 - Direction
113 - Purpose
114
115 * - 0
116 - sink
117 - Data input, connected to the TPG and camera sensors
118
119 * - 1
120 - source
121 - RGB/YUV data, connected to the FR and DS V4L2 subdevices
122
123 * - 2
124 - source
125 - RAW bayer data, connected to the FR V4L2 subdevices
126
127 The ISP is limited to both input and output resolutions between 640x480 and
128 8192x8192, and this is reflected in the ISP and resizer subdevice's .set_fmt()
129 operations.
130
131 **mali-c55 resizer fr**
132 The `mali-c55 resizer fr` subdevice has two _sink_ pads to reflect the different
133 insertion points in the hardware (either RAW or demosaiced data):
134
135 .. flat-table::
136 :header-rows: 1
137
138 * - Pad
139 - Direction
140 - Purpose
141
142 * - 0
143 - sink
144 - Data input connected to the ISP's demosaiced stream.
145
146 * - 1
147 - source
148 - Data output connected to the capture video device
149
150 * - 2
151 - sink
152 - Data input connected to the ISP's raw data stream
153
154 The data source in use is selected through the routing API; two routes each of a
155 single stream are available:
156
157 .. flat-table::
158 :header-rows: 1
159
160 * - Sink Pad
161 - Source Pad
162 - Purpose
163
164 * - 0
165 - 1
166 - Demosaiced data route
167
168 * - 2
169 - 1
170 - Raw data route
171
172
173 If the demosaiced route is active then the FR pipe is only capable of output
174 in RGB/YUV formats. If the raw route is active then the output reflects the
175 input (which may be either Bayer or RGB/YUV data).
176
177 Using the driver to capture video
178 =================================
179
180 Using the media controller APIs we can configure the input source and ISP to
181 capture images in a variety of formats. In the examples below, configuring the
182 media graph is done with the v4l-utils [1]_ package's media-ctl utility.
183 Capturing the images is done with yavta [2]_.
184
185 Configuring the input source
186 ----------------------------
187
188 The first step is to set the input source that we wish by enabling the correct
189 media link. Using the example topology above, we can select the TPG as follows:
190
191 .. code-block:: none
192
193 media-ctl -l "'lte-csi2-rx':1->'mali-c55 isp':0[0]"
194 media-ctl -l "'mali-c55 tpg':0->'mali-c55 isp':0[1]"
195
196 Configuring which video devices will stream data
197 ------------------------------------------------
198
199 The driver will wait for all video devices to have their VIDIOC_STREAMON ioctl
200 called before it tells the sensor to start streaming. To facilitate this we need
201 to enable links to the video devices that we want to use. In the example below
202 we enable the links to both of the image capture video devices
203
204 .. code-block:: none
205
206 media-ctl -l "'mali-c55 resizer fr':1->'mali-c55 fr':0[1]"
207 media-ctl -l "'mali-c55 resizer ds':1->'mali-c55 ds':0[1]"
208
209 Capturing bayer data from the source and processing to RGB/YUV
210 --------------------------------------------------------------
211
212 To capture 1920x1080 bayer data from the source and push it through the ISP's
213 full processing pipeline, we configure the data formats appropriately on the
214 source, ISP and resizer subdevices and set the FR resizer's routing to select
215 processed data. The media bus format on the resizer's source pad will be either
216 RGB121212_1X36 or YUV10_1X30, depending on whether you want to capture RGB or
217 YUV. The ISP's debayering block outputs RGB data natively, setting the source
218 pad format to YUV10_1X30 enables the colour space conversion block.
219
220 In this example we target RGB565 output, so select RGB121212_1X36 as the resizer
221 source pad's format:
222
223 .. code-block:: none
224
225 # Set formats on the TPG and ISP
226 media-ctl -V "'mali-c55 tpg':0[fmt:SRGGB20_1X20/1920x1080]"
227 media-ctl -V "'mali-c55 isp':0[fmt:SRGGB20_1X20/1920x1080]"
228 media-ctl -V "'mali-c55 isp':1[fmt:SRGGB20_1X20/1920x1080]"
229
230 # Set routing on the FR resizer
231 media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[1],2/0->1/0[0]]"
232
233 # Set format on the resizer, must be done AFTER the routing.
234 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/1920x1080]"
235
236 The downscale output can also be used to stream data at the same time. In this
237 case since only processed data can be captured through the downscale output no
238 routing need be set:
239
240 .. code-block:: none
241
242 # Set format on the resizer
243 media-ctl -V "'mali-c55 resizer ds':1[fmt:RGB121212_1X36/1920x1080]"
244
245 Following which images can be captured from both the FR and DS output's video
246 devices (simultaneously, if desired):
247
248 .. code-block:: none
249
250 yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
251 yavta -f RGB565 -s 1920x1080 -c10 /dev/video1
252
253 Cropping the image
254 ~~~~~~~~~~~~~~~~~~
255
256 Both the full resolution and downscale pipes can crop to a minimum resolution of
257 640x480. To crop the image simply configure the resizer's sink pad's crop and
258 compose rectangles and set the format on the video device:
259
260 .. code-block:: none
261
262 media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(480,270)/640x480 compose:(0,0)/640x480]"
263 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
264 yavta -f RGB565 -s 640x480 -c10 /dev/video0
265
266 Downscaling the image
267 ~~~~~~~~~~~~~~~~~~~~~
268
269 Both the full resolution and downscale pipes can downscale the image by up to 8x
270 provided the minimum 640x480 output resolution is adhered to. For the best image
271 result the scaling ratio for each direction should be the same. To configure
272 scaling we use the compose rectangle on the resizer's sink pad:
273
274 .. code-block:: none
275
276 media-ctl -V "'mali-c55 resizer fr':0[fmt:RGB121212_1X36/1920x1080 crop:(0,0)/1920x1080 compose:(0,0)/640x480]"
277 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB121212_1X36/640x480]"
278 yavta -f RGB565 -s 640x480 -c10 /dev/video0
279
280 Capturing images in YUV formats
281 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
282
283 If we need to output YUV data rather than RGB the color space conversion block
284 needs to be active, which is achieved by setting MEDIA_BUS_FMT_YUV10_1X30 on the
285 resizer's source pad. We can then configure a capture format like NV12 (here in
286 its multi-planar variant)
287
288 .. code-block:: none
289
290 media-ctl -V "'mali-c55 resizer fr':1[fmt:YUV10_1X30/1920x1080]"
291 yavta -f NV12M -s 1920x1080 -c10 /dev/video0
292
293 Capturing RGB data from the source and processing it with the resizers
294 ----------------------------------------------------------------------
295
296 The Mali-C55 ISP can work with sensors capable of outputting RGB data. In this
297 case although none of the image quality blocks would be used it can still
298 crop/scale the data in the usual way. For this reason RGB data input to the ISP
299 still goes through the ISP subdevice's pad 1 to the resizer.
300
301 To achieve this, the ISP's sink pad's format is set to
302 MEDIA_BUS_FMT_RGB202020_1X60 - this reflects the format that data must be in to
303 work with the ISP. Converting the camera sensor's output to that format is the
304 responsibility of external hardware.
305
306 In this example we ask the test pattern generator to give us RGB data instead of
307 bayer.
308
309 .. code-block:: none
310
311 media-ctl -V "'mali-c55 tpg':0[fmt:RGB202020_1X60/1920x1080]"
312 media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
313
314 Cropping or scaling the data can be done in exactly the same way as outlined
315 earlier.
316
317 Capturing raw data from the source and outputting it unmodified
318 -----------------------------------------------------------------
319
320 The ISP can additionally capture raw data from the source and output it on the
321 full resolution pipe only, completely unmodified. In this case the downscale
322 pipe can still process the data normally and be used at the same time.
323
324 To configure raw bypass the FR resizer's subdevice's routing table needs to be
325 configured, followed by formats in the appropriate places:
326
327 .. code-block:: none
328
329 media-ctl -R "'mali-c55 resizer fr'[0/0->1/0[0],2/0->1/0[1]]"
330 media-ctl -V "'mali-c55 isp':0[fmt:RGB202020_1X60/1920x1080]"
331 media-ctl -V "'mali-c55 resizer fr':2[fmt:RGB202020_1X60/1920x1080]"
332 media-ctl -V "'mali-c55 resizer fr':1[fmt:RGB202020_1X60/1920x1080]"
333
334 # Set format on the video device and stream
335 yavta -f RGB565 -s 1920x1080 -c10 /dev/video0
336
337 .. _mali-c55-3a-stats:
338
339 Capturing ISP Statistics
340 ========================
341
342 The ISP is capable of producing statistics for consumption by image processing
343 algorithms running in userspace. These statistics can be captured by queueing
344 buffers to the `mali-c55 3a stats` V4L2 Device whilst the ISP is streaming. Only
345 the :ref:`V4L2_META_FMT_MALI_C55_STATS <v4l2-meta-fmt-mali-c55-stats>`
346 format is supported, so no format-setting need be done:
347
348 .. code-block:: none
349
350 # We assume the media graph has been configured to support RGB565 capture
351 # from the mali-c55 fr V4L2 Device, which is at /dev/video0. The statistics
352 # V4L2 device is at /dev/video3
353
354 yavta -f RGB565 -s 1920x1080 -c32 /dev/video0 && \
355 yavta -c10 -F /dev/video3
356
357 The layout of the buffer is described by :c:type:`mali_c55_stats_buffer`,
358 but broadly statistics are generated to support three image processing
359 algorithms; AEXP (Auto-Exposure), AWB (Auto-White Balance) and AF (Auto-Focus).
360 These stats can be drawn from various places in the Mali C55 ISP pipeline, known
361 as "tap points". This high-level block diagram is intended to explain where in
362 the processing flow the statistics can be drawn from::
363
364 +--> AEXP-2 +----> AEXP-1 +--> AF-0
365 | +----> AF-1 |
366 | | |
367 +---------+ | +--------------+ | +--------------+ |
368 | Input +-+-->+ Digital Gain +---+-->+ Black Level +---+---+
369 +---------+ +--------------+ +--------------+ |
370 +-----------------------------------------------------------------+
371 |
372 | +--------------+ +---------+ +----------------+
373 +-->| Sinter Noise +-+ White +--+--->| Lens Shading +--+---------------+
374 | Reduction | | Balance | | | | | |
375 +--------------+ +---------+ | +----------------+ | |
376 +---> AEXP-0 (A) +--> AEXP-0 (B) |
377 +--------------------------------------------------------------------------+
378 |
379 | +----------------+ +--------------+ +----------------+
380 +-->| Tone mapping +-+--->| Demosaicing +->+ Purple Fringe +-+-----------+
381 | | | +--------------+ | Correction | | |
382 +----------------+ +-> AEXP-IRIDIX +----------------+ +---> AWB-0 |
383 +----------------------------------------------------------------------------+
384 | +-------------+ +-------------+
385 +------------------->| Colour +---+--->| Output |
386 | Correction | | | Pipelines |
387 +-------------+ | +-------------+
388 +--> AWB-1
389
390 By default all statistics are drawn from the 0th tap point for each algorithm;
391 I.E. AEXP statistics from AEXP-0 (A), AWB statistics from AWB-0 and AF
392 statistics from AF-0. This is configurable for AEXP and AWB statsistics through
393 programming the ISP's parameters.
394
395 .. _mali-c55-3a-params:
396
397 Programming ISP Parameters
398 ==========================
399
400 The ISP can be programmed with various parameters from userspace to apply to the
401 hardware before and during video stream. This allows userspace to dynamically
402 change values such as black level, white balance and lens shading gains and so
403 on.
404
405 The buffer format and how to populate it are described by the
406 :ref:`V4L2_META_FMT_MALI_C55_PARAMS <v4l2-meta-fmt-mali-c55-params>` format,
407 which should be set as the data format for the `mali-c55 3a params` video node.
408
409 References
410 ==========
411 .. [1] https://git.linuxtv.org/v4l-utils.git/
412 .. [2] https://git.ideasonboard.org/yavta.git
413 .. [3] https://developer.arm.com/Processors/Mali-C55