| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
.. include:: <isonum.txt> |
| 4 |
|
| 5 |
========================================= |
| 6 |
Rockchip Image Signal Processor (rkisp1) |
| 7 |
========================================= |
| 8 |
|
| 9 |
Introduction |
| 10 |
============ |
| 11 |
|
| 12 |
This file documents the driver for the Rockchip ISP1 that is part of RK3288 |
| 13 |
and RK3399 SoCs. The driver is located under drivers/media/platform/rockchip/ |
| 14 |
rkisp1 and uses the Media-Controller API. |
| 15 |
|
| 16 |
Revisions |
| 17 |
========= |
| 18 |
|
| 19 |
There exist multiple smaller revisions to this ISP that got introduced in |
| 20 |
later SoCs. Revisions can be found in the enum :c:type:`rkisp1_cif_isp_version` |
| 21 |
in the UAPI and the revision of the ISP inside the running SoC can be read |
| 22 |
in the field hw_revision of struct media_device_info as returned by |
| 23 |
ioctl MEDIA_IOC_DEVICE_INFO. |
| 24 |
|
| 25 |
Versions in use are: |
| 26 |
|
| 27 |
- RKISP1_V10: used at least in rk3288 and rk3399 |
| 28 |
- RKISP1_V11: declared in the original vendor code, but not used |
| 29 |
- RKISP1_V12: used at least in rk3326 and px30 |
| 30 |
- RKISP1_V13: used at least in rk1808 |
| 31 |
|
| 32 |
Topology |
| 33 |
======== |
| 34 |
.. _rkisp1_topology_graph: |
| 35 |
|
| 36 |
.. kernel-figure:: rkisp1.dot |
| 37 |
:alt: Diagram of the default media pipeline topology |
| 38 |
:align: center |
| 39 |
|
| 40 |
|
| 41 |
The driver has 4 video devices: |
| 42 |
|
| 43 |
- rkisp1_mainpath: capture device for retrieving images, usually in higher |
| 44 |
resolution. |
| 45 |
- rkisp1_selfpath: capture device for retrieving images. |
| 46 |
- rkisp1_stats: a metadata capture device that sends statistics. |
| 47 |
- rkisp1_params: a metadata output device that receives parameters |
| 48 |
configurations from userspace. |
| 49 |
|
| 50 |
The driver has 3 subdevices: |
| 51 |
|
| 52 |
- rkisp1_resizer_mainpath: used to resize and downsample frames for the |
| 53 |
mainpath capture device. |
| 54 |
- rkisp1_resizer_selfpath: used to resize and downsample frames for the |
| 55 |
selfpath capture device. |
| 56 |
- rkisp1_isp: is connected to the sensor and is responsible for all the isp |
| 57 |
operations. |
| 58 |
|
| 59 |
|
| 60 |
rkisp1_mainpath, rkisp1_selfpath - Frames Capture Video Nodes |
| 61 |
------------------------------------------------------------- |
| 62 |
Those are the `mainpath` and `selfpath` capture devices to capture frames. |
| 63 |
Those entities are the DMA engines that write the frames to memory. |
| 64 |
The selfpath video device can capture YUV/RGB formats. Its input is YUV encoded |
| 65 |
stream and it is able to convert it to RGB. The selfpath is not able to |
| 66 |
capture bayer formats. |
| 67 |
The mainpath can capture both bayer and YUV formats but it is not able to |
| 68 |
capture RGB formats. |
| 69 |
Both capture videos support |
| 70 |
the ``V4L2_CAP_IO_MC`` :ref:`capability <device-capabilities>`. |
| 71 |
|
| 72 |
|
| 73 |
rkisp1_resizer_mainpath, rkisp1_resizer_selfpath - Resizers Subdevices Nodes |
| 74 |
---------------------------------------------------------------------------- |
| 75 |
Those are resizer entities for the mainpath and the selfpath. Those entities |
| 76 |
can scale the frames up and down and also change the YUV sampling (for example |
| 77 |
YUV4:2:2 -> YUV4:2:0). They also have cropping capability on the sink pad. |
| 78 |
The resizers entities can only operate on YUV:4:2:2 format |
| 79 |
(MEDIA_BUS_FMT_YUYV8_2X8). |
| 80 |
The mainpath capture device supports capturing video in bayer formats. In that |
| 81 |
case the resizer of the mainpath is set to 'bypass' mode - it just forward the |
| 82 |
frame without operating on it. |
| 83 |
|
| 84 |
rkisp1_isp - Image Signal Processing Subdevice Node |
| 85 |
--------------------------------------------------- |
| 86 |
This is the isp entity. It is connected to the sensor on sink pad 0 and |
| 87 |
receives the frames using the CSI-2 protocol. It is responsible of configuring |
| 88 |
the CSI-2 protocol. It has a cropping capability on sink pad 0 that is |
| 89 |
connected to the sensor and on source pad 2 connected to the resizer entities. |
| 90 |
Cropping on sink pad 0 defines the image region from the sensor. |
| 91 |
Cropping on source pad 2 defines the region for the Image Stabilizer (IS). |
| 92 |
|
| 93 |
.. _rkisp1_stats: |
| 94 |
|
| 95 |
rkisp1_stats - Statistics Video Node |
| 96 |
------------------------------------ |
| 97 |
The statistics video node outputs the 3A (auto focus, auto exposure and auto |
| 98 |
white balance) statistics, and also histogram statistics for the frames that |
| 99 |
are being processed by the rkisp1 to userspace applications. |
| 100 |
Using these data, applications can implement algorithms and re-parameterize |
| 101 |
the driver through the rkisp_params node to improve image quality during a |
| 102 |
video stream. |
| 103 |
The buffer format is defined by struct :c:type:`rkisp1_stat_buffer`, and |
| 104 |
userspace should set |
| 105 |
:ref:`V4L2_META_FMT_RK_ISP1_STAT_3A <v4l2-meta-fmt-rk-isp1-stat-3a>` as the |
| 106 |
dataformat. |
| 107 |
|
| 108 |
.. _rkisp1_params: |
| 109 |
|
| 110 |
rkisp1_params - Parameters Video Node |
| 111 |
------------------------------------- |
| 112 |
The rkisp1_params video node receives a set of parameters from userspace |
| 113 |
to be applied to the hardware during a video stream, allowing userspace |
| 114 |
to dynamically modify values such as black level, cross talk corrections |
| 115 |
and others. |
| 116 |
|
| 117 |
The ISP driver supports two different parameters configuration methods, the |
| 118 |
`fixed parameters format` or the `extensible parameters format`. |
| 119 |
|
| 120 |
When using the `fixed parameters` method the buffer format is defined by struct |
| 121 |
:c:type:`rkisp1_params_cfg`, and userspace should set |
| 122 |
:ref:`V4L2_META_FMT_RK_ISP1_PARAMS <v4l2-meta-fmt-rk-isp1-params>` as the |
| 123 |
dataformat. |
| 124 |
|
| 125 |
When using the `extensible parameters` method the buffer format is defined by |
| 126 |
struct :c:type:`rkisp1_ext_params_cfg`, and userspace should set |
| 127 |
:ref:`V4L2_META_FMT_RK_ISP1_EXT_PARAMS <v4l2-meta-fmt-rk-isp1-ext-params>` as |
| 128 |
the dataformat. |
| 129 |
|
| 130 |
Capturing Video Frames Example |
| 131 |
============================== |
| 132 |
|
| 133 |
In the following example, the sensor connected to pad 0 of 'rkisp1_isp' is |
| 134 |
imx219. |
| 135 |
|
| 136 |
The following commands can be used to capture video from the selfpath video |
| 137 |
node with dimension 900x800 planar format YUV 4:2:2. It uses all cropping |
| 138 |
capabilities possible, (see explanation right below) |
| 139 |
|
| 140 |
.. code-block:: bash |
| 141 |
|
| 142 |
# set the links |
| 143 |
"media-ctl" "-d" "platform:rkisp1" "-r" |
| 144 |
"media-ctl" "-d" "platform:rkisp1" "-l" "'imx219 4-0010':0 -> 'rkisp1_isp':0 [1]" |
| 145 |
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_selfpath':0 [1]" |
| 146 |
"media-ctl" "-d" "platform:rkisp1" "-l" "'rkisp1_isp':2 -> 'rkisp1_resizer_mainpath':0 [0]" |
| 147 |
|
| 148 |
# set format for imx219 4-0010:0 |
| 149 |
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"imx219 4-0010":0 [fmt:SRGGB10_1X10/1640x1232]' |
| 150 |
|
| 151 |
# set format for rkisp1_isp pads: |
| 152 |
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":0 [fmt:SRGGB10_1X10/1640x1232 crop: (0,0)/1600x1200]' |
| 153 |
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_isp":2 [fmt:YUYV8_2X8/1600x1200 crop: (0,0)/1500x1100]' |
| 154 |
|
| 155 |
# set format for rkisp1_resizer_selfpath pads: |
| 156 |
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":0 [fmt:YUYV8_2X8/1500x1100 crop: (300,400)/1400x1000]' |
| 157 |
"media-ctl" "-d" "platform:rkisp1" "--set-v4l2" '"rkisp1_resizer_selfpath":1 [fmt:YUYV8_2X8/900x800]' |
| 158 |
|
| 159 |
# set format for rkisp1_selfpath: |
| 160 |
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "width=900,height=800," |
| 161 |
"v4l2-ctl" "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "-v" "pixelformat=422P" |
| 162 |
|
| 163 |
# start streaming: |
| 164 |
v4l2-ctl "-z" "platform:rkisp1" "-d" "rkisp1_selfpath" "--stream-mmap" "--stream-count" "10" |
| 165 |
|
| 166 |
|
| 167 |
In the above example the sensor is configured to bayer format: |
| 168 |
`SRGGB10_1X10/1640x1232`. The rkisp1_isp:0 pad should be configured to the |
| 169 |
same mbus format and dimensions as the sensor, otherwise streaming will fail |
| 170 |
with 'EPIPE' error. So it is also configured to `SRGGB10_1X10/1640x1232`. |
| 171 |
In addition, the rkisp1_isp:0 pad is configured to cropping `(0,0)/1600x1200`. |
| 172 |
|
| 173 |
The cropping dimensions are automatically propagated to be the format of the |
| 174 |
isp source pad `rkisp1_isp:2`. Another cropping operation is configured on |
| 175 |
the isp source pad: `(0,0)/1500x1100`. |
| 176 |
|
| 177 |
The resizer's sink pad `rkisp1_resizer_selfpath` should be configured to format |
| 178 |
`YUYV8_2X8/1500x1100` in order to match the format on the other side of the |
| 179 |
link. In addition a cropping `(300,400)/1400x1000` is configured on it. |
| 180 |
|
| 181 |
The source pad of the resizer, `rkisp1_resizer_selfpath:1` is configured to |
| 182 |
format `YUYV8_2X8/900x800`. That means that the resizer first crop a window |
| 183 |
of `(300,400)/1400x100` from the received frame and then scales this window |
| 184 |
to dimension `900x800`. |
| 185 |
|
| 186 |
Note that the above example does not uses the stats-params control loop. |
| 187 |
Therefore the capture frames will not go through the 3A algorithms and |
| 188 |
probably won't have a good quality, and can even look dark and greenish. |
| 189 |
|
| 190 |
Configuring Quantization |
| 191 |
======================== |
| 192 |
|
| 193 |
The driver supports limited and full range quantization on YUV formats, |
| 194 |
where limited is the default. |
| 195 |
To switch between one or the other, userspace should use the Colorspace |
| 196 |
Conversion API (CSC) for subdevices on source pad 2 of the |
| 197 |
isp (`rkisp1_isp:2`). The quantization configured on this pad is the |
| 198 |
quantization of the captured video frames on the mainpath and selfpath |
| 199 |
video nodes. |
| 200 |
Note that the resizer and capture entities will always report |
| 201 |
``V4L2_QUANTIZATION_DEFAULT`` even if the quantization is configured to full |
| 202 |
range on `rkisp1_isp:2`. So in order to get the configured quantization, |
| 203 |
application should get it from pad `rkisp1_isp:2`. |
| 204 |
|