| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
i.MX Video Capture Driver |
| 4 |
========================= |
| 5 |
|
| 6 |
Introduction |
| 7 |
------------ |
| 8 |
|
| 9 |
The Freescale i.MX5/6 contains an Image Processing Unit (IPU), which |
| 10 |
handles the flow of image frames to and from capture devices and |
| 11 |
display devices. |
| 12 |
|
| 13 |
For image capture, the IPU contains the following internal subunits: |
| 14 |
|
| 15 |
- Image DMA Controller (IDMAC) |
| 16 |
- Camera Serial Interface (CSI) |
| 17 |
- Image Converter (IC) |
| 18 |
- Sensor Multi-FIFO Controller (SMFC) |
| 19 |
- Image Rotator (IRT) |
| 20 |
- Video De-Interlacing or Combining Block (VDIC) |
| 21 |
|
| 22 |
The IDMAC is the DMA controller for transfer of image frames to and from |
| 23 |
memory. Various dedicated DMA channels exist for both video capture and |
| 24 |
display paths. During transfer, the IDMAC is also capable of vertical |
| 25 |
image flip, 8x8 block transfer (see IRT description), pixel component |
| 26 |
re-ordering (for example UYVY to YUYV) within the same colorspace, and |
| 27 |
packed <--> planar conversion. The IDMAC can also perform a simple |
| 28 |
de-interlacing by interweaving even and odd lines during transfer |
| 29 |
(without motion compensation which requires the VDIC). |
| 30 |
|
| 31 |
The CSI is the backend capture unit that interfaces directly with |
| 32 |
camera sensors over Parallel, BT.656/1120, and MIPI CSI-2 buses. |
| 33 |
|
| 34 |
The IC handles color-space conversion, resizing (downscaling and |
| 35 |
upscaling), horizontal flip, and 90/270 degree rotation operations. |
| 36 |
|
| 37 |
There are three independent "tasks" within the IC that can carry out |
| 38 |
conversions concurrently: pre-process encoding, pre-process viewfinder, |
| 39 |
and post-processing. Within each task, conversions are split into three |
| 40 |
sections: downsizing section, main section (upsizing, flip, colorspace |
| 41 |
conversion, and graphics plane combining), and rotation section. |
| 42 |
|
| 43 |
The IPU time-shares the IC task operations. The time-slice granularity |
| 44 |
is one burst of eight pixels in the downsizing section, one image line |
| 45 |
in the main processing section, one image frame in the rotation section. |
| 46 |
|
| 47 |
The SMFC is composed of four independent FIFOs that each can transfer |
| 48 |
captured frames from sensors directly to memory concurrently via four |
| 49 |
IDMAC channels. |
| 50 |
|
| 51 |
The IRT carries out 90 and 270 degree image rotation operations. The |
| 52 |
rotation operation is carried out on 8x8 pixel blocks at a time. This |
| 53 |
operation is supported by the IDMAC which handles the 8x8 block transfer |
| 54 |
along with block reordering, in coordination with vertical flip. |
| 55 |
|
| 56 |
The VDIC handles the conversion of interlaced video to progressive, with |
| 57 |
support for different motion compensation modes (low, medium, and high |
| 58 |
motion). The deinterlaced output frames from the VDIC can be sent to the |
| 59 |
IC pre-process viewfinder task for further conversions. The VDIC also |
| 60 |
contains a Combiner that combines two image planes, with alpha blending |
| 61 |
and color keying. |
| 62 |
|
| 63 |
In addition to the IPU internal subunits, there are also two units |
| 64 |
outside the IPU that are also involved in video capture on i.MX: |
| 65 |
|
| 66 |
- MIPI CSI-2 Receiver for camera sensors with the MIPI CSI-2 bus |
| 67 |
interface. This is a Synopsys DesignWare core. |
| 68 |
- Two video multiplexers for selecting among multiple sensor inputs |
| 69 |
to send to a CSI. |
| 70 |
|
| 71 |
For more info, refer to the latest versions of the i.MX5/6 reference |
| 72 |
manuals [#f1]_ and [#f2]_. |
| 73 |
|
| 74 |
|
| 75 |
Features |
| 76 |
-------- |
| 77 |
|
| 78 |
Some of the features of this driver include: |
| 79 |
|
| 80 |
- Many different pipelines can be configured via media controller API, |
| 81 |
that correspond to the hardware video capture pipelines supported in |
| 82 |
the i.MX. |
| 83 |
|
| 84 |
- Supports parallel, BT.565, and MIPI CSI-2 interfaces. |
| 85 |
|
| 86 |
- Concurrent independent streams, by configuring pipelines to multiple |
| 87 |
video capture interfaces using independent entities. |
| 88 |
|
| 89 |
- Scaling, color-space conversion, horizontal and vertical flip, and |
| 90 |
image rotation via IC task subdevs. |
| 91 |
|
| 92 |
- Many pixel formats supported (RGB, packed and planar YUV, partial |
| 93 |
planar YUV). |
| 94 |
|
| 95 |
- The VDIC subdev supports motion compensated de-interlacing, with three |
| 96 |
motion compensation modes: low, medium, and high motion. Pipelines are |
| 97 |
defined that allow sending frames to the VDIC subdev directly from the |
| 98 |
CSI. There is also support in the future for sending frames to the |
| 99 |
VDIC from memory buffers via output/mem2mem devices. |
| 100 |
|
| 101 |
- Includes a Frame Interval Monitor (FIM) that can correct vertical sync |
| 102 |
problems with the ADV718x video decoders. |
| 103 |
|
| 104 |
|
| 105 |
Topology |
| 106 |
-------- |
| 107 |
|
| 108 |
The following shows the media topologies for the i.MX6Q SabreSD and |
| 109 |
i.MX6Q SabreAuto. Refer to these diagrams in the entity descriptions |
| 110 |
in the next section. |
| 111 |
|
| 112 |
The i.MX5/6 topologies can differ upstream from the IPUv3 CSI video |
| 113 |
multiplexers, but the internal IPUv3 topology downstream from there |
| 114 |
is common to all i.MX5/6 platforms. For example, the SabreSD, with the |
| 115 |
MIPI CSI-2 OV5640 sensor, requires the i.MX6 MIPI CSI-2 receiver. But |
| 116 |
the SabreAuto has only the ADV7180 decoder on a parallel bt.656 bus, and |
| 117 |
therefore does not require the MIPI CSI-2 receiver, so it is missing in |
| 118 |
its graph. |
| 119 |
|
| 120 |
.. _imx6q_topology_graph: |
| 121 |
|
| 122 |
.. kernel-figure:: imx6q-sabresd.dot |
| 123 |
:alt: Diagram of the i.MX6Q SabreSD media pipeline topology |
| 124 |
:align: center |
| 125 |
|
| 126 |
Media pipeline graph on i.MX6Q SabreSD |
| 127 |
|
| 128 |
.. kernel-figure:: imx6q-sabreauto.dot |
| 129 |
:alt: Diagram of the i.MX6Q SabreAuto media pipeline topology |
| 130 |
:align: center |
| 131 |
|
| 132 |
Media pipeline graph on i.MX6Q SabreAuto |
| 133 |
|
| 134 |
Entities |
| 135 |
-------- |
| 136 |
|
| 137 |
imx6-mipi-csi2 |
| 138 |
-------------- |
| 139 |
|
| 140 |
This is the MIPI CSI-2 receiver entity. It has one sink pad to receive |
| 141 |
the MIPI CSI-2 stream (usually from a MIPI CSI-2 camera sensor). It has |
| 142 |
four source pads, corresponding to the four MIPI CSI-2 demuxed virtual |
| 143 |
channel outputs. Multiple source pads can be enabled to independently |
| 144 |
stream from multiple virtual channels. |
| 145 |
|
| 146 |
This entity actually consists of two sub-blocks. One is the MIPI CSI-2 |
| 147 |
core. This is a Synopsys Designware MIPI CSI-2 core. The other sub-block |
| 148 |
is a "CSI-2 to IPU gasket". The gasket acts as a demultiplexer of the |
| 149 |
four virtual channels streams, providing four separate parallel buses |
| 150 |
containing each virtual channel that are routed to CSIs or video |
| 151 |
multiplexers as described below. |
| 152 |
|
| 153 |
On i.MX6 solo/dual-lite, all four virtual channel buses are routed to |
| 154 |
two video multiplexers. Both CSI0 and CSI1 can receive any virtual |
| 155 |
channel, as selected by the video multiplexers. |
| 156 |
|
| 157 |
On i.MX6 Quad, virtual channel 0 is routed to IPU1-CSI0 (after selected |
| 158 |
by a video mux), virtual channels 1 and 2 are hard-wired to IPU1-CSI1 |
| 159 |
and IPU2-CSI0, respectively, and virtual channel 3 is routed to |
| 160 |
IPU2-CSI1 (again selected by a video mux). |
| 161 |
|
| 162 |
ipuX_csiY_mux |
| 163 |
------------- |
| 164 |
|
| 165 |
These are the video multiplexers. They have two or more sink pads to |
| 166 |
select from either camera sensors with a parallel interface, or from |
| 167 |
MIPI CSI-2 virtual channels from imx6-mipi-csi2 entity. They have a |
| 168 |
single source pad that routes to a CSI (ipuX_csiY entities). |
| 169 |
|
| 170 |
On i.MX6 solo/dual-lite, there are two video mux entities. One sits |
| 171 |
in front of IPU1-CSI0 to select between a parallel sensor and any of |
| 172 |
the four MIPI CSI-2 virtual channels (a total of five sink pads). The |
| 173 |
other mux sits in front of IPU1-CSI1, and again has five sink pads to |
| 174 |
select between a parallel sensor and any of the four MIPI CSI-2 virtual |
| 175 |
channels. |
| 176 |
|
| 177 |
On i.MX6 Quad, there are two video mux entities. One sits in front of |
| 178 |
IPU1-CSI0 to select between a parallel sensor and MIPI CSI-2 virtual |
| 179 |
channel 0 (two sink pads). The other mux sits in front of IPU2-CSI1 to |
| 180 |
select between a parallel sensor and MIPI CSI-2 virtual channel 3 (two |
| 181 |
sink pads). |
| 182 |
|
| 183 |
ipuX_csiY |
| 184 |
--------- |
| 185 |
|
| 186 |
These are the CSI entities. They have a single sink pad receiving from |
| 187 |
either a video mux or from a MIPI CSI-2 virtual channel as described |
| 188 |
above. |
| 189 |
|
| 190 |
This entity has two source pads. The first source pad can link directly |
| 191 |
to the ipuX_vdic entity or the ipuX_ic_prp entity, using hardware links |
| 192 |
that require no IDMAC memory buffer transfer. |
| 193 |
|
| 194 |
When the direct source pad is routed to the ipuX_ic_prp entity, frames |
| 195 |
from the CSI can be processed by one or both of the IC pre-processing |
| 196 |
tasks. |
| 197 |
|
| 198 |
When the direct source pad is routed to the ipuX_vdic entity, the VDIC |
| 199 |
will carry out motion-compensated de-interlace using "high motion" mode |
| 200 |
(see description of ipuX_vdic entity). |
| 201 |
|
| 202 |
The second source pad sends video frames directly to memory buffers |
| 203 |
via the SMFC and an IDMAC channel, bypassing IC pre-processing. This |
| 204 |
source pad is routed to a capture device node, with a node name of the |
| 205 |
format "ipuX_csiY capture". |
| 206 |
|
| 207 |
Note that since the IDMAC source pad makes use of an IDMAC channel, |
| 208 |
pixel reordering within the same colorspace can be carried out by the |
| 209 |
IDMAC channel. For example, if the CSI sink pad is receiving in UYVY |
| 210 |
order, the capture device linked to the IDMAC source pad can capture |
| 211 |
in YUYV order. Also, if the CSI sink pad is receiving a packed YUV |
| 212 |
format, the capture device can capture a planar YUV format such as |
| 213 |
YUV420. |
| 214 |
|
| 215 |
The IDMAC channel at the IDMAC source pad also supports simple |
| 216 |
interweave without motion compensation, which is activated if the source |
| 217 |
pad's field type is sequential top-bottom or bottom-top, and the |
| 218 |
requested capture interface field type is set to interlaced (t-b, b-t, |
| 219 |
or unqualified interlaced). The capture interface will enforce the same |
| 220 |
field order as the source pad field order (interlaced-bt if source pad |
| 221 |
is seq-bt, interlaced-tb if source pad is seq-tb). |
| 222 |
|
| 223 |
For events produced by ipuX_csiY, see ref:`imx_api_ipuX_csiY`. |
| 224 |
|
| 225 |
Cropping in ipuX_csiY |
| 226 |
--------------------- |
| 227 |
|
| 228 |
The CSI supports cropping the incoming raw sensor frames. This is |
| 229 |
implemented in the ipuX_csiY entities at the sink pad, using the |
| 230 |
crop selection subdev API. |
| 231 |
|
| 232 |
The CSI also supports fixed divide-by-two downscaling independently in |
| 233 |
width and height. This is implemented in the ipuX_csiY entities at |
| 234 |
the sink pad, using the compose selection subdev API. |
| 235 |
|
| 236 |
The output rectangle at the ipuX_csiY source pad is the same as |
| 237 |
the compose rectangle at the sink pad. So the source pad rectangle |
| 238 |
cannot be negotiated, it must be set using the compose selection |
| 239 |
API at sink pad (if /2 downscale is desired, otherwise source pad |
| 240 |
rectangle is equal to incoming rectangle). |
| 241 |
|
| 242 |
To give an example of crop and /2 downscale, this will crop a |
| 243 |
1280x960 input frame to 640x480, and then /2 downscale in both |
| 244 |
dimensions to 320x240 (assumes ipu1_csi0 is linked to ipu1_csi0_mux): |
| 245 |
|
| 246 |
.. code-block:: none |
| 247 |
|
| 248 |
media-ctl -V "'ipu1_csi0_mux':2[fmt:UYVY2X8/1280x960]" |
| 249 |
media-ctl -V "'ipu1_csi0':0[crop:(0,0)/640x480]" |
| 250 |
media-ctl -V "'ipu1_csi0':0[compose:(0,0)/320x240]" |
| 251 |
|
| 252 |
Frame Skipping in ipuX_csiY |
| 253 |
--------------------------- |
| 254 |
|
| 255 |
The CSI supports frame rate decimation, via frame skipping. Frame |
| 256 |
rate decimation is specified by setting the frame intervals at |
| 257 |
sink and source pads. The ipuX_csiY entity then applies the best |
| 258 |
frame skip setting to the CSI to achieve the desired frame rate |
| 259 |
at the source pad. |
| 260 |
|
| 261 |
The following example reduces an assumed incoming 60 Hz frame |
| 262 |
rate by half at the IDMAC output source pad: |
| 263 |
|
| 264 |
.. code-block:: none |
| 265 |
|
| 266 |
media-ctl -V "'ipu1_csi0':0[fmt:UYVY2X8/640x480@1/60]" |
| 267 |
media-ctl -V "'ipu1_csi0':2[fmt:UYVY2X8/640x480@1/30]" |
| 268 |
|
| 269 |
Frame Interval Monitor in ipuX_csiY |
| 270 |
----------------------------------- |
| 271 |
|
| 272 |
See ref:`imx_api_FIM`. |
| 273 |
|
| 274 |
ipuX_vdic |
| 275 |
--------- |
| 276 |
|
| 277 |
The VDIC carries out motion compensated de-interlacing, with three |
| 278 |
motion compensation modes: low, medium, and high motion. The mode is |
| 279 |
specified with the menu control V4L2_CID_DEINTERLACING_MODE. The VDIC |
| 280 |
has two sink pads and a single source pad. |
| 281 |
|
| 282 |
The direct sink pad receives from an ipuX_csiY direct pad. With this |
| 283 |
link the VDIC can only operate in high motion mode. |
| 284 |
|
| 285 |
When the IDMAC sink pad is activated, it receives from an output |
| 286 |
or mem2mem device node. With this pipeline, the VDIC can also operate |
| 287 |
in low and medium modes, because these modes require receiving |
| 288 |
frames from memory buffers. Note that an output or mem2mem device |
| 289 |
is not implemented yet, so this sink pad currently has no links. |
| 290 |
|
| 291 |
The source pad routes to the IC pre-processing entity ipuX_ic_prp. |
| 292 |
|
| 293 |
ipuX_ic_prp |
| 294 |
----------- |
| 295 |
|
| 296 |
This is the IC pre-processing entity. It acts as a router, routing |
| 297 |
data from its sink pad to one or both of its source pads. |
| 298 |
|
| 299 |
This entity has a single sink pad. The sink pad can receive from the |
| 300 |
ipuX_csiY direct pad, or from ipuX_vdic. |
| 301 |
|
| 302 |
This entity has two source pads. One source pad routes to the |
| 303 |
pre-process encode task entity (ipuX_ic_prpenc), the other to the |
| 304 |
pre-process viewfinder task entity (ipuX_ic_prpvf). Both source pads |
| 305 |
can be activated at the same time if the sink pad is receiving from |
| 306 |
ipuX_csiY. Only the source pad to the pre-process viewfinder task entity |
| 307 |
can be activated if the sink pad is receiving from ipuX_vdic (frames |
| 308 |
from the VDIC can only be processed by the pre-process viewfinder task). |
| 309 |
|
| 310 |
ipuX_ic_prpenc |
| 311 |
-------------- |
| 312 |
|
| 313 |
This is the IC pre-processing encode entity. It has a single sink |
| 314 |
pad from ipuX_ic_prp, and a single source pad. The source pad is |
| 315 |
routed to a capture device node, with a node name of the format |
| 316 |
"ipuX_ic_prpenc capture". |
| 317 |
|
| 318 |
This entity performs the IC pre-process encode task operations: |
| 319 |
color-space conversion, resizing (downscaling and upscaling), |
| 320 |
horizontal and vertical flip, and 90/270 degree rotation. Flip |
| 321 |
and rotation are provided via standard V4L2 controls. |
| 322 |
|
| 323 |
Like the ipuX_csiY IDMAC source, this entity also supports simple |
| 324 |
de-interlace without motion compensation, and pixel reordering. |
| 325 |
|
| 326 |
ipuX_ic_prpvf |
| 327 |
------------- |
| 328 |
|
| 329 |
This is the IC pre-processing viewfinder entity. It has a single sink |
| 330 |
pad from ipuX_ic_prp, and a single source pad. The source pad is routed |
| 331 |
to a capture device node, with a node name of the format |
| 332 |
"ipuX_ic_prpvf capture". |
| 333 |
|
| 334 |
This entity is identical in operation to ipuX_ic_prpenc, with the same |
| 335 |
resizing and CSC operations and flip/rotation controls. It will receive |
| 336 |
and process de-interlaced frames from the ipuX_vdic if ipuX_ic_prp is |
| 337 |
receiving from ipuX_vdic. |
| 338 |
|
| 339 |
Like the ipuX_csiY IDMAC source, this entity supports simple |
| 340 |
interweaving without motion compensation. However, note that if the |
| 341 |
ipuX_vdic is included in the pipeline (ipuX_ic_prp is receiving from |
| 342 |
ipuX_vdic), it's not possible to use interweave in ipuX_ic_prpvf, |
| 343 |
since the ipuX_vdic has already carried out de-interlacing (with |
| 344 |
motion compensation) and therefore the field type output from |
| 345 |
ipuX_vdic can only be none (progressive). |
| 346 |
|
| 347 |
Capture Pipelines |
| 348 |
----------------- |
| 349 |
|
| 350 |
The following describe the various use-cases supported by the pipelines. |
| 351 |
|
| 352 |
The links shown do not include the backend sensor, video mux, or mipi |
| 353 |
csi-2 receiver links. This depends on the type of sensor interface |
| 354 |
(parallel or mipi csi-2). So these pipelines begin with: |
| 355 |
|
| 356 |
sensor -> ipuX_csiY_mux -> ... |
| 357 |
|
| 358 |
for parallel sensors, or: |
| 359 |
|
| 360 |
sensor -> imx6-mipi-csi2 -> (ipuX_csiY_mux) -> ... |
| 361 |
|
| 362 |
for mipi csi-2 sensors. The imx6-mipi-csi2 receiver may need to route |
| 363 |
to the video mux (ipuX_csiY_mux) before sending to the CSI, depending |
| 364 |
on the mipi csi-2 virtual channel, hence ipuX_csiY_mux is shown in |
| 365 |
parenthesis. |
| 366 |
|
| 367 |
Unprocessed Video Capture: |
| 368 |
-------------------------- |
| 369 |
|
| 370 |
Send frames directly from sensor to camera device interface node, with |
| 371 |
no conversions, via ipuX_csiY IDMAC source pad: |
| 372 |
|
| 373 |
-> ipuX_csiY:2 -> ipuX_csiY capture |
| 374 |
|
| 375 |
IC Direct Conversions: |
| 376 |
---------------------- |
| 377 |
|
| 378 |
This pipeline uses the preprocess encode entity to route frames directly |
| 379 |
from the CSI to the IC, to carry out scaling up to 1024x1024 resolution, |
| 380 |
CSC, flipping, and image rotation: |
| 381 |
|
| 382 |
-> ipuX_csiY:1 -> 0:ipuX_ic_prp:1 -> 0:ipuX_ic_prpenc:1 -> ipuX_ic_prpenc capture |
| 383 |
|
| 384 |
Motion Compensated De-interlace: |
| 385 |
-------------------------------- |
| 386 |
|
| 387 |
This pipeline routes frames from the CSI direct pad to the VDIC entity to |
| 388 |
support motion-compensated de-interlacing (high motion mode only), |
| 389 |
scaling up to 1024x1024, CSC, flip, and rotation: |
| 390 |
|
| 391 |
-> ipuX_csiY:1 -> 0:ipuX_vdic:2 -> 0:ipuX_ic_prp:2 -> 0:ipuX_ic_prpvf:1 -> ipuX_ic_prpvf capture |
| 392 |
|
| 393 |
|
| 394 |
Usage Notes |
| 395 |
----------- |
| 396 |
|
| 397 |
To aid in configuration and for backward compatibility with V4L2 |
| 398 |
applications that access controls only from video device nodes, the |
| 399 |
capture device interfaces inherit controls from the active entities |
| 400 |
in the current pipeline, so controls can be accessed either directly |
| 401 |
from the subdev or from the active capture device interface. For |
| 402 |
example, the FIM controls are available either from the ipuX_csiY |
| 403 |
subdevs or from the active capture device. |
| 404 |
|
| 405 |
The following are specific usage notes for the Sabre* reference |
| 406 |
boards: |
| 407 |
|
| 408 |
|
| 409 |
i.MX6Q SabreLite with OV5642 and OV5640 |
| 410 |
--------------------------------------- |
| 411 |
|
| 412 |
This platform requires the OmniVision OV5642 module with a parallel |
| 413 |
camera interface, and the OV5640 module with a MIPI CSI-2 |
| 414 |
interface. Both modules are available from Boundary Devices: |
| 415 |
|
| 416 |
- https://boundarydevices.com/product/nit6x_5mp |
| 417 |
- https://boundarydevices.com/product/nit6x_5mp_mipi |
| 418 |
|
| 419 |
Note that if only one camera module is available, the other sensor |
| 420 |
node can be disabled in the device tree. |
| 421 |
|
| 422 |
The OV5642 module is connected to the parallel bus input on the i.MX |
| 423 |
internal video mux to IPU1 CSI0. It's i2c bus connects to i2c bus 2. |
| 424 |
|
| 425 |
The MIPI CSI-2 OV5640 module is connected to the i.MX internal MIPI CSI-2 |
| 426 |
receiver, and the four virtual channel outputs from the receiver are |
| 427 |
routed as follows: vc0 to the IPU1 CSI0 mux, vc1 directly to IPU1 CSI1, |
| 428 |
vc2 directly to IPU2 CSI0, and vc3 to the IPU2 CSI1 mux. The OV5640 is |
| 429 |
also connected to i2c bus 2 on the SabreLite, therefore the OV5642 and |
| 430 |
OV5640 must not share the same i2c slave address. |
| 431 |
|
| 432 |
The following basic example configures unprocessed video capture |
| 433 |
pipelines for both sensors. The OV5642 is routed to ipu1_csi0, and |
| 434 |
the OV5640, transmitting on MIPI CSI-2 virtual channel 1 (which is |
| 435 |
imx6-mipi-csi2 pad 2), is routed to ipu1_csi1. Both sensors are |
| 436 |
configured to output 640x480, and the OV5642 outputs YUYV2X8, the |
| 437 |
OV5640 UYVY2X8: |
| 438 |
|
| 439 |
.. code-block:: none |
| 440 |
|
| 441 |
# Setup links for OV5642 |
| 442 |
media-ctl -l "'ov5642 1-0042':0 -> 'ipu1_csi0_mux':1[1]" |
| 443 |
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" |
| 444 |
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" |
| 445 |
# Setup links for OV5640 |
| 446 |
media-ctl -l "'ov5640 1-0040':0 -> 'imx6-mipi-csi2':0[1]" |
| 447 |
media-ctl -l "'imx6-mipi-csi2':2 -> 'ipu1_csi1':0[1]" |
| 448 |
media-ctl -l "'ipu1_csi1':2 -> 'ipu1_csi1 capture':0[1]" |
| 449 |
# Configure pads for OV5642 pipeline |
| 450 |
media-ctl -V "'ov5642 1-0042':0 [fmt:YUYV2X8/640x480 field:none]" |
| 451 |
media-ctl -V "'ipu1_csi0_mux':2 [fmt:YUYV2X8/640x480 field:none]" |
| 452 |
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/640x480 field:none]" |
| 453 |
# Configure pads for OV5640 pipeline |
| 454 |
media-ctl -V "'ov5640 1-0040':0 [fmt:UYVY2X8/640x480 field:none]" |
| 455 |
media-ctl -V "'imx6-mipi-csi2':2 [fmt:UYVY2X8/640x480 field:none]" |
| 456 |
media-ctl -V "'ipu1_csi1':2 [fmt:AYUV32/640x480 field:none]" |
| 457 |
|
| 458 |
Streaming can then begin independently on the capture device nodes |
| 459 |
"ipu1_csi0 capture" and "ipu1_csi1 capture". The v4l2-ctl tool can |
| 460 |
be used to select any supported YUV pixelformat on the capture device |
| 461 |
nodes, including planar. |
| 462 |
|
| 463 |
i.MX6Q SabreAuto with ADV7180 decoder |
| 464 |
------------------------------------- |
| 465 |
|
| 466 |
On the i.MX6Q SabreAuto, an on-board ADV7180 SD decoder is connected to the |
| 467 |
parallel bus input on the internal video mux to IPU1 CSI0. |
| 468 |
|
| 469 |
The following example configures a pipeline to capture from the ADV7180 |
| 470 |
video decoder, assuming NTSC 720x480 input signals, using simple |
| 471 |
interweave (unconverted and without motion compensation). The adv7180 |
| 472 |
must output sequential or alternating fields (field type 'seq-bt' for |
| 473 |
NTSC, or 'alternate'): |
| 474 |
|
| 475 |
.. code-block:: none |
| 476 |
|
| 477 |
# Setup links |
| 478 |
media-ctl -l "'adv7180 3-0021':0 -> 'ipu1_csi0_mux':1[1]" |
| 479 |
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" |
| 480 |
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" |
| 481 |
# Configure pads |
| 482 |
media-ctl -V "'adv7180 3-0021':0 [fmt:UYVY2X8/720x480 field:seq-bt]" |
| 483 |
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x480]" |
| 484 |
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" |
| 485 |
# Configure "ipu1_csi0 capture" interface (assumed at /dev/video4) |
| 486 |
v4l2-ctl -d4 --set-fmt-video=field=interlaced_bt |
| 487 |
|
| 488 |
Streaming can then begin on /dev/video4. The v4l2-ctl tool can also be |
| 489 |
used to select any supported YUV pixelformat on /dev/video4. |
| 490 |
|
| 491 |
This example configures a pipeline to capture from the ADV7180 |
| 492 |
video decoder, assuming PAL 720x576 input signals, with Motion |
| 493 |
Compensated de-interlacing. The adv7180 must output sequential or |
| 494 |
alternating fields (field type 'seq-tb' for PAL, or 'alternate'). |
| 495 |
|
| 496 |
.. code-block:: none |
| 497 |
|
| 498 |
# Setup links |
| 499 |
media-ctl -l "'adv7180 3-0021':0 -> 'ipu1_csi0_mux':1[1]" |
| 500 |
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" |
| 501 |
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" |
| 502 |
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" |
| 503 |
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" |
| 504 |
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" |
| 505 |
# Configure pads |
| 506 |
media-ctl -V "'adv7180 3-0021':0 [fmt:UYVY2X8/720x576 field:seq-tb]" |
| 507 |
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/720x576]" |
| 508 |
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x576]" |
| 509 |
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x576 field:none]" |
| 510 |
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" |
| 511 |
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" |
| 512 |
# Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) |
| 513 |
v4l2-ctl -d2 --set-fmt-video=field=none |
| 514 |
|
| 515 |
Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be |
| 516 |
used to select any supported YUV pixelformat on /dev/video2. |
| 517 |
|
| 518 |
This platform accepts Composite Video analog inputs to the ADV7180 on |
| 519 |
Ain1 (connector J42). |
| 520 |
|
| 521 |
i.MX6DL SabreAuto with ADV7180 decoder |
| 522 |
-------------------------------------- |
| 523 |
|
| 524 |
On the i.MX6DL SabreAuto, an on-board ADV7180 SD decoder is connected to the |
| 525 |
parallel bus input on the internal video mux to IPU1 CSI0. |
| 526 |
|
| 527 |
The following example configures a pipeline to capture from the ADV7180 |
| 528 |
video decoder, assuming NTSC 720x480 input signals, using simple |
| 529 |
interweave (unconverted and without motion compensation). The adv7180 |
| 530 |
must output sequential or alternating fields (field type 'seq-bt' for |
| 531 |
NTSC, or 'alternate'): |
| 532 |
|
| 533 |
.. code-block:: none |
| 534 |
|
| 535 |
# Setup links |
| 536 |
media-ctl -l "'adv7180 4-0021':0 -> 'ipu1_csi0_mux':4[1]" |
| 537 |
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]" |
| 538 |
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" |
| 539 |
# Configure pads |
| 540 |
media-ctl -V "'adv7180 4-0021':0 [fmt:UYVY2X8/720x480 field:seq-bt]" |
| 541 |
media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x480]" |
| 542 |
media-ctl -V "'ipu1_csi0':2 [fmt:AYUV32/720x480]" |
| 543 |
# Configure "ipu1_csi0 capture" interface (assumed at /dev/video0) |
| 544 |
v4l2-ctl -d0 --set-fmt-video=field=interlaced_bt |
| 545 |
|
| 546 |
Streaming can then begin on /dev/video0. The v4l2-ctl tool can also be |
| 547 |
used to select any supported YUV pixelformat on /dev/video0. |
| 548 |
|
| 549 |
This example configures a pipeline to capture from the ADV7180 |
| 550 |
video decoder, assuming PAL 720x576 input signals, with Motion |
| 551 |
Compensated de-interlacing. The adv7180 must output sequential or |
| 552 |
alternating fields (field type 'seq-tb' for PAL, or 'alternate'). |
| 553 |
|
| 554 |
.. code-block:: none |
| 555 |
|
| 556 |
# Setup links |
| 557 |
media-ctl -l "'adv7180 4-0021':0 -> 'ipu1_csi0_mux':4[1]" |
| 558 |
media-ctl -l "'ipu1_csi0_mux':5 -> 'ipu1_csi0':0[1]" |
| 559 |
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_vdic':0[1]" |
| 560 |
media-ctl -l "'ipu1_vdic':2 -> 'ipu1_ic_prp':0[1]" |
| 561 |
media-ctl -l "'ipu1_ic_prp':2 -> 'ipu1_ic_prpvf':0[1]" |
| 562 |
media-ctl -l "'ipu1_ic_prpvf':1 -> 'ipu1_ic_prpvf capture':0[1]" |
| 563 |
# Configure pads |
| 564 |
media-ctl -V "'adv7180 4-0021':0 [fmt:UYVY2X8/720x576 field:seq-tb]" |
| 565 |
media-ctl -V "'ipu1_csi0_mux':5 [fmt:UYVY2X8/720x576]" |
| 566 |
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/720x576]" |
| 567 |
media-ctl -V "'ipu1_vdic':2 [fmt:AYUV32/720x576 field:none]" |
| 568 |
media-ctl -V "'ipu1_ic_prp':2 [fmt:AYUV32/720x576 field:none]" |
| 569 |
media-ctl -V "'ipu1_ic_prpvf':1 [fmt:AYUV32/720x576 field:none]" |
| 570 |
# Configure "ipu1_ic_prpvf capture" interface (assumed at /dev/video2) |
| 571 |
v4l2-ctl -d2 --set-fmt-video=field=none |
| 572 |
|
| 573 |
Streaming can then begin on /dev/video2. The v4l2-ctl tool can also be |
| 574 |
used to select any supported YUV pixelformat on /dev/video2. |
| 575 |
|
| 576 |
This platform accepts Composite Video analog inputs to the ADV7180 on |
| 577 |
Ain1 (connector J42). |
| 578 |
|
| 579 |
i.MX6Q SabreSD with MIPI CSI-2 OV5640 |
| 580 |
------------------------------------- |
| 581 |
|
| 582 |
Similarly to i.MX6Q SabreLite, the i.MX6Q SabreSD supports a parallel |
| 583 |
interface OV5642 module on IPU1 CSI0, and a MIPI CSI-2 OV5640 |
| 584 |
module. The OV5642 connects to i2c bus 1 and the OV5640 to i2c bus 2. |
| 585 |
|
| 586 |
The device tree for SabreSD includes OF graphs for both the parallel |
| 587 |
OV5642 and the MIPI CSI-2 OV5640, but as of this writing only the MIPI |
| 588 |
CSI-2 OV5640 has been tested, so the OV5642 node is currently disabled. |
| 589 |
The OV5640 module connects to MIPI connector J5. The NXP part number |
| 590 |
for the OV5640 module that connects to the SabreSD board is H120729. |
| 591 |
|
| 592 |
The following example configures unprocessed video capture pipeline to |
| 593 |
capture from the OV5640, transmitting on MIPI CSI-2 virtual channel 0: |
| 594 |
|
| 595 |
.. code-block:: none |
| 596 |
|
| 597 |
# Setup links |
| 598 |
media-ctl -l "'ov5640 1-003c':0 -> 'imx6-mipi-csi2':0[1]" |
| 599 |
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]" |
| 600 |
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" |
| 601 |
media-ctl -l "'ipu1_csi0':2 -> 'ipu1_csi0 capture':0[1]" |
| 602 |
# Configure pads |
| 603 |
media-ctl -V "'ov5640 1-003c':0 [fmt:UYVY2X8/640x480]" |
| 604 |
media-ctl -V "'imx6-mipi-csi2':1 [fmt:UYVY2X8/640x480]" |
| 605 |
media-ctl -V "'ipu1_csi0_mux':0 [fmt:UYVY2X8/640x480]" |
| 606 |
media-ctl -V "'ipu1_csi0':0 [fmt:AYUV32/640x480]" |
| 607 |
|
| 608 |
Streaming can then begin on "ipu1_csi0 capture" node. The v4l2-ctl |
| 609 |
tool can be used to select any supported pixelformat on the capture |
| 610 |
device node. |
| 611 |
|
| 612 |
To determine what is the /dev/video node correspondent to |
| 613 |
"ipu1_csi0 capture": |
| 614 |
|
| 615 |
.. code-block:: none |
| 616 |
|
| 617 |
media-ctl -e "ipu1_csi0 capture" |
| 618 |
/dev/video0 |
| 619 |
|
| 620 |
/dev/video0 is the streaming element in this case. |
| 621 |
|
| 622 |
Starting the streaming via v4l2-ctl: |
| 623 |
|
| 624 |
.. code-block:: none |
| 625 |
|
| 626 |
v4l2-ctl --stream-mmap -d /dev/video0 |
| 627 |
|
| 628 |
Starting the streaming via Gstreamer and sending the content to the display: |
| 629 |
|
| 630 |
.. code-block:: none |
| 631 |
|
| 632 |
gst-launch-1.0 v4l2src device=/dev/video0 ! kmssink |
| 633 |
|
| 634 |
The following example configures a direct conversion pipeline to capture |
| 635 |
from the OV5640, transmitting on MIPI CSI-2 virtual channel 0. It also |
| 636 |
shows colorspace conversion and scaling at IC output. |
| 637 |
|
| 638 |
.. code-block:: none |
| 639 |
|
| 640 |
# Setup links |
| 641 |
media-ctl -l "'ov5640 1-003c':0 -> 'imx6-mipi-csi2':0[1]" |
| 642 |
media-ctl -l "'imx6-mipi-csi2':1 -> 'ipu1_csi0_mux':0[1]" |
| 643 |
media-ctl -l "'ipu1_csi0_mux':2 -> 'ipu1_csi0':0[1]" |
| 644 |
media-ctl -l "'ipu1_csi0':1 -> 'ipu1_ic_prp':0[1]" |
| 645 |
media-ctl -l "'ipu1_ic_prp':1 -> 'ipu1_ic_prpenc':0[1]" |
| 646 |
media-ctl -l "'ipu1_ic_prpenc':1 -> 'ipu1_ic_prpenc capture':0[1]" |
| 647 |
# Configure pads |
| 648 |
media-ctl -V "'ov5640 1-003c':0 [fmt:UYVY2X8/640x480]" |
| 649 |
media-ctl -V "'imx6-mipi-csi2':1 [fmt:UYVY2X8/640x480]" |
| 650 |
media-ctl -V "'ipu1_csi0_mux':2 [fmt:UYVY2X8/640x480]" |
| 651 |
media-ctl -V "'ipu1_csi0':1 [fmt:AYUV32/640x480]" |
| 652 |
media-ctl -V "'ipu1_ic_prp':1 [fmt:AYUV32/640x480]" |
| 653 |
media-ctl -V "'ipu1_ic_prpenc':1 [fmt:ARGB8888_1X32/800x600]" |
| 654 |
# Set a format at the capture interface |
| 655 |
v4l2-ctl -d /dev/video1 --set-fmt-video=pixelformat=RGB3 |
| 656 |
|
| 657 |
Streaming can then begin on "ipu1_ic_prpenc capture" node. |
| 658 |
|
| 659 |
To determine what is the /dev/video node correspondent to |
| 660 |
"ipu1_ic_prpenc capture": |
| 661 |
|
| 662 |
.. code-block:: none |
| 663 |
|
| 664 |
media-ctl -e "ipu1_ic_prpenc capture" |
| 665 |
/dev/video1 |
| 666 |
|
| 667 |
|
| 668 |
/dev/video1 is the streaming element in this case. |
| 669 |
|
| 670 |
Starting the streaming via v4l2-ctl: |
| 671 |
|
| 672 |
.. code-block:: none |
| 673 |
|
| 674 |
v4l2-ctl --stream-mmap -d /dev/video1 |
| 675 |
|
| 676 |
Starting the streaming via Gstreamer and sending the content to the display: |
| 677 |
|
| 678 |
.. code-block:: none |
| 679 |
|
| 680 |
gst-launch-1.0 v4l2src device=/dev/video1 ! kmssink |
| 681 |
|
| 682 |
Known Issues |
| 683 |
------------ |
| 684 |
|
| 685 |
1. When using 90 or 270 degree rotation control at capture resolutions |
| 686 |
near the IC resizer limit of 1024x1024, and combined with planar |
| 687 |
pixel formats (YUV420, YUV422p), frame capture will often fail with |
| 688 |
no end-of-frame interrupts from the IDMAC channel. To work around |
| 689 |
this, use lower resolution and/or packed formats (YUYV, RGB3, etc.) |
| 690 |
when 90 or 270 rotations are needed. |
| 691 |
|
| 692 |
|
| 693 |
File list |
| 694 |
--------- |
| 695 |
|
| 696 |
drivers/staging/media/imx/ |
| 697 |
include/media/imx.h |
| 698 |
include/linux/imx-media.h |
| 699 |
|
| 700 |
References |
| 701 |
---------- |
| 702 |
|
| 703 |
.. [#f1] http://www.nxp.com/assets/documents/data/en/reference-manuals/IMX6DQRM.pdf |
| 704 |
.. [#f2] http://www.nxp.com/assets/documents/data/en/reference-manuals/IMX6SDLRM.pdf |
| 705 |
|
| 706 |
|
| 707 |
Authors |
| 708 |
------- |
| 709 |
|
| 710 |
- Steve Longerbeam <[email protected]> |
| 711 |
- Philipp Zabel <[email protected]> |
| 712 |
- Russell King <[email protected]> |
| 713 |
|
| 714 |
Copyright (C) 2012-2017 Mentor Graphics Inc. |