| 1 |
.. SPDX-License-Identifier: GPL-2.0 |
| 2 |
|
| 3 |
========================================================= |
| 4 |
Raspberry Pi PiSP Back End Memory-to-Memory ISP (pisp-be) |
| 5 |
========================================================= |
| 6 |
|
| 7 |
The PiSP Back End |
| 8 |
================= |
| 9 |
|
| 10 |
The PiSP Back End is a memory-to-memory Image Signal Processor (ISP) which reads |
| 11 |
image data from DRAM memory and performs image processing as specified by the |
| 12 |
application through the parameters in a configuration buffer, before writing |
| 13 |
pixel data back to memory through two distinct output channels. |
| 14 |
|
| 15 |
The ISP registers and programming model are documented in the `Raspberry Pi |
| 16 |
Image Signal Processor (PiSP) Specification document`_ |
| 17 |
|
| 18 |
The PiSP Back End ISP processes images in tiles. The handling of image |
| 19 |
tessellation and the computation of low-level configuration parameters is |
| 20 |
realized by a free software library called `libpisp |
| 21 |
<https://github.com/raspberrypi/libpisp>`_. |
| 22 |
|
| 23 |
The full image processing pipeline, which involves capturing RAW Bayer data from |
| 24 |
an image sensor through a MIPI CSI-2 compatible capture interface, storing them |
| 25 |
in DRAM memory and processing them in the PiSP Back End to obtain images usable |
| 26 |
by an application is implemented in `libcamera <https://libcamera.org>`_ as |
| 27 |
part of the Raspberry Pi platform support. |
| 28 |
|
| 29 |
The pisp-be driver |
| 30 |
================== |
| 31 |
|
| 32 |
The Raspberry Pi PiSP Back End (pisp-be) driver is located under |
| 33 |
drivers/media/platform/raspberrypi/pisp-be. It uses the `V4L2 API` to register |
| 34 |
a number of video capture and output devices, the `V4L2 subdev API` to register |
| 35 |
a subdevice for the ISP that connects the video devices in a single media graph |
| 36 |
realized using the `Media Controller (MC) API`. |
| 37 |
|
| 38 |
The media topology registered by the `pisp-be` driver is represented below: |
| 39 |
|
| 40 |
.. _pips-be-topology: |
| 41 |
|
| 42 |
.. kernel-figure:: raspberrypi-pisp-be.dot |
| 43 |
:alt: Diagram of the default media pipeline topology |
| 44 |
:align: center |
| 45 |
|
| 46 |
|
| 47 |
The media graph registers the following video device nodes: |
| 48 |
|
| 49 |
- pispbe-input: output device for images to be submitted to the ISP for |
| 50 |
processing. |
| 51 |
- pispbe-tdn_input: output device for temporal denoise. |
| 52 |
- pispbe-stitch_input: output device for image stitching (HDR). |
| 53 |
- pispbe-output0: first capture device for processed images. |
| 54 |
- pispbe-output1: second capture device for processed images. |
| 55 |
- pispbe-tdn_output: capture device for temporal denoise. |
| 56 |
- pispbe-stitch_output: capture device for image stitching (HDR). |
| 57 |
- pispbe-config: output device for ISP configuration parameters. |
| 58 |
|
| 59 |
pispbe-input |
| 60 |
------------ |
| 61 |
|
| 62 |
Images to be processed by the ISP are queued to the `pispbe-input` output device |
| 63 |
node. For a list of image formats supported as input to the ISP refer to the |
| 64 |
`Raspberry Pi Image Signal Processor (PiSP) Specification document`_. |
| 65 |
|
| 66 |
pispbe-tdn_input, pispbe-tdn_output |
| 67 |
----------------------------------- |
| 68 |
|
| 69 |
The `pispbe-tdn_input` output video device receives images to be processed by |
| 70 |
the temporal denoise block which are captured from the `pispbe-tdn_output` |
| 71 |
capture video device. Userspace is responsible for maintaining queues on both |
| 72 |
devices, and ensuring that buffers completed on the output are queued to the |
| 73 |
input. |
| 74 |
|
| 75 |
pispbe-stitch_input, pispbe-stitch_output |
| 76 |
----------------------------------------- |
| 77 |
|
| 78 |
To realize HDR (high dynamic range) image processing the image stitching and |
| 79 |
tonemapping blocks are used. The `pispbe-stitch_output` writes images to memory |
| 80 |
and the `pispbe-stitch_input` receives the previously written frame to process |
| 81 |
it along with the current input image. Userspace is responsible for maintaining |
| 82 |
queues on both devices, and ensuring that buffers completed on the output are |
| 83 |
queued to the input. |
| 84 |
|
| 85 |
pispbe-output0, pispbe-output1 |
| 86 |
------------------------------ |
| 87 |
|
| 88 |
The two capture devices write to memory the pixel data as processed by the ISP. |
| 89 |
|
| 90 |
pispbe-config |
| 91 |
------------- |
| 92 |
|
| 93 |
The `pispbe-config` output video devices receives a buffer of configuration |
| 94 |
parameters that define the desired image processing to be performed by the ISP. |
| 95 |
|
| 96 |
The format of the ISP configuration parameter is defined by |
| 97 |
:c:type:`pisp_be_tiles_config` C structure and the meaning of each parameter is |
| 98 |
described in the `Raspberry Pi Image Signal Processor (PiSP) Specification |
| 99 |
document`_. |
| 100 |
|
| 101 |
ISP configuration |
| 102 |
================= |
| 103 |
|
| 104 |
The ISP configuration is described solely by the content of the parameters |
| 105 |
buffer. The only parameter that userspace needs to configure using the V4L2 API |
| 106 |
is the image format on the output and capture video devices for validation of |
| 107 |
the content of the parameters buffer. |
| 108 |
|
| 109 |
.. _Raspberry Pi Image Signal Processor (PiSP) Specification document: https://datasheets.raspberrypi.com/camera/raspberry-pi-image-signal-processor-specification.pdf |