Debian Codegraph @

doc/spec/source/programming_guide/VPL_prg_vpp.rst

raw
1 .. SPDX-FileCopyrightText: 2019-2020 Intel Corporation
2 ..
3 .. SPDX-License-Identifier: CC-BY-4.0
4
5 .. _vid_process_procedure:
6
7 ===========================
8 Video Processing Procedures
9 ===========================
10
11 The following pseudo code shows the video processing procedure:
12
13 .. literalinclude:: ../snippets/prg_vpp.c
14 :language: c++
15 :start-after: /*beg1*/
16 :end-before: /*end1*/
17 :lineno-start: 1
18
19 Note the following key points about the example:
20
21 - The application uses the :cpp:func:`MFXVideoVPP_QueryIOSurf` function to
22 obtain the number of frame surfaces needed for input and output. The application
23 must allocate two frame surface pools: one for the input and one for the output.
24 - The video processing function :cpp:func:`MFXVideoVPP_RunFrameVPPAsync` is
25 asynchronous. The application must use the :cpp:func:`MFXVideoCORE_SyncOperation`
26 function to synchronize in order to make the output result ready.
27 - The body of the video processing procedure covers the following three scenarios:
28
29 - If the number of frames consumed at input is equal to the number of frames
30 generated at output, :term:`VPP` returns :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` when an
31 output is ready. The application must process the output frame after
32 synchronization, as the :cpp:func:`MFXVideoVPP_RunFrameVPPAsync` function is
33 asynchronous. The application must provide a NULL input at the end of the
34 sequence to drain any remaining frames.
35 - If the number of frames consumed at input is more than the number of frames
36 generated at output, VPP returns :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA` for
37 additional input until an output is ready. When the output is ready, VPP
38 returns :cpp:enumerator:`mfxStatus::MFX_ERR_NONE`. The application must process the
39 output frame after synchronization and provide a NULL input at the end of the
40 sequence to drain any remaining frames.
41 - If the number of frames consumed at input is less than the number of frames
42 generated at output, VPP returns either :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_SURFACE`
43 (when more than one output is ready), or :cpp:enumerator:`mfxStatus::MFX_ERR_NONE`
44 (when one output is ready and VPP expects new input). In both cases, the
45 application must process the output frame after synchronization and provide
46 a NULL input at the end of the sequence to drain any remaining frames.
47
48 -------------
49 Configuration
50 -------------
51
52 |vpl_short_name| configures the video processing pipeline operation based on the
53 difference between the input and output formats, specified in the
54 :cpp:struct:`mfxVideoParam` structure. The following list shows several examples:
55
56 - When the input color format is :term:`YUY2` and the output color format is
57 :term:`NV12`, |vpl_short_name| enables color conversion from YUY2 to NV12.
58 - When the input is interleaved and the output is progressive, |vpl_short_name| enables
59 deinterlacing.
60 - When the input is single field and the output is interlaced or progressive,
61 |vpl_short_name| enables field weaving, optionally with deinterlacing.
62 - When the input is interlaced and the output is single field, |vpl_short_name| enables
63 field splitting.
64
65 In addition to specifying the input and output formats, the application can
66 provide hints to fine-tune the video processing pipeline operation. The
67 application can disable filters in the pipeline by using the
68 :cpp:struct:`mfxExtVPPDoNotUse` structure, enable filters by using the
69 :cpp:struct:`mfxExtVPPDoUse` structure, and configure filters by using dedicated
70 configuration structures. See the :ref:`Configurable VPP Filters table <vpp-filters-table>`
71 for a complete list of configurable video processing filters, their IDs, and
72 configuration structures. See the :ref:`ExtendedBufferID enumerator <extendedbufferid>`
73 for more details.
74
75 |vpl_short_name| ensures that all filters necessary to convert the input format to the
76 output format are included in the pipeline. |vpl_short_name| may skip some optional
77 filters even if they are explicitly requested by the application, for example
78 due to limitations of the underlying hardware. To notify the application about
79 skipped optional filters, |vpl_short_name| returns the :cpp:enumerator:`mfxStatus::MFX_WRN_FILTER_SKIPPED`
80 warning. The application can retrieve the list of active filters by attaching
81 the :cpp:struct:`mfxExtVPPDoUse` structure to the :cpp:struct:`mfxVideoParam`
82 structure and calling the :cpp:func:`MFXVideoVPP_GetVideoParam` function. The
83 application must allocate enough memory for the filter list.
84
85 See the :ref:`Configurable VPP Filters table <vpp-filters-table>` for a full
86 list of configurable filters.
87
88 .. _vpp-filters-table:
89
90 .. list-table:: Configurable VPP Filters
91 :header-rows: 1
92 :widths: 58 42
93
94 * - **Filter ID**
95 - **Configuration Structure**
96 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_DENOISE2`
97 - :cpp:struct:`mfxExtVPPDenoise2`
98 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_MCTF`
99 - :cpp:struct:`mfxExtVppMctf`
100 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_DETAIL`
101 - :cpp:struct:`mfxExtVPPDetail`
102 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION`
103 - :cpp:struct:`mfxExtVPPFrameRateConversion`
104 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_IMAGE_STABILIZATION`
105 - :cpp:struct:`mfxExtVPPImageStab`
106 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_PROCAMP`
107 - :cpp:struct:`mfxExtVPPProcAmp`
108 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_FIELD_PROCESSING`
109 - :cpp:struct:`mfxExtVPPFieldProcessing`
110 * - :cpp:enumerator:`MFX_EXTBUFF_VPP_3DLUT`
111 - :cpp:struct:`mfxExtVPP3DLut`
112
113 The following example shows video processing configuration:
114
115 .. literalinclude:: ../snippets/prg_vpp.c
116 :language: c++
117 :start-after: /*beg2*/
118 :end-before: /*end2*/
119 :lineno-start: 1
120
121 ------------------
122 Region of Interest
123 ------------------
124
125 During video processing operations, the application can specify a region of
126 interest for each frame as shown in the following figure:
127
128 .. figure:: ../images/vpp_region_of_interest_operation.png
129 :alt: VPP region of interest operation
130
131 VPP region of interest operation
132
133 Specifying a region of interest guides the resizing function to achieve special
134 effects, such as resizing from 16:9 to 4:3, while keeping the aspect ratio intact.
135 Use the ``CropX``, ``CropY``, ``CropW``, and ``CropH`` parameters in the
136 :cpp:struct:`mfxVideoParam` structure to specify a region of interest for each frame
137 when calling :cpp:func:`MFXVideoVPP_RunFrameVPPAsync`. Note: For per-frame dynamic
138 change, the application should set the ``CropX``, ``CropY``, ``CropW``, and ``CropH``
139 parameters when calling :cpp:func:`MFXVideoVPP_RunFrameVPPAsync` per frame.
140
141 The :ref:`VPP Region of Interest Operations table <vpp-region-op-table>` shows
142 examples of VPP operations applied to a region of interest.
143
144 .. _vpp-region-op-table:
145
146 .. list-table:: VPP Region of Interest Operations
147 :header-rows: 1
148 :widths: 30 15 20 15 20
149
150 * - | **Operation**
151 |
152 |
153 - | **VPP Input**
154 | *Width X Height*
155 |
156 - | **VPP Input**
157 | *CropX, CropY,*
158 | *CropW, CropH*
159 - | **VPP Output**
160 | *Width X Height*
161 |
162 - | **VPP Output**
163 | *CropX, CropY,*
164 | *CropW, CropH*
165 * - Cropping
166 - 720 x 480
167 - 16, 16, 688, 448
168 - 720 x 480
169 - 16, 16, 688, 448
170 * - Resizing
171 - 720 x 480
172 - 0, 0, 720, 480
173 - 1440 x 960
174 - 0, 0, 1440, 960
175 * - Horizontal stretching
176 - 720 x 480
177 - 0, 0, 720, 480
178 - 640 x 480
179 - 0, 0, 640, 480
180 * - 16:9 4:3 with letter boxing at the top and bottom
181 - 1920 x 1088
182 - 0, 0, 1920, 1088
183 - 720 x 480
184 - 0, 36, 720, 408
185 * - 4:3 16:9 with pillar boxing at the left and right
186 - 720 x 480
187 - 0, 0, 720, 480
188 - 1920 x 1088
189 - 144, 0, 1632, 1088
190
191
192
193 ---------------------------
194 Multi-view Video Processing
195 ---------------------------
196
197 |vpl_short_name| video processing supports processing multiple views. For video processing
198 initialization, the application needs to attach the :cpp:struct:`mfxExtMVCSeqDesc`
199 structure to the :cpp:struct:`mfxVideoParam` structure and call the
200 :cpp:func:`MFXVideoVPP_Init` function. The function saves the view identifiers.
201 During video processing, |vpl_short_name| processes each view individually. |vpl_short_name| refers
202 to the ``FrameID`` field of the :cpp:struct:`mfxFrameInfo` structure to configure
203 each view according to its processing pipeline. If the video processing source
204 frame is not the output from the |vpl_short_name| MVC decoder, then the application needs to
205 fill the the ``FrameID`` field before calling the :cpp:func:`MFXVideoVPP_RunFrameVPPAsync`
206 function. This is shown in the following pseudo code:
207
208 .. literalinclude:: ../snippets/prg_vpp.c
209 :language: c++
210 :start-after: /*beg3*/
211 :end-before: /*end3*/
212 :lineno-start: 1
213
214 ----------------------
215 Video Processing 3DLUT
216 ----------------------
217
218 |vpl_short_name| video processing supports 3DLUT with Intel HW specific memory layout. The following pseudo code
219 shows how to create a :cpp:enumerator:`MFX_3DLUT_MEMORY_LAYOUT_INTEL_65LUT` 3DLUT surface.
220
221 .. literalinclude:: ../snippets/prg_vpp.c
222 :language: c++
223 :start-after: /*beg4*/
224 :end-before: /*end4*/
225 :lineno-start: 1
226
227 The following pseudo code shows how to create a system memory :cpp:struct:`mfx3DLutSystemBuffer` 3DLUT surface.
228
229 .. literalinclude:: ../snippets/prg_vpp.c
230 :language: c++
231 :start-after: /*beg5*/
232 :end-before: /*end5*/
233 :lineno-start: 1
234
235 The following pseudo code shows how to specify 3DLUT interpolation method :cpp:enum:`mfx3DLutInterpolationMethod`.
236
237 .. literalinclude:: ../snippets/prg_vpp.c
238 :language: c++
239 :start-after: /*beg11*/
240 :end-before: /*end11*/
241 :lineno-start: 1
242
243 ----------------
244 HDR Tone Mapping
245 ----------------
246
247 |vpl_short_name| video processing supports HDR Tone Mapping with Intel HW. The following pseudo code
248 shows how to perform HDR Tone Mapping.
249
250 The following pseudo code shows HDR to SDR.
251
252 .. literalinclude:: ../snippets/prg_vpp.c
253 :language: c++
254 :start-after: /*beg6*/
255 :end-before: /*end6*/
256 :lineno-start: 1
257
258 The following pseudo code shows SDR to HDR.
259
260 .. literalinclude:: ../snippets/prg_vpp.c
261 :language: c++
262 :start-after: /*beg7*/
263 :end-before: /*end7*/
264 :lineno-start: 1
265
266 The following pseudo code shows HDR to HDR.
267
268 .. literalinclude:: ../snippets/prg_vpp.c
269 :language: c++
270 :start-after: /*beg8*/
271 :end-before: /*end8*/
272 :lineno-start: 1
273
274 -----------------------
275 Camera RAW acceleration
276 -----------------------
277
278 |vpl_short_name| supports camera raw format processing with Intel HW. The following pseudo code
279 shows how to perform camera raw hardware acceleration. For pipeline processing initialization,
280 the application needs to attach the camera structures to the :cpp:struct:`mfxVideoParam` structure
281 and call the :cpp:func:`MFXVideoVPP_Init` function.
282
283 The following pseudo code shows camera raw processing.
284
285 .. literalinclude:: ../snippets/prg_vpp.c
286 :language: c++
287 :start-after: /*beg9*/
288 :end-before: /*end9*/
289 :lineno-start: 1
290
291 |vpl_short_name| supports AI based super resolution with Intel HW. The following pseudo code
292 shows how to perform AI based super resolution.
293
294 ---------------------------
295 AI Powered Video Processing
296 ---------------------------
297
298 The following pseudo code shows AI super resolution video processing.
299
300 .. literalinclude:: ../snippets/prg_vpp.c
301 :language: c++
302 :start-after: /*beg12*/
303 :end-before: /*end12*/
304 :lineno-start: 1
305
306 The following pseudo code shows AI frame interpolation video processing.
307
308 .. literalinclude:: ../snippets/prg_vpp.c
309 :language: c++
310 :start-after: /*beg13*/
311 :end-before: /*end13*/
312 :lineno-start: 1