| 1 |
.. SPDX-FileCopyrightText: 2019-2020 Intel Corporation |
| 2 |
.. |
| 3 |
.. SPDX-License-Identifier: CC-BY-4.0 |
| 4 |
|
| 5 |
.. _vid_process_procedure: |
| 6 |
|
| 7 |
=========================== |
| 8 |
Video Processing Procedures |
| 9 |
=========================== |
| 10 |
|
| 11 |
The following pseudo code shows the video processing procedure: |
| 12 |
|
| 13 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 14 |
:language: c++ |
| 15 |
:start-after: /*beg1*/ |
| 16 |
:end-before: /*end1*/ |
| 17 |
:lineno-start: 1 |
| 18 |
|
| 19 |
Note the following key points about the example: |
| 20 |
|
| 21 |
- The application uses the :cpp:func:`MFXVideoVPP_QueryIOSurf` function to |
| 22 |
obtain the number of frame surfaces needed for input and output. The application |
| 23 |
must allocate two frame surface pools: one for the input and one for the output. |
| 24 |
- The video processing function :cpp:func:`MFXVideoVPP_RunFrameVPPAsync` is |
| 25 |
asynchronous. The application must use the :cpp:func:`MFXVideoCORE_SyncOperation` |
| 26 |
function to synchronize in order to make the output result ready. |
| 27 |
- The body of the video processing procedure covers the following three scenarios: |
| 28 |
|
| 29 |
- If the number of frames consumed at input is equal to the number of frames |
| 30 |
generated at output, :term:`VPP` returns :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` when an |
| 31 |
output is ready. The application must process the output frame after |
| 32 |
synchronization, as the :cpp:func:`MFXVideoVPP_RunFrameVPPAsync` function is |
| 33 |
asynchronous. The application must provide a NULL input at the end of the |
| 34 |
sequence to drain any remaining frames. |
| 35 |
- If the number of frames consumed at input is more than the number of frames |
| 36 |
generated at output, VPP returns :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA` for |
| 37 |
additional input until an output is ready. When the output is ready, VPP |
| 38 |
returns :cpp:enumerator:`mfxStatus::MFX_ERR_NONE`. The application must process the |
| 39 |
output frame after synchronization and provide a NULL input at the end of the |
| 40 |
sequence to drain any remaining frames. |
| 41 |
- If the number of frames consumed at input is less than the number of frames |
| 42 |
generated at output, VPP returns either :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_SURFACE` |
| 43 |
(when more than one output is ready), or :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` |
| 44 |
(when one output is ready and VPP expects new input). In both cases, the |
| 45 |
application must process the output frame after synchronization and provide |
| 46 |
a NULL input at the end of the sequence to drain any remaining frames. |
| 47 |
|
| 48 |
------------- |
| 49 |
Configuration |
| 50 |
------------- |
| 51 |
|
| 52 |
|vpl_short_name| configures the video processing pipeline operation based on the |
| 53 |
difference between the input and output formats, specified in the |
| 54 |
:cpp:struct:`mfxVideoParam` structure. The following list shows several examples: |
| 55 |
|
| 56 |
- When the input color format is :term:`YUY2` and the output color format is |
| 57 |
:term:`NV12`, |vpl_short_name| enables color conversion from YUY2 to NV12. |
| 58 |
- When the input is interleaved and the output is progressive, |vpl_short_name| enables |
| 59 |
deinterlacing. |
| 60 |
- When the input is single field and the output is interlaced or progressive, |
| 61 |
|vpl_short_name| enables field weaving, optionally with deinterlacing. |
| 62 |
- When the input is interlaced and the output is single field, |vpl_short_name| enables |
| 63 |
field splitting. |
| 64 |
|
| 65 |
In addition to specifying the input and output formats, the application can |
| 66 |
provide hints to fine-tune the video processing pipeline operation. The |
| 67 |
application can disable filters in the pipeline by using the |
| 68 |
:cpp:struct:`mfxExtVPPDoNotUse` structure, enable filters by using the |
| 69 |
:cpp:struct:`mfxExtVPPDoUse` structure, and configure filters by using dedicated |
| 70 |
configuration structures. See the :ref:`Configurable VPP Filters table <vpp-filters-table>` |
| 71 |
for a complete list of configurable video processing filters, their IDs, and |
| 72 |
configuration structures. See the :ref:`ExtendedBufferID enumerator <extendedbufferid>` |
| 73 |
for more details. |
| 74 |
|
| 75 |
|vpl_short_name| ensures that all filters necessary to convert the input format to the |
| 76 |
output format are included in the pipeline. |vpl_short_name| may skip some optional |
| 77 |
filters even if they are explicitly requested by the application, for example |
| 78 |
due to limitations of the underlying hardware. To notify the application about |
| 79 |
skipped optional filters, |vpl_short_name| returns the :cpp:enumerator:`mfxStatus::MFX_WRN_FILTER_SKIPPED` |
| 80 |
warning. The application can retrieve the list of active filters by attaching |
| 81 |
the :cpp:struct:`mfxExtVPPDoUse` structure to the :cpp:struct:`mfxVideoParam` |
| 82 |
structure and calling the :cpp:func:`MFXVideoVPP_GetVideoParam` function. The |
| 83 |
application must allocate enough memory for the filter list. |
| 84 |
|
| 85 |
See the :ref:`Configurable VPP Filters table <vpp-filters-table>` for a full |
| 86 |
list of configurable filters. |
| 87 |
|
| 88 |
.. _vpp-filters-table: |
| 89 |
|
| 90 |
.. list-table:: Configurable VPP Filters |
| 91 |
:header-rows: 1 |
| 92 |
:widths: 58 42 |
| 93 |
|
| 94 |
* - **Filter ID** |
| 95 |
- **Configuration Structure** |
| 96 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_DENOISE2` |
| 97 |
- :cpp:struct:`mfxExtVPPDenoise2` |
| 98 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_MCTF` |
| 99 |
- :cpp:struct:`mfxExtVppMctf` |
| 100 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_DETAIL` |
| 101 |
- :cpp:struct:`mfxExtVPPDetail` |
| 102 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_FRAME_RATE_CONVERSION` |
| 103 |
- :cpp:struct:`mfxExtVPPFrameRateConversion` |
| 104 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_IMAGE_STABILIZATION` |
| 105 |
- :cpp:struct:`mfxExtVPPImageStab` |
| 106 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_PROCAMP` |
| 107 |
- :cpp:struct:`mfxExtVPPProcAmp` |
| 108 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_FIELD_PROCESSING` |
| 109 |
- :cpp:struct:`mfxExtVPPFieldProcessing` |
| 110 |
* - :cpp:enumerator:`MFX_EXTBUFF_VPP_3DLUT` |
| 111 |
- :cpp:struct:`mfxExtVPP3DLut` |
| 112 |
|
| 113 |
The following example shows video processing configuration: |
| 114 |
|
| 115 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 116 |
:language: c++ |
| 117 |
:start-after: /*beg2*/ |
| 118 |
:end-before: /*end2*/ |
| 119 |
:lineno-start: 1 |
| 120 |
|
| 121 |
------------------ |
| 122 |
Region of Interest |
| 123 |
------------------ |
| 124 |
|
| 125 |
During video processing operations, the application can specify a region of |
| 126 |
interest for each frame as shown in the following figure: |
| 127 |
|
| 128 |
.. figure:: ../images/vpp_region_of_interest_operation.png |
| 129 |
:alt: VPP region of interest operation |
| 130 |
|
| 131 |
VPP region of interest operation |
| 132 |
|
| 133 |
Specifying a region of interest guides the resizing function to achieve special |
| 134 |
effects, such as resizing from 16:9 to 4:3, while keeping the aspect ratio intact. |
| 135 |
Use the ``CropX``, ``CropY``, ``CropW``, and ``CropH`` parameters in the |
| 136 |
:cpp:struct:`mfxVideoParam` structure to specify a region of interest for each frame |
| 137 |
when calling :cpp:func:`MFXVideoVPP_RunFrameVPPAsync`. Note: For per-frame dynamic |
| 138 |
change, the application should set the ``CropX``, ``CropY``, ``CropW``, and ``CropH`` |
| 139 |
parameters when calling :cpp:func:`MFXVideoVPP_RunFrameVPPAsync` per frame. |
| 140 |
|
| 141 |
The :ref:`VPP Region of Interest Operations table <vpp-region-op-table>` shows |
| 142 |
examples of VPP operations applied to a region of interest. |
| 143 |
|
| 144 |
.. _vpp-region-op-table: |
| 145 |
|
| 146 |
.. list-table:: VPP Region of Interest Operations |
| 147 |
:header-rows: 1 |
| 148 |
:widths: 30 15 20 15 20 |
| 149 |
|
| 150 |
* - | **Operation** |
| 151 |
| |
| 152 |
| |
| 153 |
- | **VPP Input** |
| 154 |
| *Width X Height* |
| 155 |
| |
| 156 |
- | **VPP Input** |
| 157 |
| *CropX, CropY,* |
| 158 |
| *CropW, CropH* |
| 159 |
- | **VPP Output** |
| 160 |
| *Width X Height* |
| 161 |
| |
| 162 |
- | **VPP Output** |
| 163 |
| *CropX, CropY,* |
| 164 |
| *CropW, CropH* |
| 165 |
* - Cropping |
| 166 |
- 720 x 480 |
| 167 |
- 16, 16, 688, 448 |
| 168 |
- 720 x 480 |
| 169 |
- 16, 16, 688, 448 |
| 170 |
* - Resizing |
| 171 |
- 720 x 480 |
| 172 |
- 0, 0, 720, 480 |
| 173 |
- 1440 x 960 |
| 174 |
- 0, 0, 1440, 960 |
| 175 |
* - Horizontal stretching |
| 176 |
- 720 x 480 |
| 177 |
- 0, 0, 720, 480 |
| 178 |
- 640 x 480 |
| 179 |
- 0, 0, 640, 480 |
| 180 |
* - 16:9 4:3 with letter boxing at the top and bottom |
| 181 |
- 1920 x 1088 |
| 182 |
- 0, 0, 1920, 1088 |
| 183 |
- 720 x 480 |
| 184 |
- 0, 36, 720, 408 |
| 185 |
* - 4:3 16:9 with pillar boxing at the left and right |
| 186 |
- 720 x 480 |
| 187 |
- 0, 0, 720, 480 |
| 188 |
- 1920 x 1088 |
| 189 |
- 144, 0, 1632, 1088 |
| 190 |
|
| 191 |
|
| 192 |
|
| 193 |
--------------------------- |
| 194 |
Multi-view Video Processing |
| 195 |
--------------------------- |
| 196 |
|
| 197 |
|vpl_short_name| video processing supports processing multiple views. For video processing |
| 198 |
initialization, the application needs to attach the :cpp:struct:`mfxExtMVCSeqDesc` |
| 199 |
structure to the :cpp:struct:`mfxVideoParam` structure and call the |
| 200 |
:cpp:func:`MFXVideoVPP_Init` function. The function saves the view identifiers. |
| 201 |
During video processing, |vpl_short_name| processes each view individually. |vpl_short_name| refers |
| 202 |
to the ``FrameID`` field of the :cpp:struct:`mfxFrameInfo` structure to configure |
| 203 |
each view according to its processing pipeline. If the video processing source |
| 204 |
frame is not the output from the |vpl_short_name| MVC decoder, then the application needs to |
| 205 |
fill the the ``FrameID`` field before calling the :cpp:func:`MFXVideoVPP_RunFrameVPPAsync` |
| 206 |
function. This is shown in the following pseudo code: |
| 207 |
|
| 208 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 209 |
:language: c++ |
| 210 |
:start-after: /*beg3*/ |
| 211 |
:end-before: /*end3*/ |
| 212 |
:lineno-start: 1 |
| 213 |
|
| 214 |
---------------------- |
| 215 |
Video Processing 3DLUT |
| 216 |
---------------------- |
| 217 |
|
| 218 |
|vpl_short_name| video processing supports 3DLUT with Intel HW specific memory layout. The following pseudo code |
| 219 |
shows how to create a :cpp:enumerator:`MFX_3DLUT_MEMORY_LAYOUT_INTEL_65LUT` 3DLUT surface. |
| 220 |
|
| 221 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 222 |
:language: c++ |
| 223 |
:start-after: /*beg4*/ |
| 224 |
:end-before: /*end4*/ |
| 225 |
:lineno-start: 1 |
| 226 |
|
| 227 |
The following pseudo code shows how to create a system memory :cpp:struct:`mfx3DLutSystemBuffer` 3DLUT surface. |
| 228 |
|
| 229 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 230 |
:language: c++ |
| 231 |
:start-after: /*beg5*/ |
| 232 |
:end-before: /*end5*/ |
| 233 |
:lineno-start: 1 |
| 234 |
|
| 235 |
The following pseudo code shows how to specify 3DLUT interpolation method :cpp:enum:`mfx3DLutInterpolationMethod`. |
| 236 |
|
| 237 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 238 |
:language: c++ |
| 239 |
:start-after: /*beg11*/ |
| 240 |
:end-before: /*end11*/ |
| 241 |
:lineno-start: 1 |
| 242 |
|
| 243 |
---------------- |
| 244 |
HDR Tone Mapping |
| 245 |
---------------- |
| 246 |
|
| 247 |
|vpl_short_name| video processing supports HDR Tone Mapping with Intel HW. The following pseudo code |
| 248 |
shows how to perform HDR Tone Mapping. |
| 249 |
|
| 250 |
The following pseudo code shows HDR to SDR. |
| 251 |
|
| 252 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 253 |
:language: c++ |
| 254 |
:start-after: /*beg6*/ |
| 255 |
:end-before: /*end6*/ |
| 256 |
:lineno-start: 1 |
| 257 |
|
| 258 |
The following pseudo code shows SDR to HDR. |
| 259 |
|
| 260 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 261 |
:language: c++ |
| 262 |
:start-after: /*beg7*/ |
| 263 |
:end-before: /*end7*/ |
| 264 |
:lineno-start: 1 |
| 265 |
|
| 266 |
The following pseudo code shows HDR to HDR. |
| 267 |
|
| 268 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 269 |
:language: c++ |
| 270 |
:start-after: /*beg8*/ |
| 271 |
:end-before: /*end8*/ |
| 272 |
:lineno-start: 1 |
| 273 |
|
| 274 |
----------------------- |
| 275 |
Camera RAW acceleration |
| 276 |
----------------------- |
| 277 |
|
| 278 |
|vpl_short_name| supports camera raw format processing with Intel HW. The following pseudo code |
| 279 |
shows how to perform camera raw hardware acceleration. For pipeline processing initialization, |
| 280 |
the application needs to attach the camera structures to the :cpp:struct:`mfxVideoParam` structure |
| 281 |
and call the :cpp:func:`MFXVideoVPP_Init` function. |
| 282 |
|
| 283 |
The following pseudo code shows camera raw processing. |
| 284 |
|
| 285 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 286 |
:language: c++ |
| 287 |
:start-after: /*beg9*/ |
| 288 |
:end-before: /*end9*/ |
| 289 |
:lineno-start: 1 |
| 290 |
|
| 291 |
|vpl_short_name| supports AI based super resolution with Intel HW. The following pseudo code |
| 292 |
shows how to perform AI based super resolution. |
| 293 |
|
| 294 |
--------------------------- |
| 295 |
AI Powered Video Processing |
| 296 |
--------------------------- |
| 297 |
|
| 298 |
The following pseudo code shows AI super resolution video processing. |
| 299 |
|
| 300 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 301 |
:language: c++ |
| 302 |
:start-after: /*beg12*/ |
| 303 |
:end-before: /*end12*/ |
| 304 |
:lineno-start: 1 |
| 305 |
|
| 306 |
The following pseudo code shows AI frame interpolation video processing. |
| 307 |
|
| 308 |
.. literalinclude:: ../snippets/prg_vpp.c |
| 309 |
:language: c++ |
| 310 |
:start-after: /*beg13*/ |
| 311 |
:end-before: /*end13*/ |
| 312 |
:lineno-start: 1 |