Debian Codegraph @

doc/spec/source/programming_guide/VPL_prg_encoding.rst

raw
1 .. SPDX-FileCopyrightText: 2019-2020 Intel Corporation
2 ..
3 .. SPDX-License-Identifier: CC-BY-4.0
4
5 ===================
6 Encoding Procedures
7 ===================
8
9 There are two methods for shared memory allocation and handling in |vpl_short_name|:
10 external and internal.
11
12 ---------------
13 External Memory
14 ---------------
15
16 The following pseudo code shows the encoding procedure with external memory
17 (legacy mode):
18
19 .. literalinclude:: ../snippets/prg_encoding.c
20 :language: c++
21 :start-after: /*beg1*/
22 :end-before: /*end1*/
23 :lineno-start: 1
24
25 Note the following key points about the example:
26
27 - The application uses the :cpp:func:`MFXVideoENCODE_QueryIOSurf` function to
28 obtain the number of working frame surfaces required for reordering input
29 frames.
30 - The application calls the :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function
31 for the encoding operation. The input frame must be in an unlocked frame
32 surface from the frame surface pool. If the encoding output is not available,
33 the function returns the :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA` status
34 code to request additional input frames.
35 - Upon successful encoding, the :cpp:func:`MFXVideoENCODE_EncodeFrameAsync`
36 function returns :cpp:enumerator:`mfxStatus::MFX_ERR_NONE`. At this point, the
37 encoded bitstream is not yet available because the
38 :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function is asynchronous. The
39 application must use the :cpp:func:`MFXVideoCORE_SyncOperation` function to
40 synchronize the encoding operation before retrieving the encoded bitstream.
41 - At the end of the stream, the application continuously calls the
42 :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function with a NULL surface
43 pointer to drain any remaining bitstreams cached within the |vpl_short_name| encoder,
44 until the function returns :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA`.
45
46 .. note:: It is the application's responsibility to fill pixels outside of the
47 crop window when it is smaller than the frame to be encoded,
48 especially in cases when crops are not aligned to minimum coding block
49 size (16 for AVC and 8 for HEVC and VP9).
50
51 ---------------
52 Internal Memory
53 ---------------
54
55 The following pseudo code shows the encoding procedure with internal memory:
56
57 .. literalinclude:: ../snippets/prg_encoding.c
58 :language: c++
59 :start-after: /*beg2*/
60 :end-before: /*end2*/
61 :lineno-start: 1
62
63 There are several key differences in this example, compared to external memory
64 (legacy mode):
65
66 - The application does not need to call the
67 :cpp:func:`MFXVideoENCODE_QueryIOSurf` function to obtain the number of
68 working frame surfaces since allocation is done by |vpl_short_name|.
69 - The application calls the :cpp:func:`MFXMemory_GetSurfaceForEncode` function
70 to get a free surface for the subsequent encode operation.
71 - The application must call the
72 :cpp:member:`mfxFrameSurfaceInterface::Release`
73 function to decrement the reference counter of the obtained surface after
74 the call to the :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function.
75
76
77 .. _config-change:
78
79 --------------------
80 Configuration Change
81 --------------------
82
83 The application changes configuration during encoding by calling the
84 :cpp:func:`MFXVideoENCODE_Reset` function. Depending on the difference in
85 configuration parameters before and after the change, the |vpl_short_name| encoder will
86 either continue the current sequence or start a new one. If the encoder starts a
87 new sequence, it completely resets internal state and begins a new sequence with
88 the IDR frame.
89
90 The application controls encoder behavior during parameter change by attaching
91 the :cpp:struct:`mfxExtEncoderResetOption` structure to the
92 :cpp:struct:`mfxVideoParam` structure during reset. By using this structure, the
93 application instructs the encoder to start or not start a new sequence after
94 reset. In some cases, the request to continue the current sequence cannot be
95 satisfied and the encoder will fail during reset. To avoid this scenario, the
96 application may query the reset outcome before the actual reset by calling the
97 :cpp:func:`MFXVideoENCODE_Query` function with the
98 :cpp:struct:`mfxExtEncoderResetOption` attached to the
99 :cpp:struct:`mfxVideoParam` structure.
100
101 The application uses the following procedure to change encoding configurations:
102
103 #. The application retrieves any cached frames in the |vpl_short_name| encoder by calling
104 the :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function with a NULL input
105 frame pointer until the function returns
106 :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA`.
107
108 #. The application calls the :cpp:func:`MFXVideoENCODE_Reset` function with the
109 new configuration:
110
111 - If the function successfully sets the configuration, the application can
112 continue encoding as usual.
113 - If the new configuration requires a new memory allocation, the function
114 returns :cpp:enumerator:`mfxStatus::MFX_ERR_INCOMPATIBLE_VIDEO_PARAM`. The
115 application must close the |vpl_short_name| encoder and reinitialize the encoding
116 procedure with the new configuration.
117
118 ------------------------
119 External Bitrate Control
120 ------------------------
121
122 The application can make the encoder use the external Bitrate Control (BRC)
123 instead of the native bitrate control. To make the encoder use the external BRC,
124 the application should attach the :cpp:struct:`mfxExtCodingOption2` structure
125 with ``ExtBRC = MFX_CODINGOPTION_ON`` and the :cpp:struct:`mfxExtBRC` callback
126 structure to the :cpp:struct:`mfxVideoParam` structure during encoder
127 initialization. The **Init**, **Reset**, and **Close** callbacks will be invoked
128 inside their corresponding functions: :cpp:func:`MFXVideoENCODE_Init`,
129 :cpp:func:`MFXVideoENCODE_Reset`, and :cpp:func:`MFXVideoENCODE_Close`. The
130 following figure shows asynchronous encoding flow with external BRC (using
131 ``GetFrameCtrl`` and ``Update``):
132
133 .. figure:: ../images/extbrc_async.png
134 :alt: Asynchronous encoding flow with external BRC
135
136 Asynchronous encoding flow with external BRC
137
138 .. note:: ``IntAsyncDepth`` is the |vpl_short_name| max internal asynchronous encoding
139 queue size. It is always less than or equal to
140 :cpp:member:`mfxVideoParam::AsyncDepth`.
141
142 The following pseudo code shows use of the external BRC:
143
144 .. literalinclude:: ../snippets/prg_encoding.c
145 :language: c++
146 :start-after: /*beg3*/
147 :end-before: /*end3*/
148 :lineno-start: 1
149
150 ----
151 JPEG
152 ----
153
154 The application can use the same encoding procedures for JPEG/motion JPEG
155 encoding, as shown in the following pseudo code:
156
157 .. code-block:: c++
158
159 // encoder initialization
160 MFXVideoENCODE_Init (...);
161 // single frame/picture encoding
162 MFXVideoENCODE_EncodeFrameAsync (...);
163 MFXVideoCORE_SyncOperation(...);
164 // close down
165 MFXVideoENCODE_Close(...);
166
167 The application may specify Huffman and quantization tables during encoder
168 initialization by attaching :cpp:struct:`mfxExtJPEGQuantTables` and
169 :cpp:struct:`mfxExtJPEGHuffmanTables` buffers to the :cpp:struct:`mfxVideoParam`
170 structure. If the application does not define tables, then the |vpl_short_name| encoder
171 uses tables recommended in ITU-T\* Recommendation T.81. If the application does
172 not define a quantization table it must specify the
173 :cpp:member:`mfxInfoMFX::Quality` parameter. In this case, the |vpl_short_name| encoder
174 scales the default quantization table according to the specified
175 :cpp:member:`mfxInfoMFX::Quality` parameter value.
176
177 The application should properly configure chroma sampling format and color
178 format using the :cpp:member:`mfxFrameInfo::FourCC` and
179 :cpp:member:`mfxFrameInfo::ChromaFormat` fields. For example, to encode a 4:2:2
180 vertically sampled YCbCr picture, the application should set
181 :cpp:member:`mfxFrameInfo::FourCC` to :cpp:enumerator:`MFX_FOURCC_YUY2` and
182 :cpp:member:`mfxFrameInfo::ChromaFormat` to
183 :cpp:enumerator:`MFX_CHROMAFORMAT_YUV422V`. To encode a 4:4:4 sampled RGB
184 picture, the application should set :cpp:member:`mfxFrameInfo::FourCC` to
185 :cpp:enumerator:`MFX_FOURCC_RGB4` and :cpp:member:`mfxFrameInfo::ChromaFormat`
186 to :cpp:enumerator:`MFX_CHROMAFORMAT_YUV444`.
187
188 The |vpl_short_name| encoder supports different sets of chroma sampling and color formats
189 on different platforms. The application must call the
190 :cpp:func:`MFXVideoENCODE_Query` function to check if the required color format
191 is supported on a given platform and then initialize the encoder with proper
192 values of :cpp:member:`mfxFrameInfo::FourCC` and
193 :cpp:member:`mfxFrameInfo::ChromaFormat`.
194
195 The application should not define the number of scans and number of components.
196 These numbers are derived by the |vpl_short_name| encoder from the
197 :cpp:member:`mfxInfoMFx::Interleaved` flag and from chroma type. If interleaved
198 coding is specified, then one scan is encoded that contains all image
199 components. Otherwise, the number of scans is equal to number of components.
200 The encoder uses the following component IDs: “1” for luma (Y), “2” for chroma
201 Cb (U), and “3” for chroma Cr (V).
202
203 The application should allocate a buffer that is big enough to hold the encoded
204 picture. A rough upper limit may be calculated using the following equation
205 where **Width** and **Height** are width and height of the picture in pixel and
206 **BytesPerPx** is the number of bytes for one pixel:
207
208 .. code-block:: c++
209
210 BufferSizeInKB = 4 + (Width * Height * BytesPerPx + 1023) / 1024;
211
212 The equation equals 1 for a monochrome picture, 1.5 for NV12 and YV12 color
213 formats, 2 for YUY2 color format, and 3 for RGB32 color format (alpha channel is
214 not encoded).
215
216 -------------------------
217 Multi-view Video Encoding
218 -------------------------
219
220 Similar to the decoding and video processing initialization procedures, the
221 application attaches the :cpp:struct:`mfxExtMVCSeqDesc` structure to the
222 :cpp:struct:`mfxVideoParam` structure for encoding initialization. The
223 :cpp:struct:`mfxExtMVCSeqDesc` structure configures the |vpl_short_name| MVC encoder to
224 work in three modes:
225
226 - **Default dependency mode:** The application specifies
227 :cpp:member:`mfxExtMVCSeqDesc::NumView` and all other fields to zero. The
228 |vpl_short_name| encoder creates a single operation point with all views (view
229 identifier 0 : NumView-1) as target views. The first view (view identifier 0)
230 is the base view. Other views depend on the base view.
231
232 - **Explicit dependency mode:** The application specifies
233 :cpp:member:`mfxExtMVCSeqDesc::NumView` and the view dependency array, and
234 sets all other fields to zero. The |vpl_short_name| encoder creates a single operation
235 point with all views (view identifier View[0 : NumView-1].ViewId) as target
236 views. The first view (view identifier View[0].ViewId) is the base view. View
237 dependencies are defined as :cpp:struct:`mfxMVCViewDependency` structures.
238
239 - **Complete mode:** The application fully specifies the views and their
240 dependencies. The |vpl_short_name| encoder generates a bitstream with corresponding
241 stream structures.
242
243 During encoding, the |vpl_short_name| encoding function
244 :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` accumulates input frames until
245 encoding of a picture is possible. The function returns
246 :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA` for more data at input or
247 :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` if it successfully accumulated enough
248 data for encoding a picture. The generated bitstream contains the complete
249 picture (multiple views). The application can change this behavior and instruct
250 the encoder to output each view in a separate bitstream buffer. To do so, the
251 application must turn on the :cpp:member:`mfxExtCodingOption::ViewOutput` flag.
252 In this case, the encoder returns
253 :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_BITSTREAM` if it needs more bitstream
254 buffers at output and :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` when processing
255 of the picture (multiple views) has been finished. It is recommended that the
256 application provide a new input frame each time the |vpl_short_name| encoder requests a
257 new bitstream buffer. The application must submit view data for encoding in the
258 order they are described in the :cpp:struct:`mfxExtMVCSeqDesc` structure.
259 Particular view data can be submitted for encoding only when all views that it
260 depends upon have already been submitted.
261
262 The following pseudo code shows the encoding procedure:
263
264 .. literalinclude:: ../snippets/prg_encoding.c
265 :language: c++
266 :start-after: /*beg4*/
267 :end-before: /*end4*/
268 :lineno-start: 1
269
270 ---------------------------
271 Encoder Quality Information
272 ---------------------------
273
274 Video encoder supports outputting quality information. Currently, it only
275 reports the mean squared error (MSE).
276
277 - The application uses the :cpp:struct:`mfxExtQualityInfoMode` structure with
278 ``mfxQualityInfoMode = MFX_QUALITY_INFO_LEVEL_FRAME`` to initiate the request.
279 - The application uses the :cpp:struct:`mfxExtQualityInfoOutput` structure to obtain the
280 MSE result.
281
282 The following pseudo code shows an example of gathering encoder quality information:
283
284 .. literalinclude:: ../snippets/prg_encoding.c
285 :language: c++
286 :start-after: /*beg9*/
287 :end-before: /*end9*/
288 :lineno-start: 1
289
290 Note the following key points about the example:
291
292 - The application must attach the :cpp:struct:`mfxExtQualityInfoMode` structure to the
293 :cpp:struct:`mfxVideoParam` structure and call the :cpp:func:`MFXVideoENCODE_Query` to
294 check the support status of the request. The function returns
295 :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` if support is successful, or issues a warning
296 :cpp:enumerator:`mfxStatus::MFX_WRN_INCOMPATIBLE_VIDEO_PARAM` if not supported, and
297 sets :cpp:struct:`mfxExtQualityInfoMode` to
298 ``mfxQualityInfoMode = MFX_QUALITY_INFO_DISABLE``.
299 - The application uses the :cpp:func:`MFXVideoENCODE_Init` to initialize the encoder.
300 - The application can attach the :cpp:struct:`mfxExtQualityInfoOutput` structure to
301 the encoded bitstream structure before calling
302 :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function.
303 - The application must use the :cpp:func:`MFXVideoCORE_SyncOperation` function to
304 synchronize the encoding operation before retrieving the encoded bitstream and MSE
305 result.
306
307 ------------------------
308 AV1 Screen Content Tools
309 ------------------------
310
311 AV1 video encoder supports Palette Prediction and Intra Block Copy mode.
312 The following pseudo code shows an example of encoding with AV1 Screen Content tools:
313
314 .. literalinclude:: ../snippets/prg_encoding.c
315 :language: c++
316 :start-after: /*beg10*/
317 :end-before: /*end10*/
318 :lineno-start: 1
319
320 Note the following key points about the example:
321
322 - The application must attach the :cpp:struct:`mfxExtAV1ScreenContentTools` structure to
323 the :cpp:struct:`mfxVideoParam` structure and call the :cpp:func:`MFXVideoENCODE_Query`
324 to check the support status of the request. The function returns
325 :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` if support is successful, issues a warning
326 :cpp:enumerator:`mfxStatus::MFX_WRN_INCOMPATIBLE_VIDEO_PARAM` due to platform
327 limitations and updates the :cpp:struct:`mfxExtAV1ScreenContentTools` structure values
328 to the default, or returns the error :cpp:enumerator:`mfxStatus::MFX_ERR_UNSUPPORTED` if
329 not support.
330 - The application uses the :cpp:func:`MFXVideoENCODE_Init` to initialize the encoder.
331 - The application calls the :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function to
332 perform the encoding operation.
333 - The application must use the :cpp:func:`MFXVideoCORE_SyncOperation` function to
334 synchronize the encoding operation before retrieving the encoded bitstream.
335
336 ----------------------
337 Alpha Channel Encoding
338 ----------------------
339
340 The following pseudo code shows several examples of alpha channel encoding
341 with multiple input data formats:
342
343 .. literalinclude:: ../snippets/prg_encoding.c
344 :language: c++
345 :start-after: /*beg11*/
346 :end-before: /*end11*/
347 :lineno-start: 1
348
349 Note the following key points about the example:
350
351 API Definition:
352
353 - :cpp:struct:`mfxExtAlphaChannelEncCtrl`: This structure configures the alpha channel encoding. It includes fields like EnableAlphaChannelEncoding, AlphaChannelMode, and AlphaChannelBitrateRatio.
354 - :cpp:member:`mfxExtAlphaChannelEncCtrl::EnableAlphaChannelEncoding`: Set to :cpp:enumerator:`MFX_CODINGOPTION_ON` to enable alpha channel encoding.
355 - :cpp:member:`mfxExtAlphaChannelEncCtrl::AlphaChannelMode`: Defines whether the alpha is straight or pre-multiplied.
356
357 - :cpp:enumerator:`MFX_ALPHA_MODE_PREMULTIPLIED`: RGB and alpha are independent.
358 - :cpp:enumerator:`MFX_ALPHA_MODE_STRAIGHT`: RGB and alpha are linked.
359 - :cpp:member:`mfxExtAlphaChannelEncCtrl::AlphaChannelBitrateRatio`: Indicates the percentage of the bitrate allocated to the alpha channel.
360
361 Usage Instructions:
362
363 - At present, Internal Memory Allocation as described in `Memory Allocation and External Allocators <https://intel.github.io/libvpl/latest/programming_guide/VPL_prg_mem.html>`_ is not yet supported.
364 - The application queries encoding capabilities (:cpp:func:`MFXVideoENCODE_Query`) and surface requirements (:cpp:func:`MFXVideoENCODE_QueryIOSurf`).
365 - The application initializes(:cpp:func:`MFXVideoENCODE_Init`) the encoder with alpha channel encoding enabled.
366 - Alpha channel data can be provided separately or embedded within the main surface data depending on the format.
367 - For each input data type, the application configures the alpha surface accordingly before calling :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` for encoding.
368
369 - NV12_VIDEO: Uses video memory for NV12 data, with a mfxExtAlphaChannelSurface structure for video memory handle.
370 - NV12_SYS: Uses system memory for NV12 data, with a mfxExtAlphaChannelSurface structure attached to the surface.
371 - RGBA_VIDEO: Directly uses shared resources for RGBA video textures.
372 - RGBA_SYS: Not yet supported.
373
374 - The application must use the :cpp:func:`MFXVideoCORE_SyncOperation` function to synchronize the encoding operation before retrieving the encoded bitstream.
375
376 ------------------------
377 AI Assisted Encoder Ctrl
378 ------------------------
379
380 The following pseudo code shows an example of encoding with AI Assited Encoder Options:
381
382 .. literalinclude:: ../snippets/prg_encoding.c
383 :language: c++
384 :start-after: /*beg12*/
385 :end-before: /*end12*/
386 :lineno-start: 1
387
388 Note the following key points about the example:
389
390 - The application must attach the :cpp:struct:`mfxExtAIEncCtrl` structure to the
391 :cpp:struct:`mfxVideoParam` structure and call the :cpp:func:`MFXVideoENCODE_Query`
392 to check the support status of the request. The function returns
393 :cpp:enumerator:`mfxStatus::MFX_ERR_NONE` if support is successful, issues a warning
394 :cpp:enumerator:`mfxStatus::MFX_WRN_INCOMPATIBLE_VIDEO_PARAM` due to platform limitations
395 and updates the :cpp:struct:`mfxExtAIEncCtrl` structure values to the default, or returns
396 the error :cpp:enumerator:`mfxStatus::MFX_ERR_UNSUPPORTED` if not support.
397 - The application uses the :cpp:func:`MFXVideoENCODE_Init` to initialize the encoder.
398 - The application calls the :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function to perform
399 the encoding operation.
400 - The application must use the :cpp:func:`MFXVideoCORE_SyncOperation` function to
401 synchronize the encoding operation before retrieving the encoded bitstream.