Debian Codegraph @

doc/spec/source/programming_guide/VPL_prg_decoding.rst

raw
1 .. SPDX-FileCopyrightText: 2019-2020 Intel Corporation
2 ..
3 .. SPDX-License-Identifier: CC-BY-4.0
4
5 .. _decoding-proc:
6
7 ===================
8 Decoding Procedures
9 ===================
10
11 There are several approaches to decode video frames. The first one is based on
12 the internal allocation mechanism presented here:
13
14 .. literalinclude:: ../snippets/prg_decoding.c
15 :language: c++
16 :start-after: /*beg1*/
17 :end-before: /*end1*/
18 :lineno-start: 1
19
20 Note the following key points about the example:
21
22 - The application calls the :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` function
23 for a decoding operation with the bitstream buffer (bits), frame surface is
24 allocated internally by the library.
25
26 .. attention:: As shown in the example above starting with API version 2.0,
27 the application can provide NULL
28 as the working frame surface that leads to internal memory
29 allocation.
30
31 - If decoding output is not available, the function returns a status code
32 requesting additional bitstream input as follows:
33
34 - :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA`: The function needs
35 additional bitstream input. The existing buffer contains less than a frame's
36 worth of bitstream data.
37
38 - Upon successful decoding, the :cpp:func:`MFXVideoDECODE_DecodeFrameAsync`
39 function returns :cpp:enumerator:`mfxStatus::MFX_ERR_NONE`. However, the
40 decoded frame data (identified by the surface_out pointer) is not yet
41 available because the :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` function is
42 asynchronous. The application must use the
43 :cpp:func:`MFXVideoCORE_SyncOperation` or
44 :cpp:member:`mfxFrameSurfaceInterface::Synchronize` to synchronize the
45 decoding operation before retrieving the decoded frame data.
46
47 - At the end of the bitstream, the application continuously calls the
48 :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` function with a NULL bitstream
49 pointer to drain any remaining frames cached within the decoder until the
50 function returns :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_DATA`.
51
52 - When application completes the work with frame surface, it must call release
53 to avoid memory leaks.
54
55 The next example demonstrates how applications can use internally pre-allocated
56 chunk of video surfaces:
57
58 .. literalinclude:: ../snippets/prg_decoding.c
59 :language: c++
60 :start-after: /*beg2*/
61 :end-before: /*end2*/
62 :lineno-start: 1
63
64 Here the application should use the :cpp:func:`MFXVideoDECODE_QueryIOSurf`
65 function to obtain the number of working frame surfaces required to reorder
66 output frames. It is also required that
67 :cpp:func:`MFXMemory_GetSurfaceForDecode` call is done after decoder
68 initialization. In the :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` the |vpl_short_name|
69 library increments reference counter of incoming surface frame so it is required
70 that the application releases frame surface after the call.
71
72 Another approach to decode frames is to allocate video frames on-fly with help
73 of :cpp:func:`MFXMemory_GetSurfaceForDecode` function, feed the library and
74 release working surface after :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` call.
75
76 .. attention:: Please pay attention on two release calls for surfaces:
77 after :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` to decrease
78 reference counter of working surface returned by
79 :cpp:func:`MFXMemory_GetSurfaceForDecode`.
80 After :cpp:func:`MFXVideoCORE_SyncOperation` to decrease
81 reference counter of output surface returned by
82 :cpp:func:`MFXVideoDECODE_DecodeFrameAsync`.
83
84 .. literalinclude:: ../snippets/prg_decoding.c
85 :language: c++
86 :start-after: /*beg6*/
87 :end-before: /*end6*/
88 :lineno-start: 1
89
90
91 The following pseudo code shows the decoding procedure according to the legacy
92 mode with external video frames allocation:
93
94 .. literalinclude:: ../snippets/prg_decoding.c
95 :language: c++
96 :start-after: /*beg3*/
97 :end-before: /*end3*/
98 :lineno-start: 1
99
100 Note the following key points about the example:
101
102
103 - The application can use the :cpp:func:`MFXVideoDECODE_DecodeHeader` function
104 to retrieve decoding initialization parameters from the bitstream. This step
105 is optional if the data is retrievable from other sources such as an
106 audio/video splitter.
107
108 - The :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` function can return following
109 status codes in addition to the described above:
110
111 - :cpp:enumerator:`mfxStatus::MFX_ERR_MORE_SURFACE`: The function needs one
112 more frame surface to produce any output.
113 - :cpp:enumerator:`mfxStatus::MFX_ERR_REALLOC_SURFACE`: Dynamic resolution
114 change case - the function needs a bigger working frame surface (work).
115
116 The following pseudo code shows the simplified decoding procedure:
117
118 .. literalinclude:: ../snippets/prg_decoding.c
119 :language: c++
120 :start-after: /*beg4*/
121 :end-before: /*end4*/
122 :lineno-start: 1
123
124 .. _simplified-decoding-procedure:
125
126 |vpl_short_name| API version 2.0 introduces a new decoding approach. For simple use cases,
127 when the user wants to decode a stream and does not want to set additional
128 parameters, a simplified procedure for the decoder's initialization has been
129 proposed. In this scenario it is possible to skip explicit stages of a
130 stream's header decoding and the decoder's initialization and instead to perform
131 these steps implicitly during decoding of the first frame. This change also
132 requires setting the additional field :cpp:member:`mfxBitstream::CodecId` to
133 indicate codec type. In this mode the decoder allocates
134 :cpp:struct:`mfxFrameSurface1` internally, so users should set the input surface
135 to zero.
136
137 -----------------------
138 Bitstream Repositioning
139 -----------------------
140
141 The application can use the following procedure for bitstream reposition during
142 decoding:
143
144 #. Use the :cpp:func:`MFXVideoDECODE_Reset` function to reset the |vpl_short_name|
145 decoder.
146 #. Optional: If the application maintains a sequence header that correctly
147 decodes the bitstream at the new position, the application may insert the
148 sequence header to the bitstream buffer.
149 #. Append the bitstream from the new location to the bitstream buffer.
150 #. Resume the decoding procedure. If the sequence header is not inserted in the
151 previous steps, the |vpl_short_name| decoder searches for a new sequence header before
152 starting decoding.
153
154 -----------------------
155 Broken Streams Handling
156 -----------------------
157
158 Robustness and the capability to handle a broken input stream is an important
159 part of the decoder.
160
161 First, the start code prefix (ITU-T\* H.264 3.148 and ITU-T H.265 3.142) is
162 used to separate NAL units. Then all syntax elements in the bitstream are parsed
163 and verified. If any of the elements violate the specification, the input
164 bitstream is considered invalid and the decoder tries to re-sync (find the next
165 start code). Subsequent decoder behavior is dependent on which syntax element is
166 broken:
167
168 * SPS header is broken: return
169 :cpp:enumerator:`mfxStatus::MFX_ERR_INCOMPATIBLE_VIDEO_PARAM` (HEVC decoder
170 only, AVC decoder uses last valid).
171 * PPS header is broken: re-sync, use last valid PPS for decoding.
172 * Slice header is broken: skip this slice, re-sync.
173 * Slice data is broken: corruption flags are set on output surface.
174
175 Many streams have IDR frames with ``frame_num != 0`` while the specification
176 says that “If the current picture is an IDR picture, frame_num shall be equal to
177 0” (ITU-T H.265 7.4.3).
178
179 VUI is also validated, but errors do not invalidate the whole SPS. The decoder
180 either does not use the corrupted VUI (AVC) or resets incorrect values to
181 default (HEVC).
182
183 .. note:: Some requirements are relaxed because there are many streams which
184 violate the strict standard but can be decoded without errors.
185
186 Corruption at the reference frame is spread over all inter-coded pictures that
187 use the reference frame for prediction. To cope with this problem you must
188 either periodically insert I-frames (intra-coded) or use the intra-refresh
189 technique. The intra-refresh technique allows recovery from corruptions within a
190 predefined time interval. The main point of intra-refresh is to insert a cyclic
191 intra-coded pattern (usually a row) of macroblocks into the inter-coded
192 pictures, restricting motion vectors accordingly. Intra-refresh is often used in
193 combination with recovery point SEI, where the ``recovery_frame_cnt`` is derived
194 from the intra-refresh interval. The recovery point SEI message is well
195 described at ITU-T H.264 D.2.7 and ITU-T H.265 D.2.8. If decoding starts from AU
196 associated with this SEI message, then the message can be used by the decoder to
197 determine from which picture all subsequent pictures have no errors. In
198 comparison to IDR, the recovery point message does not mark reference pictures
199 as "unused for reference".
200
201 Besides validation of syntax elements and their constraints, the decoder also
202 uses various hints to handle broken streams:
203
204 * If there are no valid slices for the current frame, then the whole frame is
205 skipped.
206 * The slices which violate slice segment header semantics (ITU-T H.265 7.4.7.1)
207 are skipped. Only the ``slice_temporal_mvp_enabled_flag`` is checked for now.
208 * Since LTR (Long Term Reference) stays at DPB until it is explicitly
209 cleared by IDR or MMCO, the incorrect LTR could cause long standing visual
210 artifacts. AVC decoder uses the following approaches to handle this:
211
212 * When there is a DPB overflow in the case of an incorrect MMCO command that
213 marks the reference picture as LT, the operation is rolled back.
214 * An IDR frame with ``frame_num != 0`` can’t be LTR.
215
216 * If the decoder detects frame gapping, it inserts "fake"’" (marked as
217 non-existing) frames, updates FrameNumWrap (ITU-T H.264 8.2.4.1) for reference
218 frames, and applies the Sliding Window (ITU-T H.264 8.2.5.3) marking process.
219 Fake frames are marked as reference, but since they are marked as
220 non-existing, they are not used for inter-prediction.
221
222 --------------------
223 VP8 Specific Details
224 --------------------
225
226 Unlike other |vpl_short_name| supported decoders, VP8 can accept only a complete frame as
227 input. The application should provide the complete frame accompanied by the
228 :cpp:enumerator:`MFX_BITSTREAM_COMPLETE_FRAME` flag. This is the single specific
229 difference.
230
231 ----
232 JPEG
233 ----
234
235 The application can use the same decoding procedures for JPEG/motion JPEG
236 decoding, as shown in the following pseudo code:
237
238 .. code-block:: c++
239
240 // optional; retrieve initialization parameters
241 MFXVideoDECODE_DecodeHeader(...);
242 // decoder initialization
243 MFXVideoDECODE_Init(...);
244 // single frame/picture decoding
245 MFXVideoDECODE_DecodeFrameAsync(...);
246 MFXVideoCORE_SyncOperation(...);
247 // optional; retrieve meta-data
248 MFXVideoDECODE_GetUserData(...);
249 // close
250 MFXVideoDECODE_Close(...);
251
252
253 The :cpp:func:`MFXVideoDECODE_Query` function will return
254 :cpp:enumerator:`mfxStatus::MFX_ERR_UNSUPPORTED` if the input bitstream contains
255 unsupported features.
256
257 For still picture JPEG decoding, the input can be any JPEG bitstreams that
258 conform to the ITU-T Recommendation T.81 with an EXIF or JFIF header. For
259 motion JPEG decoding, the input can be any JPEG bitstreams that conform to the
260 ITU-T Recommendation T.81.
261
262 Unlike other |vpl_short_name| decoders, JPEG decoding supports three different output
263 color formats: :term:`NV12`, :term:`YUY2`, and :term:`RGB32`. This support
264 sometimes requires internal color conversion and more complicated
265 initialization. The color format of the input bitstream is described by the
266 :cpp:member:`mfxInfoMFX::JPEGChromaFormat` and
267 :cpp:member:`mfxInfoMFX::JPEGColorFormat` fields. The
268 :cpp:func:`MFXVideoDECODE_DecodeHeader` function usually fills them in. If the
269 JPEG bitstream does not contains color format information, the application
270 should provide it. Output color format is described by general |vpl_short_name|
271 parameters: the :cpp:member:`mfxFrameInfo::FourCC` and
272 :cpp:member:`mfxFrameInfo::ChromaFormat` fields.
273
274 Motion JPEG supports interlaced content by compressing each field (a half-height
275 frame) individually. This behavior is incompatible with the rest of the |vpl_short_name|
276 transcoding pipeline, where |vpl_short_name| requires fields to be in odd and even lines
277 of the same frame surface. The decoding procedure is modified as follows:
278
279 - The application calls the :cpp:func:`MFXVideoDECODE_DecodeHeader` function
280 with the first field JPEG bitstream to retrieve initialization parameters.
281 - The application initializes the |vpl_short_name| JPEG decoder with the following
282 settings:
283
284 - The ``PicStruct`` field of the :cpp:struct:`mfxVideoParam` structure set to
285 the correct interlaced type, :cpp:enumerator:`MFX_PICSTRUCT_FIELD_TFF` or
286 :cpp:enumerator:`MFX_PICSTRUCT_FIELD_BFF`, from the motion JPEG header.
287 - Double the ``Height`` field in the :cpp:struct:`mfxVideoParam` structure as
288 the value returned by the :cpp:func:`MFXVideoDECODE_DecodeHeader` function
289 describes only the first field. The actual frame surface should contain both
290 fields.
291
292 - During decoding, the application sends both fields for decoding in the
293 same :cpp:struct:`mfxBitstream`. The application should also set
294 :cpp:member:`mfxBitstream::DataFlag` to
295 :cpp:enumerator:`MFX_BITSTREAM_COMPLETE_FRAME`. |vpl_short_name| decodes both fields
296 and combines them into odd and even lines according to |vpl_short_name| convention.
297
298 By default, the :cpp:func:`MFXVideoDECODE_DecodeHeader` function returns the
299 ``Rotation`` parameter so that after rotation, the pixel at the
300 first row and first column is at the top left. The application can overwrite the
301 default rotation before calling :cpp:func:`MFXVideoDECODE_Init`.
302
303 The application may specify Huffman and quantization tables during decoder
304 initialization by attaching :cpp:struct:`mfxExtJPEGQuantTables` and
305 :cpp:struct:`mfxExtJPEGHuffmanTables` buffers to the :cpp:struct:`mfxVideoParam`
306 structure. In this case, the decoder ignores tables from bitstream and uses
307 the tables specified by the application. The application can also retrieve these
308 tables by attaching the same buffers to :cpp:struct:`mfxVideoParam` and calling
309 :cpp:func:`MFXVideoDECODE_GetVideoParam` or
310 :cpp:func:`MFXVideoDECODE_DecodeHeader` functions.
311
312 -------------------------
313 Multi-view Video Decoding
314 -------------------------
315
316 The |vpl_short_name| MVC decoder operates on complete MVC streams that contain all
317 view and temporal configurations. The application can configure the |vpl_short_name|
318 decoder to generate a subset at the decoding output. To do this, the application
319 must understand the stream structure and use the stream information to configure
320 the decoder for target views.
321
322 The decoder initialization procedure is as follows:
323
324 #. The application calls the :cpp:func:`MFXVideoDECODE_DecodeHeader` function to
325 obtain the stream structural information. This is done in two steps:
326
327 #. The application calls the :cpp:func:`MFXVideoDECODE_DecodeHeader` function
328 with the :cpp:struct:`mfxExtMVCSeqDesc` structure attached to the
329 :cpp:struct:`mfxVideoParam` structure. At this point, do not allocate
330 memory for the arrays in the :cpp:struct:`mfxExtMVCSeqDesc` structure. Set
331 the ``View``, ``ViewId``, and ``OP`` pointers to NULL and set
332 ``NumViewAlloc``, ``NumViewIdAlloc``, and ``NumOPAlloc`` to zero. The
333 function parses the bitstream and returns
334 :cpp:enumerator:`mfxStatus::MFX_ERR_NOT_ENOUGH_BUFFER` with the correct
335 values for ``NumView``, ``NumViewId``, and ``NumOP``. This step can be
336 skipped if the application is able to obtain the ``NumView``,
337 ``NumViewId``, and ``NumOP`` values from other sources.
338 #. The application allocates memory for the ``View``, ``ViewId``, and ``OP``
339 arrays and calls the :cpp:func:`MFXVideoDECODE_DecodeHeader` function
340 again. The function returns the MVC structural information in the
341 allocated arrays.
342
343 #. The application fills the :cpp:struct:`mfxExtMVCTargetViews` structure to
344 choose the target views, based on information described in the
345 :cpp:struct:`mfxExtMVCSeqDesc` structure.
346 #. The application initializes the |vpl_short_name| decoder using the
347 :cpp:func:`MFXVideoDECODE_Init` function. The application must attach both
348 the :cpp:struct:`mfxExtMVCSeqDesc` structure and the
349 :cpp:struct:`mfxExtMVCTargetViews` structure to the
350 :cpp:struct:`mfxVideoParam` structure.
351
352 In the above steps, do not modify the values of the
353 :cpp:struct:`mfxExtMVCSeqDesc` structure after the
354 :cpp:func:`MFXVideoDECODE_DecodeHeader` function, as the |vpl_short_name| decoder uses the
355 values in the structure for internal memory allocation. Once the application
356 configures the |vpl_short_name| decoder, the rest of the decoding procedure remains
357 unchanged. As shown in the pseudo code below, the application calls the
358 :cpp:func:`MFXVideoDECODE_DecodeFrameAsync` function multiple times to obtain
359 all target views of the current frame picture, one target view at a time. The
360 target view is identified by the ``FrameID`` field of the
361 :cpp:struct:`mfxFrameInfo` structure.
362
363 .. literalinclude:: ../snippets/prg_decoding.c
364 :language: c++
365 :start-after: /*beg5*/
366 :end-before: /*end5*/
367 :lineno-start: 1
368
369 ---------------------------------------------------
370 Combined Decode with Multi-channel Video Processing
371 ---------------------------------------------------
372
373 The |vpl_short_name| exposes interface for making decode and video processing operations
374 in one call. Users can specify a number of output processing channels and
375 multiple video filters per each channel. This interface supports only internal
376 memory allocation model and returns array of processed frames through
377 :cpp:struct:`mfxSurfaceArray` reference object as shown by the example:
378
379 .. literalinclude:: ../snippets/prg_decoding_vpp.c
380 :language: c++
381 :start-after: /*beg1*/
382 :end-before: /*end1*/
383 :lineno-start: 1
384
385 It's possible that different video processing channels may have different
386 latency:
387
388 .. literalinclude:: ../snippets/prg_decoding_vpp.c
389 :language: c++
390 :start-after: /*beg2*/
391 :end-before: /*end2*/
392 :lineno-start: 1
393
394 Application can match decoded frame w/ specific VPP channels using
395 :cpp:member:`mfxFrameData::TimeStamp`, :cpp:member:mfxFrameData::FrameOrder` and
396 :cpp:member:`mfxFrameInfo::ChannelId`.
397
398 Application can skip some or all channels including decoding output with help of
399 `skip_channels` and `num_skip_channels` parameters as follows: application fills
400 `skip_channels` array with `ChannelId`s to disable output of correspondent
401 channels. In that case :cpp:member:`surf_array_out` would contain only surfaces
402 for the remaining channels. If the decoder's channel and/or impacted VPP
403 channels don't have output frame(s) for the current call (for instance, input
404 bitstream doesn't contain complete frame or deinterlacing/FRC filter have delay)
405 `skip_channels` parameter is ignored for this channel.
406
407 If application disables all channels the SDK returns NULL as
408 :cpp:struct:`mfxSurfaceArray`.
409
410 If application doesn't need to disable any channels it sets `num_skip_channels`
411 to zero, `skip_channels` is ignored when `num_skip_channels` is zero.
412
413 If application doesn't need to make scaling or cropping operations it has to set the following fields
414 :cpp:member:`mfxFrameInfo::Width`, :cpp:member:`mfxFrameInfo::Height`,
415 :cpp:member:`mfxFrameInfo::CropX`, :cpp:member:`mfxFrameInfo::CropY`
416 :cpp:member:`mfxFrameInfo::CropW`, :cpp:member:`mfxFrameInfo::CropH` of the VPP channel to zero.
417 In that case output surfaces have the original decoded resolution and cropping. The operation supports
418 bitstreams with resolution change without need of :cpp:func:`MFXVideoDECODE_VPP_Reset` call.
419
420 .. note:: Even if more than one input compressed frame is consumed, the
421 :cpp:func:`MFXVideoDECODE_VPP_DecodeFrameAsync` produces only one decoded
422 frame and correspondent frames from VPP channels.
423
424 ---------------------------------------------------
425 AsyncDepth Specific Details
426 ---------------------------------------------------
427
428 `AsyncDepth` means how many asynchronous operations an application performs before the application explicitly
429 synchronizes the result, application will set this parameter and call :cpp:func:`MFXVideoDECODE_DecodeFrameAsync`,
430 the number of submitted frames is specified by `AsyncDepth`, then call :cpp:func:`MFXVideoCORE_SyncOperation` to synchronize
431 the result of all frames.
432
433 The following pseudo code shows the simplified asynchronous and synchronous procedure:
434
435 .. literalinclude:: ../snippets/prg_decoding.c
436 :language: c++
437 :start-after: /*beg7*/
438 :end-before: /*end7*/
439 :lineno-start: 1
440
441 According to memory utilization and performance, two modes are recommend for decoder pipeline:
442
443 - For smallest memory footprint: the application should set AsyncDepth to 1 and then call :cpp:func:`MFXVideoCORE_SyncOperation`
444 after each frame. This will have poor performance, since the GPU will not be kept fully occupied.
445 - For best performance: the application should set AsyncDepth to 4 and plan to :cpp:func:`MFXVideoCORE_SyncOperation` after every
446 4th frame, which generally will keep the GPU close to fully occupied.