Debian Codegraph @

doc/spec/source/appendix/VPL_apnds_c.rst

raw
1 .. SPDX-FileCopyrightText: 2019-2020 Intel Corporation
2 ..
3 .. SPDX-License-Identifier: CC-BY-4.0
4
5 .. _stream_vid_conf_features:
6
7 =========================================
8 Streaming and Video Conferencing Features
9 =========================================
10
11 The following sections address some aspects of additional requirements that
12 streaming or video conferencing applications may use in the encoding or
13 transcoding process. See the :ref:`config-change` section for additional
14 information.
15
16 ----------------------
17 Dynamic Bitrate Change
18 ----------------------
19
20 The |vpl_short_name| encoder supports dynamic bitrate change according to bitrate control mode
21 and HRD conformance requirements. If HRD conformance is required, for example if
22 the application sets the ``NalHrdConformance`` option in the
23 :cpp:struct:`mfxExtCodingOption` structure to ON, the only allowed
24 bitrate control mode is VBR. In this mode, the application can change the
25 ``TargetKbps`` and ``MaxKbps`` values of the :cpp:struct:`mfxInfoMFX` structure
26 by calling the :cpp:func:`MFXVideoENCODE_Reset` function. This sort of change in
27 bitrate usually results in the generation of a new keyframe and sequence header.
28 There are exceptions, such as if HRD information is absent in the stream.
29 In this scenario, the change of ``TargetKbps`` does not require a change in the
30 sequence header and as a result the encoder does not insert a keyframe.
31
32 If HRD conformance is not required, for example if the application turns off
33 the ``NalHrdConformance`` option in the :cpp:struct:`mfxExtCodingOption`
34 structure, all bitrate control modes are available. In CBR and AVBR modes the
35 application can change ``TargetKbps``. In VBR mode the application can change
36 ``TargetKbps`` and ``MaxKbps`` values. This sort of change in bitrate will not
37 result in the generation of a new keyframe or sequence header.
38
39 The |vpl_short_name| encoder may change some initialization parameters provided by the
40 application during initialization. That in turn may lead to incompatibility
41 between the parameters provided by the application during reset and the working set
42 of parameters used by the encoder. For this reason, it is strongly recommended
43 to retrieve the actual working parameters using the
44 :cpp:func:`MFXVideoENCODE_GetVideoParam` function before making any changes to
45 bitrate settings.
46
47 In all modes, |vpl_short_name| encoders will respond to the bitrate changes as quickly as
48 the underlying algorithm allows, without breaking other encoding restrictions
49 such as HRD compliance if it is enabled. How quickly the actual bitrate can catch
50 up with the specified bitrate is implementation dependent.
51
52 Alternatively, the application may use the :term:`CQP` encoding mode to perform
53 customized bitrate adjustment on a per-frame base. The application may use any
54 of the encoded or display order modes to use per-frame CQP.
55
56 -------------------------
57 Dynamic Resolution Change
58 -------------------------
59
60 The |vpl_short_name| encoder supports dynamic resolution change in all bitrate control modes.
61 The application may change resolution by calling the :cpp:func:`MFXVideoENCODE_Reset`
62 function. The application may decrease or increase resolution up to the size
63 specified during encoder initialization.
64
65 Resolution change always results in the insertion of a key IDR frame and a new
66 sequence parameter set in the header. The only exception is the |vpl_short_name| VP9 encoder
67 (see section for `Dynamic reference frame scaling`_). The |vpl_short_name| encoder does not
68 guarantee HRD conformance across the resolution change point.
69
70 The |vpl_short_name| encoder may change some initialization parameters provided by the
71 application during initialization. That in turn may lead to incompatibility of
72 parameters provide by the application during reset and working set of parameters
73 used by the encoder. Due to this potential incompatibility, it is strongly
74 recommended to retrieve the actual working parameters set by
75 :cpp:func:`MFXVideoENCODE_GetVideoParam` function before making any resolution
76 change.
77
78 -------------------------------
79 Dynamic Reference Frame Scaling
80 -------------------------------
81
82 The VP9 standard allows changing the resolution without the insertion of a
83 keyframe. This is possible because the VP9 encoder has the built-in capability
84 to upscale and downscale reference frames to match the resolution of the frame
85 being encoded. By default the |vpl_short_name| VP9 encoder inserts a keyframe when the
86 application does `Dynamic Resolution Change`_. In this case, the first frame
87 with a new resolution is encoded using inter prediction from the scaled reference
88 frame of the previous resolution. Dynamic scaling has the following limitations,
89 described in the VP9 specification:
90
91 * The resolution of any active reference frame cannot exceed 2x the resolution
92 of the current frame.
93 * The resolution of any active reference frame cannot be smaller than 1/16 of
94 the current frame resolution.
95
96 In the case of dynamic scaling, the |vpl_short_name| VP9 encoder always uses a single active
97 reference frame for the first frame after a resolution change. The VP9
98 encoder has the following limitations for dynamic resolution change:
99
100 * The new resolution should not exceed 16x the resolution of the current frame.
101 * The new resolution should be less than 1/2 of current frame resolution.
102
103 The application may force insertion of a keyframe at the point of resolution
104 change by invoking encoder reset with :cpp:member:`mfxExtEncoderResetOption::StartNewSequence`
105 set to :cpp:enumerator:`MFX_CODINGOPTION_ON`. If a keyframe is inserted, the
106 dynamic resolution limitations are not enforced.
107
108 Note that resolution change with dynamic reference scaling is compatible with
109 multiref (:cpp:member:`mfxInfoMFX::NumRefFrame` > 1). For multiref
110 configuration, the |vpl_short_name| VP9 encoder uses multiple references within stream pieces
111 of the same resolution and uses a single reference at the place of resolution
112 change.
113
114 ---------------------------
115 Forced Keyframe Generation
116 ---------------------------
117
118 |vpl_short_name| supports forced keyframe generation during encoding. The application
119 can set the FrameType parameter of the :cpp:struct:`mfxEncodeCtrl` structure to
120 control how the current frame is encoded, as follows:
121
122 - If the |vpl_short_name| encoder works in the display order, the application can enforce any
123 current frame to be a keyframe. The application cannot change the frame type
124 of already buffered frames inside the encoder.
125 - If the |vpl_short_name| encoder works in the encoded order, the application must specify
126 exact frame type for every frame. In this way, the application can enforce the
127 current frame to have any frame type that the particular coding standard allows.
128
129 .. _sec_reference_list_selection:
130
131 ------------------------
132 Reference List Selection
133 ------------------------
134
135 During streaming or video conferencing, if the application can obtain feedback
136 about how well the client receives certain frames, the application may need to
137 adjust the encoding process to use or not use certain frames as reference. This
138 section describes how to fine-tune the encoding process based on client feedback.
139
140 The application can specify the reference window size by specifying the
141 :cpp:member:`mfxInfoMFX::NumRefFrame` parameter during encoding initialization.
142 Certain platforms may have limits on the the size of the reference window. Use the
143 :cpp:func:`MFXVideoENCODE_GetVideoParam` function to retrieve the current working
144 set of parameters.
145
146 During encoding, the application can specify the actual reference list lengths
147 by attaching the :cpp:struct:`mfxExtAVCRefListCtrl` structure to the
148 :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function. ``NumRefIdxL0Active``
149 specifies the length of the reference list L0 and ``NumRefIdxL1Active`` specifies
150 the length of the reference list L1. These two numbers must be less than or equal
151 to the :cpp:member:`mfxInfoMFX::NumRefFrame` parameter during encoding initialization.
152
153 The application can instruct the |vpl_short_name| encoder to use or not use certain reference
154 frames. To do this, there is a prerequisite that the application uniquely
155 identify each input frame by setting the :cpp:member:`mfxFrameData::FrameOrder`
156 parameter. The application then specifies the preferred reference frame list
157 ``PreferredRefList`` and/or the rejected frame list ``RejectedRefList``, and
158 attaches the :cpp:struct:`mfxExtAVCRefListCtrl` structure to the
159 :cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function. The two lists fine-tune how
160 the encoder chooses the reference frames for the current frame.
161 The encoder does not keep ``PreferredRefList`` and the application must send it
162 for each frame if necessary. There are limitations as follows:
163
164 - The frames in the lists are ignored if they are out of the reference window.
165 - If by going through the lists, the |vpl_short_name| encoder cannot find a reference frame
166 for the current frame, the encoder will encode the current frame without
167 using any reference frames.
168 - If the GOP pattern contains B-frames, the |vpl_short_name| encoder may not be able to follow
169 the :cpp:struct:`mfxExtAVCRefListCtrl` instructions.
170
171 ---------------------------------
172 Low Latency Encoding and Decoding
173 ---------------------------------
174
175 The application can set :cpp:member:`mfxVideoParam::AsyncDepth` = 1 to disable
176 any decoder buffering of output frames, which is aimed to improve the transcoding
177 throughput. With :cpp:member:`mfxVideoParam::AsyncDepth` = 1, the application
178 must synchronize after the decoding or transcoding operation of each frame.
179
180 The application can adjust :cpp:member:`mfxExtCodingOption::MaxDecFrameBuffering`
181 during encoding initialization to improve decoding latency. It is recommended to
182 set this value equal to the number of reference frames.
183
184 ------------------------------------------------
185 Reference Picture Marking Repetition SEI Message
186 ------------------------------------------------
187
188 The application can request writing the reference picture marking repetition SEI
189 message during encoding initialization by setting ``RefPicMarkRep`` of the
190 :cpp:struct:`mfxExtCodingOption` structure. The reference picture marking
191 repetition SEI message repeats certain reference frame information in the output
192 bitstream for robust streaming.
193
194 The |vpl_short_name| decoder will respond to the reference picture marking repetition SEI
195 message if the message exists in the bitstream and compare it to the reference
196 list information specified in the sequence/picture headers. The decoder will
197 report any mismatch of the SEI message with the reference list information in
198 the :cpp:member:`mfxFrameData::Corrupted` field.
199
200 .. _sec_long_term_reference_frame:
201
202 -------------------------
203 Long Term Reference Frame
204 -------------------------
205
206 The application may use long term reference frames to improve coding efficiency
207 or robustness for video conferencing applications. The application controls the
208 long term frame marking process by attaching the :cpp:struct:`mfxExtAVCRefListCtrl`
209 extended buffer during encoding. The |vpl_short_name| encoder itself never marks a frame as
210 long term.
211
212 There are two control lists in the :cpp:struct:`mfxExtAVCRefListCtrl` extended
213 buffer. The ``LongTermRefList`` list contains the frame orders (the ``FrameOrder``
214 value in the :cpp:struct:`mfxFrameData` structure) of the frames that should be
215 marked as long term frames. The ``RejectedRefList`` list contains the frame order
216 of the frames that should be unmarked as long term frames. The application can
217 only mark or unmark the frames that are buffered inside the encoder. Because of
218 this, it is recommended that the application marks a frame when it is submitted
219 for encoding. The application can either explicitly unmark long term reference
220 frames or wait for the IDR frame. When the IDR frame is reached, all long term
221 reference frames will be unmarked.
222
223 The |vpl_short_name| encoder puts all long term reference frames at the end of a reference
224 frame list. If the number of active reference frames (the ``NumRefIdxL0Active``
225 and ``NumRefIdxL1Active`` values in the :cpp:struct:`mfxExtAVCRefListCtrl`
226 extended buffer) is less than than the total reference frame number (the
227 ``NumRefFrame`` value in the :cpp:struct:`mfxInfoMFX` structure during the
228 encoding initialization), the encoder may ignore some or all long term
229 reference frames. The application may avoid this by providing a list of preferred
230 reference frames in the ``PreferredRefList`` list in the
231 :cpp:struct:`mfxExtAVCRefListCtrl` extended buffer. In this case, the encoder
232 reorders the reference list based on the specified list.
233
234 --------------------
235 Temporal Scalability
236 --------------------
237
238 The application may specify the temporal hierarchy of frames by using the
239 :cpp:struct:`mfxExtAvcTemporalLayers` extended buffer during the encoder
240 initialization in the display order encoding mode. |vpl_short_name| inserts the prefix
241 NAL unit before each slice with a unique temporal and priority ID. The temporal
242 ID starts from zero and the priority ID starts from the ``BaseLayerPID`` value.
243 |vpl_short_name| increases the temporal ID and priority ID value by one for each
244 consecutive layer.
245
246 If the application needs to specify a unique sequence or picture parameter set
247 ID, the application must use the :cpp:struct:`mfxExtCodingOptionSPSPPS`
248 extended buffer, with all pointers and sizes set to zero and valid
249 ``SPSId`` and ``PPSId`` fields. The same SPS and PPS ID will be used for all
250 temporal layers.
251
252 Each temporal layer is a set of frames with the same temporal ID. Each layer is
253 defined by the ``Scale`` value. The scale for layer N is equal to the ratio
254 between the frame rate of subsequent temporal layers with a temporal ID less
255 than or equal to N and the frame rate of the base temporal layer. The application
256 may skip some temporal layers by specifying the ``Scale`` value as zero. The
257 application should use an integer ratio of the frame rates for two consecutive
258 temporal layers.
259
260 For example, a video sequence with 30 frames/second is typically separated by
261 three temporal layers that can be decoded as 7.5 fps (base layer), 15 fps (base
262 and first temporal layer) and 30 fps (all three layers). In this scenario,
263 ``Scale`` should have the values {1,2,4,0,0,0,0,0}.