| 1 |
.. SPDX-FileCopyrightText: 2019-2020 Intel Corporation |
| 2 |
.. |
| 3 |
.. SPDX-License-Identifier: CC-BY-4.0 |
| 4 |
|
| 5 |
.. _stream_vid_conf_features: |
| 6 |
|
| 7 |
========================================= |
| 8 |
Streaming and Video Conferencing Features |
| 9 |
========================================= |
| 10 |
|
| 11 |
The following sections address some aspects of additional requirements that |
| 12 |
streaming or video conferencing applications may use in the encoding or |
| 13 |
transcoding process. See the :ref:`config-change` section for additional |
| 14 |
information. |
| 15 |
|
| 16 |
---------------------- |
| 17 |
Dynamic Bitrate Change |
| 18 |
---------------------- |
| 19 |
|
| 20 |
The |vpl_short_name| encoder supports dynamic bitrate change according to bitrate control mode |
| 21 |
and HRD conformance requirements. If HRD conformance is required, for example if |
| 22 |
the application sets the ``NalHrdConformance`` option in the |
| 23 |
:cpp:struct:`mfxExtCodingOption` structure to ON, the only allowed |
| 24 |
bitrate control mode is VBR. In this mode, the application can change the |
| 25 |
``TargetKbps`` and ``MaxKbps`` values of the :cpp:struct:`mfxInfoMFX` structure |
| 26 |
by calling the :cpp:func:`MFXVideoENCODE_Reset` function. This sort of change in |
| 27 |
bitrate usually results in the generation of a new keyframe and sequence header. |
| 28 |
There are exceptions, such as if HRD information is absent in the stream. |
| 29 |
In this scenario, the change of ``TargetKbps`` does not require a change in the |
| 30 |
sequence header and as a result the encoder does not insert a keyframe. |
| 31 |
|
| 32 |
If HRD conformance is not required, for example if the application turns off |
| 33 |
the ``NalHrdConformance`` option in the :cpp:struct:`mfxExtCodingOption` |
| 34 |
structure, all bitrate control modes are available. In CBR and AVBR modes the |
| 35 |
application can change ``TargetKbps``. In VBR mode the application can change |
| 36 |
``TargetKbps`` and ``MaxKbps`` values. This sort of change in bitrate will not |
| 37 |
result in the generation of a new keyframe or sequence header. |
| 38 |
|
| 39 |
The |vpl_short_name| encoder may change some initialization parameters provided by the |
| 40 |
application during initialization. That in turn may lead to incompatibility |
| 41 |
between the parameters provided by the application during reset and the working set |
| 42 |
of parameters used by the encoder. For this reason, it is strongly recommended |
| 43 |
to retrieve the actual working parameters using the |
| 44 |
:cpp:func:`MFXVideoENCODE_GetVideoParam` function before making any changes to |
| 45 |
bitrate settings. |
| 46 |
|
| 47 |
In all modes, |vpl_short_name| encoders will respond to the bitrate changes as quickly as |
| 48 |
the underlying algorithm allows, without breaking other encoding restrictions |
| 49 |
such as HRD compliance if it is enabled. How quickly the actual bitrate can catch |
| 50 |
up with the specified bitrate is implementation dependent. |
| 51 |
|
| 52 |
Alternatively, the application may use the :term:`CQP` encoding mode to perform |
| 53 |
customized bitrate adjustment on a per-frame base. The application may use any |
| 54 |
of the encoded or display order modes to use per-frame CQP. |
| 55 |
|
| 56 |
------------------------- |
| 57 |
Dynamic Resolution Change |
| 58 |
------------------------- |
| 59 |
|
| 60 |
The |vpl_short_name| encoder supports dynamic resolution change in all bitrate control modes. |
| 61 |
The application may change resolution by calling the :cpp:func:`MFXVideoENCODE_Reset` |
| 62 |
function. The application may decrease or increase resolution up to the size |
| 63 |
specified during encoder initialization. |
| 64 |
|
| 65 |
Resolution change always results in the insertion of a key IDR frame and a new |
| 66 |
sequence parameter set in the header. The only exception is the |vpl_short_name| VP9 encoder |
| 67 |
(see section for `Dynamic reference frame scaling`_). The |vpl_short_name| encoder does not |
| 68 |
guarantee HRD conformance across the resolution change point. |
| 69 |
|
| 70 |
The |vpl_short_name| encoder may change some initialization parameters provided by the |
| 71 |
application during initialization. That in turn may lead to incompatibility of |
| 72 |
parameters provide by the application during reset and working set of parameters |
| 73 |
used by the encoder. Due to this potential incompatibility, it is strongly |
| 74 |
recommended to retrieve the actual working parameters set by |
| 75 |
:cpp:func:`MFXVideoENCODE_GetVideoParam` function before making any resolution |
| 76 |
change. |
| 77 |
|
| 78 |
------------------------------- |
| 79 |
Dynamic Reference Frame Scaling |
| 80 |
------------------------------- |
| 81 |
|
| 82 |
The VP9 standard allows changing the resolution without the insertion of a |
| 83 |
keyframe. This is possible because the VP9 encoder has the built-in capability |
| 84 |
to upscale and downscale reference frames to match the resolution of the frame |
| 85 |
being encoded. By default the |vpl_short_name| VP9 encoder inserts a keyframe when the |
| 86 |
application does `Dynamic Resolution Change`_. In this case, the first frame |
| 87 |
with a new resolution is encoded using inter prediction from the scaled reference |
| 88 |
frame of the previous resolution. Dynamic scaling has the following limitations, |
| 89 |
described in the VP9 specification: |
| 90 |
|
| 91 |
* The resolution of any active reference frame cannot exceed 2x the resolution |
| 92 |
of the current frame. |
| 93 |
* The resolution of any active reference frame cannot be smaller than 1/16 of |
| 94 |
the current frame resolution. |
| 95 |
|
| 96 |
In the case of dynamic scaling, the |vpl_short_name| VP9 encoder always uses a single active |
| 97 |
reference frame for the first frame after a resolution change. The VP9 |
| 98 |
encoder has the following limitations for dynamic resolution change: |
| 99 |
|
| 100 |
* The new resolution should not exceed 16x the resolution of the current frame. |
| 101 |
* The new resolution should be less than 1/2 of current frame resolution. |
| 102 |
|
| 103 |
The application may force insertion of a keyframe at the point of resolution |
| 104 |
change by invoking encoder reset with :cpp:member:`mfxExtEncoderResetOption::StartNewSequence` |
| 105 |
set to :cpp:enumerator:`MFX_CODINGOPTION_ON`. If a keyframe is inserted, the |
| 106 |
dynamic resolution limitations are not enforced. |
| 107 |
|
| 108 |
Note that resolution change with dynamic reference scaling is compatible with |
| 109 |
multiref (:cpp:member:`mfxInfoMFX::NumRefFrame` > 1). For multiref |
| 110 |
configuration, the |vpl_short_name| VP9 encoder uses multiple references within stream pieces |
| 111 |
of the same resolution and uses a single reference at the place of resolution |
| 112 |
change. |
| 113 |
|
| 114 |
--------------------------- |
| 115 |
Forced Keyframe Generation |
| 116 |
--------------------------- |
| 117 |
|
| 118 |
|vpl_short_name| supports forced keyframe generation during encoding. The application |
| 119 |
can set the FrameType parameter of the :cpp:struct:`mfxEncodeCtrl` structure to |
| 120 |
control how the current frame is encoded, as follows: |
| 121 |
|
| 122 |
- If the |vpl_short_name| encoder works in the display order, the application can enforce any |
| 123 |
current frame to be a keyframe. The application cannot change the frame type |
| 124 |
of already buffered frames inside the encoder. |
| 125 |
- If the |vpl_short_name| encoder works in the encoded order, the application must specify |
| 126 |
exact frame type for every frame. In this way, the application can enforce the |
| 127 |
current frame to have any frame type that the particular coding standard allows. |
| 128 |
|
| 129 |
.. _sec_reference_list_selection: |
| 130 |
|
| 131 |
------------------------ |
| 132 |
Reference List Selection |
| 133 |
------------------------ |
| 134 |
|
| 135 |
During streaming or video conferencing, if the application can obtain feedback |
| 136 |
about how well the client receives certain frames, the application may need to |
| 137 |
adjust the encoding process to use or not use certain frames as reference. This |
| 138 |
section describes how to fine-tune the encoding process based on client feedback. |
| 139 |
|
| 140 |
The application can specify the reference window size by specifying the |
| 141 |
:cpp:member:`mfxInfoMFX::NumRefFrame` parameter during encoding initialization. |
| 142 |
Certain platforms may have limits on the the size of the reference window. Use the |
| 143 |
:cpp:func:`MFXVideoENCODE_GetVideoParam` function to retrieve the current working |
| 144 |
set of parameters. |
| 145 |
|
| 146 |
During encoding, the application can specify the actual reference list lengths |
| 147 |
by attaching the :cpp:struct:`mfxExtAVCRefListCtrl` structure to the |
| 148 |
:cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function. ``NumRefIdxL0Active`` |
| 149 |
specifies the length of the reference list L0 and ``NumRefIdxL1Active`` specifies |
| 150 |
the length of the reference list L1. These two numbers must be less than or equal |
| 151 |
to the :cpp:member:`mfxInfoMFX::NumRefFrame` parameter during encoding initialization. |
| 152 |
|
| 153 |
The application can instruct the |vpl_short_name| encoder to use or not use certain reference |
| 154 |
frames. To do this, there is a prerequisite that the application uniquely |
| 155 |
identify each input frame by setting the :cpp:member:`mfxFrameData::FrameOrder` |
| 156 |
parameter. The application then specifies the preferred reference frame list |
| 157 |
``PreferredRefList`` and/or the rejected frame list ``RejectedRefList``, and |
| 158 |
attaches the :cpp:struct:`mfxExtAVCRefListCtrl` structure to the |
| 159 |
:cpp:func:`MFXVideoENCODE_EncodeFrameAsync` function. The two lists fine-tune how |
| 160 |
the encoder chooses the reference frames for the current frame. |
| 161 |
The encoder does not keep ``PreferredRefList`` and the application must send it |
| 162 |
for each frame if necessary. There are limitations as follows: |
| 163 |
|
| 164 |
- The frames in the lists are ignored if they are out of the reference window. |
| 165 |
- If by going through the lists, the |vpl_short_name| encoder cannot find a reference frame |
| 166 |
for the current frame, the encoder will encode the current frame without |
| 167 |
using any reference frames. |
| 168 |
- If the GOP pattern contains B-frames, the |vpl_short_name| encoder may not be able to follow |
| 169 |
the :cpp:struct:`mfxExtAVCRefListCtrl` instructions. |
| 170 |
|
| 171 |
--------------------------------- |
| 172 |
Low Latency Encoding and Decoding |
| 173 |
--------------------------------- |
| 174 |
|
| 175 |
The application can set :cpp:member:`mfxVideoParam::AsyncDepth` = 1 to disable |
| 176 |
any decoder buffering of output frames, which is aimed to improve the transcoding |
| 177 |
throughput. With :cpp:member:`mfxVideoParam::AsyncDepth` = 1, the application |
| 178 |
must synchronize after the decoding or transcoding operation of each frame. |
| 179 |
|
| 180 |
The application can adjust :cpp:member:`mfxExtCodingOption::MaxDecFrameBuffering` |
| 181 |
during encoding initialization to improve decoding latency. It is recommended to |
| 182 |
set this value equal to the number of reference frames. |
| 183 |
|
| 184 |
------------------------------------------------ |
| 185 |
Reference Picture Marking Repetition SEI Message |
| 186 |
------------------------------------------------ |
| 187 |
|
| 188 |
The application can request writing the reference picture marking repetition SEI |
| 189 |
message during encoding initialization by setting ``RefPicMarkRep`` of the |
| 190 |
:cpp:struct:`mfxExtCodingOption` structure. The reference picture marking |
| 191 |
repetition SEI message repeats certain reference frame information in the output |
| 192 |
bitstream for robust streaming. |
| 193 |
|
| 194 |
The |vpl_short_name| decoder will respond to the reference picture marking repetition SEI |
| 195 |
message if the message exists in the bitstream and compare it to the reference |
| 196 |
list information specified in the sequence/picture headers. The decoder will |
| 197 |
report any mismatch of the SEI message with the reference list information in |
| 198 |
the :cpp:member:`mfxFrameData::Corrupted` field. |
| 199 |
|
| 200 |
.. _sec_long_term_reference_frame: |
| 201 |
|
| 202 |
------------------------- |
| 203 |
Long Term Reference Frame |
| 204 |
------------------------- |
| 205 |
|
| 206 |
The application may use long term reference frames to improve coding efficiency |
| 207 |
or robustness for video conferencing applications. The application controls the |
| 208 |
long term frame marking process by attaching the :cpp:struct:`mfxExtAVCRefListCtrl` |
| 209 |
extended buffer during encoding. The |vpl_short_name| encoder itself never marks a frame as |
| 210 |
long term. |
| 211 |
|
| 212 |
There are two control lists in the :cpp:struct:`mfxExtAVCRefListCtrl` extended |
| 213 |
buffer. The ``LongTermRefList`` list contains the frame orders (the ``FrameOrder`` |
| 214 |
value in the :cpp:struct:`mfxFrameData` structure) of the frames that should be |
| 215 |
marked as long term frames. The ``RejectedRefList`` list contains the frame order |
| 216 |
of the frames that should be unmarked as long term frames. The application can |
| 217 |
only mark or unmark the frames that are buffered inside the encoder. Because of |
| 218 |
this, it is recommended that the application marks a frame when it is submitted |
| 219 |
for encoding. The application can either explicitly unmark long term reference |
| 220 |
frames or wait for the IDR frame. When the IDR frame is reached, all long term |
| 221 |
reference frames will be unmarked. |
| 222 |
|
| 223 |
The |vpl_short_name| encoder puts all long term reference frames at the end of a reference |
| 224 |
frame list. If the number of active reference frames (the ``NumRefIdxL0Active`` |
| 225 |
and ``NumRefIdxL1Active`` values in the :cpp:struct:`mfxExtAVCRefListCtrl` |
| 226 |
extended buffer) is less than than the total reference frame number (the |
| 227 |
``NumRefFrame`` value in the :cpp:struct:`mfxInfoMFX` structure during the |
| 228 |
encoding initialization), the encoder may ignore some or all long term |
| 229 |
reference frames. The application may avoid this by providing a list of preferred |
| 230 |
reference frames in the ``PreferredRefList`` list in the |
| 231 |
:cpp:struct:`mfxExtAVCRefListCtrl` extended buffer. In this case, the encoder |
| 232 |
reorders the reference list based on the specified list. |
| 233 |
|
| 234 |
-------------------- |
| 235 |
Temporal Scalability |
| 236 |
-------------------- |
| 237 |
|
| 238 |
The application may specify the temporal hierarchy of frames by using the |
| 239 |
:cpp:struct:`mfxExtAvcTemporalLayers` extended buffer during the encoder |
| 240 |
initialization in the display order encoding mode. |vpl_short_name| inserts the prefix |
| 241 |
NAL unit before each slice with a unique temporal and priority ID. The temporal |
| 242 |
ID starts from zero and the priority ID starts from the ``BaseLayerPID`` value. |
| 243 |
|vpl_short_name| increases the temporal ID and priority ID value by one for each |
| 244 |
consecutive layer. |
| 245 |
|
| 246 |
If the application needs to specify a unique sequence or picture parameter set |
| 247 |
ID, the application must use the :cpp:struct:`mfxExtCodingOptionSPSPPS` |
| 248 |
extended buffer, with all pointers and sizes set to zero and valid |
| 249 |
``SPSId`` and ``PPSId`` fields. The same SPS and PPS ID will be used for all |
| 250 |
temporal layers. |
| 251 |
|
| 252 |
Each temporal layer is a set of frames with the same temporal ID. Each layer is |
| 253 |
defined by the ``Scale`` value. The scale for layer N is equal to the ratio |
| 254 |
between the frame rate of subsequent temporal layers with a temporal ID less |
| 255 |
than or equal to N and the frame rate of the base temporal layer. The application |
| 256 |
may skip some temporal layers by specifying the ``Scale`` value as zero. The |
| 257 |
application should use an integer ratio of the frame rates for two consecutive |
| 258 |
temporal layers. |
| 259 |
|
| 260 |
For example, a video sequence with 30 frames/second is typically separated by |
| 261 |
three temporal layers that can be decoded as 7.5 fps (base layer), 15 fps (base |
| 262 |
and first temporal layer) and 30 fps (all three layers). In this scenario, |
| 263 |
``Scale`` should have the values {1,2,4,0,0,0,0,0}. |