Debian Codegraph @

doc/surface_sharing_apis_overview.md

raw
1 # Intel® Video Processing Library (Intel® VPL) Surface Sharing - Overview of new APIs
2
3 Intel® VPL API 2.10 introduces new interfaces for sharing surfaces between Intel® VPL runtime and other applications, frameworks, and graphics APIs. This functionality is only supported when using the internal memory management model.
4
5 **Importing** a surface enables Intel® VPL to access raw video data as input to encode or VPP operations without first mapping the data to system memory and then copying it to a surface allocated by Intel® VPL runtime using ``mfxFrameSurfaceInterface::Map``.
6
7 **Exporting** a surface similarly enables the application to access raw video data which is the output of decode or VPP operations and which was allocated by Intel® VPL runtime, without first mapping to system memory using ``mfxFrameSurfaceInterface::Map``.
8
9 The following provides an overview of the new APIs. Additional information and runnable sample code may be found in the examples directory (examples/api2x/hello-sharing-*).
10
11 - Note: The following APIs are defined under the ONEVPL_EXPERIMENTAL macro. This macro must be defined at build time for these APIs to be accessible. Behavior of these APIs may change in future releases. Backwards compatibility and future presence of experiemental APIs is not guaranteed.
12
13 ## Importing and exporting surfaces between Intel® VPL runtime and external frameworks
14
15 ### New header file mfxmemory.h
16
17 The following function declarations are moved from mfxvideo.h into mfxmemory.h. For backward compatibility, mfxvideo.h #includes mfxmemory.h
18 ```
19 MFXMemory_GetSurfaceForDecode
20 MFXMemory_GetSurfaceForEncode
21 MFXMemory_GetSurfaceForVPP (with alias: MFXMemory_GetSurfaceForVPPIn)
22 MFXMemory_GetSurfaceForVPPOut
23 ```
24 ### New function - surface IMPORT
25
26 A new interface structure ```mfxMemoryInterface``` is defined, which contains function pointers filled in by the Intel® VPL runtime. New functions are then called by the application (i.e. a callback interface).
27
28 The application requests the interface using ```MFXVideoCORE_GetHandle()```. Inputs to GetHandle() are the mfxSession handle and an enum of type ```mfxHandleType```. The output is a generic handle (```mfxHDL``` = ```void*```) which the application then casts to the appropriate interface structure.
29
30 Detailed description:
31
32 - Defines a new value for ```mfxHandleType``` named ```MFX_HANDLE_MEMORY_INTERFACE```, which is used to request an interface of type ```mfxMemoryInterface```.
33
34 ```
35 MFX_HANDLE_MEMORY_INTERFACE = 1001, /*!< Pointer to interface of type mfxMemoryInterface. */
36 ```
37
38 - Defines a new callback interface mfxMemoryInterface in mfxmemory.h
39 Initially this will include fields for ```Context```, ```Version```, and one new function ```ImportFrameSurface```
40
41 ```
42 /* Specifies memory interface. */
43 typedef struct mfxMemoryInterface {
44 mfxHDL Context; /*!< The context of the memory interface. User should not touch (change, set, null) this pointer. */
45 mfxStructVersion Version; /*!< The version of the structure. */
46
47 /*!
48 @brief
49 Imports an application-provided surface into mfxFrameSurface1 which may be used as input for encoding or video processing.
50
51 @param[in] memory_interface Valid memory interface.
52 @param[in] surf_component Surface component type. Required for allocating new surfaces from the appropriate pool.
53 @param[in,out] external_surface Pointer to the mfxSurfaceXXX object describing the surface to be imported. All fields in
54 mfxSurfaceHeader must be set by the application. mfxSurfaceHeader::SurfaceType is
55 read by Intel® VPL runtime to determine which particular mfxSurfaceXXX structure is supplied.
56 For example, if mfxSurfaceXXX::SurfaceType == MFX_SURFACE_TYPE_D3D11_TEX2D, then the handle
57 will be interpreted as an object of type mfxSurfaceD3D11Tex2D. The application should
58 set or clear other fields as specified in the corresponding structure description.
59 After successful import, the value of mfxSurfaceHeader::SurfaceFlags will be replaced with the actual
60 import type. It can be used to determine which import type (with or without copy) took place in the case
61 of initial default setting, or if multiple import flags were OR'ed.
62 All external sync operations on the ext_surface must be completed before calling this function.
63 @param[out] imported_surface Pointer to a valid mfxFrameSurface1 object containing the imported frame.
64 imported_surface may be passed as an input to Encode or VPP processing operations.
65
66 @return
67 MFX_ERR_NONE The function completed successfully.\n
68 MFX_ERR_NULL_PTR If ext_surface or imported_surface are NULL.\n
69 MFX_ERR_INVALID_HANDLE If the corresponding session was not initialized.\n
70 MFX_ERR_UNSUPPORTED If surf_component is not one of [MFX_SURFACE_COMPONENT_ENCODE, MFX_SURFACE_COMPONENT_VPP_INPUT], or if
71 mfxSurfaceHeader::SurfaceType is not supported by Intel® VPL runtime for this operation.\n
72
73 @since This function is available since API version 2.10.
74 */
75
76 /* For reference with Export flow please search for mfxFrameSurfaceInterface::Export. */
77 mfxStatus (MFX_CDECL *ImportFrameSurface)(struct mfxMemoryInterface* memory_interface, mfxSurfaceComponent surf_component, mfxSurfaceHeader* external_surface, mfxFrameSurface1** imported_surface);
78
79 mfxHDL reserved[16];
80 } mfxMemoryInterface;
81 ```
82
83 - Defines a new macro ```MFXGetMemoryInterface``` which is a shortcut to calling ```MFXVideoCORE_GetHandle()``` with handle type ```MFX_HANDLE_MEMORY_INTERFACE```.
84
85 ```
86 /*! Alias for returning interface of type mfxMemoryInterface. */
87 #define MFXGetMemoryInterface(session, piface) MFXVideoCORE_GetHandle((session), MFX_HANDLE_MEMORY_INTERFACE, (mfxHDL *)(piface))
88 ```
89
90 ### New function - surface EXPORT
91 A new function ```Export()``` is added to the ```mfxFrameSurfaceInterface``` callback API.
92
93 ```
94 typedef struct mfxFrameSurfaceInterface {
95 ...
96
97 /*! @brief
98 If successful returns an exported surface, which is a refcounted object allocated by runtime. It could be exported with or without copy, depending
99 on export flags and the possibility of such export. Exported surface is valid throughout the session, as long as the original mfxFrameSurface1
100 object is not closed and the refcount of exported surface is not zero.
101
102 @param[in] surface Valid surface.
103 @param[in] export_header Description of export: caller should fill in SurfaceType (type to export to) and SurfaceFlags (allowed export modes).
104 @param[out] exported_surface Exported surface, allocated by runtime, user needs to decrement refcount after usage for object release.
105 After successful export, the value of mfxSurfaceHeader::SurfaceFlags will contain the actual export mode.
106
107
108 @return
109 MFX_ERR_NONE If no error. \n
110 MFX_ERR_NULL_PTR If export surface or surface is NULL. \n
111 MFX_ERR_UNSUPPORTED If requested export is not supported. \n
112 MFX_ERR_NOT_IMPLEMENTED If requested export is not implemented. \n
113 MFX_ERR_UNKNOWN Any internal error.
114 */
115
116 /* For reference with Import flow please search for mfxMemoryInterface::ImportFrameSurface. */
117 mfxStatus (MFX_CDECL *Export)(mfxFrameSurface1* surface, mfxSurfaceHeader export_header, mfxSurfaceHeader** exported_surface);
118
119 ...
120 } mfxFrameSurfaceInterface;
121 ```
122
123 ### New enumerated types
124
125 - Defines a new enum ```mfxSurfaceComponent``` to distinguish which internal surface pool frames are imported/exported to/from. This is required in order for Intel® VPL runtime to allocate and assign surfaces from the proper internal frame pool.
126
127 ```
128 /*! The mfxSurfaceComponent enumerator specifies the internal surface pool to use when importing surfaces. */
129 typedef enum {
130 MFX_SURFACE_COMPONENT_UNKNOWN = 0, /*!< Unknown surface component. */
131
132 MFX_SURFACE_COMPONENT_ENCODE = 1, /*!< Shared surface for encoding. */
133 MFX_SURFACE_COMPONENT_DECODE = 2, /*!< Shared surface for decoding. */
134 MFX_SURFACE_COMPONENT_VPP_INPUT = 3, /*!< Shared surface for VPP input. */
135 MFX_SURFACE_COMPONENT_VPP_OUTPUT = 4, /*!< Shared surface for VPP output. */
136 } mfxSurfaceComponent;
137 ```
138
139 - Defines a new enum ```mfxSurfaceType``` which will be part of the common description (header) that describes external surfaces. New definitions will be added to mfxSurfaceType as new external APIs are supported.
140
141 ```
142 /*! The mfxSurfaceType enumerator specifies the surface type described by mfxSurfaceHeader. */
143 typedef enum {
144 MFX_SURFACE_TYPE_UNKNOWN = 0, /*!< Unknown surface type. */
145
146 MFX_SURFACE_TYPE_D3D11_TEX2D = 2, /*!< D3D11 surface of type ID3D11Texture2D. */
147 MFX_SURFACE_TYPE_VAAPI = 3, /*!< VA-API surface. */
148 MFX_SURFACE_TYPE_OPENCL_IMG2D = 4, /*!< OpenCL 2D image (cl_mem). */
149 } mfxSurfaceType;
150 ```
151
152 - Defines a new list of enums prefixed with ```MFX_SURFACE_FLAG_``` which will be used as both inputs and outputs to Intel® VPL import/export operations. To avoid need for explicit cast when OR'ing flags together, it is not a typedef.
153 ```
154 /*! This enumerator specifies the sharing modes which are allowed for importing or exporting shared surfaces. */
155 enum {
156 MFX_SURFACE_FLAG_DEFAULT = 0x0000, /*!< Default is SHARED import or export. */
157
158 MFX_SURFACE_FLAG_IMPORT_SHARED = 0x0010, /*!< Import frames directly by mapping a shared native handle from an application-provided surface to an internally-allocated surface. */
159 MFX_SURFACE_FLAG_IMPORT_COPY = 0x0020, /*!< Import frames by copying data from an application-provided surface to an internally-allocated surface. */
160
161 MFX_SURFACE_FLAG_EXPORT_SHARED = 0x0100, /*!< Export frames directly by mapping a shared native handle from an internally-allocated surface to an application-provided surface. */
162 MFX_SURFACE_FLAG_EXPORT_COPY = 0x0200, /*!< Export frames by copying data from an internally-allocated surface to an application-provided surface. */
163 };
164 ```
165 - As inputs, these flags indicate whether the runtime should attempt to import/export surfaces with zero-copy ("shared") and/or by creating a copy ("copy").
166 - As outputs, these flages indicate what operation Intel® VPL runtime performed (shared mapping or copy).
167 - Flags may be OR'd together to indicate, e.g. "attempt zero-copy sharing if possible, otherwise doing a copy is acceptable".
168
169 ### New data structures - Common
170
171 Each supported external API will require a new data structure which contains the necessary info to permit sharing/copying between Intel® VPL and the external framework. The contents will vary for each external API, but same basic approach is followed as with Intel® VPL extension buffers: define a common "header" structure which is always the first element in the API-specific data structure, followed by additional elements which are particular to each API (frame handle, device context, etc.)
172
173 Defines a new data structure ```mfxSurfaceHeader``` which will be included as the first element in each API-specific surface description structure (below).
174
175 ```
176 typedef struct {
177 mfxSurfaceType SurfaceType; /*!< Set to the MFX_SURFACE_TYPE enum corresponding to the specific structure. */
178 mfxU32 SurfaceFlags; /*!< Set to the MFX_SURFACE_FLAG enum (or combination) corresponding to the allowed import / export mode(s). Multiple flags may be combined with OR.
179 Upon a successful Import or Export operation, this field will indicate the actual mode used.*/
180
181 mfxU32 StructSize; /*!< Size in bytes of the complete mfxSurfaceXXX structure. */
182
183 mfxU16 NumExtParam; /*!< The number of extra configuration structures attached to the structure. */
184 mfxExtBuffer** ExtParam; /*!< Points to an array of pointers to the extra configuration structures; see the ExtendedBufferID enumerator for a list of extended configurations. */
185
186 mfxU32 reserved[6];
187 } mfxSurfaceHeader;
188 ```
189
190 Defines a new data structure ```mfxSurfaceInterface``` which includes a set of callback functions to control refcounting and synchronization of imported/exported surfaces. The runtime fills in this structure with hooks to the appropriate function calls, which are implemented in the runtime.
191 ```
192 typedef struct mfxSurfaceInterface {
193 mfxSurfaceHeader Header; /*!< Exported surface header. Contains description of current surface. */
194
195 mfxStructVersion Version; /*!< The version of the structure. */
196
197 mfxHDL Context; /*!< The context of the exported surface interface. User should not touch (change, set, null) this pointer. */
198
199 /*! @brief
200 Increments the internal reference counter of the surface. The surface is not destroyed until the surface is released using the mfxSurfaceInterface::Release function.
201 mfxSurfaceInterface::AddRef should be used each time a new link to the surface is created (for example, copy structure) for proper surface management.
202
203 @param[in] surface Valid surface.
204
205 @return
206 MFX_ERR_NONE If no error. \n
207 MFX_ERR_NULL_PTR If surface is NULL. \n
208 MFX_ERR_INVALID_HANDLE If mfxSurfaceInterface->Context is invalid (for example NULL). \n
209 MFX_ERR_UNKNOWN Any internal error.
210
211 */
212 mfxStatus (MFX_CDECL *AddRef)(struct mfxSurfaceInterface* surface);
213
214 /*! @brief
215 Decrements the internal reference counter of the surface. mfxSurfaceInterface::Release should be called after using the
216 mfxSurfaceInterface::AddRef function to add a surface or when allocation logic requires it. For example, call
217 mfxSurfaceInterface::Release to release a surface obtained with the mfxFrameSurfaceInterface::Export function.
218
219 @param[in] surface Valid surface.
220
221 @return
222 MFX_ERR_NONE If no error. \n
223 MFX_ERR_NULL_PTR If surface is NULL. \n
224 MFX_ERR_INVALID_HANDLE If mfxSurfaceInterface->Context is invalid (for example NULL). \n
225 MFX_ERR_UNDEFINED_BEHAVIOR If Reference Counter of surface is zero before call. \n
226 MFX_ERR_UNKNOWN Any internal error.
227 */
228 mfxStatus (MFX_CDECL *Release)(struct mfxSurfaceInterface* surface);
229
230 /*! @brief
231 Returns current reference counter of exported surface.
232
233 @param[in] surface Valid surface.
234 @param[out] counter Sets counter to the current reference counter value.
235
236 @return
237 MFX_ERR_NONE If no error. \n
238 MFX_ERR_NULL_PTR If surface or counter is NULL. \n
239 MFX_ERR_INVALID_HANDLE If mfxSurfaceInterface->Context is invalid (for example NULL). \n
240 MFX_ERR_UNKNOWN Any internal error.
241 */
242 mfxStatus (MFX_CDECL *GetRefCounter)(struct mfxSurfaceInterface* surface, mfxU32* counter);
243
244 /*! @brief
245 This function is only valuable for surfaces which were exported in sharing mode (without a copy).
246 Guarantees readiness of both the data (pixels) and any original mfxFrameSurface1 frame's meta information (for example corruption flags) after a function completes.
247
248 Instead of MFXVideoCORE_SyncOperation, users may directly call the mfxSurfaceInterface::Synchronize function after the corresponding
249 Decode or VPP function calls (MFXVideoDECODE_DecodeFrameAsync or MFXVideoVPP_RunFrameVPPAsync).
250 The prerequisites to call the functions are:
251
252 @li The main processing functions return MFX_ERR_NONE.
253 @li A valid surface object.
254
255 @param[in] surface Valid surface.
256 @param[out] wait Wait time in milliseconds.
257
258 @return
259 MFX_ERR_NONE If no error. \n
260 MFX_ERR_NULL_PTR If surface is NULL. \n
261 MFX_ERR_INVALID_HANDLE If any of surface is not valid object . \n
262 MFX_WRN_IN_EXECUTION If the given timeout is expired and the surface is not ready. \n
263 MFX_ERR_ABORTED If the specified asynchronous function aborted due to data dependency on a previous asynchronous function that did not complete. \n
264 MFX_ERR_UNKNOWN Any internal error.
265 */
266 mfxStatus (MFX_CDECL *Synchronize)(struct mfxSurfaceInterface* surface, mfxU32 wait);
267
268 mfxHDL reserved[11];
269 } mfxSurfaceInterface;
270 ```
271
272 ### New data structures - framework-specific
273
274 Defines a set of new data structures of the form ```mfxSurfaceXXX```, where each XXX corresponds to an external framework such as DirectX11 (``mfxSurfaceD3D11Tex2D``), OpenCL (``mfxSurfaceOpenCLImg2D``), etc. Note that each struct begins with mfxSurfaceInterface, so any component may parse this structure by reading the first value (```SurfaceInterface.Header.SurfaceType```) and then casting the object to the corresponding ```mfxSurfaceXXX``` definition.
275
276 ```
277 typedef struct {
278 mfxSurfaceInterface SurfaceInterface;
279
280 mfxHDL texture2D; /*!< Pointer to texture, type ID3D11Texture2D */
281 mfxHDL reserved[7];
282 } mfxSurfaceD3D11Tex2D;
283 ```
284
285 ```
286 typedef struct {
287 mfxSurfaceInterface SurfaceInterface;
288
289 mfxHDL vaDisplay; /*!< Object of type VADisplay. */
290 mfxU32 vaSurfaceID; /*!< Object of type VASurfaceID. */
291 mfxU32 reserved1;
292
293 mfxHDL reserved[6];
294 } mfxSurfaceVAAPI;
295 ```
296
297 ```
298 typedef struct {
299 mfxSurfaceInterface SurfaceInterface;
300
301 mfxHDL ocl_context; /*!< Pointer to OpenCL context, type cl_context */
302 mfxHDL ocl_command_queue; /*!< Pointer to OpenCL command queue, type cl_command_queue */
303
304 mfxHDL ocl_image[4]; /*!< Pointer to OpenCL 2D images, type cl_mem */
305 mfxU32 ocl_image_num; /*!< Number of valid images (planes), depends on color format */
306
307 mfxHDL reserved[8];
308 } mfxSurfaceOpenCLImg2D;
309 ```
310
311 ```
312 /*!
313 Optional extension buffer, which can be attached to mfxSurfaceHeader::ExtParam
314 (second parameter of mfxFrameSurfaceInterface::Export) in order to pass OCL parameters
315 during mfxFrameSurface1 exporting to OCL surface.
316 If buffer is not provided all resources will be created by Intel® VPL RT internally.
317 */
318 typedef struct {
319 mfxExtBuffer Header; /*!< Extension buffer header. Header.BufferId must be equal to MFX_EXTBUFF_EXPORT_SHARING_DESC_OCL. */
320
321 mfxHDL ocl_context; /*!< Pointer to OpenCL context, type cl_context */
322 mfxHDL ocl_command_queue; /*!< Pointer to OpenCL command queue, type cl_command_queue */
323
324 mfxHDL reserved[8];
325 } mfxExtSurfaceOpenCLImg2DExportDescription;
326 ```
327
328 ## Querying for supported frameworks
329
330 Prior to session creation, applications may query the surface sharing capabilities of each implementation. This is possible by defining a new ```mfxImplCapsDeliveryFormat``` value:
331
332 ```
333 MFX_IMPLCAPS_SURFACE_TYPES = 5, /*!< Deliver capabilities as mfxSurfaceTypesSupported structure. */
334 ```
335
336 To access surface sharing capabilities, Intel® VPL dispatcher will call MFXQueryImplsDescription(format = MFX_IMPLCAPS_SURFACE_TYPES), and each runtime which supports the surface sharing APIs will return an array of mfxSurfaceTypesSupported structures (one per implementation) which descripbe the supported surface types, along with the associated components and sharing modes that each supports. The description structure is defined as follows:
337
338 ```
339 typedef struct {
340 mfxStructVersion Version; /*!< Version of the structure. */
341
342 mfxU16 NumSurfaceTypes; /*!< Number of supported surface types. */
343 struct surftype {
344 mfxSurfaceType SurfaceType; /*!< Supported surface type. */
345 mfxU32 reserved[6]; /*!< Reserved for future use. */
346 mfxU16 NumSurfaceComponents; /*!< Number of supported surface components. */
347 struct surfcomp {
348 mfxSurfaceComponent SurfaceComponent; /*!< Supported surface component. */
349 mfxU32 SurfaceFlags; /*!< Supported surface flags for this component (may be OR'd). */
350 mfxU32 reserved[7]; /*!< Reserved for future use. */
351 } *SurfaceComponents;
352 } *SurfaceTypes;
353
354 mfxU32 reserved[4]; /*!< Reserved for future use. */
355 } mfxSurfaceTypesSupported;
356 ```
357
358 As with other Intel® VPL implementation capabilities reports, an application may use this information in two ways:
359 1) It can call MFXEnumImplementations(format = MFX_IMPLCAPS_SURFACE_TYPES) either before or after session creation, and then use the contents of the returned mfxSurfaceTypesSupported structures to inform the construction of media processing pipelines.
360 2) It can call MFXSetConfigFilterProperty() prior to session creation, which causes the dispatcher to filter out (remove) any implementations which do not support the requested surface sharing operations. See an example in the code snippets below.
361
362 ## Code Snippets
363
364 ### Using MFXSetConfigFilterProperty() to require surface sharing capabilities
365 ```
366 // this example requires that the implementation provide D3D11 sharing for decode with EXPORT_SHARED mode (zero-copy)
367 // note that all three properties should be applied to the same mfxConfig object to avoid any ambiguity in
368 // describing the implementation capabilities
369
370 mfxConfig config = MFXCreateConfig(loader);
371 mfxVariant var = {};
372 var.Version.Version = (mfxU16)MFX_VARIANT_VERSION;
373
374 var.Type = MFX_VARIANT_TYPE_U32;
375 var.Data.U32 = MFX_SURFACE_TYPE_D3D11_TEX2D;
376 MFXSetConfigFilterProperty(config, (mfxU8 *)"mfxSurfaceTypesSupported.surftype.SurfaceType", var);
377
378 var.Type = MFX_VARIANT_TYPE_U32;
379 var.Data.U32 = MFX_SURFACE_COMPONENT_DECODE;
380 MFXSetConfigFilterProperty(config, (mfxU8 *)"mfxSurfaceTypesSupported.surftype.surfcomp.SurfaceComponent", var);
381
382 var.Type = MFX_VARIANT_TYPE_U32;
383 var.Data.U32 = MFX_SURFACE_FLAG_EXPORT_SHARED;
384 MFXSetConfigFilterProperty(config, (mfxU8 *)"mfxSurfaceTypesSupported.surftype.surfcomp.SurfaceFlags", var);
385
386 MFXCreateSession(loader, 0, &session);
387 ```
388
389 ### Exporting a frame decoded in Intel® VPL to a D3D11 texture
390 ```
391 // decode frame
392 sts = MFXVideoDECODE_DecodeFrameAsync(session, (isDrainingDec ? NULL : &fileInfo->bitstream), NULL, &pmfxDecOutSurface, &syncpD);
393
394 // run VPP on frame
395 sts = MFXVideoVPP_ProcessFrameAsync(session, pmfxDecOutSurface, &pmfxVPPOutSurface);
396
397 // release decoded frame (decrease reference count) after passing to VPP
398 pmfxDecOutSurface->FrameInterface->Release(pmfxDecOutSurface);
399
400 // wait for frame processing to complete
401 do {
402 sts = pmfxVPPOutSurface->FrameInterface->Synchronize(pmfxVPPOutSurface, 100);
403 } while (sts == MFX_WRN_IN_EXECUTION);
404
405 // set header, all other fields should be empty
406 mfxSurfaceHeader export_header = {};
407 export_header.SurfaceType = MFX_SURFACE_TYPE_D3D11_TEX2D;
408 export_header.SurfaceFlags = MFX_SURFACE_FLAG_EXPORT_SHARED;
409
410 // export mfxFrameSurface1 to a D3D11 texture, zero-copy enabled
411 mfxSurfaceD3D11Tex2D *exported_surface = nullptr;
412 sts = pmfxVPPOutSurface->FrameInterface->Export(pmfxVPPOutSurface, export_header, (mfxSurfaceHeader **)(&exported_surface));
413
414 // render the original D3D11 texture (output of VPP)
415 // RenderFrame() is a function which renders the exported surface to the display
416 ID3D11Texture2D *pTex2D = exported_surface->texture2D;
417 sts = RenderFrame(pTex2D);
418
419 // release the exported surface
420 sts = exported_surface->SurfaceInterface.Release(&(exported_surface->SurfaceInterface));
421
422 // release VPP output frame
423 pmfxVPPOutSurface->FrameInterface->Release(pmfxVPPOutSurface);
424 ```
425
426 ### Importing a D3D11 texture as a Intel® VPL frame for encoding
427 ```
428 // get interface with import function (only call once per session)
429 mfxMemoryInterface *memoryInterface = nullptr;
430 sts = MFXGetMemoryInterface(session, &memoryInterface);
431
432 // CaptureFrame is a function which allocates and returns a D3D11 texture to encode
433 // (for example: desktop duplication)
434 ID3D11Texture2D *pTex2D;
435 sts = CaptureFrame(&pTex2D);
436
437 // import D3D11 surface into mfxFrameSurface1
438 // set header.surfaceType, device, and texture2D, all other fields should be empty
439 mfxSurfaceD3D11Tex2D extSurfD3D11 = {};
440 extSurfD3D11.SurfaceInterface.Header.SurfaceType = MFX_SURFACE_TYPE_D3D11_TEX2D;
441 extSurfD3D11.SurfaceInterface.Header.SurfaceFlags = (MFX_SURFACE_FLAG_IMPORT_SHARED | MFX_SURFACE_FLAG_IMPORT_COPY);
442 extSurfD3D11.SurfaceInterface.Header.StructSize = sizeof(mfxSurfaceD3D11Tex2D);
443
444 // pointer to the texture to import
445 extSurfD3D11.texture2D = pTex2D;
446
447 // import D3D11 texture, zero-copy preferred, copy okay
448 mfxFrameSurface1 *pmfxEncSurface = nullptr;
449 sts = memoryInterface->ImportFrameSurface(memoryInterface, MFX_SURFACE_COMPONENT_ENCODE, (mfxSurfaceHeader *)extSurfD3D11, &pmfxEncSurface);
450
451 // encode
452 sts = MFXVideoENCODE_EncodeFrameAsync(session, NULL, (isDrainingEnc ? NULL : pmfxEncSurface), &fileInfo->bitstream, &syncp);
453
454 // release imported surface
455 pmfxEncSurface->FrameInterface->Release(pmfxEncSurface);
456 ```
457
458 ### Exporting a frame decoded in Intel® VPL to a VAAPI surface
459 ```
460 // decode frame
461 sts = MFXVideoDECODE_DecodeFrameAsync(session, (isDrainingDec ? NULL : &fileInfo->bitstream), NULL, &pmfxDecOutSurface, &syncpD);
462
463 // run VPP on frame
464 sts = MFXVideoVPP_ProcessFrameAsync(session, pmfxDecOutSurface, &pmfxVPPOutSurface);
465
466 // release decoded frame (decrease reference count) after passing to VPP
467 pmfxDecOutSurface->FrameInterface->Release(pmfxDecOutSurface);
468
469 // wait for frame processing to complete
470 do {
471 sts = pmfxVPPOutSurface->FrameInterface->Synchronize(pmfxVPPOutSurface, 100);
472 } while (sts == MFX_WRN_IN_EXECUTION);
473
474 // set header, all other fields should be empty
475 mfxSurfaceHeader export_header = {};
476 export_header.SurfaceType = MFX_SURFACE_TYPE_VAAPI;
477 export_header.SurfaceFlags = MFX_SURFACE_FLAG_EXPORT_SHARED;
478
479 // export mfxFrameSurface1 from Intel® VPL to a VAAPI surface, zero-copy enabled
480 mfxSurfaceVAAPI *exported_surface = nullptr;
481 sts = pmfxVPPOutSurface->FrameInterface->Export(pmfxVPPOutSurface, export_header, (mfxSurfaceHeader **)(&exported_surface));
482
483 // save the original VAAPI texture (output of VPP)
484 // SaveFrame() is a function which saves the exported surface to a file
485 VASurfaceID surfID = exported_surface->vaSurfaceID;
486 sts = SaveFrame(surfID);
487
488 // release the exported surface
489 sts = exported_surface->SurfaceInterface.Release(&(exported_surface->SurfaceInterface));
490
491 // release VPP output frame
492 pmfxVPPOutSurface->FrameInterface->Release(pmfxVPPOutSurface);
493 ```
494
495 ### Importing a VAAPI surface as a Intel® VPL frame for encoding
496
497 ```
498 // get interface with import function (only call once per session)
499 mfxMemoryInterface *memoryInterface = nullptr;
500 sts = MFXGetMemoryInterface(session, &memoryInterface);
501
502 // CaptureFrame is a function which allocates and returns a VAAPI texture to encode
503 // (for example: desktop duplication)
504 VASurfaceID vaSurfaceID;
505 sts = CaptureFrame(&vaSurfaceID);
506
507 // import VAAPI surface into mfxFrameSurface1
508 // set header.surfaceType, device, and texture2D, all other fields should be empty
509 mfxSurfaceVAAPI extSurfVAAPI = {};
510 extSurfVAAPI.SurfaceInterface.Header.SurfaceType = MFX_SURFACE_TYPE_VAAPI;
511 extSurfVAAPI.SurfaceInterface.Header.SurfaceFlags = (MFX_SURFACE_FLAG_IMPORT_SHARED | MFX_SURFACE_FLAG_IMPORT_COPY);
512 extSurfVAAPI.SurfaceInterface.Header.StructSize = sizeof(mfxSurfaceVAAPI);
513
514 // pointer to the surface to import and the display context of type VADisplay
515 extSurfVAAPI.vaDisplay = GetVADisplay();
516 extSurfVAAPI.vaSurfaceID = vaSurfaceID;
517
518 // import VAAPI surface, zero-copy preferred, copy okay
519 mfxFrameSurface1 *pmfxEncSurface = nullptr;
520 sts = memoryInterface->ImportFrameSurface(memoryInterface, MFX_SURFACE_COMPONENT_ENCODE, (mfxSurfaceHeader *)extSurfVAAPI, &pmfxEncSurface);
521
522 // encode
523 sts = MFXVideoENCODE_EncodeFrameAsync(session, NULL, (isDrainingEnc ? NULL : pmfxEncSurface), &fileInfo->bitstream, &syncp);
524
525 // release imported surface
526 pmfxEncSurface->FrameInterface->Release(pmfxEncSurface);
527 ```
528
529 ### Initializing OpenCL extension functions
530 ```
531 // to import/export with OpenCL, the application must check for the presence of and
532 // retreive the addresses of the following functions from the OpenCL runtime
533 // this only happens one time, during initialization of the OpenCL context
534 // these function pointers will be used in the code snippets which follow
535
536 clCreateFromD3D11Texture2DKHR_fn pfn_clCreateFromD3D11Texture2DKHR;
537 clEnqueueAcquireD3D11ObjectsKHR_fn pfn_clEnqueueAcquireD3D11ObjectsKHR;
538 clEnqueueReleaseD3D11ObjectsKHR_fn pfn_clEnqueueReleaseD3D11ObjectsKHR;
539
540 cl_platform_id platform_id = GetPlatformID(); // platform ID for the device used to create the OpenCL context
541
542 pfn_clCreateFromD3D11Texture2DKHR = clGetExtensionFunctionAddressForPlatform(platform_id, (const char *)"clCreateFromD3D11Texture2DKHR");
543 pfn_clEnqueueAcquireD3D11ObjectsKHR = clGetExtensionFunctionAddressForPlatform(platform_id, (const char *)"clEnqueueAcquireD3D11ObjectsKHR");
544 pfn_clEnqueueReleaseD3D11ObjectsKHR = clGetExtensionFunctionAddressForPlatform(platform_id, (const char *)"clEnqueueReleaseD3D11ObjectsKHR");
545 ```
546
547 ### Exporting a frame decoded in Intel® VPL to an OpenCL image for processing
548 - Note: Currently supported on Windows only. Exported images will be provided as shared surfaces created with the clCreateFromD3D11Texture2DKHR extension.
549 ```
550 // decode frame
551 sts = MFXVideoDECODE_DecodeFrameAsync(session, (isDrainingDec ? NULL : &fileInfo->bitstream), NULL, &pmfxDecOutSurface, &syncpD);
552
553 // run VPP on frame
554 sts = MFXVideoVPP_ProcessFrameAsync(session, pmfxDecOutSurface, &pmfxVPPOutSurface);
555
556 // release decoded frame (decrease reference count) after passing to VPP
557 pmfxDecOutSurface->FrameInterface->Release(pmfxDecOutSurface);
558
559 // wait for frame processing to complete
560 do {
561 sts = pmfxVPPOutSurface->FrameInterface->Synchronize(pmfxVPPOutSurface, 100);
562 } while (sts == MFX_WRN_IN_EXECUTION);
563
564 // set header, all other fields should be empty
565 mfxSurfaceHeader export_header = {};
566 export_header.SurfaceType = MFX_SURFACE_TYPE_OPENCL_IMG2D;
567 export_header.SurfaceFlags = MFX_SURFACE_FLAG_EXPORT_SHARED;
568 export_header.NumExtParam = 1;
569
570 // provide extension buffer with the OpenCL context and command queue which
571 // were initialized by the application
572 // the runtime needs this information to confirm that the exported surface is accessible
573 // in this context and command queue
574 mfxExtSurfaceOpenCLImg2DExportDescription export_header_buf = {};
575 export_header_buf.Header.BufferId = MFX_EXTBUFF_EXPORT_SHARING_DESC_OCL;
576 export_header_buf.ocl_context = GetOpenCLContext();
577 export_header_buf.ocl_command_queue = GetOpenCLCommandQueue();
578 export_header_buf.Header.BufferSz = sizeof(mfxExtSurfaceOpenCLImg2DExportDescription);
579
580 // attach the extension buffer to mfxSurfaceHeader
581 std::vector<mfxExtBuffer *> extBufs;
582 extBufs.push_back((mfxExtBuffer *)&export_header_buf);
583 export_header.ExtParam = extBufs.data();
584
585 // export mfxFrameSurface1 from Intel® VPL to an OpenCL image, zero-copy enabled
586 mfxSurfaceOpenCLImg2D *extSurfOCL = nullptr;
587 sts = pmfxVPPOutSurface->FrameInterface->Export(pmfxVPPOutSurface, export_header, (mfxSurfaceHeader **)&extSurfOCL);
588
589 // access the exported OpenCL image
590 // in this example the video format = NV12 and consists of two planes (0 = Y and 1 = interleaved UV)
591 cl_mem clBufInY = (cl_mem)extSurfOCL->ocl_image[0]
592 cl_mem clBufInUV = (cl_mem)extSurfOCL->ocl_image[1];
593
594 // lock the shared surface for OCL exclusive use
595 cl_mem surfaces[2] = { clBufInY, clBufInUV };
596 clError = m_pfn_clEnqueueAcquireD3D11ObjectsKHR(m_clqueue, 2, surfaces, 0, NULL, NULL);
597
598 // pass the exported images as arguments to OpenCL kernels
599 clError = clSetKernelArg(m_clkernelY, 0, sizeof(cl_mem), &clBufInY);
600 clError = clSetKernelArg(m_clkernelUV, 0, sizeof(cl_mem), &clBufInUV);
601
602 // enqueue the kernels for processing
603 cl_event kernelEvents[2] = {};
604 clError = clEnqueueNDRangeKernel(m_clqueue, m_clkernelY, 2, NULL, m_GlobalWorkSizeY, m_LocalWorkSizeY, 0, NULL, &kernelEvents[0]);
605 clError = clEnqueueNDRangeKernel(m_clqueue, m_clkernelUV, 2, NULL, m_GlobalWorkSizeUV, m_LocalWorkSizeUV, 0, NULL, &kernelEvents[1]);
606
607 // wait for kernels to finish (also flushes command queue)
608 clError = clWaitForEvents(sizeof(kernelEvents) / sizeof(kernelEvents[0]), kernelEvents);
609
610 // unlock the shared exported surface from OCL exclusive use
611 clError = pfn_clEnqueueReleaseD3D11ObjectsKHR(m_clqueue, 2, surfaces, 0, NULL, NULL);
612
613 // release the exported surface
614 sts = extSurfOCL->SurfaceInterface.Release(&(extSurfOCL->SurfaceInterface));
615
616 // release VPP output frame
617 pmfxVPPOutSurface->FrameInterface->Release(pmfxVPPOutSurface);```
618 ```
619
620 ### Importing an OpenCL image as a Intel® VPL frame for encoding
621 - Note: Currently supported on Windows only.
622 ```
623 // get interface with import function (only call once per session)
624 mfxMemoryInterface *memoryInterface = nullptr;
625 sts = MFXGetMemoryInterface(session, &memoryInterface);
626
627 // allocate an OCL surface (NV12) the first time it's needed
628
629 // Y plane
630 cl_mem clBufInY = nullptr;
631 cl_image_format src_format = {};
632 cl_image_desc src_image_desc = {};
633
634 src_format.image_channel_order = CL_R;
635 src_format.image_channel_data_type = CL_UNORM_INT8;
636 src_image_desc.image_width = width;
637 src_image_desc.image_height = height;
638 src_image_desc.image_type = CL_MEM_OBJECT_IMAGE2D;
639 clBufInY = clCreateImage(m_clcontext, CL_MEM_READ_WRITE | CL_MEM_ALLOC_HOST_PTR, &src_format, &src_image_desc, 0, &error);
640
641 // UV plane
642 cl_mem clBufInUV = nullptr;
643 src_format.image_channel_order = CL_RG;
644 src_format.image_channel_data_type = CL_UNORM_INT8;
645 src_image_desc.image_width /= 2;
646 src_image_desc.image_height /= 2;
647 clBufInUV = clCreateImage(m_clcontext, CL_MEM_READ_WRITE | CL_MEM_ALLOC_HOST_PTR, &src_format, &src_image_desc, 0, &error);
648
649 // now load data into the OpenCL image and do some processing...
650 // (see the Export example for main steps in running kernels)
651
652 // once OpenCL processing is complete, import the surface
653
654 // set header and ocl_context, ocl_flags is optional, all other fields should be empty
655 mfxSurfaceOpenCLImg2D extSurfOCL = {};
656 extSurfOCL.SurfaceInterface.Header.SurfaceType = MFX_SURFACE_TYPE_OPENCL_IMG2D;
657
658 // Specify either SHARED or COPY as permitted import modes
659 // NOTE: we expect that runtime only supports copy mode in current implementation
660 // After calling ImportFrameSurface, we can check extSurfOCL.SurfaceInterface.Header.SurfaceFlags to see what
661 // import mode was actually used by the runtime.
662 extSurfOCL.SurfaceInterface.Header.SurfaceFlags = (MFX_SURFACE_FLAG_IMPORT_SHARED | MFX_SURFACE_FLAG_IMPORT_COPY);
663 extSurfOCL.SurfaceInterface.Header.StructSize = sizeof(mfxSurfaceOpenCLImg2D);
664
665 // pass the OpenCL context and command queue which were initialized by the application
666 extSurfOCL.ocl_context = GetOpenCLContext();
667 extSurfOCL.ocl_command_queue = GetOpenCLCommandQueue();
668
669 extSurfOCL.ocl_image_num = 2;
670 extSurfOCL.ocl_image[0] = clBufInY;
671 extSurfOCL.ocl_image[1] = clBufInUV;
672
673 // import the OpenCL surface
674 mfxFrameSurface1 *pmfxEncSurface = nullptr;
675 sts = memoryInterface->ImportFrameSurface(encCtx->memoryInterface, MFX_SURFACE_COMPONENT_ENCODE, (mfxSurfaceHeader *)(&extSurfOCL), &pmfxEncSurface);
676
677 // encode
678 sts = MFXVideoENCODE_EncodeFrameAsync(session, NULL, (isDrainingEnc ? NULL : pmfxEncSurface), &fileInfo->bitstream, &syncp);
679
680 // release OCL surfaces (decrease refcount)
681 clReleaseMemObject(clBufInY);
682 clReleaseMemObject(clBufInUV);
683
684 // release imported surface
685 pmfxEncSurface->FrameInterface->Release(pmfxEncSurface);
686 ```
687
688 ## Reference counting and release of imported surfaces
689
690 The following describes details the proper release procedure and sequence for surface in import/shared mode
691
692 If resources required for importing were created by user, then user should release it.
693
694 * The exact moment of possible deallocation of the original surface is dependent on the type of Import and the type of surface. For some types (i.e. for those who support refcounts - DXVA), it is possible to release right after successful Import.
695
696 For DXVA, GPU RT adds refcount on imported d3d11 texture, so user can release original texture right after import.
697
698 For VAAPI, user can’t delete original VAsurface, because it will obviously invalidate imported surface. So user must wait till destruction of imported `mfxFrameSurface1` first.
699
700 For example,
701 ```
702 surfaceForImport.vaSurfaceID = vaSurfaceID;
703 ...
704 m_memoryInterface->ImportFrameSurface(m_memoryInterface, MFX_SURFACE_COMPONENT_ENCODE, &surfaceForImport.SurfaceInterface.Header, &pmfxEncSurface);
705 ```
706
707 In general, user should destroy `surfaceForImport` after destruction of `pmfxEncSurface` (it is destroyed when refcount reaches zero, refcount decreased by `Release()` calls) then it is 100% safe.
708
709 But in some cases, like DXVA textures, user may destroy surfaceForImport earlier.
710
711 When Encoder does frame reordering (it happens, when there are B-frames in GOP), then user must keep reordered mfxFrameSurfaces alive, until they are encoded. To understand when it is safe to delete the mfxFrameSurface – you need to check value of `mfxFrameSurface1::Data::Locked`.
712
713 When it is `0`, then component doesn’t need this mfxFrameSurface anymore. So it is safe to delete or, much better, reuse it.
714
715 It is essential for legacy allocation model (external allocator), when we managing surface pool by hands.
716 Also it can be used for Shared imported native handles without refcounts (like VAsurfaces), i.e. it is right moment to release `mfxFrameSurface1` object then native handle. Order matters, original handle should be destroyed after `mfxFrameSurface1` object.
717
718 For refcounted case management of `mfxFrameSurface1` objects is much simpler. Encoder (Decoder and VPP do the same) will `AddRef()` surface inside, so it is safe to `Release()` `mfxFrameSurface1` object right after submission disregard the returned status. `mfxFrameSurface1` will not be deleted, because Encoder holds it.
719
720 Important note: there is no dependency between original handle and `mfxFrameSurface1` object for copy imported surface. They could be managed (i.e. deleted) independently.
721
722 For Export case (for both Copy and Shared type) original `mfxFrameSurface1` and Exported `mfxSurfaceXXX` can be managed (i.e. `Release()`) independently, Intel® VPL RT manages dependency and keeps all required resources alive.