Render API (original) (raw)

MDK supports rendering video via OpenGL, D3D11/12, Vulkan and Metal. You can choose which api to use at runtime via Player.setRenderAPI(RenderAPI* api, void* vo_opaque = nullptr).

OpenGL is the default api if you don't call setRenderAPI(). To use another api, a concrete RenderAPI subclass object is required. For example to use d3d11

D3D11RenderAPI ra;
// set ra.context, ra.rtv if provided by app
player.setRenderAPI(&ra);

4 Rendering Modes in MDK

Render To Texture

Render target is set by user, we start render pass internally

OpenGL

No need to call player.setRenderAPI() if fbo is correct

GLRenderAPI ra{};
ra.fbo =;
player.setRenderAPI(&ra, this);

D3D11

D3D11RenderAPI ra{};
ra.context = immediateContext; // immediateContext provied by app, can be null if rtv is not null
ra.rtv = renderTargetView; // ID3D11Texture2D* or ID3D11RenderTargetView* provided by app, can not be null
player.setRenderAPI(&ra);

To use the texture as video encoder/processer input, must ensure current draw is completed

IDXSync

class __declspec(uuid("1c5a6669-0875-4212-9c0f-afefb0482146")) IDXSync : public IUnknown { public: // sync to producer, wait on consumer. virtual HRESULT producerReady(UINT subresource) = 0; virtual HRESULT consumerWait(IUnknown* pCtxOrCmdQ, UINT subresource, DWORD cpuWaitMs = 0) = 0;

// sync to consumer, wait on producer. virtual HRESULT consumerRelease(IUnknown* pCtxOrCmdQ, UINT subresource) = 0; virtual HRESULT producerWait(UINT subresource, DWORD cpuWaitMs = 0) = 0; };

ComPtr<IDXSync> s;
UINT size = sizeof(IDXSync*);
// res is ID3D11Resource
if (SUCCEEDED(res->GetPrivateData(__uuidof(IDXSync), &size, s.ReleaseAndGetAddressOf()))) {
    s->consumerWait(immediateContext, subresource/* usually 0*/);
}

D3D12

D3D12RenderAPI ra{};
ra.cmdQueue = ;
ra.rt = ;
player.setRenderAPI(&ra, this);

Vulkan

VulkanRenderAPI ra{};
ra.device = ;
ra.phy_device = ;
ra.opaque = this;
ra.rt = ;
ra.renderTargetInfo = [](void* opaque, int* w, int* h, VkFormat* fmt, VkImageLayout* layout) {
    return 1;
};
ra.currentCommandBuffer = [](void* opaque) -> VkCommandBuffer{};
player.setRenderAPI(&ra, this);

Metal

MetalRenderAPI ra{};
ra.texture = ;
ra.device = ;
ra.cmdQueue = ;
player.setRenderAPI(&ra, this);

Examples:

Render in a Foreign Render Pass

In this mode, render pass and render target are setup in user code. It's named "Foreign Context". Usually you have to render a frame in a correct context in the rendering thread of the gui toolkit or app. For example, call Player.renderVideo() in QOpenGLWidget::paintGL() or similar events for Qt Apps, in GLSurfaceView.RendereronDrawFrame(GL10 gl) for android apps. The callback of Player.setRenderCallback(callback) will be called if a new frame is ready to display, you can notify your gui toolkit or app in this callback.

You need to call only

OpenGL

No need to call player.setRenderAPI(), we can obtain the implicit global command queue(context)

D3D11

To use a RenderAPI other than OpenGL, a concrete RenderAPI subclass object with some valid members is required.

D3D11RenderAPI ra{};
ra.context = immediateContext; // immediateContext provied by app
player.setRenderAPI(&ra);

D3D12

D3D12RenderAPI ra{};
ra.cmdQueue = ...;
ra.colorFormat = ...;
ra.opaque = this;
ra.currentCommandList = [](const void* opaque){ ... };
player.setRenderAPI(&ra, this);

Vulkan

VulkanRenderAPI ra{};
ra.device = ;
ra.phy_device = ;
ra.graphics_family = ;
ra.graphics_queue = ; // optional but recommended
ra.opaque = this;
ra.render_pass = vkrpnat->renderPass;
ra.renderTargetInfo = [](void* opaque, int* w, int* h, VkFormat* fmt, VkImageLayout* layout) {
    *w = ;
    *h = ;
    *fmt = VK_FORMAT_R8G8B8A8_UNORM; // MUST match render pass
    *layout = VK_IMAGE_LAYOUT_SHADER_READ_ONLY_OPTIMAL;
    return 2;
};
ra.currentCommandBuffer = [](void* opaque) {};
m_player->setRenderAPI(&ra, this);

Metal

MetalRenderAPI ra{};
ra.opaque = this;
ra.currentCommand = [](const void** enc, const void** cmdBuf, const void *opaque) {
    *enc = ;
    *cmdBuf = ;
};
ra.device = ;
ra.cmdQueue = ;
ra.colorFormat =;        // MUST match render pass
ra.depthStencilFormat =; // MUST match render pass
player.setRenderAPI(&ra, this);

Examples:

Render On a Platform Surface

MDK use another project UGS to create and drive a rendering loop for a platform native surface. Call updateNativeSurface() with such a surface value when surface changes, or a null value to create internally. A null value also works for X11 Window or wl_egl_window* or gbm_surface* on linux, EGL_DISPMANX_WINDOW_T* on raspberry pi(legacy driver).

No need to call setRenderCallback() and renderVideo().

To use a RenderAPI other than OpenGL, a concrete RenderAPI with default initialized members is enough. For example a D3D11RenderAPI requires context and rtv member can be null. But you can also set RenderAPI Render Context Creation Options fields(see RenderAPI.h comments). And setRenderAPI() MUST be called before the first Player.updateNativeSurface()

D3D11RenderAPI ra;
player.setRenderAPI(&ra);
// ...
player.updateNativeSurface(....);

Examples:

Render On a Surface Created Internally

Use Player.createSurface() to create variant kinds of surface supported by current OS.

To use a RenderAPI other than OpenGL, a concrete RenderAPI with default initialized members MUST be set via setRenderAPI() before createSurface()

examples: