Region Capture (original) (raw)

Abstract

This document introduces an API for cropping a video track derived from display-capture of the current tab.

This First Public Working Draft represents the direction the Web Real-Time Communications Working Group intends to explore to solve the use case of partial capture of browsing contexts. The Working Group is particularly interested in feedback on how well this direction matches the said use case from potential adopters of the API.

Definitions

This document uses the definition of the following concepts from [[SCREEN-CAPTURE]]:display-surface and browser [=display-surface=].

Use Cases

Generic Use-Case

Complex applications often comprise multiple [=documents=] in distinct [=iframes=], all displayed within the same [=browsing context=]. Consider such an application. Assume one of these documents, CAPTURING-DOC uses {{MediaDevices/getDisplayMedia()}} orgetViewportMedia to capture the entire current [=browsing context=]. If this document then wishes to crop the video track to the coordinates of some sub-sectionCAPTURE-TARGET of a collaborating document CAPTURED-DOC, how canCAPTURING-DOC do so performantly and reliably? Recall especially that changes in layout due to scrolling, zooming or window resizing present additional challenges.

Practical Use-Case

Consider a combo-application consisting of two major parts hosted in different iframes within the same tab - a video-conferencing application and a productivity-suite application. Assume the video-conferencing uses existing/upcoming APIs such as {{MediaDevices/getDisplayMedia()}} and/or getViewportMedia and captures the entire tab. Now it needs to crop away everything other than a particular section of the productivity-suite. It needs to crop away its own video-conferencing content, any speaker notes and other private and/or irrelevant content in the productivity-suite, before transmitting the resulting cropped video remotely.

Moreover, consider that it is likely that the two collaborating applications are cross-origin from each other. They can post messages, but all communication is asynchronous, and it's easier and more performant if information is transmitted sparingly between them. That precludes solutions involving posting of entire frames, as well as solutions which are too slow to react to changes in layout (e.g. scrolling, zooming and window-size changes).

It is worthwhile to note that most applications would likey prefer to usegetViewportMedia in such scenarios. However, as of this writing,getViewportMedia is still unspecified and unimplemented. It will have non-trivial requirements whose adoption will take some time and effort. As such, many applications will likely use a combination of {{MediaDevices/getDisplayMedia()}} and Region Capture for some time to come.

Solution Overview

The region-capture mechanism comprises two parts:

[=CropTarget production=]: A mechanism for tagging an {{Element}} as a potential target for the [=cropping mechanism=].
[=Cropping mechanism=]: A mechanism for instructing the user agent to start cropping a video track to the contours of a previously [=tagging|tagged=] {{Element}}, or to stop such cropping and revert a track to its [=uncropped=] state.

We define two crop-states for video tracks - cropped anduncropped. Tracks start out [=uncropped=], and may turn to [=cropped=] when {{BrowserCaptureMediaStreamTrack/cropTo}} is successfully called on them.

CropTarget Production

CropTarget Motivation

The [=cropping mechanism=] presented in this document ({{BrowserCaptureMediaStreamTrack/cropTo}}) relies on [=Crop-session Target=] rather than on direct node references. This serves a dual purpose.

It allows cropping by one document to coordinates specified in another document.
[=Tagging=] an {{Element}} as a potential crop-target allows the user agent to avoid unnecessary work on all other elements, like the calculation of bounding boxes and sending such coordinates cross-process.

CropTarget Definition

CropTarget is an intentionally empty, opaque identifier. Its purpose is to be handed to {{BrowserCaptureMediaStreamTrack/cropTo}} as input.

      [Exposed=(Window,Worker), Serializable]
      interface CropTarget {
        [Exposed=Window, SecureContext] static Promise<CropTarget> fromElement(Element element);
      };

There is no consensus yet on whether {{CropTarget/fromElement}} should be exposed beyond secure contexts.

fromElement()

Calling {{CropTarget/fromElement}} with an {{Element}} of a supported type associates that {{Element}} with a {{CropTarget}}. This {{CropTarget}} may be used as input to {{BrowserCaptureMediaStreamTrack/cropTo}}. We define avalid CropTarget as one returned by a call to {{CropTarget.fromElement()}} in a document that is stillactive.

When {{CropTarget/fromElement}} is called with a given element, the user agent [=create a CropTarget|creates a CropTarget=] with element as input. The user agent MUST return a {{Promise}} p. The user agent MUST resolvep only after it has finished all the necessary internal propagation of state associated with the new {{CropTarget}}, at which point the user agent MUST be ready to receive the new {{CropTarget}} as a valid parameter to {{BrowserCaptureMediaStreamTrack/cropTo}}.

When cloning an {{Element}} on which {{CropTarget/fromElement}} was previously called, the clone is not associated with any {{CropTarget}}. If {{CropTarget/fromElement}} is later called on the clone, a new {{CropTarget}} will be assigned to it.

There is no consensus yet on whether producing a {{CropTarget}} should be done by invoking an asynchronous method like {{CropTarget.fromElement()}}, or a {{CropTarget}} constructor that accepts an {{Element}} as input. This is further discussed onissue #17.

To create a CropTarget with element as input, run the following steps:

Let cropTarget be a new object of type {{CropTarget}}.
Let weakRef be a weak reference to element.
[=Create a CropTarget|Create=]cropTarget.[[\Element]] initialized to weakRef.
cropTarget keeps a weak reference to the element it represents. In other words, cropTarget will not prevent garbage collection of its element.

{{CropTarget}} objects are serializable. The [=serialization steps=], givenvalue, serialized, and a boolean forStorage, are:

If forStorage is true, throw with new {{DOMException}} object whose {{DOMException/name}} attribute has the value {{"DataCloneError"}}.
Set serialized.[[\CropTargetElement]] tovalue.{{CropTarget/[[Element]]}}.

The [=deserialization steps=], given serialized and value are:

Set value.{{CropTarget/[[Element]]}} toserialized.[[\CropTargetElement]].

Cropping Mechanism

BrowserCaptureMediaStreamTrack

Recall that, as per [[SCREEN-CAPTURE]], when {{MediaDevices/getDisplayMedia()}} is called, it returns a {{Promise}}<{{MediaStream}}>, and that this {{MediaStream}} contains exactly one video track, whose type is {{MediaStreamTrack}}.

We specify that if the user chooses to capture a [=browser=] [=display-surface=], the user agent MUST instantiate the video track as either {{MediaStreamTrack}}, or as some sub-class of {{MediaStreamTrack}}, and that {{BrowserCaptureMediaStreamTrack/cropTo}} MUST be exposed on this track. For simplicity's sake, this document assumes that a subclass called {{BrowserCaptureMediaStreamTrack}} is used by the user agent.

The track MUST be initially [=uncropped=].

      [Exposed = Window]
      interface BrowserCaptureMediaStreamTrack : MediaStreamTrack {
        Promise<undefined> cropTo(CropTarget? cropTarget);
        BrowserCaptureMediaStreamTrack clone();
      };

cropTo()

Calls to this method instruct the user agent to start/stop cropping a video track to the bounding client rectangle of cropTarget.{{CropTarget/[[Element]]}}. Since the track is restricted to the visible viewport of the [=display-surface=], the captured area will be the intersection of the visible viewport and the element bounding client rectangle. Whenever {{BrowserCaptureMediaStreamTrack/cropTo}} is invoked, the user agent MUST execute the following algorithm:

If cropTarget is neither a [=valid CropTarget=] nor null, the user agent MUST return a {{Promise}} [=rejected=] with an {{UnknownError}}.
Let p be a new {{Promise}}.
Run the following steps in parallel:
1. If cropTarget is neither {{undefined}} nor a [=valid CropTarget=], reject p with a {{NotAllowedError}} and abort these steps.
2. If cropTarget is either {{undefined}} or a [=valid CropTarget=], the user agent MUST update [=this=] video track's [=crop-state=] according tocropTarget:
  - If cropTarget is set to {{undefined}}, the user agent MUST stop cropping. [=This=] video track reverts to the [=uncropped=] state.
  - If cropTarget is a [=valid CropTarget=], the user agent MUST start cropping [=this=] video track to the contours of the element referenced by this {{CropTarget}}. This means that for each new frame produced on the track, the user agent calculates the bounding box of the pixels belonging to the element, and crops the frame to the coordinates of this bounding box.
3. Call the track's state before this method invocation PRE-STATE, and after this method invocation POST-STATE. The user agent MUST resolve p when it is guaranteed that no more frames cropped (or uncropped) according to PRE-STATE will be delivered to the application, and that any additional frames delivered to the application will therefore be cropped (or uncropped) according to eitherPOST-STATE or a later state.
  The timing of the cropTo promise resolution and the timing of the actual cropping of video frames is observable to JavaScript through MediaStreamTrack transforms. It is expected that the first newly cropped video frame will be enqueued on the MediaStreamTrack ReadableStream just after the cropTo promise is resolved.
Return p.

clone()

When a {{BrowserCaptureMediaStreamTrack}} is cloned, the user agent MUST produce a track which is initially [=uncropped=], regardless of the [=crop-state=] of the original track.

Crop-Session Lifetime

Crop-Session Definitions

We define an {{Element}} for which a {{CropTarget}} was produced (through a call to {{CropTarget/fromElement}}) as a potential crop-target.

We define a [=potential crop-target=] which is targeted by a successful call to {{BrowserCaptureMediaStreamTrack/cropTo}} as the crop-session target.

Consider a frame produced on a [=cropped=] video track. The user agent calculates the intersection of (i) the [=top-level browsing context=]'s viewport and (ii) the bounding box of all pixels belonging to the [=crop-session target=]. This intersection is defined as the crop-session target's coordinates for that frame.

Crop-Session Edge Cases

Consider a video track VT [=cropped=] to a given [=crop-session target=]TARGET. We define the behavior of the crop-session of the VT in the face of changes undergone by TARGET.

Empty Crop-Target

We define as an empty crop-session target the case where a [=crop-session target=] is attached to the DOM, yet consists of zero pixels which are drawn inside of the [=top-level browsing context|top-level browsing context's=] viewport.

Some examples of when this could happen include:

The [=crop-session target=] consists of zero pixels.
The [=browsing context|browsing context's=] viewport has been scrolled and the [=crop-session target=] now lies outside of the viewport.

The user agent MUST NOT produce new frames on tracks with an [=empty crop-session target=]. For such a track, the user agent MUST resume the production of frames if the track either become [=uncropped=], or if its [=crop-session target=] stops being [=empty crop-session target|empty=].

Disconnected Crop-Session Target

We define as disconnected crop-session target a [=crop-session target=] that had been detached from the DOM.

The difference between an [=empty crop-session target=] and a [=disconnected crop-session target=], is that a [=disconnected crop-session target|disconnected=] one may becomeunreachable, in which case it would not produce any new frames. Nevertheless, the user agent MUST treat a [=disconnected crop-session target=] the same way it treats an [=empty crop-session target=]. The application may call {{BrowserCaptureMediaStreamTrack/cropTo}} on the track with either {{undefined}} or a new {{CropTarget}}, thereby allowing the production of frames on the track to be resumed.

Sample Code

Code in the capture-target:

      const mainContentArea = navigator.getElementById('mainContentArea');
      const cropTarget = await CropTarget.fromElement(mainContentArea);
      sendCropTarget(cropTarget);

      function sendCropTarget(cropTarget) {
        // Can send the crop-target to another document in this tab
        // using postMessage() or using any other means.
        // Possibly there is no other document, and this is just consumed locally.
      }

Code in the capturing-document:

      async function startCroppedCapture(cropTarget) {
        const stream = await navigator.mediaDevices.getDisplayMedia();
        const [track] = stream.getVideoTracks();
        if (!!track.cropTo) {
          handleError(stream);
          return;
        }
        await track.cropTo(cropTarget);
        transmitVideoRemotely(track);
      }