The WebM Project | WebM Container Guidelines (original) (raw)

Last modified: 2023-10-16

Show Contents

• Overview
• Goals
  • Short Term
  • Long Term
• Naming
• HTML5 Video Type Parameters
  • Video Codec
  • Audio Codecs
  • canPlayType function
• Smart Client
• WebM Guidelines
  • Demuxer and Muxer Guidelines
  • Muxer Guidelines
    * VP8/VP9 Alternate Reference Frames
    * Encode Example
    * Decode Example
  • Demuxer Guidelines
  • WebVTT Guidelines
    * Storing WebVTT data in a WebM Track
    * Storing WebVTT cue in a WebM Block
    * General WebVTT Notes
    * WebVTT Chapter cues
• Implementation Details
  • EBML Basics
  • Global Elements (Used throughout the format)
  • Segment
  • Meta Seek Information
  • Segment Information
  • Cluster
  • Track
  • Colour
    * Location of the Colour Element in an MKV File
    * Location of the Colour Element in a Stream
  • Cueing Data
• Attachment
• Chapters
• Tagging
  • Target Types
  • VP9 Codec Feature Metadata (CodecPrivate)
    * Codec Feature Format
    * ID Byte Format
    * Length Format
    * VP9 Feature: Profile
    * VP9 Feature: Level
    * VP9 Feature: Bit Depth
    * VP9 Feature: Chroma Subsampling
  • Official Tags

Overview

WebM is a digital multimedia container file format promoted by the open-source WebM Project. It comprises a subset of the Matroska multimedia container format.

Goals

Short Term

• Select a container format that's suitable for VP8/VP9 and ideal for the open web.
• Make it easy for web content providers to create and distribute VP8/VP9 videos.

Long Term

• Foster the popularity of our open format so video can be enjoyed everywhere, with no effort by users.

Naming

| | | | | ----------------------- | -------------------- | | Container Format Name | WebM | | Filename Extension | .webm | | MIME-type | video/webm | | Audio-only MIME-type | audio/webm | | Uniform Type Identifier | org.webmproject.webm | | Video Codec Name | VP8 or VP9 | | Audio Codec Name | Vorbis or Opus |

HTML5 Video Type Parameters

Video Codec

• VP8: vp8.X codec is vp8, bitstream is version X.
  • VP8 currently only has bitstream version 0.
  • This will also match to a FourCC of VP8X for applications that need a FourCC.
• VP9: vp9.X codec is vp9, bitstream is version X.
  • VP9 currently only has bitstream version 0.
  • This will also match to a FourCC of VP9X for applications that need a FourCC.

The above codec strings will eventually be deprecated in favor of the format designed for use with MP4 containers .

Audio Codecs

• Vorbis
• Opus
• Audio-only files SHOULD have a mime of "audio/webm"

canPlayType function

• canPlayType('video/webm') SHOULD return maybe
• canPlayType('audio/webm') SHOULD return maybe
• canPlayType('video/webm; codecs="vp8, vorbis"') SHOULD return probably
• canPlayType('video/webm; codecs="vp8, opus"') SHOULD return probably
• canPlayType('video/webm; codecs="vp9, vorbis"') SHOULD return probably
• canPlayType('video/webm; codecs="vp9, opus"') SHOULD return probably
• canPlayType('audio/webm; codecs="vorbis"') SHOULD return probably
• canPlayType('audio/webm; codecs="opus"') SHOULD return probably

Note that, currently, the attribute string value "vp8" may also be expressed as "vp8.0", and "vp9" may be expressed as "vp9.0".

Smart Client

One of the major goals is to allow content creators to have advanced playback capabilities, such as fast seeking and fast start using only an HTTP server. To achieve this, the WebM file format guidelines below should be followed when creating content.

WebM Guidelines

These guidelines are currently for file streaming over an HTTP connection, and indicate the areas where WebM is strict, relative to the more permissive Matroska specification .

Demuxer and Muxer Guidelines

• DocType element SHOULD be "webm".
• Video codec SHOULD be VP8 or VP9.
  • The Codec ID SHOULD be "V_VP8" or "V_VP9".
  • There is no CodecPrivate data for VP8. For VP9, CodecPrivate SHOULD contain a list of specific VP9 codec features (Level, Profile, Bit Depth and Chroma Subsampling values) as described in VP9 Codec Feature Format .
• Audio codec SHOULD be Vorbis or Opus.
  • Matroska CodecID specification
  • The Codec ID SHOULD be "A_VORBIS" or "A_OPUS".
• Initial WebM release does not support subtitles.
  • WHATWG / W3C RFC will release guidance on subtitles and other overlays in HTML5 <video> in the near future. WebM intends to follow that guidance. Ref: WebVTT .
• DocReadTypeVersion SHOULD follow the Matroska specification.
  • Example: Files with v2 elements should have a DocReadTypeVersion of 2.

Muxer Guidelines

Muxers should treat all guidelines marked SHOULD in this section as MUST. This will foster consistency across WebM files in the real world.

• WebM SHOULD contain the SeekHead element.
  • Reason: Allows the client to know if the file contains a Cues element.
• WebM files SHOULD include a keyframe-only Cues element.
  • The Cues element SHOULD contain only video key frames, to decrease the size of the file header.
  • It is recommended that the Cues element be before any clusters, so that the client can seek to a point in the data that has not yet been downloaded in a single seek operation. Ref: a tool that will put the Cues at the front.
• All absolute (block + cluster) timecodes MUST be monotonically increasing.
  • All timecodes are associated with the start time of the block.
• The TimecodeScale element SHOULD be set to a default of 1.000.000 nanoseconds.
  • Reason: Allows every cluster to have blocks with positive values up to 32.767 seconds.
• Key frames SHOULD be placed at the beginning of clusters.
  • Having key frames at the beginning of clusters should make seeking faster and easier for the client.
• Audio blocks that contain the video key frame's timecode SHOULD be in the same cluster as the video key frame block.
• Audio blocks that have same absolute timecode as video blocks SHOULD be written before the video blocks.
• WebM files MUST only support pixels for the DisplayUnit element.

VP8/VP9 Alternate Reference Frames

When enabled, the VP8 and VP9 encoders will at their discretion inject a new frame -- an alternate reference (AR) frame -- into the output, prior to the frame that depends on it. There will be at MOST 1 frame added between I/P-frames. The dependent frame (D) will always be a P-frame. The AR will be marked with the invisible flag by the codec SDK. This frame MUST be decoded before D, but will produce no output on its own.

The encoder will currently set the AR's timestamp as close as possible to the previous frame while attempting to provide a timestamp that is strictly increasing. In cases where the time base given to the encoder at configure time is not granular enough to allow for this, the AR will share the same timestamp as D, but SHOULD be treated as having no duration.

Ideally the AR's timestamp should be as close as possible to frame D-1 to allow the decoder as much time as possible to decode AR before needing to display D.

Encode Example

| | | | | | | ------ | --- | -- | --- | | Input | F 0 | | F 1 | | Output | I/P | AR | D | | PTS | 0 | 1 | 2 |

Decode Example

| | | | | | | ------ | --- | -- | --- | | Input | I/P | AR | D | | Output | F 0 | | F 1 | | PTS | 0 | | 2 |

Demuxer Guidelines

• The demuxer MUST only open webm DocType files.
• Once the demuxer deems the header and metadata of the file to be valid for a WebM file and the player starts playing the file, the demuxer MUST do all it can to parse the file, so playback can occur as correctly as possible.
• Seeking will be disabled if the webm file does not have a key frame Cues element.
  • The project is considering support for seeking without a Cues element.
• If the Cues element is not at the beginning of the file its retrieval should be deferred to allow playback to start as soon as possible.

WebVTT Guidelines

Storing WebVTT data in a WebM Track

The contents of the WebVTT file is stored as its own WebM track. The information that would appear as attributes of the HTML5 track tag can be embedded in WebM Track element as follows:

• The TrackType sub-element value is 0x11 for WebVTT SUBTITLES and CAPTIONS, and 0x21 for WebVTT DESCRIPTIONS, and METADATA.
• The label attribute is stored as the Name sub-element.
• The srclang attribute is stored as the Language sub-element.

Per the convention (see the Matroska Codec Specifications ) used for flavors of a particular video or audio codec, the CodecID for a WebVTT track is "D_WEBVTT/ kind ", where kind is one of SUBTITLES, CAPTIONS, DESCRIPTIONS, or METADATA.

Storing WebVTT cue in a WebM Block

The WebVTT cues are stored as the data portion of Block elements in the track, per the formatting described below. All WebVTT data stored within a WebM Block MUST be encoded as UTF-8. The timestamp of the WebM block and its duration are synthesized from the start and end times specified on the timestamp of the WebVTT cue. A BlockGroup element (not a SimpleBlock) MUST be used to contain the Block element, in order to also use a BlockDuration element, which is necessary to losslessly encode the original timestamp of the WebVTT cue.

If the WebVTT cue includes a WebVTT cue identifier then the WebVTT cue identifier is written to the WebM Block followed by a WebVTT line terminator . If the WebVTT cue does not have a WebVTT cue identifier then a WebVTT line terminator is written to the WebM Block. The empty line is used to distinguish that there was no WebVTT cue identifier in the original WebVTT cue.

The WebVTT cue timings is not written to the WebM Block. The start and end time of the WebVTT cue is synthesized from the start time and duration of the WebM Block.

If the WebVTT cue includes a WebVTT cue settings then the WebVTT cue settings is written to the WebM Block followed by a WebVTT line terminator. If the WebVTT cue does not have a WebVTT cue settings then a WebVTT line terminator is written to the WebM Block.

The cue payload is then written to the WebM Block.

General WebVTT Notes

Note that no WebVTT data is stored in the CodecPrivate element of the WebM Track header. All WebVTT cues are stored as Block elements for the track.

The timestamps for WebVTT cues can overlap in time. This is how roll-up captions work: multiple cues are rendered simultaneously, and when the top cue expires, the other cues move up and a new cue appears at the bottom. The WebM block timestamps must therefore be allowed to be monotonically increasing (a requirement already needed for the WebM container to support VP8/VP9 alt-ref frames), and the duration for a block MUST be allowed to overlap the start time of the next block.

WebVTT Chapter cues

WebVTT chapter cues are used for navigation and so they are handled differently, because they must all be together and immediately available. For this reason, WebVTT chapter cues should not be embedded the same as for timed cues (a representation that would vitiate their use for navigation); instead they should be converted to Matroska chapters (see the Matroska Chapter Specifications ) and embedded that way. Matroska chapters are a superset of WebVTT chapter cues and therefore the conversion is lossless.

Implementation Details

At initial release, WebM supports a subset of the Matroska specification. Support for additional Matroska functionality will be under consideration as the project matures.

Following is a more detailed description of the currently supported elements, and the features still being evaluated.

EBML Basics

Global Elements (Used throughout the format)

Segment

Meta Seek Information

Segment Information

Cluster

Track

Colour

See illustration and further description below.

Location of the Colour Element in an MKV File

The figure below illustrates the position of the Colour element (and its children) in the structure of an MKV file. This element shall always appear within a Video element that is a child of a TrackEntry element. A TrackEntry element is always encapsulated in a Tracks element, which is a first-level element. The Tracks element is typically located after the SeekHead and Info structures, and before any Cluster data, as shown below.

Note that SeekHead and Info elements are first-level elements, thus may be ordered arbitrarily in an MKV file. Such elements shall appear only once per video stream in any MKV file.

EBML
Segment
├── SeekHead
│   ├── Seek
│   ├── Seek
│   ├── Seek
│   └── Seek
├── Void
├── Info
├── Tracks
│   └── TrackEntry
│       └── Video
│           └── Colour                      <===
│               ├── MasteringMetadata       <===
│               ├── MatrixCoefficients      <===
│               ├── ...                     <===
├── Cues
│   ├── CuePoint
│   │   └── CueTrackPositions
│   ├── CuePoint
│   │   └── CueTrackPositions
│   ├── CuePoint
│   │   └── CueTrackPositions
...
├── Cluster
│   ├── SimpleBlock
│   ├── SimpleBlock
│   ├── SimpleBlock
│   ├── SimpleBlock
│   └── SimpleBlock
├── Cluster
│   ├── SimpleBlock
│   ├── SimpleBlock
│   ├── SimpleBlock
│   ├── SimpleBlock
│   └── SimpleBlock

Location of the Colour Element in a Stream

In a DASH-based streaming application, the video decoder will receive the Colour MKV metadata elements

• in the first chunk of the video bitstream served to the playback device, at the beginning of a playback.
• in the first chunk of the video bitstream served to the playback device, at the beginning of a change in resolution.

Cueing Data

Attachment

Chapters

Tagging

• The Tags element SHOULD be placed at the end of the file to allow for trivial updates.
• TagName SHOULD appear before the tag data, i.e., TagString , TagBinary .
• A SimpleTag SHOULD NOT contain other SimpleTag s.
• When concatenating multiple files, new Segments should be used to avoid merging tags common to each file.

Target Types

See the Matroska Tag Specifications .

VP9 Codec Feature Metadata (CodecPrivate)

VP9 Levels, Profiles, Bit Depth and Chroma Subsampling values are specific to VP9, so the data is stored in the CodecPrivate element as a list of codec features.

Codec Feature Format

 0                   1                   2                   3
 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0 1
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+
|    ID Byte    |     Length    |                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+                               |
|                                                               |
:               Bytes 1..Length of Codec Feature                :
|                                                               |
+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+-+

Each codec feature is defined by the binary format of ID byte, length, and data.

ID Byte Format

ID byte will be an unsigned byte.

 0 1 2 3 4 5 6 7
+-+-+-+-+-+-+-+-+
|X|    ID       |
+-+-+-+-+-+-+-+-+

X bit

Reserved for future expansion (currently MUST be set to 0 ).

ID bits

These bits signify the identification bits of the VP9 codec feature.

Length Format

Length will be stored as an unsigned 8-bit integer.

VP9 Feature: Profile

ID = 1 , Length = 1

The data is an unsigned 8-bit integer that specifies the number of decoded bits per channel. The VP9 profile in the bitstream is canonical and should match the VP9 profile set in the container. Possible values are from 0-3 . All other values are undefined. See the draft VP9 spec for value definitions.

VP9 Feature: Level

ID = 2 , Length = 1

The data is an unsigned 8-bit integer that specifies the VP9 level. Possible values are:

| | | | | -- | --------- | | 10 | Level 1 | | 11 | Level 1.1 | | 20 | Level 2 | | 21 | Level 2.1 | | 30 | Level 3 | | 31 | Level 3.1 | | 40 | Level 4 | | 41 | Level 4.1 | | 50 | Level 5 | | 51 | Level 5.1 | | 52 | Level 5.2 | | 60 | Level 6 | | 61 | Level 6.1 | | 62 | Level 6.2 |

All other values are currently undefined. See VP9 Levels for definitions.

VP9 Feature: Bit Depth

ID = 3, Length = 1

The data is an unsigned 8-bit integer that specifies the number of decoded bits per channel. Possible values are 8 , 10 , and 12 .

VP9 Feature: Chroma Subsampling

ID = 4 , Length = 1

The data is an unsigned 8-bit integer that specifies the chroma subsampling. Possible values are:

| | | | | - | ------------------------------- | | 0 | 4:2:0 vertical | | 1 | 4:2:0 colocated with luma (0,0) | | 2 | 4:2:2 | | 3 | 4:4:4 |

All other values are currently undefined.

Official Tags

Work in-progress. See the wiki .