The 'haptics' Top-level Media Type (original) (raw)

MEDIAMAN Y. K. Muthusamy Internet-Draft
Intended status: Standards Track C. Ullrich Expires: 28 January 2024 27 July 2023

               The 'haptics' Top-level Media Type
                 draft-ietf-mediaman-haptics-05

Abstract

This memo serves to register and document the 'haptics' top-level media type, under which subtypes for representation formats for haptics may be registered. This document also serves as a registration for a set of subtypes, which are representative of some existing subtypes already in use.

Status of This Memo

This Internet-Draft is submitted in full conformance with the provisions of BCP 78 and BCP 79.

Internet-Drafts are working documents of the Internet Engineering Task Force (IETF). Note that other groups may also distribute working documents as Internet-Drafts. The list of current Internet- Drafts is at https://datatracker.ietf.org/drafts/current/.

Internet-Drafts are draft documents valid for a maximum of six months and may be updated, replaced, or obsoleted by other documents at any time. It is inappropriate to use Internet-Drafts as reference material or to cite them other than as "work in progress."

This Internet-Draft will expire on 28 January 2024.

This document is subject to BCP 78 and the IETF Trust's Legal Provisions Relating to IETF Documents (https://trustee.ietf.org/ license-info) in effect on the date of publication of this document. Please review these documents carefully, as they describe your rights and restrictions with respect to this document. Code Components extracted from this document must include Revised BSD License text as described in Section 4.e of the Trust Legal Provisions and are provided without warranty as described in the Revised BSD License.

Muthusamy & Ullrich Expires 28 January 2024 [Page 1] RFC XXXX The 'haptics' Top-level Media Type July 2023

Table of Contents

Introduction . . . . . . . . . . . . . . . . . . . . . . . . 2 1.1. Terminology . . . . . . . . . . . . . . . . . . . . . . . 3
Background and Justification . . . . . . . . . . . . . . . . 3 2.1. MPEG ISOBMFF . . . . . . . . . . . . . . . . . . . . . . 3 2.2. Haptic Sub-modalities . . . . . . . . . . . . . . . . . . 4 2.3. Another Human Sense . . . . . . . . . . . . . . . . . . . 4 2.4. Commercial Uptake . . . . . . . . . . . . . . . . . . . . 4 2.5. Haptic Data Formats in Use . . . . . . . . . . . . . . . 5 2.6. Haptic Subtypes (envisioned standards) . . . . . . . . . 6 2.7. 'application' top-level type not suitable . . . . . . . . 6
Security Considerations . . . . . . . . . . . . . . . . . . . 7
IANA Considerations . . . . . . . . . . . . . . . . . . . . . 8 4.1. Definition and Encoding . . . . . . . . . . . . . . . . . 8 4.2. Registration Procedure . . . . . . . . . . . . . . . . . 8 4.3. Subtype Registrations . . . . . . . . . . . . . . . . . . 8 4.3.1. IVS Haptics Type . . . . . . . . . . . . . . . . . . 8 4.3.2. HJIF Haptics Type . . . . . . . . . . . . . . . . . . 9 4.3.3. HMPG Haptics Type . . . . . . . . . . . . . . . . . . 10
Normative References . . . . . . . . . . . . . . . . . . . . 11
Informative References . . . . . . . . . . . . . . . . . . . 12 Authors' Addresses . . . . . . . . . . . . . . . . . . . . . . . 13
Introduction

The term 'haptics' refers to the generation of touch-related sensations in a device or interface. Haptics is widely used in consumer devices in order to provide touch-based feedback to users. The most common use of haptics is in mobile devices, where it is used to provide feedback to users interacting with the touchscreen, e.g., typing on a virtual keyboard. Haptic technologies are unlike audio and visual enabling technologies in the sense that they require some form of actuation in order to create a tactile sensation. For mobile phones and game controllers, these actuators are typically small vibrating motors. For large touchscreens in vehicles, these actuators can be specialized piezoelectric materials. Haptic capabilities are found in nearly every modern smartphone and game and virtual reality controller, making these devices an ideal target for enhanced media experiences.

Internet Media Types [RFC6838] are used to label content carried over Internet protocols. This document defines a new top-level type 'haptics' according to [TOPLEVEL]. This top-level type indicates that the content specifies haptic data. Under this top-level type, different representation formats of haptics may be registered.

Muthusamy & Ullrich Expires 28 January 2024 [Page 2] RFC XXXX The 'haptics' Top-level Media Type July 2023

1.1. Terminology

The keywords MUST, MUST NOT, REQUIRED, SHALL, SHALL NOT, SHOULD,SHOULD NOT, RECOMMENDED, NOT RECOMMENDED, MAY, and OPTIONAL in this document are to be interpreted as described in BCP 14 [RFC2119] [RFC8174] when, and only when, they appear in all capitals, as shown here.

Background and Justification

Haptic signals provide an additional layer of entertainment and sensory immersion for the user, when combined with audio and video signals. Haptic tracks, in separate files, can be combined with audio/video files and played back in sync to provide an overall immersive media experience (audio, visual, tactile) for the user. More recently, haptic tracks embedded in standard file formats such as ISOBMFF (ISO Base Media File Format), enable playback of the haptic signals over one or more actuators, simultaneously with audio and video playback [ISOBMFF-IS]. Haptic signals are also part of media streams that use RTP, such as those for streaming games, XR, and wearables.

2.1. MPEG ISOBMFF

Historically, there has not been a registration of formats for haptics. However, haptics was proposed as a first-order media type (at the same level as audio and video) in ISOBMFF in April 2020. The proposal has since progressed to International Standard, and was published in January 2022 [ISOBMFF-IS]. Haptics is officially part of the ISO/IEC 14496-12 (ISOBMFF) standard, 7th Edition. Given this development, a strong case can be made for haptics to be added to the list of top-level media types recognized by the IETF.

We envision the following designations for haptics in mp4 files, once the top-level type 'haptics' is registered:

'haptics/mp4' - mp4 files with just haptic tracks and no audio or video in them (e.g., streaming games, haptics files for haptic vests, belts, gloves, etc.)
'video/mp4' - mp4 files with video, audio, and haptics (to ensure consistency with existing mp4 files with video content)
'audio/mp4' - mp4 files with audio and haptics (to ensure consistency with existing mp4 files with audio content without any video)

Muthusamy & Ullrich Expires 28 January 2024 [Page 3] RFC XXXX The 'haptics' Top-level Media Type July 2023

2.2. Haptic Sub-modalities

There are multiple sub-modalities of haptics:

Vibrotactile (touch, vibration)
Kinesthetic (force feedback)
Surface (surface friction)
Spatial, non-contact (ultrasound)
Thermal (temperature)

Therefore, designating 'haptics' as a top-level media type enables the definition of data formats pertaining to these sub-modalities in a more streamlined manner. This would not be possible if 'haptics' were to be placed under other top-level types like 'audio', 'video', or 'application'.

2.3. Another Human Sense

The top-level media type 'audio' pertains to the human sense of hearing, the top-level media type 'video' pertains to the human sense of seeing, so it only makes sense for the (equally important) human sense of touch to be represented by another top-level media type 'haptics'. Placing 'haptics' under 'audio' or 'video' is not reflective of the kinds of files or use cases that would need haptics but have nothing whatsoever to do with audio or video.

2.4. Commercial Uptake

Haptics is rapidly becoming a standard feature of consumer electronic devices. For example:

iPhone (206+ million units sold in 2020): native support for haptic encoded data
Android (1.38+ billion units sold in 2020): API support of haptic buffers
W3C (HTML vibration API [W3C-Vibration]): Optionally supported in mobile web browsers. W3C has also defined vibration extensions for gamepads [W3C-Gamepad]
Game consoles (39+ million units sold in 2019): MS Xbox, Sony PlayStation, Nintendo Switch, etc.

Muthusamy & Ullrich Expires 28 January 2024 [Page 4] RFC XXXX The 'haptics' Top-level Media Type July 2023

XR devices (9+ million units sold in 2019): OpenXR haptic API

Haptic media is expected to be commonly exchanged between these devices. Since they represent the majority of CE devices, a strong case can be made for 'haptics' as a top-level media type.

2.5. Haptic Data Formats in Use

There are multiple instances of existing haptic data formats that will live as sub-types under the proposed 'haptics' top-level media type. While these subtypes have not been registered with IANA or standardized (yet), the prevalence of these haptic data formats in a large number of devices around the world, pre-dating the standardization of haptic tracks in ISOBMFF, provides a compelling argument for 'haptics' to be designated as a top-level media type:

'ahap': The AHAP haptic data format [AHAP] is currently the standard encoding on all iOS devices + iOS connected game peripherals. The format has seen usage and adoption beyond Apple devices as well, with decoders available for Android and other XR systems.
'ogg': Google has introduced a proprietary extension to the OGG format in the latest version of Android 11. This encoding enables haptic media to be stored in OGG files.
'ivs': The IVS haptic data format [MPEG-Haptics-Encoder] is in use:
- In mobile phones from LG Electronics (specifically, the models V30, V40, and the newest V50) that are sold worldwide
- In gaming phones from ASUS (specifically, models ROG, ROG Phone II, ROG Phone 3) that are sold worldwide
'hapt': The HAPT haptic data format is currently a vendor-specific format that is in use:
- In mobile haptic advertising (for W3C devices)
- The following Japanese game developers use the HAPT format as part of Immersion's TouchSense SDK:
  
  o KLAB: https://www.klab.com/en/ (https://www.klab.com/en/)
  
  o Craft&Meister: http://www.crafts-meister.co.jp/pc/ company_en.html (http://www.crafts-meister.co.jp/pc/ company_en.html)

Muthusamy & Ullrich Expires 28 January 2024 [Page 5] RFC XXXX The 'haptics' Top-level Media Type July 2023

  -  Tencent is using the TouchSense SDK for their popular social
     media application QQ and live streaming application NOW:
     Immersion-Announces-Tencent-Licenses-TouchSense-Technology-
     Deliver
     (https://www.businesswire.com/news/home/20171026006443/en/
     Immersion-Announces-Tencent-Licenses-TouchSense%C2%AE-
     Technology-Deliver)

Given the widespread use of these subtypes, it makes sense for 'haptics' to be a top-level media type.

2.6. Haptic Subtypes (envisioned standards)

The MPEG ISOBMFF proposal included an informative annex of known haptic coding formats with proposed FourCC codes for them. These codes are not registered yet, but the plan is indeed to standardize these haptic coding formats in the near future. Once standardized, these types should also be registered as subtypes of the 'haptics' top-level media type:

'hmpg': the MPEG-I haptics streamable binary coding format described in ISO/IEC DIS 23090-31: Haptics coding [MPEG-Haptics-Coding]
'hjif': the MPEG-I haptics JSON-based interchange format described in ISO/IEC DIS 23090-31: Haptics coding [MPEG-Haptics-Coding]
IEEE P1918.1.1 vibrotactile coding standard [IEEE-P191811] being developed under the IEEE Tactile Internet initiative as part of the 5G URLL profile. Format name is yet to be finalized.
Enumerated effects haptic coding format (based on MIDI). Format name is yet to be finalized.
Audio-to-vibe haptic coding format (automatic audio to vibration conversion algorithms). Format name is yet to be finalized.

2.7. 'application' top-level type not suitable

From the above arguments, it is clear that haptics does not really belong under any other media type. To reiterate, there are three main reasons why the 'haptics' media type does not fit under the 'application' top-level type:

haptics connects to a sensory system, touch/motion, directly, and is more specific than the abstract 'application' type, and

Muthusamy & Ullrich Expires 28 January 2024 [Page 6] RFC XXXX The 'haptics' Top-level Media Type July 2023

'application' has historically been used for applications, i.e., code, which means it is viewed and treated with great care for security. 'haptics' is not code, just as 'audio' and 'video' are not code either.
haptics is a property of a media stream, it is not an application under any normal definition. As such, it should be its own type.

Security Considerations

Haptics are interpreted data structures that represent collections of different media rendering instructions intended to be decoded and rendered on target device hardware. Haptic data can be represented as collections of signal data and/or descriptive text in XML/JSON or a similar format. Signal data is typically not executed by endpoint processors and represents minimal security risk. Descriptive text is typically parsed and represented in memory using standard XML data structures. This data is utilized to construct one or more signals that are sent to the endpoint device hardware.

Because of the media/rendering nature of the data path for haptic coded data the security profile of haptic data is expected to be largely consistent with the security profile of visual and audio media data.

As with any synthesized media data (audio, video, and haptics), there is a security risk associated with execution of commands based on the descriptive encoding either through its inherent extensibility or through the insertion of arbitrary executable data in the descriptive format itself. Indeed, media rendering systems are normally implemented with a mix of user and kernel space execution since these media must ultimately make their way to a hardware system. In theory, malicious instructions present in descriptive haptic media have the potential to execute arbitrary code in kernel space, effectively bypassing system permissions structures and/or execution sandboxes.

Haptics, audio, and video media have widespread use and careful attention should be paid by operating system and device driver implementors to ensure that synthesis and rendering signal paths do not provide attack surfaces for malicious payloads.

Thermal haptic devices (that provide a sensation of heat) and kinesthetic haptic devices (that provide force feedback) could potentially injure users if the heat or force, respectively, are not properly controlled or inadvertently exceed safety levels. Implementors need to ensure that adequate measures are taken to prevent such scenarios.

Muthusamy & Ullrich Expires 28 January 2024 [Page 7] RFC XXXX The 'haptics' Top-level Media Type July 2023

These security considerations apply to the subtype registrations described in this document as well as all future haptics registrations.

IANA Considerations

This specification registers a new top-level type, 'haptics', and requests IANA to add it to the registry of top-level types specified in [TOPLEVEL], adds it as an alternative value of "Type Name" in the media types registration form [Media-Type-Registration], and registers several subtypes for it.

4.1. Definition and Encoding

'haptics' as the primary media content type indicates that the content identified by it requires a certain haptics subsystem such as low-level haptics APIs, which in turn will require hardware capabilities such as one or more actuators to render the haptics media. The 'haptics' media type does not provide any specific information about the underlying data format and how the haptics information should be interpreted -- the subtypes defined within a 'haptics' tree name the specific haptic formats. Unrecognized subtypes of 'haptics' should be treated as 'application/octet- stream'. Implementations may still pass unrecognized subtypes to the haptics subsystem and associated rendering hardware.

4.2. Registration Procedure

New haptics formats should be registered using the online form [Media-Type-Registration]. [RFC6838] should be consulted on registration procedures. In particular, the haptics specification should preferably be freely available.

Note that new subtypes may define parameters. If an implementation does not recognize a parameter sub-value in the comma- separated list, it should ignore the sub-value and continue processing the other sub-values in the list.

4.3. Subtype Registrations

In this section, the initial entries under the top-level 'haptics' media type are specified. They also serve as examples for future registrations.

4.3.1. IVS Haptics Type

Type name: haptics

Muthusamy & Ullrich Expires 28 January 2024 [Page 8] RFC XXXX The 'haptics' Top-level Media Type July 2023

Subtype name: ivs

Required parameters: N/A

Optional parameters: N/A

Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32

Interoperability considerations: The IVS format is a device- independent haptic effect coding based on the XML format. It is designed to enable interoperability between distinct physical endpoints. Not all devices may be able to render all effects present in an IVS file.

Security considerations: See Section 3 of RFC XXXX.

[Note to RFC Editor: Please replace XXXX with the number of this RFC.]

Published specification: ISO/IEC JTC 1/SC 29/WG 2 N 0072 "Encoder Input Format for MPEG Haptics" [MPEG-Haptics-Encoder].

Applications that use this media type: All applications that are able to create, edit, or display haptic media content.

Additional information:

File extension(s): Haptic file extensions used for IVS files: .ivs
Macintosh file type code(s): (no code specified)
Macintosh Universal Type Identifier code: N/A
Fragment Identifier: N/A
Deprecated Alias: N/A

Person & email address to contact for further information: Yeshwant Muthusamy(yeshwant@yeshvik.com)

Change controller: ISO/IEC JTC1/SC 29/WG 7 (MPEG 3D Graphics and Haptic Coding)

4.3.2. HJIF Haptics Type

Type name: haptics

Subtype name: hjif

Muthusamy & Ullrich Expires 28 January 2024 [Page 9] RFC XXXX The 'haptics' Top-level Media Type July 2023

Required parameters: N/A

Optional parameters: N/A

Encoding considerations: 8bit if UTF-8; binary if UTF-16 or UTF-32

Interoperability considerations: The HJIF format is a human-readable haptic effect coding based on the JSON format. It is designed as an interchange format for temporal and spatial haptic effects. The haptic effects may target specific parts of the human body and may be associated with a reference device description allowing haptic rendering software to adapt the effects to available hardware.

Security considerations: See Section 3 of RFC XXXX.

[Note to RFC Editor: Please replace XXXX with the number of this RFC.]

Published specification: ISO/IEC DIS 23090-31: Haptics coding [MPEG-Haptics-Coding].

Applications that use this media type: All applications that are able to create, edit, or display haptic media content.

Additional information:

File extension(s): Haptic file extensions used for HJIF files: .hjif
Macintosh file type code(s): (no code specified)
Macintosh Universal Type Identifier code: N/A
Fragment Identifier: N/A
Deprecated Alias: N/A

Person & email address to contact for further information: Yeshwant Muthusamy(yeshwant@yeshvik.com)

Change controller: ISO/IEC JTC1/SC 29/WG 7 (MPEG 3D Graphics and Haptic Coding)

4.3.3. HMPG Haptics Type

Type name: haptics

Subtype name: hmpg

Muthusamy & Ullrich Expires 28 January 2024 [Page 10] RFC XXXX The 'haptics' Top-level Media Type July 2023

Required parameters: N/A

Optional parameters: N/A

Encoding considerations: binary

Interoperability considerations: The HMPG format is a streamable binary haptic effect coding. It is designed to enable efficient coding of temporal and spatial haptic effects. The haptic effects may target specific parts of the human body and may be associated with a reference device description allowing haptic rendering software to adapt the effects to available hardware.

Security considerations: See Section 3 of RFC XXXX.

[Note to RFC Editor: Please replace XXXX with the number of this RFC.]

Published specification: ISO/IEC DIS 23090-31: Haptics coding [MPEG-Haptics-Coding].

Applications that use this media type: All applications that are able to create, edit, or display haptic media content.

Additional information:

File extension(s): Haptic file extensions used for HMPG files: .hmpg
Macintosh file type code(s): (no code specified)
Macintosh Universal Type Identifier code: N/A
Fragment Identifier: N/A
Deprecated Alias: N/A

Person & email address to contact for further information: Yeshwant Muthusamy(yeshwant@yeshvik.com)

Change controller: ISO/IEC JTC1/SC 29/WG 7 (MPEG 3D Graphics and Haptic Coding)

Normative References

Muthusamy & Ullrich Expires 28 January 2024 [Page 11] RFC XXXX The 'haptics' Top-level Media Type July 2023

[RFC2119] Bradner, S., "Key words for use in RFCs to Indicate Requirement Levels", BCP 14, RFC 2119, DOI 10.17487/RFC2119, March 1997, https://www.rfc-editor.org/info/rfc2119.

[RFC8174] Leiba, B., "Ambiguity of Uppercase vs Lowercase in RFC 2119 Key Words", BCP 14, RFC 8174, DOI 10.17487/RFC8174, May 2017, https://www.rfc-editor.org/info/rfc8174.

[RFC6838] Freed, N., Klensin, J., and T. Hansen, "Media Type Specifications and Registration Procedures", BCP 13, RFC 6838, DOI 10.17487/RFC6838, January 2013, https://www.rfc-editor.org/info/rfc6838.

[TOPLEVEL] Dürst, M. J., "Guidelines for the Definition of New Top- Level Media Types", Work in Progress, Internet-Draft, draft-ietf-mediaman-toplevel-03, 26 March 2023, <https://datatracker.ietf.org/doc/draft-ietf-mediaman- toplevel/03/>.

Informative References

[ISOBMFF-IS] "ISO/IEC 14496-12 (7th Edition) Information technology — Coding of audio-visual objects — Part 12: ISO base media file format", https://www.iso.org/standard/83102.html.

[MPEG-Haptics-Encoder] "Encoder Input Format for MPEG Haptics", <https://www.mpegstandards.org/standards/ Explorations/40/>.

[AHAP] "Apple Haptic Audio Pattern", <https://developer.apple.com/documentation/corehaptics/ representing_haptic_patterns_in_ahap_files>.

[MPEG-Haptics-Coding] "ISO/IEC DIS 23090-31 Information Technology — Coded representation of immersive media — Part 31: Haptics coding", https://www.iso.org/standard/86122.html.

[W3C-Vibration] "W3C Vibration API (Second Edition)", https://www.w3.org/TR/vibration/.

[W3C-Gamepad] "W3C Gamepad Extensions", https://w3c.github.io/gamepad/extensions.html.

Muthusamy & Ullrich Expires 28 January 2024 [Page 12] RFC XXXX The 'haptics' Top-level Media Type July 2023

[IEEE-P191811] "P1918.1.1 - Haptic Codecs for the Tactile Internet", https://standards.ieee.org/project/1918_1_1.html.

[Media-Type-Registration] "IANA, Application for a Media Type", http://www.iana.org/form/media-types.

Authors' Addresses

Yeshwant K. Muthusamy 600 Longwood Drive Allen, TX 75013 United States of America Phone: +1 469-854-9836 Email: yeshwant@yeshvik.com

Chris Ullrich 311 Court Ave Ventura, CA 93003 United States of America Phone: +1 805-320-0774 Email: chrisullrich@gmail.com

Muthusamy & Ullrich Expires 28 January 2024 [Page 13]