Hand and Body Pose Detection in iOS 14 Will Provide New Ways to Interact With Your iPhone Without Touching the Display (original) (raw)
Starting in iOS 14 and macOS Big Sur, developers will be able to add the capability to detect human body and hand poses in photos and videos to their apps using Apple's updated Vision framework, as explained in this WWDC 2020 session.
This functionality will allow apps to analyze the poses, movements, and gestures of people, enabling a wide variety of potential features. Apple provides some examples, including a fitness app that could automatically track the exercise a user performs, a safety-training app that could help employees use correct ergonomics, and a media-editing app that could find photos or videos based on pose similarity.
Hand pose detection in particular promises to deliver a new form of interaction with apps. Apple's demonstration showed a person holding their thumb and index finger together and then being able to draw in an iPhone app without touching the display.
Additionally, apps could use the framework to overlay emoji or graphics on a user's hands that mirror the specific gesture, such as a peace sign.
Another example is a camera app that automatically triggers photo capture when it detects the user making a specific hand gesture in the air.
The framework is capable of detecting multiple hands or bodies in one scene, but the algorithms might not work as well with people who are wearing gloves, bent over, facing upside down, or wearing overflowing or robe-like clothing. The algorithm can also experience difficulties if a person is close to edge of the screen or partially obstructed.
Similar functionality is already available through ARKit, but it is limited to augmented reality sessions and only works with the rear-facing camera on compatible iPhone and iPad models. With the updated Vision framework, developers have many more possibilities.
Popular Stories
Apple Announces New 14-Inch MacBook Pro With M5 Chip
Wednesday October 15, 2025 6:07 am PDT by Joe Rossignol
Apple today updated the 14-inch MacBook Pro base model with its new M5 chip, which is also available in updated iPad Pro and Vision Pro models. In addition, the base 14-inch MacBook Pro can now be configured with up to 4TB of storage on Apple's online store, whereas the previous model maxed out at 2TB. However, the maximum amount of unified RAM available for this model remains 32GB. Like...
Apple Debuts New iPad Pro With M5 Chip, Faster Charging, and More
Apple today announced the next-generation iPad Pro, featuring the custom-designed M5, C1X, and N1 chips. The M5 chip has up to a 10-core CPU, with four performance cores and six efficiency cores. It features a next-generation GPU with Neural Accelerator in each core, allowing the new iPad Pro to deliver up to 3.5x the AI performance than the previous model, and a third-generation ray-tracing ...
Apple Said to Cut iPhone Air Production Amid Underwhelming Sales
Apple plans to cut production of the iPhone Air amid underwhelming sales performance, Japan's Mizuho Securities believes (via The Elec). The Japanese investment banking and securities firm claims that the iPhone 17 Pro and iPhone 17 Pro Max are seeing higher sales than their predecessors during the same period last year, while the standard iPhone 17 is a major success, performing...
Here's Everything Apple Announced Today
Wednesday October 15, 2025 3:54 pm PDT by Juli Clover
We didn't get a second fall event this year, but Apple did unveil updated products with a series of press releases that went out today. The M5 chip made an appearance in new MacBook Pro, Vision Pro, and iPad Pro models. Subscribe to the MacRumors YouTube channel for more videos. We've rounded up our coverage and highlighted the main feature changes for each device below. MacBook Pro M5...
Apple's Next Rumored Products: New HomePod Mini, Apple TV, and More
Thursday October 16, 2025 9:13 am PDT by Joe Rossignol
Apple on Wednesday updated the 14-inch MacBook Pro, iPad Pro, and Vision Pro with its next-generation M5 chip, but previous rumors have indicated that the company still plans to announce at least a few additional products before the end of the year. The following Apple products have at one point been rumored to be updated in 2025, although it is unclear if the timeframe for any of them has...
New 14-Inch MacBook Pro Has Two Key Upgrades Beyond the M5 Chip
Thursday October 16, 2025 8:31 am PDT by Joe Rossignol
Apple on Wednesday updated the 14-inch MacBook Pro base model with an M5 chip, and there are two key storage-related upgrades beyond that chip bump. First, Apple says the new 14-inch MacBook Pro offers up to 2× faster SSD performance than the equivalent previous-generation model, so read and write speeds should get a significant boost. Apple says it is using "the latest storage technology," ...
Apple Updates Vision Pro With M5 Chip, Dual Knit Band, and 120Hz Support
Wednesday October 15, 2025 6:14 am PDT by Joe Rossignol
Apple today updated the Vision Pro headset with its next-generation M5 chip for faster performance, and a more comfortable Dual Knit Band. The M5 chip has a 10-core CPU, a 10-core GPU with Neural Accelerators, and a 16-core Neural Engine, and we have confirmed the Vision Pro still has 16GB of RAM. With the M5 chip, the Vision Pro offers faster performance and longer battery life compared...
New MacBook Pro Does Not Include a Charger in the Box in Europe
Wednesday October 15, 2025 6:59 am PDT by Joe Rossignol
The new 14-inch MacBook Pro with an M5 chip does not include a charger in the box in European countries, including the U.K., Ireland, Germany, Italy, France, Spain, the Netherlands, Norway, and others, according to Apple's online store. In the U.S. and all other countries outside of Europe, the new MacBook Pro comes with Apple's 70W USB-C Power Adapter, but European customers miss out....
iOS 26.0.2 Update for iPhones Coming Soon
Apple's software engineers continue to internally test iOS 26.0.2, according to MacRumors logs, which have been a reliable indicator of upcoming iOS versions. iOS 26.0.2 will be a minor update that addresses bugs and/or security vulnerabilities, but we do not know any specific details yet. The update will likely be released by the end of next week. Last month, Apple released iOS 26.0.1,...
AirPods Max 2: Everything We Know So Far
Apple's AirPods Max have now been available for almost five years, so what do we know about the second-generation version? According to Apple supply chain analyst Ming-Chi Kuo, the new AirPods Max will be lighter than the current ones, but exactly how much is as yet known. The current AirPods Max weigh 0.85 pounds (386.2 grams), excluding the charging case, making it one of the heavier...