Everything you need to know about Apple's new mixed-reality headset
Apple is releasing a new computing device you wear on your head.
Welcome to the inaugural edition of HHHYPERSPACE, a new blog for technologists to explore the fast-moving frontiers of technology. Let’s kick it off with a look at Apple’s moves into spatial computing. Let’s explore what Apple provides in the Vision Pro specs, who and what it’s for, and Apple’s overall vision for it.
After a slew of incremental updates in last June’s WWDC keynote, CEO Tim Cook had "one more thing" to announce. Apple finally announced its long-rumored AR+VR device, Vision Pro – and it was impressive. It was announced months in advance of its early 2024 arrival to allow partners and developers time to take advantage of this new paradigm and build out the ecosystem at launch. Early reviewers have consistently marveled at the quality of the device and display, and how intuitive and responsive the eye and hand tracking is.
Its arrival was expected around April by most, but as we turned our calendars to 2024, Apple announced it was being released on February 2 in the US, with presale starting today.
This post recaps a number of details given at WWDC and now:
WWDC: “One More Thing” in the WWDC keynote (or see the supercut)
WWDC: visionOS developer page and info sessions
SPONSOR: This post is brought to by HHHYPERGROWTH, my newsletter that dives into the investment view over SaaS tech companies across industries like AI/ML, data & analytics, next-gen security, developer tooling, and the hyperscalers. I dive deep into companies (including Snowflake, Samsara, Cloudflare, Zscaler, CrowdStrike, Datadog, and more) to cover platform positioning, product announcements, competitive landscape, growth vectors, market adjacencies, and earnings takes. The premium service provides weekly posts on these topics, while the free newsletter comes out once a month (straight from that premium content). Go sign up!
For more details, my blog post Apple’s new mixed-reality headset (at the separate HHHYPERGROWTH blog) has more background on the Apple Vision Pro announcement at WWDC, its announced design & specs, and my initil take on Apple’s vision for the product. (This piece samples from it heavily, but updated with information at launch.)
Launch
I found it interesting that Apple released the Vision Pro with a tweet by the CEO and a PR — no event was scheduled. I feel the lack of event signals they aren’t expecting this to be a huge hit with consumers - it is a high-end “Pro” VR/AR device for enterprise work & content creation — though will certainly appeal to high-end consumers with features like immersive 3D entertainment and spatial video.
It’s not like they didn’t want the release to be hidden, however, as they aired an ad for it at the national college championship game on January 8th dubbed “Get Ready”.
Pre-orders began today, and it starts to ship on Friday, February 2. Due dates have already slipped into March. (Likely due to both strong demand and limited initial supply.) You can also go into an Apple retail store at release for a 25 minute demo, provided first-come first-served (no reservations). [Hopefully the demos continue beyond the opening weekend.]
Who/what is this for?
First off, the ideal user is one already heavily enmeshed in Apple’s iCloud ecosystem (across Photos, Mail, Calendar, Notes, Music, Podcasts, Apple Pay, et al). This is how Vision Pro best shares content with other devices via wifi. It can also use AirPlay to show the display “through your eyes” to a Mac or iPad, and can Bluetooth to Airpods and existing Apple accessories like the Magic Keyboard, Trackpad, and Mouse.
As an enterprise work device:
This is for creatives in the Mac ecosystem (video, audio, podcast, music) that would benefit from spatial control and display.
This is for spatial content creators and developers creating spatial apps, as a Pro line with dedicated compute.
It provides a private & portable workspace, with infinite display (like a laptop + endless displays on your face).
It allows connecting to existing Mac systems as a private & portable spatial display and control (eyes, hands, voice).
For consumers:
It is a high-end AR/VR headset that leapfrogs competition in specs and input control.
It can be used as an immersive high-end solo entertainment system for 2D and 3D content (as a 4K infinite display with spatial audio), in a private & portable device.
It can act as a spatial video capture and playback system for reliving memories in 3D. (However, I feel this use case will improve more as other consumer-grade devices emerge from Apple, so you can share these 3D videos with others.)
It’s also for cutting-edge consumer tech fans who have $3500 to burn.
If you want a VR social gaming rig, you should look elsewhere (like the Meta Quest 3) until the visionOS ecosystem develops further and more spatial games appear on the platform.
Another missing focus is fitness. While this is a portable device, Apple clearly intends it to be used in a stationary position. I expect future consumer-friendly editions to connect to Watch and Fitness+.
Specs
The specs in the PR matched what we learned 7 months ago at WWDC, which are applicable to any use case.
It is a spatial computer (or is that facial computer?) that runs their new visionOS operating system, which is natively controlled through eyes, hands, and voice.
It is mixed reality, where users can use passthrough AR to view their environment, or completely immerse themselves in VR. Users can wrap themselves in provided Environments during video and music playback with the twist of the digital crown.
It has an ultra-high resolution micro-OLED 4K panel per eye, which combines into a single 23M pixel 4K HDR display. This gives users a seemingly infinite display, and supports Dolby Vision format in video playback.
It has a modular design, with various swappable headbands and light seal cushions to get the right fit. You need to scan your face in-store when you order it, or by using an iPhone/iPad during online orders. It doesn’t work with glasses, and those with prescriptions will need optical lens inserts. (Or, just use contacts.)
It has advanced spatial audio from dual-driver audio pods in the headband to complete the immersive experience with directional sound. You can also connect AirPods Pros (2nd gen) for complete privacy. It supports Lossless Audio and Dolby Atmos formats in music and video playback.
It is built upon Apple Silicon, including the M2 chip that now powers all Mac lines, which includes their Neural Engine for onboard ML. Mark Gurman of Bloomberg noted the M2 chip is the 10 GPU/8 CPU model that powers higher-end MacBook Airs and low-end MacBook Pros (until the M3 revamp in October).
The outside and inside the headset have a combined 12 cameras (forward, down, and side), 5 sensors (LiDAR, IR), and 6 microphones. Apple created a new R1 chip specifically focused on capturing and processing real-time information from these as you move your head, eyes, and hands, and use your voice. The result is a silky-smooth hyper-realistic passthrough AR and VR experience.
An external OLED display called EyeSight shows your “eyes” (from your digital avatar) to help combat VR isolation. People interacting with you can see your eye expressions, and they “break through” the display (via passthrough AR) so you can see them.
As always, Apple is concerned about the security & privacy of the device. They added a new Optic ID for user authentication, app purchases, and Apple Pay (as TouchID/FaceID do).
SharePlay allows for using AirPlay to transmit the display to a connected iPad or Mac, so others can see what you are viewing. Guest Mode allows sharing selected apps and experiences with others using the headset. (However, allowing others to use the device on a regular basis might require getting other light seals for the proper fit.)
The external (wired) battery is good for 2 hours, or 2.5 hours if doing video playback, browsing, FaceTime, or video capture. Or you plug that battery into a USB-C plug for continual power.
Pricing is $3499, starting at 256GB of storage, with options for 512GB and 1TB (increasing +$200 each tier).
The box includes 2 types of headbands, custom light seal cushions (based on the fit of your face), plus a lens cover and polish cloth.
Apple offers AppleCare+ for 2 years at $499, which I consider a must for Apple products.
Add-ons at purchase include custom lens inserts from Zeiss (priced at $99 for readers or $150 for prescription), a carrying case ($199), and additional batteries ($199). And don’t forget those Bluetooth accessories like Magic Keyboard and Trackpad.
It is available in the US starting on February 2, and requires a US-based Apple account to activate. Canada and the UK are expected later in 2024. It can be delivered to home or in-store, with an appointment for proper fitting and training.
Reviews seem to universally point to how precise and intuitive the eye and hand tracking are. Key to the responsiveness is the custom R1 chip, which processes sensor, camera, and mic data within 12ms. This high processing rate helps Vision Pro avoid the nausea effect common in VR.
Reviewers are also impressed with the quality of the 4K display. Apple has been able to leverage its OLED experience in the iPhone and Watch to compress pixels into micro-OLED, stuffing 64 pixels into the space of one iPhone pixel. The unit combines two 4K displays per eye into an overall 23M pixels. This minified display helps Vision Pro avoid the screendoor effect common in VR.
However, a primary complaint from reviewers has been around the weight of it — likely from the aluminum housing and tempered glass of the lens. Several tech reviewers have found the newly included Dual Band to be more comfortable in distributing the weight. One reviewer noted how her discomfort with the woven single band was alleviated by switching to the dual (over-head) band:
“…the Vision Pro unit I tried on today came with a strap that you stretch and ends up at the back of your head. It was wide, ridged and soft, and I at first thought it would be very comfortable. But 15 minutes into my experience, I started to feel weighed down by the device, and five more minutes later, I was in pain. … When I finally told the company’s staff about my issues, they changed the strap to one that had two loops, with one that went over the top of my head. That Dual Loop band felt much better for weight distribution, and it didn’t keep slipping down my hair. It’s worth pointing out that Apple did first perform a scan to determine my strap size, and they chose the Medium for me. I also had to keep turning a dial on the back right to make everything feel more snug, so I had some control over how tightly the device sat. Basically, you’ll have quite a lot of options to adapt the Vision Pro to your head.”
Several early WWDC reviewers noted that Apple stated the weight as 1-1.5lb. A report by a VR research outfit in November showed it inline with other popular models. Meta’s new Quest 3 made width improvements over the Quest 2 in how wide the unit is (how much hangs off the face). I wonder if the width of the unit is a contributor to the discomfort that same feel after prolonged use, as the weight is all centered on the forehead.
visionOS
Apple has created a new spatial operating system for the Vision Pro dubbed visionOS, as well as an associated App Store.
There are no external controllers needed to control the device. Similar to Microsoft's HoloLens, you control it through your head, eyes, hand gestures, and voice. You don't need to use exaggerated movements, and can remain standing in place or seated as you control it. Sensors over your eyes allow your gaze to control the focus (say, selecting an app from the home screen or changing focus to another window). You control the click-and-touch actions via hand gestures, which seem similar to existing touch gestures on the Mac trackpad and iPhone/iPad screens. Search bars and other text inputs allow using your voice.
A session at WWDC on spatial input methods provided more details on how the device is controlled via eyes and hands. Apple has developed a new set of common gestures based on hand actions to trigger UI interactions within native apps.
Beyond that, apps can leverage a Direct Touch mode for more precision on specific actions, such as manipulating a virtual object or mirroring physical activities.
As far as traditional inputs, users can type on a virtual keyboard, or use voice input to input text. It also supports connecting with Bluetooth accessories such as Magic Keyboard, Magic Mouse, and Magic Trackpad already used in the Mac line.
If using the device for work needs (writing emails or text documents) the Bluetooth accessories seem a must. Reviews about the virtual keyboard show it as one of the weaker areas of the device experience.
Another reviewer was really not a fan:
"This was one of the more frustrating aspects of the demo for me. Although there were several typing options – hunting and pecking with your fingers, using eye control to select keys, or just using Siri – none of them felt adequate for anything resembling extended use. … The floating keyboard here clearly needs improvement.”
Capabilities
Typical Mac apps include the familiar Notes, Calendar, FaceTime, Mail, Messages, Photos, Safari, Freeform, Music, Mindfulness, Keynote, Numbers, App Store, iCloud, and other compatible apps.
For work:
Beyond the native Apple apps and native apps from partners, users can run 1M+ apps from the existing iOS/iPad ecosystem.
Users can use the device as an external 4K display and controller for existing Mac systems, allowing users to use their eyes and hands to interact with Mac apps running on the connected system.
Users create a digital avatar to collaborate with others in group video apps, including the native FaceTime as well as apps from launch partners Zoom, Microsoft Teams, and Cisco Webex.
Other productivity apps mentioned were Slack, JigSpace (3D object presentations), and Fantastical (calendar/contacts).
Expect Apple content tools like Final Cut Pro and Logic Pro to come out with native apps, but you can likely use the iPad versions of these apps in the meantime. Adobe Creative Cloud and Microsoft Office were mentioned at WWDC but there is no sign yet of their native apps.
For entertainment:
As just mentioned, the digital avatar can be used in FaceTime for video calls.
Users can watch 2D/3D entertainment content from AppleTV+, Disney+, Max, and other launch partners. This includes setting the mood in immersive Environments in Cinema Mode.
Content platforms providing support include Apple TV+, MLS Season Pass, Disney+, ESPN, NBA, MLB, PGA Tour, Max, Discovery+, Amazon Prime Video, Paramount+, Peacock, Pluto TV, Tubi, Fubo, Crunchyroll, Red Bull TV, IMAX, TikTok, MUBI. (Sadly, no Youtube, Netflix, or Spotify.)
250 titles are available via Apple Arcade, including NBA 2K24 and spatial games like Super Fruit Ninja. More are coming, as Unity has committed to supporting the platform.
The VisionPro can take and show spatial (3D) videos, as well as spatial videos taken on the iPhone 15 Pro line (which was just recently enabled). Reviews suggest this works well for up-close subjects more than landscapes. It also can show Panorama shots in Photos as a wrapped immersive vista.
Very related to this device is Apple’s new Immersive Video format, which records in 180-degree 3D video with spatial audio. Apple has been shooting AppleTV+ content with this for a while now in anticipation of this release (see this leaked photo of the camera on set). This new format is enabling TV/movie content to be shot with a 8K HDR spatial video and audio, and will hopefully be leveraged in more entertainment and sport events as well. (Early demos included a view hovering over the court at an NBA game, which garnered rave reviews.)
AppleTV+ will have over 150 3D movies, including several shot in the new Immersive Video format, including an Alicia Keys private performance, Adventure (extreme sports), Wild Life (animals), and Prehistoric Planet (from Planet Earth).
Disney+ provides 4 immersive Environments (for Pixar, Marvel, and Star Wars fans) and 150+ 3D movies, including the latest Avatar, Star Wars, Avengers, Spiderman, Elemental, Encanto, and more to come. 3D movies from other partners included Super Mario Brothers and Dune.
The device also includes mixed-reality experiences, including Encounter Dinosaurs mentioned reviewer demos where a dinosaur emerges from a portal that opens in a wall, and a butterfly lands on your finger.
Travel Mode allows the device to stabilize content while on planes, to help prevent any nausea. (One assumes this applies to cars and trains, too.)
Concerns from here?
Content, content, content, content. Apple needs partners and developers to step up with more mixed-reality experiences and content for enterprise and consumer use.
We haven’t seen many Vision Pro-related announcements from other app partners beyond Disney and a handful of others. Apple mentioned Adobe and Microsoft at WWDC but no word from them yet.
Is the weight of the unit going to allow it to be used for longer lengths of time? (A half day of work? A full-length extended movie?)
Apple appears to have nailed the hand and eye tracking, however they need to greatly improve Siri from here. I hope to see them leverage LLMs to rapidly improve it, as Amazon is doing with Alexa. [More on that in a future post.]
The virtual keyboard looks like it doesn’t work that well, and needs improvement. For now, use an external Magic Keyboard for long-form typing. Expect a visionOS update soon to fix this.
Disappointingly, consumer content services like Netflix, Youtube, and Spotify seem to all be actively making their iPad apps not support running on Vision Pro. This is likely related to displeasure with Apple’s 30% services cut.
Apple’s vision
I feel some reviewers are being myopic about the overall vision that Apple is presenting.
Vision Pro is an all-new mixed reality device that competes with similarly-priced high-end enterprise-grade devices like Magic Leap 2, Varjo XR-3, and Microsoft HoloLens 2, as well as the mid-level enterprise-focused XR devices Meta Quest Pro and Lenovo ThinkReality VRX.
It is a standalone device that provides a new computing paradigm Apple dubs "spatial computing". Apple envisions this being the next evolution in computing devices, as a mixed-reality device that can extend to use cases beyond what our current laptops, smartphones, tablets, and smartwatches are capable of – morphing their features into an infinite virtual spatial environment.
This is a high-end computing device, not a consumer-grade one that focuses solely on entertainment/gaming. I think it best to think of Vision Pro as an evolution of the Mac and iPad – where Apple is moving towards the Quest, but not as a direct competitor to the Quest. This is not a product one would expect to be adopted en masse. It is the first step in a multi-year strategy, where other iterations and future products may eventually converge with Meta's "Metaverse"-driven vision. As is typical of Apple, they set the bar high with a top-of-the-line product right out of the gate.
Think of Vision Pro as combining all of Apple's high-end products in one. It acts as a MacBook Air laptop, iPhone/iPad (minus the cellular connection), dual 4K displays with an AppleTV, and dual AirPods together, all worn on your head.
This device is the culmination of all of Apple's past decade-plus of work on its various product lines, including its specialized chips, displays, microphones, cameras, sensors, and bands. This extensive expertise allowed Apple to instantly leap into being a top-of-the-line product in its coming release, and Apple clearly expects a fuller ecosystem to form by the time it launches early next year.
It is built atop their custom M2 chip (which now powers every Mac line), and a new R1 specialty chip for real-time spatial processing over an array of sensors, mics, and cameras (from existing tech across their devices), including a digital crown (from the Watch), spatial audio (from the HomePod and AirPods), and AR & spatial mapping (from ARKit & RealityKit in the iPhone & iPad).
As Apple is prone to do as it enters new markets (as it has done across the audio player, smartphone, tablet, and smartwatch markets), Vision Pro is now poised to dominate the market in high-end mixed reality. It is priced starting at $3499, with add'l charges for the optional magnetic optical lens from Zeiss for vision correction. And perhaps the "starting price" given hints at flexibility in the amount of compute and memory it has. [To compare it to Apple's other high-end computing products: the new Mac Pro with M2 Ultra starts at $5999, and their Pro Display XDR (6K monitor) at $4999.]
Like the iPad, it comes with a virtual keyboard and dictation capabilities, plus works with existing Bluetooth devices like Magic Keyboard, Magic Trackpad, AirPods, and game controllers. As they have done with the Apple Watch, expect a variety of headbands and other accessories from Apple and 3rd parties, including fashion-oriented ones. (Some are already appearing.)
As always, Apple will continue to iterate from here. This first iteration is geared for developers, content creators, enterprise AR/VR, and high-end Apple consumers addicted to cutting-edge tech. As hinted at by Pro in the name, there are future iterations in the works, including a more budget-focused “Air” model (non-Pro).
I expect their Vision line to be a heavy focus over the next few years as their product flywheel takes effect and drives the acceptance and adoption of this emerging paradigm. I expect other related XR devices to emerge (perhaps consumer-based ones honed on gaming and fitness), and Apple might revitalize its smart glasses ambitions from what it learns. We will likely see the newer M3 chip just unveiled in October to be used in the next iterations.
Despite its high-end feature set that compares to enterprise-focused competitors like HoloLens 2 and Magic Leap 2, most can't help but compare it to the price of the lower-end consumer-focused Quest line. Yes, it is way more expensive ($3499 vs $499 Quest 3), but this is not just an entertainment and gaming VR environment – it is a high-end feature-rich standalone generalized computing device that can likely replace laptops, external displays, tablets, and (single viewer) TV/audio setups.
This is the first new product line and OS from Apple since the Watch hit in 2015. (I'm not counting the HomePod from 2018, which doesn't have its own OS and App Store.) As they did in the mobile (smartphones and tablets) and wrist (smartwatches) markets, Apple is stepping into a new computing direction as a major new platform – not as the first to market, but as the best hardware and software combined experience. XR is an area that Microsoft has been focused on for over 7 years now (HoloLens first appeared in early 2016, and their current HoloLens 2 is now 3.5 years old), and Magic Leap and Oculus even longer (founded in 2010 and 2012). Countless other startups have emerged from there, like Varjo, Lynx, XREAL, and Vuzix.
I've long been interested in the shifts toward greater adoption of XR and ultimately "the Metaverse", but after a decade-plus of progress, the market hasn't shown up beyond consumer gaming and a handful of enterprise uses. The existing XR market feels like it is losing steam (see the rise and fall of the interest in the Metaverse, rumors that Microsoft halted development of HoloLens, Oculus's elusive profits and mass adoption, Magic Leap's constant struggles) – but we now have Apple entering with a new vision and direction, with an extremely high-end product that vaults to the top of the pack. What we see now is merely the pre-game of a multi-year focus. I feel that Apple's moves will greatly boost interest in mixed reality devices, followed now by several more years of innovation and iteration that will help bring it more and more to the masses.
The Vision Pro sets the bar extremely high up front for Apple, launching them with a top-of-the-line mixed reality headset that serves as a standalone facial computer. They are making it clear that this is a multi-year plan as they iterate and explore new directions in their Vision line from here. This feels like an initial enterprise-grade device that can be used to create spatial content for future devices, once it becomes more consumer-grade.
Thanks again for joining me on this new blog for technologists, to explore new frontiers in AI, AR/VR, cloud, robotics, space, and more.
Again, see my blog post Apple’s new mixed-reality headset (at the separate HHHYPERGROWTH blog) for more background on the Apple Vision Pro announcement at WWDC, its announced design & specs, and my take on Apple’s vision for the product. That is an example of the type of long-form content that will appear on this new blog. Beyond that, I’ll have short-form curated news posts with my take on the various moves in these directions.
-muji