Android XR to Enable Phone-Like Camera Access

There’s a lot of buzz in the XR (Extended Reality) world about passthrough camera access. Everyone knows where Meta, Apple, and Pico stand, but what about Google and its Android XR plans? After having a direct chat with the company, I can confirm they’re gearing up to introduce a solution similar to how things work on their phones. Keep reading to learn more about what’s happening!

The Camera Access Dilemma

Let’s backtrack a bit to get everyone on the same page. Most of today’s standalone VR headsets are essentially MR (Mixed Reality) headsets. They provide users with a full-color passthrough vision using front-facing cameras to overlay digital elements onto the real world. This functionality is key to delightful mixed-reality apps like Cubism, Starship Home, and Pencil, to name a few.

The system relies on camera feed to deliver this passthrough view. As developers, we really wish we could tap into these camera feeds ourselves. Having that capability means we could inject AI and computer vision algorithms to enrich the user experience. I’ve mentioned this before—it’s essential for realizing the full potential of mixed reality. With camera access, applications can truly grasp the environment around us. Last year, I hacked my way to some level of camera access on the Quest, which helped me create an AI + MR application for interior design. It simply wouldn’t have been possible otherwise.

However, great as it sounds, privacy concerns loom large. Handing over camera access to developers could pose risks if someone with ill intent gains access. They might, for example, silently capture images, run AI analysis, and extract sensitive information from their surroundings, be it personal IDs or credit cards. Plus, there’s the risk of sneaking images of people’s faces or bodies for unauthorized use.

It’s a delicate matter: there’s a need to balance user privacy with unlocking mixed reality’s full potential.

The Stance of XR Companies

In the initial days, the industry freely allowed camera access without much restriction. If you’ve followed my work, you might recall our experiments with camera textures on the Vive Focus back in 2019. We explored ARuco marker tracking, sound reactivity, and more.

But as mixed reality began gaining traction, these companies quickly tightened the reins, citing privacy issues. Brands like Meta, Pico, HTC, and Apple have since restricted developer access to camera feeds.

This paradigm became the norm until the XR developer community rallied for a change, pointing out the necessity of this feature. Influential figures like Cix Liv, Michael Gschwandtner, and myself began advocating collectively for a transparent user-controlled way to access camera frames. It begged the question—why the paranoia around XR when our phones, ubiquitous in our daily lives, allow camera access through simple permissions?

Our calls haven’t gone unheard. Meta is set to roll out a "Passthrough API" early this year. But where does Google’s Android XR fit into all this?

Android XR: Treating Headset Like a Phone

Android powers most of the world’s smartphones. Currently, Android developers can request camera access with user permission. Once granted, they just specify which camera they want to use and can go ahead with capturing frames.

Google’s aim is to make Android XR as compatible with existing Android apps as possible. Although there hasn’t been official confirmation, I reached out to a Google spokesperson for clarity. Here’s a direct excerpt from our email exchange on camera access for Android XR:

"Similar to any Android app, a developer can use existing camera frames with user permission for XR."

This explains they’re sticking with existing Android behavior. Much like how it works on phones, developers can access the main world-facing camera or the selfie camera using standard Android camera APIs like Camera2 and CameraX. Permissions need to be requested, much as you would do with any Android app.

Applications that tap into the selfie camera get an avatar stream created by avatar services on the headset, using tracking data like head movement or facial expressions. This approach aligns with Apple’s Vision Pro by mixing real-world captures with synthesized content, for a great cohesive experience.

A Word on Android XR’s Paths

Android developers have the option to use existing classes like CameraX to handle XR-based headsets. These allow developers to capture frames, store media, or implement machine learning—all permissible on glasses and headsets as well. This is big news for the XR space!

However, while the front camera is available, the rear camera provides only a virtual avatar, similar to Apple’s Vision Pro. This is a clever move by Google, offering a realistic blend of flexibility and safety.

Google’s decision ensures consistency across different Android applications, making certain camera-accessing apps behave just the same on this new platform. I’m pleased with the coherent permission system between phone, headset, and glasses.

That said, if you’ve been pondering on accessing raw camera streams, I might not have great news:

"At the moment, we are not providing a path for applications to access the non-standard (e.g., forward facing camera, reconstructed inward camera) sensor data."

The response suggests limitations on accessing non-standard camera streams for now, though there’s a chance of opening access for enterprise needs later.

For Unity developers, some challenges remain since CameraX and Camera2 are native Android classes. Fortunately, you can use the WebcamTexture class to grab camera frames, or rely on JNI-based solutions for accessing specific CameraX functions, potentially through community-developed libraries.

Note on Android XR

Remember, Android XR is still in its preview phase, so changes could happen before a full release. While unlikely, it’s worth noting any developments could alter current plans.

The Path Forward for Camera Access

With Google and Meta opening up camera access, it’s likely other big players will follow. 2025 looks likely to be a pivotal year for groundbreaking mixed reality tools. I’m excited to see what developers will bring to life!

(Header image is inspired by Samsung)

Disclaimer: This blog contains advertisements and affiliate links for support. Clicking on an affiliate link makes me happy as it generates a small commission. Full disclosure is available here.

Share the Love

If you found this post helpful, spread the word with fellow innovators.