Android XR to Enable Camera Access Similar to Smartphones

Passthrough camera access has become a buzzworthy topic in the XR community lately. While we already know where Meta, Apple, and Pico stand on this matter, many are curious about Google’s direction with Android XR. After chatting with the company, I can reveal that their solution will mirror the one found on phones. Let’s dive deeper into this!

## The Camera Access Dilemma

For anyone not quite following yet, let’s cover the basics. The latest standalone VR headsets double as MR headsets, giving users an RGB passthrough view of the world via front cameras. This feature paves the way for delightful mixed reality applications like Cubism, Starship Home, and Pencil.

The cameras capture frames that the operating system uses for passthrough, and developers are eager to get their hands on those too. By accessing these camera frames, developers can use AI and computer vision algorithms to enrich reality for users. As I expressed in an earlier post, I’m an advocate for camera access as it’s crucial for achieving genuine mixed reality. Thanks to a clever workaround on Quest, I even managed to create a prototype app combining AI and MR for interior design—a venture that wouldn’t have been possible otherwise.

Though this seems promising, there’s a glaring concern: privacy. Giving camera access to unscrupulous developers raises risks, such as snapping images of a user’s environment silently and extracting personal details, like ID cards or bank information, potentially leading to misuse.

It’s crucial to strike a balance between safeguarding users’ privacy and unleashing the full potential of mixed reality.

## The Behavior of the XR Companies

Initially, many XR companies provided unrestricted camera access. Long-time followers might recall my camera texture experiments on Vive Focus with the NTW team circa 2019—exploring diminished reality, Aruco marker tracking, and sound reactivity.

Then, as mixed reality gained traction, companies became cautious, restricting camera frame access due to privacy concerns. Meta, Pico, HTC, and even Apple, all followed suit, limiting developers’ access.

However, as the XR developer community recognized the importance of this feature, they began to pressure XR manufacturers to allow camera access. Advocates, including myself, Cix Liv, and Michael Gschwandtner, pushed for clear and transparent camera frame access, arguing it should be no different from phone camera access, which only requires user permission.

This pressure led some XR companies to reconsider. Meta, for instance, plans to roll out a “Passthrough API” early this year. But what about Google’s Android XR?

## Android XR to Treat the Headset Like a Phone

With Android being the dominant OS for phones globally, developing apps for it involves requesting camera access, contingent on user consent. Google is aiming to make Android XR similarly compatible, allowing developers to request access to camera streams just like they would on phones.

After an extended email exchange with a Google spokesperson, I got confirmation. Here’s what they shared on accessing camera frames for XR:

“Developers can request camera access just as they would on Android apps. For the primary world-facing camera, developers can request camera_id=0, aligning with the back camera on phones. Similarly, camera_id=1 would grant access to the selfie-camera, functioning through Android Camera APIs like Camera2 and CameraX.”

This compatibility means Android developers can utilize familiar classes like CameraX on Android XR-based headsets, enabling media saving, ML analysis, and more. This development is fantastic!

It’s noteworthy, however, that what’s accessed through the headset’s “selfie camera” is simply a reconstructed avatar—similar to Apple’s Vision Pro. This ensures that Android XR maintains coherence with standard Android phone operations, a strategy I find quite intuitive.

## A Little Caveat About Android XR

Bear in mind that Android XR is still in its preview phase, with no officially released headsets. The details discussed could potentially change upon full release. While unlikely, it’s important to consider.

## The Opening Up of Camera Access

With Google and Meta leading the charge in opening camera access, it’s possible other XR companies may soon follow. It looks like 2025 might be a watershed year for mixed reality innovation, and I can’t wait to see what the developer community comes up with!

(Header image inspired by Samsung)

Disclaimer: This blog includes ads and affiliate links to support itself. If you click an affiliate link, I earn a small commission on purchases. You can find my full disclosure here.

Share this with fellow innovators.