Building an Interior Design Visualizer for visionOS Using the Main Camera Enterprise API

When visionOS 1 was announced two years ago, I saw a lot of developer indignation about how we couldn’t access the raw camera frames of what the Vision Pro sees. I’m personally supportive of that limitation, but there were (and are) a lot of devs out there who really want to see everything their users see, mine that data, and monetize it.

Other, ads-based headsets will do that en masse soon enough – but Vision Pro devs eventually did get access to raw camera frames last year with visionOS 2…just with some major caveats and hoops to jump through.

visionOS Enterprise APIs

At WWDC 2024 Apple announced the visionOS Enterprise APIs. Apple started with six Enterprise APIs in mid-2024 and quietly added a seventh (UVC Device Access) in late-2024, I suspect on account of working closely with some medical device companies.

There are some ho-hum Enterprise API capabilities, and then there are some big ones like Main Camera Access. You want your app to have access to everything your user sees? Apple may bless you with that capability. If you ask nicely.

Enterprise APIs Come With a Major Distribution Catch

Dissemination of any product that uses any Enterprise API is currently limited to not-on-the-App-Store methods. That leaves such apps to be distributed as proprietary in-house apps or custom apps doled out via Apple Business Manager. Furthermore, Apple scrutinizes each and every Enterprise-API-enabled app and grants that limited distribution on a per-app basis.

I asked nicely and was allowed to tap into the Main Camera Access for local prototyping. As soon as I got my Enterprise API entitlement last July I was quick to do nothing with it for 10 months. And then I slapped this together:

I’m going to paint my room “Memphis Raines Storm”

You can check out the full diminished reality demo with sound over here.

I believe there is colossal potential for interior design, home improvement, architecture, and construction companies to create useful tools with Main Camera Access. Even given that the API is super-barebones right now.

Meager Beginnings

I’ve been spoiled by years of valuable depth maps, ARFrames, and segmentationBuffers that came – just as a matter of course – with iOS’s ARKit.

What do you get with visionOS’s Main Camera Access?

An image.

That’s it. No depth data baked into the frame. No segmented anything. Just a plain ol’ image. Have fun!

I did have fun. Sampling colors and textures from the main camera is a simple, yet substantial, capability.

For this specific use case of interior design visualization, I think the product would be stronger if the Enterprise API capability were removed and a curation of wallpapers & flooring textures were preloaded. By avoiding use of an Enterprise API, wider distribution (such as it is) could be achieved via the App Store.

My gut says that for such a product to be truly impressive, visionOS will need to make some striking advancements in the algorithms that dynamically shade virtual objects based on real-world lighting. However, the somewhat unlifelike and unconvincing rendering of the diminished reality, virtually-painted walls might be attributable to my haphazard shader graphs and material settings. Said differently: Perhaps a more comprehensive understanding of ShaderGraphMaterials on my part would yield more realistic effects.

Still, I’m sure serious people doing serious work could come up with many actual business reasons to sample the camera images, or process them, or analyze them. Riffing on the demo above and off the top of my head:

Some Potential Use Cases

1. Interior Design: Perfectly extract the color of a specific portion of a piece of art, or centerpiece material, or heirloom fabric and then use that color elsewhere in the space
2. Architecture: Sample a neighborhood’s textures, colors, stonework, brickwork, or historical materials, and then apply them to a new building to visualize harmonization
3. Home Improvement: Sample fading paint for ordering precisely matched touch-up paint
4. Construction: Capture a material’s variations to compare against examples of known defects, or to document inspections

Using the eye dropper tool on a photo may be today’s approach to accomplish these tasks. I don’t expect today’s heavy hardware to replace much of such activity. But I do think there will be future scenarios where removing intermediary steps by looking at something and immediately taking an action will be preferable to manually “processing” an image file.

Putting it All Together

There’s nothing to stop you from collating frames from the main camera along with data from other ARKit providers into some amalgamation. For example: Using some rough projection math I was able to approximate RoomAnchor corners and annotate them on the live camera frames using an ugly shader graph.

I get the sense that there is a lot more coming to the CameraFrames vended by this API. I would love to get even a fraction of ARFrame’s data tacked on here. I think each layer of additional information made available opens up a multitude of use cases – and opportunities for enjoyable mucking about.