Remixing the Real World With Twin

I just launched my first visionOS app, Twin – Twin uses your iPhone Pro or iPad Pro to create 3D models of your stuff and then automatically syncs those models to your Vision Pro. In your Vision Pro you can position those models wherever you want and see them even while you're completely immersed. You can see a demo of Twin here, or here's a quick look at the visionOS side:

Never lose sight of that keyboard, even when you're visiting Haleakalā

I originally started working on Twin when I noticed I really liked working in visionOS environments, but my main complaint about using environments was I couldn't see my keyboard while fully immersed. I'm sure many people noticed the lack of keyboard passthrough. So I set about fixing that issue. Good thing I just spent plenty of time learning all about iOS's photogrammetry tool, Object Capture.

So I started building...and then somewhere along the way I got obsessed with an idea.

Applying filters to images and videos and audio has become a mainstay of art, culture, technology, and daily life over the past many years. Apple developers have used CIFilter for just such purposes; apply a sepia filter to an image to give it that old-timey feel or oversaturate a video to give it an 80's feel.

The .usdz models created by Object Capture, and therefore the iOS/iPadOS versions of my app Twin, capture the texture of your stuff. That is to say: there is underlying image data that makes the 3D replica of my keyboard actually look like my real-world keyboard. Same shape, same colors, same textures.

Eventually, the reason I was building Twin was ostensibly to see specific objects while fully immersed in an environment...but in actuality the reason was because I wanted to filter my 3D objects.

This is probably related to me getting very attached to things I love. Over the years I've purchased the same product in different colorways many times. Video game controllers, shoes, shirts, phones, handheld consoles, keyboards, synthesizers, you name it. Embarrassingly, many years ago I would drool over Colorware versions of products I loved. I never pulled the trigger on a Colorware product...but I am inclined towards wanting different versions of my stuff. Someday I'll get my real-life car that can change colors, but let's start with a virtual keyboard.

This seemed like a prime use case for that newfangled spatial computing. Or maybe some good ol' augmented reality will do.

I initially pursued having different .usdz's for the different model versions. I tried falling back to SceneKit to no avail – there are perilously few ways to write to a .usdz file using native Apple SDKs, but SceneKit is one of them. Even when I found a way to bridge the SceneKit-unfriendly PhysicallyBasedMaterial to an SCNMaterial, the texture mapping was all wrong. I then turned to RealityKit, but encountered the same problem (in addition to encountering the roadblock of trying to write to a .usdz file from RealityKit).

Then I turned to where I think I will be spending many, many hours in the future: Reality Composer Pro and its ShaderGraph.

Here's a frantic video of Twin filters remixing my stuff on my Vision Pro:

View your things in ways that are impossible in the real world

This is Big

I'm fine with some of the glimpses we've seen of what the future holds for Apple Vision – starting with the Vision Pro and expanding from there (I think a significant reduction in headset/glasses weight is needed before the vast majority of people would even consider picking up a Vision, and I don't expect that for several years...so let's gloss over the tremendous prerequisite that the hardware needs to "get there"). I think huge, floating, virtual screens are cool. I think the display industry will transform in the next couple decades as we transition from buying a bunch of big screens we put on desks & entertainment units to buying significantly fewer, insanely-high-definition screens we place millimeters from our eyes.

But...we already had screens. Making screens instantly customizable and weightless is cool. But it's not that exciting to me.

I think what will initially lead to increased adoption of the Vision Pro, and eventually bolster Apple's services revenue, are sports and live events as experienced via Apple Immersive Video and similar products.

But I think what will cause people and companies to really start using augmented reality headsets/glasses – and when it will "click" for most people – is when spatial computing apps enable unique and novel things that were largely impossible on other platforms.

Twin's first filters are gaudy and ostentatious. Twin's 3D models don't follow their real-world counterparts; they are stationary and must be manually placed to overlap with their real-world twin (Apple decided visionOS 1.0 is not allowed to utilize ARKit's object tracking capabilities...I'm hoping that changes in the near future with visionOS 2.0 or 3.0). And lastly, a filtered model can only be seen by a single person: the user.

But these filters are my first step towards giving people the tools to augment or remix their world via a visionOS app. I think there are profound implications (and opportunities) in enabling people to easily + instantly + cheaply modify and reskin and accessorize and personalize their things and their space.

Even if I'm totally wrong – Twin was fun to build and I like having a Godzilla materialize on my desk psychedelic pixel by psychedelic pixel.