explore visionOS development with Tony from 1984

Using visionOS Image Tracking to Sync a Paper Map to the Lord of the Rings Movies

Last year I was in Albuquerque and happened across the excellent artist behind Myths and Maps. He was selling "aged" maps of Middle Earth. That is a nice combo of several of my "magic words."

I instantly knew what I wanted to do with my new, old map: Sync up an augmented reality version of the map, replete with pins representing the members of the Fellowship of the Ring, with the extended edition movies. I pictured a coffee-table-sized map where I could follow where the characters were as I watched the movie. A demo with sound is here, or perhaps this will get the gist across:

A much less magical, infinitely less useful Marauder's Map

When I started researching the feasibility of this demo, the first stop had the least to do with spatial computing: Can I (legally) get the video files of these movies?

Yes! All it takes is a 20-year-old external DVD drive that I hoarded at my parents' place. My oversight (or laziness) in not including an internal DVD drive in the gaming PC I built in ~2003/2004 finally paid off!

I own the Extended Edition versions of the movies on Amazon, but those are worthless digital versions. So I broke out the DVD copies I also own and ripped my first DVD in over two decades.

If you're looking to do the same, generally:

  1. Use Mac's Disk Utility to copy a .dmg file of the DVD to your computer
  2. Use VLC to convert that .dmg to .mp4
  3. Use DDRescue if you happen across any damaged DVDs that throw random errors
  4. That's it

I successfully ripped the video files using decades-old technology – now for that augmented reality part.

If you're unfamiliar with Reality Composer Pro and you're interested in visionOS development: I highly recommend just sitting down and giving it a go. I put off diving in for months because I didn't like how quickly Reality Composer got shoved aside and left to fester (i.e. the pedestrian, non-Pro version). RIP.

But Reality Composer Pro is quite powerful and rather fun. Try out some visionOS particles using only code, and then give it another go in Reality Composer Pro. Peruse the (admittedly extremely limited) materials library. And especially give the Shader Graph a go. At one point I had a Custom Geometry Modifier connected to the Middle Earth map to provide it with some 3D, topographical elements...but I eventually nixed that since the lettering and artwork of this specific map is not meant to be represented as a three dimensional object.

This demo does use visionOS's Image Tracking – there isn't much to it and the Apple sample project should be all you need to get going. I will say...the low frequency of updates provided by ImageTrackingProvider was quite surprising. ARKit on iOS was tracking images effectively in real-time ~6-7 years ago. Perhaps visionOS 2.0 will bring us a bounty of improvements next week? Perhaps I'm too optimistic.

Lastly, on my "developer-designed" pins: Spline has been my go-to recently for various light 3D-modelling tasks. I was a huge fan. I'm not a huge fan of them recently 2.5x'ing their monthly subscription cost. I need more .USDZ tools. I'll keep my eye on EndlessTools, but in the meantime I just exported .glb files from Spline (for free) and converted them (for free) using Convert3D.

Probably the most intense part of this project? That would be skimming through all 727 minutes of the trilogy to annotate who was where at what timestamp, and translating those locations to meaningless coordinates that are only relevant for my specific map at its specific size in this specific demo.

But it was worth it (I guess? maybe?) because the entire time I was excited.

Opportunities

I've probably listened to the commentary tracks for 3-6 movies max, and those were my favorite movies. Are new commentary tracks, by the creators and artists behind movies, even being made anymore?

There's assuredly a business reason for that.

But I'd say there's a business case for monetizing a new type of content for mega-franchises. For some anecdotal evidence: See the multiple copies of the same movies I own, see the multiple times I've gone back to see them again in the theater, see the Hobbiton tour I went on in New Zealand, etc.

Some very simplistic ideas immediately sprang to mind.

Maps

Game of Thrones. Lord of the Rings. Harry Potter. Masters of the Air (and really any war based story or documentary). Give me a minimap for all of them.

Next-Gen X-ray

I just watched Dark (eh...probably don't do that) and an interactive or responsive family tree next to my screen would have been very useful. Same for understanding the degree of incestuousness found in Game of Thrones and House of the Dragon. I understand how Cersei and Jaime are related. But I could really use a visual family tree to help me understand just how disgusted I should be by the Targaryen incest.

Mystery show? Give me an insane red-string-pinboard, please.

3D Environments

I think my favorite part of my Quest 2 was the home environments. If some Weta artists created a living, breathing, "background", virtual version of the Shire that I could hang out in – I would live there.

Next Level Stuff

For better or worse, a lot of the franchises and stories that have millions of megafans are complex. Plenty of worldbuilding has taken place.

What I want is to be engrossed in a story about one of my favorite worlds when I notice an unexpected name pops up on my map.

"Wait...why did Euron just show up on the corner of the map during this Book-1-era story?!"

These are mostly pipe dreams. For now. The planning, capital, artistry, and licensing/rights required would be tremendous just to take full advantage of today's technology.

Here's hoping lighter glasses are on more faces soon – and that precipitates immersive adventures in our favorite universes.