An Open Source Look at iOS 27 and visionOS 27 Object Tracking

Just here for the code? Here’s my open source visionOS 27 and iOS 27 object tracking demo.

It’s been two years since object tracking arrived in visionOS 2 when I listed the many caveats & gotchas that came with building object tracking experiences.

The good news: WWDC26 gifted us with object tracking updates that obviate two of the caveats & limitations I ran up against (i.e. caveats 6 and 7, which were both summarily addressed by high-frame-rate tracking), while also expanding to a new platform: iOS. The end result: tracking of objects is faster and more accurate.

The middling news: The bulk of my caveats listed in 2024 still stand – there hasn’t been a change to the type, texture, and size of objects that can be used as inputs for object tracking.

I took the excuse of improved performance to explore a simplified accessibility use case I wondered about in 2024: Could the Vision Pro be my eyes?

I built a project that runs different functions on separate devices: an iPhone and a Vision Pro. After calibrating the devices by having them look at the same object, they both understand the space around the Vision Pro. The iPhone user can then drag three courses onto the table to make a meal, and the Vision Pro user can listen for auditory cues to match the courses’ placements. No peeking necessary.

So: Yup! It can. This demo really does need sound, which you can watch on YouTube, but here’s a muted clip of that demo:

The demo also shows a side-by-side comparison of the worst-case object tracking setup vs. the best-case set of parameters

Before we get to the fun part: What’s actually new in object tracking this year?

iOS 27 & visionOS 27 Object Tracking Improvements

We got five meaty upgrades in this year’s releases:

1. Object tracking is now available on iOS (previously, iOS was limited to object detection – the dullard [slower, less useful] version of object tracking)
2. High frame rate tracking is available for everyone on visionOS 27, no Enterprise API entitlement needed
3. macOS 27’s Create ML now has an extended training mode, which takes significantly longer but results in a more accurate tracking model
4. A new metric coordinate system provides poses unaffected by display correction, handy when very precise measurements are needed such as for medical use cases
5. All models trained on macOS 27’s Create ML finish training faster and have some non-zero tracking accuracy improvement compared to models trained in years past

For visionOS, high-frame-rate tracking is the real standout. The virtual object, or where the OS understands the tracked object to be, won’t be perfectly glued to the real world object…but things are looking up. A side-by-side comparison (as shown in the demo on YouTube) makes it obvious how much better tracking is now. High frame rate tracking can be enabled on models trained years ago or brand new models. The new setting is power-hungry though, and I butted up against a limitation I didn’t see in any Apple documentation: On visionOS 27, only six models max can have high-frame-rate tracking enabled at a time. Attempting to load an immersive space with seven models being tracked at high frame rates resulted in a crash using Xcode 27 Beta 6.

For iOS, you can use the same .referenceobject files you trained for visionOS. There’s no retraining needed. And there’s also no high-frame-rate flag on iOS: a file in ARWorldTrackingConfiguration.trackingObjects is tracked at 60 frames per second, faster than my M2 Vision Pro’s 30 fps object tracking.

I haven’t yet spent enough time directly comparing models that were trained using standard mode versus the new extended mode to see what the latter buys us. What I can say: Apple wasn’t kidding when they said extended mode training takes significantly more time. On my 2024 MacBook Pro I saw training times seven times longer when training in extended mode compared to standard mode for the exact same 3D object.

Object Tracking Training Durations

These are the times I saw and some of my unscientific, small-sample-size conclusions:

Object	Trained	Trainer	Mode	Angles	Mac	Time
Blue carton (Fairlife 2%)	Jun 2024	macOS 15	Standard	All Angles	M2 Ultra	12 h 38 m
Cap’n Crunch	Jun 2024	macOS 15	Standard	All Angles	M2 Ultra	12 h 39 m
Dutch oven (2024 demo only)	Jun 2024	macOS 15	Standard	Upright	M2 Ultra	11 h 29 m
Red carton (Fairlife whole)	Aug 2026	macOS 27	Extended	Upright	M4 Max	29 h 00 m
Quaker protein granola	Aug 2026	macOS 27	Extended	Upright	M4 Max	28 h 16 m
Oreo bars	Aug 2026	macOS 27	Standard	Upright	M2 Ultra	4 h 35 m
Apple sauce	Aug 2026	macOS 27	Standard	Upright	M4 Max	4 h 09 m
Chobani Greek yogurt	Sep 2026	macOS 27	Standard	Upright	M4 Max	4 h 02 m
Fruit Roll-Ups	Sep 2026	macOS 27	Standard	Upright	M2 Ultra	4 h 41 m
Oreo bars (control run)	Sep 2026	macOS 27	Standard	All Angles	M2 Ultra	4 h 51 m

Comparison	Times	Result
Extended vs Standard	29 h 00 m and 28 h 16 m vs 4 h 09 m and 4 h 02 m	~7x longer
macOS 27 vs macOS 15	4 h 35 m vs 11 h 29 m	~2.5x faster
All Angles viewing angles vs Upright	4 h 51 m vs 4 h 35 m	~6% longer
M4 Max 128 GB vs M2 Ultra 192 GB	4 h 09 m and 4 h 02 m vs 4 h 35 m and 4 h 41 m	~12% faster

The two machines used for training:

• 2024 MacBook Pro, M4 Max, 128 GB
• 2023 Mac Studio, M2 Ultra, 192 GB

A nice improvement, for anyone training object tracking models in 2026 and beyond, is that Create ML now appears to train models a full 2.5 times faster than it did in 2024.

So…What Can We Do With This?

I’ve been thinking about how the Apple Vision Pro could be useful for accessibility purposes (due in no small part to my legal blindness, this NYMag article, and the latent capabilities of the device surfaced when trawling Apple documentation). With faster, more capable object tracking now available I wondered: Could the Vision Pro help me perform some simple tasks with my eyes closed?

That question is at the heart of my demo project. The naive answer: Yes, an Apple Vision Pro (or similar hardware) can serve as my eyes in a novelty, contrived situation. I don’t know how many blind people are interested in having an aide dictate what they eat, where such boxed food goes on a table, and then have the same aide walk away or watch without helping. My guess is: Not many.

And I read that NYMag article two years ago – maybe an app came out long ago that is already doing some version of what my demo does.

But there are elements here of dividing tasks (object selection & setting a placement goal vs. retrieval & guiding actual objects to their goal) and doing so using auditory cues “alone” (emitted by a $3,700 device) that I find interesting.

As I go deafer and deafer (largely due to thousands of concerts attended decades ago, moronically, without earplugs), I find myself getting more and more excited by old-people gadgets. Things for olds like me and my parents and others with accessibility needs. Things like hearing aid glasses.

I’m all here for bringing more novel accessibility capabilities to more people, as soon as possible. We’re all going to need them sooner or later!