Apple Space Capture — the future of the Metaverse

The last two years I’ve been spending a lot of time exploring how Apples new LiDAR scanner can be used to create digital twins by instantly scanning an environment or an object.

One year ago Apple released a technology called Apple Object Capture that works in a different way, using photographs and cloud-based AI to create 3D geometry.

I am becoming more and more convinced that there is a revolution around the corner, that Apple will announce soon. And it will be called Apple Space Capture.

So first, the difference between the LiDAR scanning method and photo based capture.

What the LIDAR does is using a system like radar or sonar to bounce beams off objects nearby. This lets the computer understand the geometry, its position, size and distance. It then combines this with photos taken by the camera to create 3d models.
The problem with this is while it’s fast, it’s inexact. Scanning with LiDAR is like painting a room with a brush that is constantly full with paint. The first time you paint a wall, you may get a good result. But if you point the scanner toward the wall again, you may mess up the nice smooth wall you had the first time, because the brush won’t stop leaking paint.

LiDAR is great for capturing a room if you have a good technique. But when it encounters geometry that is too complex, the results get really messy. The main problem is that LiDAR is not a smart tool. It does not understand what a wall is.

A year ago Apple released a brand new software called Apple Object Capture. AOC works in a whole different way. It takes a series of photos of an object and sends them to the cloud, where they are analyzed by an AI. Based on the photos, the AI understands the shape of the objects, and creates the correct geometry. And as the AI does this, it gets better at it. When AOC was released, it needed 150 images to make a 3D model. A month later, the requirement was lowered to 50. The Apple system is getting smarter, needing less data to understand geometry.

Apple is going to want to create their own way of creating the Metaverse. Because of course they will. And with the technology they have already developed they can and will create a system that could create perfect digital copies of any space. And that is just the beginning.

Apple Object Capture looks at an object from the outside in. To make a 3D model, you simply walk around the object or rotate it, creating a series of photos from every angle. By analyzing the series of photos the AI understands how it is shaped.

Here’s an example of the same environment scanned with both LiDAR and AOC, using the excellent app Polycam. As you can see the LiDAR scan correctly captures the general shape of the room. But the statue in the middle is too complex, so the scanner just skips it.

Here’s the same environment captured with Apple Object Capture. As you can see, it makes a good 3D model of the complex statue. But it does not look at the room. This is not because it cannot see the room. It is because the AI is looking at the statue.

I am convinced this will change, and soon. I think the Apple technology has matured enough that within the year, a new technology will be released that looks at the entire environment, and creates walls, ceilings and floors of any space it sees. But there is a big difference. And that is that this system will UNDERSTAND it is looking at a wall, or a ceiling, or a floor. Just like the human brain, it will be able to use that understanding to infer information it doesn’t have or can see. A floor stretches all the way to the wall. That’s how floors work. If the part of the floor you can see is wood, all of the floor will probably be wood.

My bet is that this technology will be called Apple Space Capture. And ASC will truly revolutionize the way real life environments can be recreated in virtual reality.

Why revolutionize? Well, the thing is, this system is learning. Which means as it gets smarter, it will need less images to understand a space. It will be able to understand photos that are not taken at the same time, in the same lighting conditions, from correct angles. It can create a detailed 3D model of any environment, using just a few photos. Like… your old family photos, for instance.

Now, I wonder if there is any way Apple could get their hands on a ginormous database of home photos taken the last 20 years… Hmmmm.

Because that would mean that in the future, any space you’ve ever been and taken a few photos of could be made into a 3D model you could walk around in. Think about that for a while. The home you grew up in. The church you married in. Any space you’d ever been and taken a few photos of could be instantly converted into a fully immersive 3D environment. And this is with technology that exists now, today.

I know from experience that there is a huge difference between looking at a photo or an image compared to actually standing in a space using Virtual or Augumented reality. And so I, for one, welcome our new Apple Metaverse Overlords.