In June 2023, Tim Cook introduced the Vision Pro as Apple’s first spatial computer; This computer seamlessly blends digital content with the physical world, while allowing users to stay present and connected to others. According to Cook, spatial computing will be the next big era of technology after desktop and mobile computing.

But spatial computing is still a vague concept. Is it augmented reality! Tim Cook also insisted that Vision Pro is an augmented reality (AR) device. Or is it more complex and includes mixed reality(MR), or does it encompass all of these concepts and more?

What is Spatial Computing?

Spatial computing is related to humans and the spaces surrounding us. Any computational system that manages the interaction between humans and the spaces surrounding them is a form of spatial computing, regardless of the technologies used in it, and no matter how simple the interaction.

Thus, the system that turns off the lights when you leave the room is considered a trivial form of spatial computing. Mixed reality applications that build a 3D model of the environment and enable us to interact with it in real-time while taking into account all its physical laws are considered among the most advanced form of spatial computing.

Although credit for the popularization of the term spatial computing goes to Tim Cook; But it was precisely defined in 2003 when Simon Greenwold published a thesis about that, defining it as: “human interaction with a machine in which the machine retains and manipulates referents to real objects and spaces”. 

Greenwald has argued that even automatically flushing toilets is a primitive form of spatial computing because the toilet senses the user’s movement as it moves away and then flushes itself.

Spatial computing is a concept that encompasses a large number of concepts, such as virtual, augmented, and mixed reality, and all the techniques used to implement them, including artificial intelligence techniques, computer vision, 3D modeling among others.

How does Spatial Computing work?

Mixed reality is currently the most advanced application in spatial computing, and the Vision Pro is the most advanced spatial computing device.

The device first builds a 3D model of the surrounding environment using data from various sources, including cameras and different types of sensors. It keeps updating the 3D model in real-time, reflecting changes like people moving through the space.

We can then interact with the reflection of real spaces in the 3D model, so that digital content blends seamlessly with the real world; We can watch a movie on the wall, place a Word document on the table, and an email application on the window.

More advanced interaction with the environment requires not only a 3D representation, but also creating a digital twin of the environment which encompasses a representation of the physical laws. So that throwing a small ball should go exactly as if we throw it in the real world, which is far from being available in Quest 3 or Vision Pro because it requires a lot of computation power.

The Future of spatial computing

Spatial computing offers a rich and unique experience of human interaction with digital content. Its applications range from education, training and work to games and entertainment.

Widespread adoption depends on several factors: price, ease of use and benefits in terms of enjoyment or productivity.

It is clear from the feedback on the Apple Vision Pro headset that we are still at the beginning; Many complain about the discomfort and the lack of productivity, but the majority agreed on the immersive experience of watching 3D movies.

Therefore, the future of spatial computing hinges on software advancements that take into account the preferences and behavior of each user, and hardware innovations that make it comfortable, light, easy to use.

This may require a change in design, materials, batteries, using of simpler wearable devices, faster and more accurate AI models to better interpret human voices, eyes and hands movements and even the processing method [It may use the cloud to perform complex computation].

The price will decrease, the content and use cases will increase, it is only a matter of time.

After Apple launched its headset, spatial computing became a new territory for competition between major tech giants like Apple and Meta. Competition has always been the impetus for developing better products and building creative digital experiences.