Before the launch of PICO 4 Ultra, whenever PICO was mentioned, the first thing that came to mind was: a great VR gaming console. This also led many users to overlook the more significant MR capabilities in PICO 4 Ultra.

This was understandable, after all, whether it was the previous PICO Neo 3 or PICO 4, their official positioning was always as a “VR headset.” It wasn’t until PICO 4 Ultra that a new positioning - MR headset - gradually emerged 👀.

Compared to VR, which is basically only used for gaming, MR undoubtedly has much more room for imagination - after all, no matter how good games are, we won’t dedicate all our time to them. Applications are the main theme of our internet life.

So, as the first device bearing the “MR” title, aiming to let you “shine and immerse,” what kind of environmental awareness capabilities does PICO 4 Ultra offer to developers and users?

Let’s start with the conclusion: After the release of PICO 4 Ultra, developers can access more spatial awareness capabilities from the PICO system. (This can be glimpsed from the PICO Developer Documentation changelog)

Before PICO 4 Ultra, developers only had access to video see-through, spatial anchors, and scene calibration functions.

Video see-through needs no explanation - it allows developers to let users see their surrounding environment in their applications, enabling quick switching from VR mode to MR mode.

Spatial anchors refer to coordinates provided by the system that allow developers to fix virtual objects at positions in the real world corresponding to these coordinates.

This is a basic capability in the MR world. The anchors provided by PICO 4 Ultra will build an anchor map around the anchor point. This means that even if you leave the current room, the previous anchors won’t drift when you return - this is the crucial memory recall capability of spatial anchors.

Using such a stable anchor capability, some very basic placement apps can be created. For example, a cute weather balloon that can be placed next to a table will quietly wait for you to play with it by the table, no matter how you move:

The anchors provided by PICO 4 Ultra will build an anchor map around the anchor point. This means that even if you leave the current room, the previous anchors won’t drift when you return.

Scene calibration is where the system provides developers with the positions of the ceiling, floor, and walls in the current room. Developers can use this data to get a more accurate room layout for more precise placement of virtual objects. For example, based on the positions of room walls and ceiling, it’s easy to create a relaxation app that uses the ceiling as a screen for when you’re lying down:

Note that before enabling this feature, users need to actively scan their current room (this needs to be enabled in Settings -> Room Calibration):

Of course, for better user experience, developers can also actively call the scanning process through PICO’s API in their applications, guiding users through the system UI to complete the scan and use this data in their applications.

Now, with the arrival of PICO SDK 3.0, PICO provides us with more MR capabilities, including shared spatial anchors and real-time mesh.

Feynman once said that if we only know the name of something, we essentially know nothing about it. So what exactly do these PICO MR capabilities mean?

First is shared spatial anchors, which is most closely related to the spatial anchors mentioned earlier. With this feature, when Xiao Shuai places a weather balloon on their desk, Xiao Mei, who is also wearing a PICO 4 Ultra, can see the same weather balloon in the same location.

Not only that, but based on this capability, whether it’s playing a board game with friends:

Or designing something with colleagues, it’s all possible:

Behind all this, PICO quietly handles the uploading and synchronization of local spatial anchors.

With spatial mesh, more imaginative scenarios will be discovered. Spatial mesh refers to PICO 4 Ultra’s ability to create 3D models of areas captured by the camera, gradually generating a series of triangular meshes to simulate the surface shapes of the real world.

In actual scenarios, if we display these triangular meshes specifically, it would look like this:

You might ask, what’s the use of this? Well, it’s actually quite significant.

First, having the shape of the real world allows us to add any filter to reality. For instance, you could instantly enter a Matrix-like world:

Or, if you prefer, you could create a slightly suspenseful app where you can only see the world around you when you clap your hands:

If you want, you can also make virtual objects interact with the real world in a more realistic way, like this almost indistinguishable silk:

It’s precisely because of these new ways of perceiving the real world that PICO 4 Ultra can significantly optimize some existing experiences.

For example, previously, when first putting on PICO and entering a game, we had to manually adjust the height of the safety zone recognized by PICO. Now, using this capability, the safety zone can automatically detect height, requiring only final confirmation from us:

For spatial audio, which is crucial to the experience, mixed reality spatial audio can be achieved by adding preset models to the real world’s spatial mesh to adjust effects like audio reflection based on the real scene. This gives you different reverb effects in different rooms, further enhancing the immersive experience of mixed reality.

Additionally, in the original VR mode, users’ range of movement was fixed, and once users stepped out of the VR safety boundary, they couldn’t interact with virtual objects even if they hadn’t touched any real objects. This obviously isn’t intuitive for MR applications where users can directly see the real world.

Fortunately, using spatial mesh, PICO 4 Ultra can provide new MR safety protection functionality for MR applications. In MR safety protection, the system only makes objects transparent to warn users when there are large virtual objects blocking their vision and they’re approaching obstacles.

Moreover, when the controller approaches the edge of virtual objects, the virtual objects around the controller will automatically become transparent.

Well, these are all the new MR capabilities that PICO 4 Ultra currently possesses. With these enhanced MR capabilities, the positioning as an “MR headset” is basically secured. Now it’s up to developers to quickly bring their creative ideas to life. Let’s wait and see 👀.

Author

SpatialOnee