Cameron may have never imagined that the 3D wave initiated by Avatar would reignite years later in the XR field in the form of spatial videos and spatial photos.

This time, the main characters in the filming are no longer those movie teams that always produce big productions, but rather every ordinary yet extraordinary creator.

That’s right, it’s you and me who are reading this article.

Why do you say that?

Let’s not mention anything else, just take a look at the iPhone. From 15 to 16, there have been relentless hardware upgrades, focusing on the buttons and photography.

To enable better spatial photography, Apple even made the “contrary to tradition” decision to adjust the camera module of the iPhone 16 series from a diagonal arrangement to a parallel one 🫣.

This fully demonstrates the importance of spatial video to Apple.

With Apple taking the lead, various manufacturers have followed suit. Look, Meta next door hurriedly added the ability to view spatial videos and panoramic images to Quest.

However, after experiencing it, when it comes to spatial content, we have to use Xu Xin’s famous saying to evaluate it—PICO 4 Ultra is still too comprehensive.

This is comprehensive, not only in terms of playback but also includes content production and interaction details among other aspects.

Play: Worry-free

Let’s talk about playback.

To play the spatial video already recorded on the iPhone on PICO, simply open the PICO App on the iPhone and select Import from Album.

Done. Next, you can directly immerse yourself in the spatial content captured on your phone using PICO.

You don’t have to worry about finding a player to be compatible with playing MV-HEVC, the new format launched by Apple; PICO has quietly taken care of everything for you.

All you have to do is put on the PICO and enjoy it all.

Don’t worry about the content taking up more space; the MV-HEVC format is inherently a format with a very high compression ratio.

Compared to the previous Side By Side stereoscopic video, MV-HEVC intelligently only saves the main image and the residual values used to reconstruct the secondary image, as well as various parameters needed to restore the stereoscopic effect.

Talk is cheap? Here we go 3, 2, 1, let’s connect

In this demo provided by Apple, we can see a side-by-side original video of 14.8MB, which has been compressed to 3.8MB after MV-HEVC encoding, rounding that means it has been compressed by five times!

In addition, the MV-HEVC compatibility is also very good; devices that do not support stereoscopic playback will treat it as a regular video, meaning that after shooting, it can also be viewed as a regular video on a mobile phone.

High compression ratio, high compatibility; if we say that the standard encoding format for future VR and immersive video is it, it would not be an exaggeration at all.

What?

Bought a new iPhone but haven’t tried this feature yet? Let’s review how to take spatial photos and record spatial videos for Apple Vision Pro using the iPhone camera, along with the new spatial video feature on iOS 17.2. Since you’ve already bought it, you might as well enjoy it a bit more for that ten bucks, right? 😉

According to reliable information, in order to make spatial video playback smoother on PICO, PICO has once again collaborated deeply with Qualcomm to jointly develop and has made significant modifications to the existing video codec firmware, drivers, and Android Codec2 framework of the 8Gen2 to support the full link of MV-HEVC encoding and decoding.

In addition to encoding and decoding, PICO has also quietly optimized some playback details behind the scenes.

If you have read our previous popular science articles, you would know that the sense of three-dimensionality in spatial video comes from the parallax created by the images received by the left and right eyes.

The principle is very simple, but in practice, unreasonable parallax and viewing sizes can easily cause discomfort for users.

To ensure user experience, PICO automatically adjusts the parallax based on the metadata in the video when playing spatial videos.

For example, during playback, PICO adjusts the image playback scale based on the overall depth of the shooting scene, modifying the image scale and left-right parallax offset to ensure that close-up viewing does not easily lead to fatigue, and that distant views do not produce stretching distortion.

Oh right, it’s not just spatial videos; on PICO, the spatial photos and panoramic photos you take can also be automatically imported for viewing on PICO, just like spatial videos.

At this point, compared to the neighboring Meta, which can only import panoramic photos and spatial videos, PICO has thoughtfully taken a step further.

Production: With Care

If it can only play spatial content, then that would be too boring.

As an MR device equipped with two big-eyed glasses from Water Wang Wang, how could PICO resist the urge to take photos themselves?

Isn’t this pair of big eyes watery? 👀

After all, compared to the two lenses on a smartphone, the distance between the two lenses on a headset device is closer to the distance between human eyes, and both lenses have the same specifications, resulting in a stronger sense of three-dimensionality. Moreover, holding a smartphone for an extended period can easily lead to fatigue.

However, wearing the PICO for an extended period may cause the battery to drain first.

However, wearing it on the head is different from wearing it on the hand; the frequent shaking of the head itself can cause the image to shake violently as well. A few years ago, a first-person filmed masterpiece, “Hardcore Henry,” is a typical example.

Look at this violently shaking first-person perspective; it is definitely excellent material for experiencing 3D dizziness.

Not only that, but there is also the jelly effect that comes with the shaking image.

In simple terms, the jelly effect occurs because the current mainstream cameras use line-by-line exposure. This leads to unexpected distortions in the final image when the content in the frame moves quickly, faster than the line-by-line exposure speed. In spatial video shooting, the negative experience caused by the jelly effect becomes even more pronounced due to the enlarged frame in front of us.

PICO has considered both of these issues and is also working hard to address them.

Thanks to the various sensors on the PICO device, the PICO 4 Ultra utilizes 6DOF information combined with images and the graphics processing unit (IMU) for joint stabilization, along with pose prediction of future image frames, intelligently planning the optimal stabilization path to make the captured footage more stable.

The solution to the jelly effect in the PICO 4 Ultra employs a fine mesh overlay, high-precision removal of the jelly effect in images, combined with anti-shake technology to optimize overall performance.

With the support of anti-shake technology, it’s still not enough; if any of the aspects such as image quality, frame rate, color, or audio have shortcomings, the final production experience will be compromised.

After all, as content that can only be experienced on a headset, any aspect that brings a poor experience is magnified right in front of your face.

In these aspects, the PICO 4 Ultra has also been thoroughly refined.

With a deep understanding of Qualcomm’s chip, the PICO 4 Ultra can record spatial videos at a frame rate of 60fps, with a single-eye resolution of 2048 * 1536, achieving clear point-to-point display at a 105° FOV.

In terms of color, for ordinary photographers, the most important aspect of spatial video is a sense of realism, which often comes from the colors in the image.

The quality of color grading in the same scene can significantly affect the final output

The PICO 4 Ultra uses a 3DLut solution, incorporating elements such as memory colors and preferred colors to adapt to various ambient lighting conditions, making the playback images more aligned with the public’s impression of everyday scenes, enhancing user enjoyment and comfort while viewing.

In terms of audio, with the four microphones on the PICO 4 Ultra, the PICO 4 Ultra can record audio in Ambisonics format through model training, allowing users to be more immersed while watching spatial videos.

In subsequent OTA upgrades, the PICO 4 Ultra will also support MFNR, the first noise reduction capability of the Qualcomm platform. This technology can achieve 0 latency capture effects, with a fully hardened algorithm, ISP, and a separate visual computing chip working together, resulting in faster multi-frame processing speeds and lower power consumption.

Interaction: Thoughtful

Finally, it’s the interaction details.

Although much cheaper than the Apple Vision Pro, the PICO 4 Ultra also strives to match the Apple Vision Pro in terms of interaction with spatial content.

You can also watch independently in windowed mode. In this mode, PICO will blur around the edges of the window to reduce the sense of depth conflict and ghosting.

You can also use immersive mode, in which the dynamic blurred edges that spread outward compensate for the insufficient recording FOV and enhance the sense of immersion, while the built-in super-resolution algorithm of PICO 4 Ultra can better restore image clarity.

In terms of interaction, to allow users to focus more on shooting, the PICO 4 Ultra has removed the “shoot” button found in traditional camera applications. There is no need to aim a ray at the button to click and trigger the photo; instead, users can simply press the Trigger key to take photos and record videos, allowing them to concentrate on the subject being captured.

This design is also carried over to the switching between photo and video modes. The buttons for the two modes no longer require precise clicks; instead, as long as they are associated with this “hot zone,” switching can be done easily.

Note that this man’s name is Xiaoshuai. The hot zone in the image does not require very precise aiming with the controller; as long as you are generally near the hot zone, you can select it directly.

In addition, if you enable the screen-off shooting mode and remove the headset, the camera function can still operate normally, allowing you to take photos and record videos while holding the headset.

With the powerful ecosystem of Douyin, after you finish shooting with the PICO 4 Ultra, you can share it to Douyin with one click, allowing more people to see your work.

In subsequent OTA upgrades, you can even edit spatial content and perform 3D effects editing directly on the PICO 4 Ultra. It can be said that from production to creation to viewing, PICO has provided a basically complete one-stop process.

Future: Promising

In fact, when spatial video first came out, its unique sense of immersion was like a new type of “time capsule.” Every time I revisit spatial videos shot a few months ago, I feel a sense of time travel back to the scene at that moment.

A new video format requires not only support from the playback end but also support from the production end; in addition to the release of everyday content, it also needs to have the leading role of large productions.

In this regard, PICO has an inherent advantage, not only possessing consumer and production products like the PICO 4 Ultra but also professional works such as “My Millionaire Wife.”

To encourage creators, PICO has also launched activities like the PICO China Mountain and Sea Tour Message in a Bottle Program to inspire everyone to unleash their creativity, with a theme that aligns well with the scenario of spatial video—nostalgia.

In the year 2024, where short and long videos have become stable, spatial video is one of the few unfinished rooms in the stable video creation building 🤩. Grab your PICO or the iPhone in your hand and start shooting!