Google built a rig with five Pixel 3s to improve the phone’s Portrait Mode photos

While most recent phones come with dual rear cameras to capture depth and deliver Portrait Mode-style shots with the background separated from the foreground, Google’s Pixel phones achieve the same effect with a single lens. That applies to the new Pixel 3 as well, and the company’s now explained how this functionality works – and why it used a rig consisting of five phones to perfect it.

For Portrait Mode-style shots on the Pixel 2, Google used a neural network-powered Phase Detection Autofocus (PDAF) system. It works on a concept called Parallax, in which the camera captures two images from slightly different angles and calculates the movement to perceive depth. But that sometimes doesn’t work, because the camera movement in the user’s hand isn’t enough to allow for shots from different angles.

To fix that, Google made a learning algorithm that corrects the depth perceived by the PDAF system. “Specifically, we train a convolutional neural network, written in TensorFlow, that takes as input the PDAF pixels and learns to predict depth. This new and improved ML-based method of depth estimation is what powers Portrait Mode on the Pixel 3,” Google’s research scientist Rahul Garg said.

The most fascinating thing about these improvements is the method Google used to build the learning algorithm. It made a monster “Frankenphone” rig with five Pixel 3 phones, and used a Wi-Fi-based software tool to make sure all of them captured images simultaneously. This was necessary to capture an object from different angles to teach the algorithm. 

The company’s engineers and researchers took photos with this rig as if they were using just one device. This helped them train the algorithm to predict the depth and separate objects better.

Credit: Google