“Hallucinating” a face, new software could have ID’d Boston bomber

Facial recognition technology available to law enforcement officials in April failed to identify Dzhokhar Tsarnaev from security camera images in time to prevent a public appeal—and the subsequent murder of an MIT police officer and shootout between the Tsarnaev brothers and police. But experimental technology from Carnegie-Mellon University’s CyLab Biometrics Center came tantalizingly close. The software being developed there could soon make facial recognition software much more powerful in generating leads for law enforcement.

The technology, called single image super-resolution, will be highlighted tonight on PBS in an episode of NOVA that looks into the science behind the Boston Marathon bomber manhunt. In a phone interview with Ars, Dr. Marios Savvides, the director of the CyLab Biometrics Center, said that the new technology could generate results much more detailed than those made by traditional image enhancement approaches. “The traditional methods yield about a 2 times to 4 times improvement” in the resolution of a facial image, he said. “This method gets us 16 times the resolution.”

Faces from pixels

CyLab, which has participated in a number of federally sponsored biometrics projects, had submitted a proposal to the Department of Justice to perform a trial of the new technology just a month before the Boston bombing, Savvides said. “We were working on the system,” he said, “but we weren’t even ready for a demonstration.”

But as the FBI published the photos of the suspects on Thursday, April 18, Savvides’ team decided to try to help authorities in making an identification of the suspect that would later turn out to be Dzhokhar Tsarnaev.

“What the FBI released is what they had and couldn’t do anything with,” Savvides said. “After [the Tsarnaevs] were ID’d on April 19, more images came to light. But the images available on April 18 were very bad. The imagery was too small for current technology—the information just wasn’t there [in the images].”

Hoping to aid law enforcement, Savvides’ team took the released photo from the FBI website and ran it through an early version of the enhancement software. The software is a machine learning system “trained” with a database of 30,000 faces presented in multiple resolutions. The algorithms constructed through training can draw from the system’s experience and reconstruct an approximation of a face based on patterns within facial images, with as little as six pixels between the eyes of a suspect. As a result of the training, the software can essentially reconstruct a face based on the relationship between pixels and human-assisted identification of facial landmarks, producing what Savvides called a “hallucination” of the individual’s face from negligible amounts of image data.

To assist the algorithm, Savvides and his team took the one mostly front-facing image of Dzhokhar Tsarnaev and estimated where “landmarks” on his face were, giving the software 79 distinct points to use as a starting point. By Friday morning, they had a result. The CyLab team sent off an image, but the gun battle in Watertown had already happened and the suspects had been identified.

The image “hallucinated” by CyLab’s single-image super resolution software, compared to an extraction from a high-resolution image of Dzhokhar Tsarnaev.

Credit: Carnegie-Mellon University

After the fact, Savvides’ team tested to see how successful their efforts would have been if they had been able to apply the image to an image database. Embedding a higher-resolution face-on image of Dzhokhar Tsarnaev’s photo into a database of 50,000 mug shots provided by the Pinellas County Sheriff’s Department, they used the software-constructed image to check if they could get a match. Filtering results by “soft biometrics”—such as age range, gender, and race—Tsarnaev came up in the search as the 11th most likely match. Using similar approaches, the image was thrown at a database of 1 million mugs used in a National Institute of Standards and Technology facial recognition challenge in 2005; Tsarnaev was the 19th result.

A screenshot of the search results using the extrapolated image of Dzhokhar Tsarnaev against a 50,000 image mug shot database. Tsarnaev’s high-resolution mug shot ranked at 11 in the search.

Credit: Carnegie Mellon University

Pwned by computers

Savvides said that his team is working on further reducing the potential errors that come with human input of “soft biometrics” by automating image classification at capture—deciding gender, age, and race data based on algorithms rather than human input to maintain consistency and improve searches.

CyLabs is working on even more sophisticated methods of visual biometric identification, too. Savvides said the team is constructing facial recognition systems that can make an identification based solely on eyes and eyebrows. Another area of research is how to take a view of half a face—such as a profile view—and create a full face-on view based on computer learning that guesses at the other half, including facial asymmetry.

Facial recognition systems are already becoming effective at assigning age and race; other systems use “multimode” biometrics such as gait and other pattern analysis to take fairly accurate guesses at nationality. But as NOVA’s Miles O’Brien put it in a conversation with Ars, “faces aren’t snowflakes”—they don’t provide the sort of absolute identification that fingerprints and DNA do. Still, there is one visual biometric that does offer something close to the accuracy of fingerprints—iris recognition.