MIT researchers have developed a model that recovers valuable data lost from images and video that have been “collapsed” into lower dimensions.
The model could be used to recreate video from motion-blurred images, or from new types of cameras that capture a person’s movement around corners but only as vague one-dimensional lines. While more testing is needed, the researchers think this approach could someday could be used to convert 2D medical images into more informative — but more expensive — 3D body scans, which could benefit medical imaging in poorer nations.
Captured visual data often collapses data of multiple dimensions of time and space into one or two dimensions, called “projections.” For example, X-rays collapse three-dimensional data about anatomical structures into a flat image. Or, consider a long-exposure shot of stars moving across the sky: The stars, whose position is changing over time, appear as blurred streaks in the still shot.
Likewise, “corner cameras,” recently invented at MIT, detect moving people around corners. These could be useful for, say, firefighters finding people in burning buildings. But the cameras aren’t exactly user-friendly. Currently they only produce projections that resemble blurry, squiggly lines, corresponding to a person’s trajectory and speed.
The researchers invented a “visual deprojection” model that uses a neural network to “learn” patterns that match low-dimensional projections to their original high-dimensional images and videos. Given new projections, the model uses what it’s learned to recreate all the original data from a projection.
In experiments, the model synthesized accurate video frames showing people walking, by extracting information from single, one-dimensional lines similar to those produced by corner cameras. The model also recovered video frames from single, motion-blurred projections of digits moving around a screen, from the popular Moving MNIST dataset.
News Source: MIT
