Scene Completion Using Millions of Photographs

Abstract
What can you do with a million images? In this paper we present a
new image completion algorithm powered by a huge database of
photographs gathered from the Web. The algorithm patches up
holes in images by finding similar image regions in the database
that are not only seamless but also semantically valid. Our chief
insight is that while the space of images is effectively infinite, the
space of semantically differentiable scenes is actually not that large.
For many image completion tasks we are able to find similar scenes
which contain image fragments that will convincingly complete the
image. Our algorithm is entirely data-driven, requiring no annotations
or labelling by the user. Unlike existing image completion
methods, our algorithm can generate a diverse set of results for each
input image and we allow users to select among them. We demonstrate
the superiority of our algorithm over existing image completion
approaches.

Introduction
Every once in a while, we all wish we could erase something from
our old photographs. A garbage truck right in the middle of a
charming Italian piazza, an ex-boyfriend in a family photo, a political
ally in a group portrait who has fallen out of favor [King 1997].
Other times, there is simply missing data in some areas of the image.
An aged corner of an old photograph, a hole in an image-based
3D reconstruction due to occlusion, a dead bug on the camera lens.
Image completion (also called inpainting or hole-filling) is the task
of filling in or replacing an image region with new image data such
that the modification can not be detected.