We present a method for taking an urban scene reconstructed from a large Internet photo collection and reasoning about its change in appearance over time. Our approach estimates when individual 3D points in the scene existed, then uses spatial and temporal affinity between points to segment the scene into spatio-temporal clusters. The result is a set of spatio-temporal objects that often correspond to meaningful units, such as billboards, signs, street art, and other dynamic scene elements, along with estimates of when each existed.

Large collections of Internet photos contain a rich record of places around the world. Recent work in structure from motion (SfM) has made it possible to reconstruct 3D models of landmarks and cities from millions of crowd-sourced photos. However, these reconstructions treat the scene as static, ignoring the fact that the real world changes over time. Buildings are renovated, signs are replaced, murals appear and fade. In this paper, we address the problem of automatically recovering the temporal structure of a scene from unstructured photo collections.

The key challenge is that Internet photos are unstructured: they are taken at different times, by different people, from different viewpoints, under different conditions. There is no explicit temporal ordering, and the timestamps (when available) can be noisy. We propose a principled approach that leverages 3D geometry to connect observations across time and uses appearance-based reasoning to estimate when scene elements changed.

Our method takes as input a 3D reconstruction of a scene computed using structure from motion from Internet photos. Each 3D point in the reconstruction is observed by a set of images, each with an associated timestamp. We formulate the problem of estimating the temporal existence interval of each point as a probabilistic inference problem. Specifically, for each 3D point, we estimate a start time and end time during which the corresponding scene element was present.

The key insight is that nearby 3D points that change at the same time likely belong to the same physical object. We exploit this by defining a Markov Random Field (MRF) over the 3D points, where the unary potentials capture appearance consistency across time and the pairwise potentials encourage spatially and temporally coherent labeling. We solve this MRF efficiently using graph cuts.

After estimating the temporal existence intervals for individual points, we perform spatio-temporal clustering to group points into coherent objects. Two points are connected if they are spatially proximate (within a threshold distance) and have overlapping temporal intervals. Connected components in this graph define our spatio-temporal objects. This simple yet effective approach discovers objects such as changing storefronts, construction sites, seasonal decorations, and rotating art installations.

We evaluate our method on four urban scenes reconstructed from Internet photo collections: Times Square (New York), Trafalgar Square (London), Piccadilly Circus (London), and the Roman Forum (Rome). These scenes span up to 10 years of photos and contain rich temporal dynamics. For each scene, we reconstruct a 3D model using SfM and apply our method to recover temporal structure.

Our method successfully identifies major scene changes with high precision. In Times Square, it correctly detects the replacement of billboards and the renovation of building facades over a multi-year period. Quantitative evaluation using manually annotated ground truth shows that our approach achieves over 80% precision in detecting true scene changes, with significantly fewer false positives than baseline methods that rely on simple appearance comparison without spatial reasoning.

We have presented Scene Chronology, a method for recovering the temporal structure of a scene from large Internet photo collections. By combining 3D reconstruction, probabilistic inference, and spatio-temporal clustering, our approach identifies meaningful changes in urban environments over multi-year timescales. Our results demonstrate that the rich temporal information embedded in crowd-sourced photos can be effectively harnessed for understanding how places change.