We present an approach towards streaming perception, the problem of real-time understanding of dynamic visual scenes that considers both accuracy and latency as a joint optimization target. Current benchmarks evaluate detectors on static images without considering the delay between image capture and result availability. In a streaming setting, this delay means that by the time results are available, the world has moved on, rendering the results outdated. We introduce a meta-benchmark that jointly evaluates the accuracy and latency of any detection algorithm by simulating the streaming setting, and show that the optimal detector under streaming evaluation is very different from the optimal detector under traditional offline evaluation.

The dominant paradigm in perception for autonomous systems treats vision as a sequence of independent frames: capture an image, process it through a detector, and output results. But in the real world, perception is inherently a streaming process — the environment continuously changes while the algorithm is computing. A detector that takes 100ms to process a frame will return results that describe the world 100ms in the past. For a vehicle traveling at 60 mph, this corresponds to nearly 3 meters of travel, making the results potentially dangerous for downstream planning.

We argue that the community needs a fundamental rethinking of how perception systems are evaluated. Standard benchmarks like COCO and Cityscapes evaluate detectors on static snapshots, rewarding accuracy without penalizing latency. This creates a perverse incentive to build increasingly complex and slow models. We propose a streaming evaluation protocol that measures the accuracy of a detector's output at the time it becomes available, against the ground truth at that same time, naturally incorporating the cost of latency.

We formalize the streaming evaluation as follows: given a continuous video stream, a detector processes frames and produces outputs with some latency. The streaming AP (sAP) evaluates each output against the ground truth at the time the output becomes available, not when the input frame was captured. This means a fast but less accurate detector may score higher than a slow but more accurate one, because the fast detector's results are more temporally aligned with reality. We further introduce forecasting-based methods that predict where objects will be at a future time, compensating for computational delay.

We evaluate a range of detectors under the streaming setting using the Argoverse-HD dataset. Under standard evaluation, heavier models (e.g., Cascade R-CNN with ResNeXt-101) achieve the highest AP. However, under streaming evaluation, lighter models (e.g., SSD with MobileNet) achieve comparable or better streaming AP due to their lower latency. The ranking of detectors is almost completely reversed: the best offline detector drops to near the bottom in the streaming ranking. Adding simple linear motion forecasting improves streaming AP by 2-5 points across most models.

We also analyze the accuracy-latency trade-off in more detail. By varying the backbone architecture and input resolution, we trace out a Pareto frontier in the accuracy-latency space. Under streaming evaluation, the optimal operating point shifts dramatically towards faster models. We find that reducing input resolution is often more beneficial than using a lighter backbone, as it provides a larger latency reduction per unit of accuracy loss. These findings have significant implications for the design of real-time perception systems.

We have introduced the problem of streaming perception and proposed a principled framework for jointly evaluating the accuracy and latency of perception systems. Our key finding is that the optimal detector under streaming conditions is fundamentally different from the optimal detector under traditional evaluation, challenging the current research paradigm. We hope this work will inspire the community to rethink the design and evaluation of real-time perception systems, giving equal consideration to latency as to accuracy.