We present a technique for optical sensing of vibrations at frequencies up to 63kHz — far beyond the frame rate of the cameras used. Our approach leverages a dual-shutter camera system consisting of a rolling shutter (RS) camera and a global shutter (GS) camera, both operating at only 130 frames per second. The RS camera captures distorted laser speckle patterns that encode high-frequency vibration information due to the sequential row-by-row exposure of the rolling shutter. The GS camera simultaneously acquires an undistorted reference speckle pattern. By comparing the RS and GS speckle images, we recover dense, per-pixel vibration signals from multiple vibrating sources in the scene simultaneously. We demonstrate applications including recovering audio from speech and music, and analyzing vibration modes of physical objects such as tuning forks.

The key insight is that a rolling shutter acts as a natural temporal encoder: because each row of the sensor is exposed at a slightly different time, high-frequency vibrations leave spatially varying signatures across the rows of a single RS frame. However, decoding these signatures is ill-posed without a reference, since the speckle pattern itself is spatially random. The global shutter camera provides exactly this missing reference, enabling robust vibration recovery even in the presence of noise, multiple sources, and complex vibration patterns.

Vibration sensing plays a critical role in a wide range of applications, from structural health monitoring and industrial quality control to audio surveillance and musical acoustics analysis. Traditional vibration sensors, such as accelerometers and microphones, require physical contact with or close proximity to the vibrating object, which limits their applicability in scenarios where the target is remote, fragile, or hazardous. Non-contact vibration measurement via optical methods is thus highly desirable, and has been an active area of research in the fields of computer vision and computational photography.

Existing optical approaches to vibration sensing broadly fall into two categories. High-speed cameras can directly capture vibrations at thousands of frames per second, but they are expensive, bulky, and produce massive amounts of data that are difficult to process in real time. Stroboscopic methods achieve temporal super-resolution by varying the illumination frequency, but they require the vibration to be periodic and can only measure one frequency at a time. More recently, the "visual microphone" approach of Davis et al. recovers sound from subtle visual motion in video, but it is limited to frequencies below half the camera frame rate due to the Nyquist limit, and requires computationally expensive motion magnification.

In this paper, we propose a fundamentally different approach. We observe that the rolling shutter mechanism found in most CMOS sensors can be repurposed as a high-frequency temporal sampling device. In a rolling shutter camera, each row of pixels is exposed at a slightly different time. If the scene contains a vibrating object illuminated by a laser, the resulting speckle pattern will shift differently for each row, creating a spatially encoded record of the temporal vibration signal. By pairing this with a global shutter camera that captures the same speckle pattern without temporal encoding, we obtain a clean reference that enables us to decode vibration frequencies up to 63kHz using sensors running at merely 130Hz. Our system can simultaneously sense vibrations from multiple independent sources in a single scene, effectively functioning as an optical microphone array.

Visual vibration analysis has been explored extensively in the computer vision community. The seminal work by Davis et al. introduced the "visual microphone" concept, recovering intelligible speech from high-speed video of everyday objects such as chip bags and plant leaves. Subsequent improvements by Davis et al. and Wadhwa et al. used phase-based motion processing to amplify subtle motions. However, these methods are fundamentally limited by the Nyquist sampling theorem — vibration frequencies above half the camera frame rate cannot be recovered. While some works have explored temporal aliasing to partially overcome this limit, they require strong prior assumptions about the vibration signal and cannot handle multiple simultaneous sources.

The rolling shutter (RS) effect has traditionally been considered an imaging artifact that causes undesirable distortions such as wobble, skew, and partial exposure. Most prior work focuses on correcting or removing RS distortions. However, a few recent works have begun to exploit RS readout as a source of temporal information. Ait-Aider et al. used RS distortions for motion estimation of rigid bodies, and Saurer et al. showed that RS can aid in visual odometry. Most relevant to our work, Pundlik et al. demonstrated that RS cameras can sense acoustic vibrations up to a few kHz by observing laser speckle shifts across rows. Our work significantly extends this line of research by introducing the dual-shutter framework, which enables robust recovery at far higher frequencies and from multiple simultaneous sources.

Laser speckle imaging is a well-established technique in optical metrology. When coherent laser light illuminates a rough surface, the scattered light produces a random interference pattern known as a speckle pattern. Surface displacement causes the speckle pattern to shift proportionally. Laser Doppler Vibrometers (LDVs) represent the gold standard for non-contact vibration measurement, achieving frequency ranges up to MHz. However, LDVs are point sensors — they measure vibration at a single spatial point and must be scanned across the surface to build a spatial map, making full-field measurement time-consuming. Our approach combines the spatial parallelism of camera-based imaging with temporal resolution approaching that of dedicated vibration instruments.

In a rolling shutter camera, the sensor does not expose all pixels simultaneously. Instead, rows are exposed sequentially from top to bottom, with a constant time delay between consecutive rows known as the row readout time t_r. For a sensor with N rows, the total readout time is T = N · t_r, and the k-th row is exposed at time t_k = k · t_r relative to the first row. When a surface vibrates at frequency f, the laser speckle pattern observed at the sensor shifts in proportion to the instantaneous displacement. Consequently, different rows of the RS image observe the speckle pattern at different phases of the vibration, effectively converting a temporal signal into a spatial pattern across the sensor rows.

The maximum vibration frequency that can be sensed is determined by the Nyquist criterion applied to the spatial sampling of the rolling shutter. Since N rows sample the vibration over the total readout time T, the effective sampling rate is f_s = N / T = 1 / t_r. For a typical CMOS sensor with a row readout time of approximately 8 microseconds, this yields an effective sampling rate of 125kHz, and a maximum recoverable frequency of approximately 63kHz. This is roughly 500 times the frame rate of the camera, demonstrating the remarkable temporal resolution gain achieved by exploiting the rolling shutter mechanism.

When a rough surface vibrates, the scattered laser speckle pattern undergoes a lateral shift proportional to the surface displacement. Formally, if the surface at a point undergoes displacement d(t) at time t, the speckle pattern observed at the sensor shifts by Δx(t) = M · d(t), where M is a magnification factor determined by the imaging geometry and the optical system. In the rolling shutter image, the k-th row observes a speckle shift of Δx(k) = M · d(k · t_r). The challenge is to recover the displacement signal d(t) from the spatially varying speckle pattern in a single RS frame.

Recovering the displacement signal from a rolling shutter speckle image alone is fundamentally ill-posed: the speckle pattern is a random, high-contrast pattern whose spatial structure is unrelated to the vibration. Without knowing the original (undisplaced) speckle pattern, it is impossible to distinguish between spatial features caused by vibration and those inherent to the speckle itself. This is where the global shutter camera becomes essential. The GS camera captures the same speckle pattern at a single instant in time, providing a spatially undistorted reference. Since both cameras view the same scene from nearly identical viewpoints, the GS image serves as a ground-truth baseline against which row-by-row shifts in the RS image can be measured.

The speckle shift between the RS and GS images is estimated using normalized cross-correlation (NCC) computed on small local patches centered around each pixel. For each row k in the RS image, a horizontal strip is cross-correlated with the corresponding strip in the GS image, yielding a sub-pixel displacement estimate Δx(k). By performing this operation for all N rows, we obtain a one-dimensional displacement signal sampled at the effective rate of 1/t_r. The use of NCC ensures robustness to local intensity variations and speckle contrast changes that may arise from differences in the optical paths of the two cameras.

A significant advantage of the dual-shutter approach is its inherent ability to sense vibrations from multiple sources simultaneously. Since the cross-correlation is computed independently for each spatial location in the image, different regions of the sensor naturally correspond to different objects in the scene. When multiple vibrating objects are present, each object's vibration signal is spatially localized in the displacement map, enabling simultaneous recovery without any source separation algorithm. This spatial multiplexing capability is analogous to a microphone array but with far denser spatial sampling — one "microphone" per pixel.

Once the per-row displacement signal Δx(k) is obtained from cross-correlation, the vibration frequency content is recovered via spectral analysis. Applying a discrete Fourier transform (DFT) to the displacement signal yields the vibration spectrum, with frequency resolution determined by Δf = 1/T = 1/(N · t_r). For a sensor with 1024 rows and t_r = 8μs, the frequency resolution is approximately 122 Hz, sufficient for most vibration analysis and audio recovery applications.

To recover time-varying vibration signals rather than static spectra, the system processes consecutive RS-GS frame pairs. Each frame pair yields one snapshot of the displacement signal, and the sequence of snapshots over time provides the temporal evolution of the vibration. A short-time Fourier transform (STFT) applied across frames produces a spectrogram that reveals how the vibration frequency content changes over time. For audio recovery applications, the displacement signal from each frame is treated as a short segment of the acoustic waveform, and consecutive segments are concatenated with overlap-add processing to reconstruct a continuous audio signal.

The practical implementation requires careful calibration of the dual-camera system. The two cameras must be spatially co-registered so that corresponding pixels observe the same scene point. This is achieved through stereo calibration followed by image rectification and warping. Additionally, the temporal synchronization between the RS and GS cameras must be precise — a synchronization error of even a few microseconds could introduce systematic bias in the displacement estimates. The system uses a hardware trigger to synchronize the start of each frame acquisition, achieving sub-microsecond timing accuracy. The laser illumination is provided by a continuous-wave laser coupled with a beam expander to illuminate the region of interest.

The experimental prototype consists of a FLIR Grasshopper3 rolling shutter camera and a FLIR Blackfly S global shutter camera, both fitted with identical 25mm lenses and mounted on a custom bracket to minimize the baseline between them. The cameras operate at 130 frames per second with a row readout time of approximately 7.85 microseconds, yielding a theoretical maximum recoverable frequency of 63.7 kHz. Illumination is provided by a 532nm continuous-wave laser with a beam expander. The system is controlled by a PC that issues synchronized hardware triggers to both cameras and records the image pairs.

In the first set of experiments, we analyze the vibration modes of tuning forks. A standard 440 Hz (A4) tuning fork is struck and imaged by the dual-shutter system. The recovered vibration spectrum shows a clear peak at 440 Hz with harmonics visible at 880 Hz, 1320 Hz, and higher. By computing the vibration spectrum at each spatial location, we generate a spatial vibration mode map that reveals the expected pattern of nodes and antinodes along the tuning fork prongs. When two tuning forks of different frequencies (440 Hz and 523 Hz) are simultaneously struck and placed in the scene, the system successfully separates and identifies both fundamental frequencies and their spatial distribution, demonstrating the multi-source capability.

The most compelling demonstration is audio recovery from speech and music. A loudspeaker playing recorded speech is placed behind a thin reflective surface (such as a piece of aluminum foil or a plastic bag). The vibrations induced on the surface by the sound waves are captured by the dual-shutter system. From the recovered displacement signals, we reconstruct intelligible audio that preserves the speaker's identity and prosody. Similarly, for music played through a speaker, the recovered audio faithfully reproduces the melody and harmonic structure. We also demonstrate recovery of live human speech by directing the laser at a surface near the speaker — the recovered signal, while noisier, remains intelligible and captures the fundamental frequency and formant structure of the speech.

To validate the system's capability at the upper frequency limit, we test with ultrasonic vibration sources operating at 20kHz, 40kHz, and beyond. Using a piezoelectric transducer driven at known frequencies, we confirm that the system accurately recovers vibration frequencies up to approximately 63kHz, consistent with the theoretical prediction. At frequencies approaching the Nyquist limit, the spectral peaks become broader due to the finite window length, but remain clearly identifiable. Below 50Hz, the system's frequency resolution is insufficient for accurate recovery, representing the practical lower bound. This frequency range of approximately 50Hz to 63kHz covers the vast majority of acoustic and structural vibration applications.

We have presented Dual-Shutter Optical Vibration Sensing, a technique that repurposes the rolling shutter mechanism — traditionally considered an imaging defect — as a high-frequency temporal encoder for vibration measurement. By pairing a rolling shutter camera with a global shutter camera, both operating at a modest 130 frames per second, we achieve vibration sensing at frequencies up to 63kHz, representing a roughly 500-fold improvement over the camera frame rate. The system enables simultaneous sensing of multiple vibration sources with dense spatial sampling, effectively functioning as an optical microphone array with one sensor per pixel.

Our work opens several promising research directions. Extending the approach to work with ambient light instead of laser illumination would dramatically broaden its applicability, though this requires overcoming the much weaker displacement signals in natural scenes. Combining our spatial encoding with learned priors from deep neural networks could enable vibration recovery from even noisier measurements. The dual-shutter principle could also be implemented in a single camera with a programmable shutter, reducing the system complexity. We believe this work demonstrates the broader principle that computational photography techniques can transform sensor limitations into sensing capabilities, suggesting that many other "defects" in imaging systems may be repurposed for novel sensing modalities.