ToF ReSTIR: Time-of-Flight Rendering with Spatio-temporal Reservoir Resampling

Abstract

We present a novel spatio-temporal reuse framework for time-resolved light transport, enabling efficient Monte Carlo rendering of time-of-flight (ToF) phenomena such as time-gated imaging and transient light capture. Existing ToF rendering methods are computationally expensive, scale poorly to complex dynamic scenes, and are therefore unsuitable for applications with strict latency constraints. To address this limitation, we draw inspiration from \emph{ReSTIR}, a reuse-based technique for steady-state real-time rendering, and adapt its core principles to interactive-rate ToF simulation. However, naively applying existing ReSTIR methods to ToF rendering leads to severe inefficiency, as reused paths frequently violate optical path-length constraints and thus contribute little or no signal. We overcome this challenge by introducing a path reuse formulation that explicitly enforces physically valid optical path lengths. The key idea is \emph{path-length-aware shift mapping}, a geometric transformation based on Newton’s method that adjusts reused light paths to satisfy temporal gating constraints, inspired by specular manifold exploration in steady-state caustics rendering. The resulting framework substantially improves the efficiency of ToF rendering across a wide range of scenarios, including complex scenes with glossy or specular materials and dynamic motion. Our method supports both time-gated and transient rendering at interactive frame rates, enabling simulation under practical latency constraints. We demonstrate the effectiveness of our approach through two downstream applications, including shape reconstruction and navigation.

Time-of-Flight Imaging and Rendering

Time-of-flight (ToF) imaging measures scene geometry by observing when light returns to the sensor after being emitted from a source. Instead of measuring only the total steady-state radiance, ToF cameras capture a time-resolved light transport signal, where different path lengths arrive at different times.

Example transient light propagation in time-of-flight imaging. Video adapted from this YouTube video .

In standard rendering, the image is obtained by integrating contributions over all light paths over path space $\mathcal{P}$:

$ I = \int_\mathcal{P} f(\bar{\mathbf{x}}) \, d\bar{\mathbf{x}}. $

In ToF rendering, this steady-state integral is replaced by a time-gated path integral, where each path contribution is weighted according to its path length:

$ I_\tau = \int_\mathcal{P} f(\bar{\mathbf{x}}) W_\tau(|\bar{\mathbf{x}}|) \, d\bar{\mathbf{x}}. $

Our goal is to evaluate above ToF path integral efficiently in an interactive-rate using ReSTIR-based method.

Path-Length-Aware Shift-Mapping

ToF rendering requires sampled light paths to satisfy strict path-length constraints imposed by the temporal gate. Conventional ReSTIR reuse strategies often violate these constraints, leading to inefficient estimates with a lot of invalid samples that falls outside of the time gate. Our method introduces path-length-aware shift mapping that preserves temporal consistency while enabling efficient sample reuse.

Overview of proposed path-length-aware reuse strategies. (A) Naive path reuse from conventional steady-state ReSTIR ends up going outside of the valid time gate. To address this issue, we introduce path-length-aware shift mapping, illustrated by the red arrow. (B) Illustration of a base path in the left pixel (blue), the naively reused path in the right pixel (orange), and the corrected path produced by our method (green), which satisfies the temporal constraint.

Baseline shift mapping (hybrid shift mapping [Lin et al. 2022]) transforms the base path $ \bar{\mathbf{x}} $ (blue) into an offset path $ \bar{\mathbf{y}} $ (orange), with the reconnection vertex $ \mathbf{p} $ at $ \mathbf{y}_4 = \mathbf{x}_4 $. We further perturb the offset path $ \bar{\mathbf{y}} $ to $ \bar{\mathbf{y}}' $ (green) to satisfy the path-length constraint, $ \ell(\bar{\mathbf{y}}') = \ell(\bar{\mathbf{x}}) + \Delta \ell $. Rather than perturbing the entire offset path, we apply path-length-aware shift mapping only at the reconnection vertex, similar to Sawhney et al. (2022).

Searching on path-length manifold is very similar to that of specular reflection. However, unlike specular reflection, which uniquely determines the reconnection vertex through geometric constraints, the path-length constraint alone leaves one remaining degree of freedom. We therefore introduce an additional constraint, referred to as a gauge, to select a unique solution on the feasible contour. Our average-gradient search efficiently identifies a stable reconnection point satisfying this gauge.

Specular materials impose a 2D constraint that uniquely determines $\mathbf{p}'$ (as a function of $\xi'$) on the surface given $\mathbf{p}_1$ and $\mathbf{p}_2$. In contrast, the path-length constraint is only 1D, meaning that feasible $\mathbf{p}'$ lie on an equi-distance contour on the surface. To further reduce the remaining degree of freedom, we introduce an additional constraint, referred to as a gauge in this paper.

Methods for imposing an additional constraint. (A) A fixed-axis gauge is reversible and easy to implement, but it may require traveling a long distance to find a valid solution. (B) Tracing along the gradient reduces the displacement, but it is not guaranteed to be reversible. (C) Our average-gradient method uses gradients at both the start and end points, ensuring reversibility. (D) Gradient flow is also bijective and yields small displacements, but it is not based on Newton’s method and requires many small steps.

ToF ReSTIR: Time-of-Flight Rendering with Spatio-temporal Reservoir Resampling

We propose a real-time Time-of-Flight (ToF) rendering method inspired by ReSTIR, augmented with a novel path-length-aware shift mapping.

Abstract

Time-of-Flight Imaging and Rendering

Path-Length-Aware Shift-Mapping

BibTeX