Supplementary material

This supplementary material complements the main paper by providing full videos, technical details and results with our method trained on another backbone, WAN.

Here we show the full videos used to make fig. 1.

Here we show training data samples, complementing fig. 4. Each sample displays the video alongside camera trajectory visualizations from three perspectives: top view, side view, and a 3D view. The cameras are color-coded to represent their position in time: the first camera is purple and the last one is red.

Here we show the full derivation of the $\varphi(\cdot)$ function used in eq. 2, which is used to enforce the yaw to be relative to the first frame, while preserving the absolute pitch and roll.
We take inspiration from the Look At matrix, frequently used in computer graphics to remove the "roll" component from a rotation matrix when specifying a direction in which to "look at". In our case, we slightly modify this procedure to take into account the gravity down vector as the direction and obtain a matrix $\mathbf{R}_{\text{no_yaw}}$ that has a null yaw component for the first frame, while still encoding the pitch and roll.
We use the following right-handed coordinate convention: +x is right ($\mathbf{r}$), +y is down ($\mathbf{d}$), +z is forward ($\mathbf{f}$).

$$\begin{align*} \mathbf{d} &\gets \mathbf{R}_{\text{pano},0}^{-1} [0~1~0]^\top && \text{ $\triangleright$ Compute the down direction vector $\mathbf{d}$ in camera space. } \\ \mathbf{r} &\gets \mathbf{d} \times [0~0~1]^\top && \text{ $\triangleright$ Compute the right direction $\mathbf{r}$ vector using the down and temporary forward vector. } \\ \mathbf{r} &\gets \frac{\mathbf{r}}{\|\mathbf{r}\|} && \text{ $\triangleright$ Normalize the right direction vector $\mathbf{r}$. } \\ \mathbf{f} &\gets \mathbf{r} \times \mathbf{d} && \text{ $\triangleright$ Compute the forward direction vector $\mathbf{f}$. } \\ \mathbf{R}_{\text{no_yaw}} &\gets \begin{bmatrix} \mathbf{r}^\top \\ \mathbf{d}^\top \\ \mathbf{f}^\top \end{bmatrix} && \text{ $\triangleright$ Compute the matrix containing the absolute pitch and roll for the first frame. } \\ \varphi(\mathbf{R}_{\text{pano},0}) &\gets \mathbf{R}_{\text{no_yaw}} \mathbf{R}_{\text{pano},0}^{-1} && \text{ $\triangleright$ Compute the matrix $\varphi(\mathbf{R}_{\text{pano},0})$, which can be applied to an extrinsic matrix to remove its yaw component. } \end{align*}$$

For completeness, here is equation 2, which computes the absolute camera extrinsics. Note that we slightly abuse notation by assuming rotation matrices are in homogeneous coordinates to match the \(4\times 4\) dimensions of extrinsics matrices.

$$ \mathbf{E}_{\text{abs},f} = \varphi(\mathbf{R}_{\text{pano},0})\mathbf{R}_{\text{pano},f} \mathbf{E}_{\text{rel},f} \; . $$

Here we show results where we trained a different diffusion model, WAN 2.2 (5B), using the same data and procedure, adjusted to the conditioning mechanism of WAN.

Training details

We randomly initialize a new camera encoder block composed of two convolutional layers, taking as input the absolute Plücker rays and outputting a feature map that is added to the patchified noisy latents, just before the first DiT layer. We train the camera encoder and finetune the DiT model for 70,400 iterations on 4 A100 (80 GB) GPUs.
In the following table, we show the quantitative results of this model. Training on this more powerful backbone yields better absolute and relative rotation error, but leads to a slight decrease in CLIP, FID and FVD score. We show qualitative results for both backbones, in the "Additional Qualitative Results" section below.

Quantitative results

Here, we supply additional details and statistics on our evaluation benchmark, SpatialVID-extreme, extending sec. 4.3 of the paper.

Random rotation trajectories

We randomly sample a start roll from \([-40°, 40°]\), and a start pitch from \([-90°, 90°]\). The final roll and pitch are sampled the same way, and the end yaw is sampled from \([-180°, 180°]\). The intermediate rotation are interpolated using spherical linear interpolation. Since the method used to compute absolute orientation metrics (Perspective Fields) is never trained with rolls going beyond 45° in magnitude, we resample a random rotation trajectory if an intermediate frame goes outside the roll bounds of \([-40°, 40°]\).

Evaluation dataset statistics

Here we show that the original SpatialVID-HQ dataset provides limited diversity in absolute orientation and relative rotations. Our new evaluation benchmark, SpatialVID-extreme, provides a broader coverage of Euler angles and total angular distance.

For the AC3D baselines, we provide the model the absolute camera orientation through text, as mentioned in sec. 4.4 of the paper. Here, we show the code used to generate these camera descriptions.
We first take the absolute camera extrinsics \(E_\text{abs}\) for the video and convert them to Euler angles. We then describe textually only the first frame's pitch and roll and the last frame's yaw, pitch and roll. We omit describing the in-between frames, since the trajectories are linear and to avoid overwhelming the model.

def get_camera_description_from_absolute_c2w(c2w_absolute):
    euler_angles = c2w_to_pitch_roll_yaw(c2w_absolute)
    first_pitch = euler_angles['pitch'][0].item()
    first_roll = euler_angles['roll'][0].item()
    first_yaw = euler_angles['yaw'][0].item()
    last_pitch = euler_angles['pitch'][-1].item()
    last_roll = euler_angles['roll'][-1].item()
    last_yaw = euler_angles['yaw'][-1].item()
    
    
    def describe_angle_shot(pitch):
        """Describe pitch angle shot."""
        pitch_rounded = round(pitch)
        if -5 <= pitch_rounded <= 5:
            return "near straight-on shot"
        elif pitch_rounded > 0:
            if 5 < pitch_rounded <= 20:
                return f"small tilt-up of {pitch_rounded} degrees"
            elif 20 < pitch_rounded <= 45:
                return f"large tilt-up of {pitch_rounded} degrees"
            else:
                return f"extreme tilt-up of {pitch_rounded} degrees"
        else:
            abs_pitch = abs(pitch_rounded)
            if 5 < abs_pitch <= 20:
                return f"small tilt-down of {abs_pitch} degrees"
            elif 20 < abs_pitch <= 45:
                return f"large tilt-down of {abs_pitch} degrees"
            else:
                return f"extreme tilt-down of {abs_pitch} degrees"
    
    def describe_dutch_angle(roll):
        """Describe roll (Dutch angle) with clockwise/counterclockwise."""
        roll_rounded = round(roll)
        if -5 <= roll_rounded <= 5:
            return "near level shot"
        
        abs_roll = abs(roll_rounded)
        if abs_roll <= 20:
            magnitude = "small"
        elif abs_roll <= 45:
            magnitude = "large"
        else:
            magnitude = "extreme"
        
        # Positive roll is counterclockwise, negative is clockwise
        direction = "counterclockwise" if roll_rounded > 0 else "clockwise"
        
        return f"a {magnitude} Dutch angle tilted {direction} {abs_roll} degrees"
    
    def describe_yaw(yaw):
        """Describe yaw (pan) direction."""
        yaw_rounded = round(yaw)
        
        abs_yaw = abs(yaw_rounded)
        direction = "right" if yaw_rounded > 0 else "left"
        return f"pan of {abs_yaw} degrees turned {direction}"
    
    # Build start description
    start_parts = []
    start_pitch_desc = describe_angle_shot(first_pitch)
    start_roll_desc = describe_dutch_angle(first_roll)
    
    if start_pitch_desc != "near straight-on shot":
        start_parts.append(start_pitch_desc)
    if start_roll_desc != "near level shot":
        start_parts.append(start_roll_desc)
    
    # Build end description
    end_parts = []
    end_yaw_desc = describe_yaw(last_yaw)
    end_pitch_desc = describe_angle_shot(last_pitch)
    end_roll_desc = describe_dutch_angle(last_roll)
    
    if end_yaw_desc:
        end_parts.append(end_yaw_desc)
    if end_pitch_desc != "near straight-on shot":
        end_parts.append(end_pitch_desc)
    if end_roll_desc != "near level shot":
        end_parts.append(end_roll_desc)
    
    # Construct final description
    description_parts = []
    

    start_text = "The camera starts at " if len(start_parts) == 1 else "The camera starts with "
    description_parts.append(start_text + ", and ".join(start_parts))
    
    end_text = "The camera ends with " + ", ".join(end_parts)
    description_parts.append(end_text)
    
    return ". ".join(description_parts)[:-1]
        

After obtaining the camera description, we concatenate it after the regular prompt. Refer to the following qualitative results for examples of camera descriptions.

Here are additional qualitative results, complementing fig. 7. We show 25 randomly-sampled videos from our SpatialVID-extreme dataset. The black overlay represents the input camera orientation and the red overlay represents the estimated camera orientation (obtained using Perspective Fields and VGGT), as described in the paper. For "AC3D + cam. text" baselines, we further provide the Camera prompt to the model to indicate the absolute camera orientation to the model.