spatial-boost

Component Identification

Old Design (Standard VLM)

• Vision encoder produces flat feature representations
• No explicit spatial reasoning capability
• Spatial understanding encoded implicitly in learned features
• Single-channel attention (no spatial-specific processing)

New Design (SpatialBoost) Parallel attention channel dedicated to spatial understanding, merged with original via trainable mixture factor.

Motivation & Problem Statement

While VLMs excel at semantic understanding, spatial reasoning—particularly depth, relative positions, and precise distances—remains challenging. Adding spatial capability requires injecting new knowledge without disrupting pre-trained visual understanding (catastrophic forgetting risk). Language-guided reasoning provides interpretable spatial knowledge while LLM decoders generate natural spatial descriptions.

The Modification

Hierarchical Spatial Reasoning via Multi-Turn Chain-of-Thought

The framework structures spatial understanding across three levels, querying progressively more complex spatial relationships:

Level 1: Pixel-level depth queries
  Question: "What is the depth at position (x, y)?"
  or "Which point is closer: (x1, y1) or (x2, y2)?"
  Output: Absolute or relative depth predictions

Level 2: Object-level spatial relationships
  Question: "Is [Object A] on the left side of [Object B]?"
  or "What 3D bounding box contains [Object]?"
  Uses 3D bounding boxes to compute relationships
  Output: Spatial relationship confirmation/measurement

Level 3: Scene-level distance reasoning
  Question: "How far apart are [Object A] and [Object B]?"
  Synthesizes Level 1 & 2 results for precise distance
  Output: Absolute distance or relative positioning

Each turn builds on prior reasoning steps, creating interpretable chain-of-thought decomposition.

Dual-Channel Attention Mechanism

Standard vision encoder attention modified to preserve and extend knowledge:

# Original frozen attention
Attn(x) = softmax(Q·K^T / √d) · V

# New spatial attention channel
Attn+(x) = softmax(Q+·K+^T / √d) · V+
# Where Q+, K+, V+ are trained on spatial tasks

# Merged output with learnable mixture
Attn_final(x) = α · Attn(x) + (1 - α) · Attn+(x)
# α is trainable mixture weight per layer

Training Strategy:

# Forward pass
for layer in vision_encoder:
    # Original channel (frozen)
    x = layer.attention(x)

    # New spatial channel (trained)
    x_spatial = layer.attention_spatial(x)

    # Merge with learned weight
    α = layer.mixture_weight
    x = α * x + (1 - α) * x_spatial

# Backward pass: only α and spatial channel parameters update
# Original parameters remain frozen (stop_gradient=True)

This dual-channel design prevents catastrophic forgetting: pre-trained semantic features remain available while new spatial pathways develop.

Architectural Enhancements

For each transformer layer in vision encoder:

1. Add parallel attention layer (Attn+)
2. Add trainable mixture weight (α)
3. Keep original attention frozen
4. Use LLM decoder to generate spatial descriptions

The frozen original channel ensures:

• Semantic understanding preserved
• Robust fallback if spatial reasoning fails
• Minimal parameter overhead (only α and new weights)

Fine-tuning Configuration

Frozen Components:

• Original attention parameters (Q, K, V transformations)
• All other vision encoder layers remain frozen
• Pre-trained knowledge fully preserved

Trainable Components:

• Spatial attention parameters (Q+, K+, V+)
• Mixture weights (α) per layer
• LLM decoder parameters
• LoRA adapters if using parameter-efficient fine-tuning

Loss Function: Combine three supervision signals:

• Spatial description accuracy (LLM output matches ground truth)
• Depth prediction accuracy (pixel-level depth supervision)
• Spatial relationship classification (object-level spatial labels)

Performance Characteristics

Spatial Understanding Improvement:

• Pixel-level depth prediction accuracy
• Object-level spatial relationship F1
• Scene-level distance estimation MAE

Preservation of Original Capability:

• Semantic understanding maintained (original attention frozen)
• Visual fidelity preserved (no degradation on non-spatial tasks)
• Inference latency increased by dual-channel overhead (~15-20% per layer)

Conditions of Applicability

Works well when:

• Spatial reasoning is critical (autonomous driving, robotics, scene understanding)
• Dense 3D annotations available (pixel-level depths, bounding boxes)
• Model capacity sufficient for dual-channel attention
• Base model is already high-quality (adding spatial to weaker models may not help)

Less optimal when:

• Pure semantic understanding is sufficient (no spatial reasoning needed)
• Limited 3D supervision available (difficult to train spatial channel)
• Inference latency critical (dual-channel adds overhead)
• Parameter count constraints (new channel increases model size)

Drop-In Replacement Checklist

• Plug-and-play with standard VLM architectures (adds attention channel)
• No changes to downstream components (output format unchanged)
• Inference compatible with existing VLM pipelines
• Can warm-start from pre-trained vision encoder (frozen weights)
• Spatial outputs optional (graceful degradation if spatial channel fails)
• Works with any 3D annotation format (pixel depths, bounding boxes, distances)