visor

Benchmarking Spatial Relationships in Text-to-Image Generation — Gokhale et al. (2022) (arXiv:2212.10015, 2022)

What this evaluates

Evaluates whether text-to-image models correctly render spatial relationships between objects mentioned in a prompt. It disentangles object generation accuracy from spatial correctness to reveal model biases like object priority and merging.

Datasets

• SR2D — total ?; splits: (unstated); repo https://github.com/microsoft/VISOR

Metrics

• VISOR (primary) — range: [0, 1]
  • Measures spatial relationship accuracy. VISORuncond scores all prompts. VISORcond scores only prompts where both objects are correctly detected. VISORn variants score specific spatial relations.
• OA — range: [0, 1]
  • Object Accuracy: percentage of prompts where both mentioned objects are correctly detected in the generated image.
• Δ_s — range: other
  • Average difference between metric score for original prompt t and flipped prompt t_flip: E_t[s^t - s^t_flip]. Measures sensitivity to spatial relationships.

Input / output format

Input: Text prompt t specifying two objects and their spatial relationship.

Output: Generated image x from the text-to-image model.

Scoring recipe

def compute_metrics(prompt, image):
    obj_a, obj_b, rel = parse_prompt(prompt)
    detected = detect_objects(image)
    a_present = obj_a in detected
    b_present = obj_b in detected
    rel_detected = detect_spatial_relation(image)
    
    oa = 1.0 if (a_present and b_present) else 0.0
    visor_cond = 1.0 if (a_present and b_present and rel_detected == rel) else 0.0
    visor_uncond = 1.0 if (rel_detected == rel) else 0.0
    return oa, visor_cond, visor_uncond

Common pitfalls

• Existing metrics like CLIPScore and FID ignore text or spatial relations, yielding Δ_s ≈ 0.
• Object Accuracy (OA) penalizes spatial failures if one object is missing, conflating generation and spatial reasoning.
• Human studies show high inter-annotator agreement but reveal models often merge objects (32.46% for DALLE-v2).

Evidence (verbatim from paper)

We generate N=4 images for each text prompt from our SR2D dataset, to obtain 126,720 images per model and compare performance in terms of OA, VISOR, VISORcond, and VISOR1/2/3/4.

Citation

@misc{gokhale2022visor,
  title={Benchmarking Spatial Relationships in Text-to-Image Generation},
  author={Gokhale et al. (2022)},
  year={2022},
  note={arXiv:2212.10015}
}

• arXiv: 2212.10015