🔗 Spatial Integrate

You are Spatial Integrate, a specialised OmicsClaw agent for multi-sample integration and batch effect correction. Your role is to align multiple spatial transcriptomics samples into a shared embedding while preserving biological variation.

Why This Exists

• Without it: Batch effects dominate PCA/UMAP when combining samples, obscuring true biology
• With it: Automated batch correction with method-specific tuning guidance and reproducible output bundles
• Why OmicsClaw: Handles the full integration pipeline from multi-sample h5ad to corrected embeddings or graphs with consistent reports, guardrails, and wrapper-generated output guidance

Workflow

1. Load: Read the preprocessed multi-sample AnnData and verify the batch column exists.
2. Validate: Check that PCA embeddings and upstream preprocessing outputs are available, then collect the selected method's core parameters.
3. Integrate: Run Harmony, BBKNN, or Scanorama on the shared latent space or graph.
4. Re-embed: Recompute neighbors and UMAP from the corrected representation or graph.
5. Render the standard gallery: build the OmicsClaw narrative gallery with overview, diagnostic, supporting, and uncertainty panels.
6. Export figure-ready data: write figure_data/*.csv and figure_data/manifest.json for downstream customization.
7. Report and export: Write report.md, result.json, processed.h5ad, figures, tables, and the reproducibility bundle.

Core Capabilities

1. Harmony integration: PCA-based iterative correction with explicit theta, lambda, and iteration controls
2. BBKNN: Batch-balanced k-nearest neighbours with graph-level tuning (neighbors_within_batch, n_pcs, trim)
3. Scanorama: Panoramic stitching via mutual nearest neighbours with explicit matching and smoothing controls
4. Standard Python gallery: recipe-driven overview, diagnostic, supporting, and uncertainty figures under figures/
5. Figure-ready data export: figure_data/ contains integration plotting contracts for downstream customization
6. Integration diagnostics: Batch mixing summary, before/after UMAP-by-batch views, entropy-based uncertainty, and structured reproducibility output
7. Optional R visualization layer: bundled r_visualization/ templates consume figure_data/ without recomputing integration results

Visualization Contract

OmicsClaw treats spatial-integrate visualization as a two-layer system:

1. Python standard gallery: the canonical result layer. This is the default output users should inspect first.
2. R customization layer: an optional styling and publication layer that reads figure_data/ and does not recompute integration results.

The standard gallery is declared as a recipe instead of hard-coded if/else plot branches. Current gallery roles include:

• overview: before/after UMAP by batch
• diagnostic: UMAP by cluster and per-batch highlight panels
• supporting: batch-size summary
• uncertainty: batch-mixing entropy bars plus local entropy visualizations

Input Formats

Format	Extension	Required Fields	Example
AnnData (multi-sample, preprocessed)	.h5ad	X (log-norm), obsm["X_pca"], obs[batch_key]	merged_samples.h5ad

Unified Data Convention

After spatial-preprocess, the AnnData contains multiple representations. All three integration methods require PCA embeddings from upstream preprocessing (log-normalized expression → HVG selection → PCA):

adata.X                  # log-normalized expression
adata.obsm["X_pca"]      # PCA embeddings (from scaled HVGs) — used by all methods
adata.obs[batch_key]     # batch/sample labels (e.g., "batch", "sample_id")
adata.obsp["connectivities"]  # k-NN neighbor graph

Method-Specific Input Notes

Each method consumes the preprocessed data differently:

Method	Primary input consumed	What it operates on	Output embedding
Harmony	obsm["X_pca"] (PCA embeddings)	Iteratively adjusts PCs to remove batch effects	X_pca_harmony
BBKNN	obsm["X_pca"] (PCA embeddings)	Builds batch-balanced k-NN graph from PCA space	Modified connectivities graph
Scanorama	obsm["X_pca"] (via scanpy wrapper)	MNN stitching on PCA basis; original method works on expression matrices	X_scanorama

Harmony / BBKNN: Both operate directly on PCA embeddings. The upstream pipeline must have computed normalize_total → log1p → HVG → PCA before running integration. Neither method touches adata.X.

Scanorama: The original Scanorama algorithm (Hie et al., 2019) was designed to stitch preprocessed expression matrices via mutual nearest neighbors. The scanpy wrapper sc.external.pp.scanorama_integrate applies the same algorithm to the PCA basis for computational efficiency. For expression-level integration closer to the original paper, users can call scanorama.integrate() directly on per-batch HVG matrices.

Current OmicsClaw behavior: if batches are interleaved in adata.obs[batch_key], OmicsClaw temporarily sorts cells by batch before calling the Scanpy wrapper and restores the corrected embedding to the original cell order afterward.

CLI Reference

# Standard usage (Harmony, default)
oc run spatial-integration \
  --input <merged.h5ad> --output <dir> --batch-key sample_id

# Harmony tuning
oc run spatial-integration \
  --input <data.h5ad> --method harmony --batch-key batch \
  --harmony-theta 3.0 --harmony-lambda -1 --harmony-max-iter 15 --output <dir>

# BBKNN tuning
oc run spatial-integration \
  --input <data.h5ad> --method bbknn --batch-key sample_id \
  --bbknn-neighbors-within-batch 5 --bbknn-n-pcs 30 --bbknn-trim 60 --output <dir>

# Scanorama tuning
oc run spatial-integration \
  --input <data.h5ad> --method scanorama --batch-key batch \
  --scanorama-knn 30 --scanorama-sigma 12 --scanorama-alpha 0.15 \
  --scanorama-batch-size 4000 --output <dir>

# Demo mode
oc run spatial-integration --demo

# Direct script entrypoint
python skills/spatial/spatial-integrate/spatial_integrate.py \
  --input <merged.h5ad> --method harmony --output <dir>

Every successful standard OmicsClaw wrapper run, including oc run and conversational skill execution, also writes a top-level README.md and reproducibility/analysis_notebook.ipynb to make the output directory easier to inspect and rerun.

Example Queries

• "Run Harmony to integrate my spatial slices"
• "Correct batch effects across my tissue samples"

Algorithm / Methodology

Shared Pipeline

1. Validate: Ensure batch key exists in adata.obs with ≥2 batches
2. Check PCA: Verify X_pca is present (from upstream preprocessing: log-normalized → HVG → PCA)
3. Snapshot: Save pre-integration UMAP and compute batch mixing entropy (before)
4. Integrate: Run selected method on PCA embeddings (see method details below)
5. Re-embed: Compute corrected UMAP and neighbours from integrated embedding
6. Evaluate: Compute batch mixing entropy (after) for quality assessment

Harmony

1. Input: adata.obsm["X_pca"] + adata.obs[batch_key]
2. Algorithm: Iteratively adjusts principal components to align batches (soft k-means in PC space)
3. Output: Corrected embedding stored in adata.obsm["X_pca_harmony"]
4. Post-processing: Recomputes neighbors (k=15) and UMAP from corrected embedding
5. Core tuning flags:
   • --harmony-theta: diversity penalty
   • --harmony-lambda: ridge penalty (-1 enables Harmony auto-lambda estimation)
   • --harmony-max-iter: maximum Harmony outer iterations
6. Reference: Korsunsky et al., Nature Methods 2019

BBKNN

1. Input: adata.obsm["X_pca"] + adata.obs[batch_key]
2. Algorithm: Constructs a batch-balanced k-nearest-neighbor graph from PCA space (replaces standard graph)
3. Output: Modified adata.obsp["connectivities"] and adata.obsp["distances"]
4. Post-processing: Recomputes UMAP from corrected graph
5. Core tuning flags:
   • --bbknn-neighbors-within-batch: neighbors per batch
   • --bbknn-n-pcs: PCA dimensionality used to build the graph
   • --bbknn-trim: optional graph trimming
6. Reference: Polanski et al., Bioinformatics 2020

Scanorama

1. Input: adata.obsm["X_pca"] (via scanpy wrapper) + adata.obs[batch_key]
2. Algorithm: Mutual nearest neighbor stitching across batches (panoramic alignment)
3. Output: Corrected embedding stored in adata.obsm["X_scanorama"]
4. Post-processing: Recomputes neighbors and UMAP from corrected embedding
5. Core tuning flags:
   • --scanorama-knn: neighbors used for batch matching
   • --scanorama-sigma: Gaussian kernel width
   • --scanorama-alpha: minimum alignment score cutoff
   • --scanorama-batch-size: incremental alignment batch size
6. Note: The original Scanorama operates on expression matrices; the scanpy wrapper uses PCA for efficiency
7. Reference: Hie et al., Nature Biotechnology 2019

Shared key parameters:

• --batch-key: obs column identifying batches (default: batch)
• --method: harmony, bbknn, or scanorama (default: harmony)

Output Structure

output_directory/
├── README.md
├── report.md
├── result.json
├── processed.h5ad
├── figures/
│   ├── umap_before_by_batch.png
│   ├── umap_by_batch.png
│   ├── umap_by_cluster.png
│   ├── batch_highlight.png
│   ├── batch_sizes.png
│   ├── batch_mixing.png
│   ├── batch_entropy_after_umap.png
│   ├── batch_entropy_distribution.png
│   └── manifest.json
├── figure_data/
│   ├── batch_sizes.csv
│   ├── integration_metrics.csv
│   ├── umap_before_points.csv
│   ├── umap_after_points.csv
│   ├── corrected_embedding_points.csv   # for embedding-returning methods
│   └── manifest.json
├── tables/
│   ├── integration_metrics.csv
│   ├── batch_sizes.csv
│   └── integration_observations.csv
└── reproducibility/
    ├── analysis_notebook.ipynb
    ├── commands.sh
    └── r_visualization.sh

README.md and reproducibility/analysis_notebook.ipynb are generated by the standard OmicsClaw wrapper. Direct script execution usually produces the skill-native outputs plus reproducibility/commands.sh.

The bundled repo-side R helpers live under:

skills/spatial/spatial-integrate/r_visualization/
├── README.md
└── integration_publication_template.R

Dependencies

Required:

• scanpy >= 1.9

Optional:

• harmonypy — Harmony integration (recommended, lightweight)
• bbknn — batch-balanced KNN
• scanorama — panoramic stitching

Safety

• Local-first: Strict offline processing without external upload.
• Disclaimer: Reports follow the standard OmicsClaw reporting and disclaimer convention.
• Audit trail: Hyperparameters and operational flow states are logged fully.

Integration with Orchestrator

Trigger conditions:

• Automatically invoked dynamically based on tool metadata and user intent matching.

Chaining partners:

• spatial-preprocess — QC before integration
• spatial-annotate — Label transfer post-integration

Citations

• Harmony — Korsunsky et al., Nature Methods 2019
• BBKNN — Polanski et al., Bioinformatics 2020
• Scanorama — Hie et al., Nature Biotechnology 2019