🔬 Spatial Preprocess

You are Spatial Preprocess, the OmicsClaw skill for loading raw spatial transcriptomics data and converting it into a downstream-ready AnnData under the current scanpy_standard workflow. The wrapper now exposes a unified Python gallery, figure-ready exports, and optional R-side customization without changing the underlying preprocessing assumptions.

Why This Exists

Without it: users manually combine loading, QC, normalization, HVG selection, PCA, neighbors, and clustering with inconsistent defaults.
With it: one command produces a standardized processed.h5ad, QC-aware figures, clustering summaries, figure-ready CSV exports, and reproducibility helpers.
Upstream boundary: if the user still has FASTQ pairs plus barcode-coordinate metadata, they should run spatial-raw-processing first.
Why OmicsClaw: the wrapper preserves raw counts, records the actual effective QC thresholds, keeps platform-specific loading separate from the scientific preprocessing steps, and uses one stable output contract for downstream tools.

Core Capabilities

Current workflow: scanpy_standard: QC, normalization, seurat_v3 HVG selection, PCA, neighbors, UMAP, and Leiden clustering.
Multi-platform loading: Visium directories / h5 / h5ad, Xenium zarr / h5ad, and converted .h5ad inputs for MERFISH / Slide-seq / seqFISH.
QC filtering: spot or cell filtering using min_genes, min_cells, max_mt_pct, and optional max_genes.
Tissue-aware presets: optional wrapper-level presets for common tissues such as brain, heart, tumor, and lung.
Raw-count preservation: raw counts are stored in both adata.layers["counts"] and adata.raw before normalization.
Graph embedding: PCA, neighbor graph, and UMAP.
Clustering: primary Leiden clustering plus optional multi-resolution sweeps.
Standard Python gallery: emits a recipe-driven preprocessing gallery with clustering overview, QC diagnostics, PCA guidance, and threshold-context panels.
Figure-ready exports: writes figure_data/ CSVs plus a manifest so downstream tools can restyle the same preprocessing result without recomputing QC or clustering.
Structured exports: writes report.md, result.json, summary tables, reproducibility helpers, and an optional R visualization entrypoint.

Input Formats

Format	Extension	Required Fields	Example
OmicsClaw raw spatial counts	`.h5ad`	`raw_counts.h5ad` emitted by `spatial-raw-processing`	`raw_counts.h5ad`
AnnData raw	`.h5ad`	count matrix in `X`; spatial coordinates recommended	`raw_visium.h5ad`
10x Visium directory	directory	Space Ranger-style output	`visium_output/`
10x feature matrix	`.h5` / `.hdf5`	filtered feature matrix	`filtered_feature_bc_matrix.h5`
Xenium zarr	`.zarr` or directory	Xenium export readable by `anndata.read_zarr`	`xenium_sample.zarr`
Converted MERFISH / Slide-seq / seqFISH	`.h5ad`	expression matrix, ideally with coordinates	`merfish_converted.h5ad`
Demo	n/a	`--demo` flag	built-in Visium-like demo

Upstream Boundary

If the user still has sequencing-level inputs such as FASTQ pairs plus an IDs barcode-coordinate file and STAR reference files, this is not the correct entry point. Run spatial-raw-processing first, then pass the resulting raw_counts.h5ad into spatial-preprocess.

Current Loader Behavior

Data type	Current wrapper behavior
`visium`	tries `sc.read_visium()` first, then 10x matrix fallbacks
`xenium`	supports `.h5ad` and `.zarr`
`slide_seq`	current wrapper expects converted `.h5ad`
`merfish`	current wrapper expects converted `.h5ad`
`seqfish`	current wrapper expects converted `.h5ad`
`generic`	expects `.h5ad`

Post-Preprocess Data Convention

After preprocessing, the processed object typically contains:

adata.layers["counts"]         # preserved raw counts
adata.raw                      # raw-count snapshot before normalization
adata.X                        # log-normalized expression
adata.var["highly_variable"]   # HVG mask
adata.obsm["X_pca"]            # PCA embedding
adata.obsm["X_umap"]           # UMAP embedding
adata.obsp["connectivities"]   # neighbor graph
adata.obsp["distances"]        # neighbor distances
adata.obs["leiden"]            # primary Leiden clusters
adata.obs["leiden_res_*"]      # optional multi-resolution results

Tissue-Specific QC Presets

When --tissue is set, OmicsClaw auto-fills default QC thresholds. Explicit CLI parameters still take precedence.

Tissue	max_mt_pct	min_genes	max_genes	Notes
pbmc	5	200	2500	low mitochondrial fraction in blood cells
brain	10	200	6000	higher complexity in neuronal tissue
heart	50	200	5000	cardiomyocytes are mitochondria-rich
tumor	20	200	5000	heterogeneous tissue composition
liver	15	200	4000	large hepatocytes
kidney	15	200	4000	mitochondria-active tubular cells
lung	15	200	5000	mixed epithelial and immune content
gut	20	200	5000	high epithelial turnover
skin	10	200	4000	keratinocyte-rich tissue
muscle	30	200	5000	elevated mitochondrial burden

Workflow

Load: detect platform type and load the dataset.
QC metrics: compute n_genes_by_counts, total_counts, and mitochondrial percentage.
Filter: apply min_genes, min_cells, max_mt_pct, and optional max_genes.
Preserve counts: store raw counts in layers["counts"] and adata.raw.
Normalize: run normalize_total(target_sum=1e4) followed by log1p.
Select HVGs: run highly_variable_genes(..., flavor="seurat_v3", layer="counts").
Embed: scale HVGs, compute PCA, build neighbors, and compute UMAP.
Cluster: run the primary Leiden clustering and optional resolution sweep.
Render the standard gallery: build the OmicsClaw preprocessing gallery with clustering overview, QC maps, PCA variance guidance, and threshold-context panels.
Export figure-ready data: write figure_data/*.csv and figure_data/manifest.json for downstream customization.
Export: write summary tables, report.md, result.json, processed.h5ad, and reproducibility helpers.

Visualization Contract

OmicsClaw treats spatial-preprocess visualization as a two-layer system:

Python standard gallery: the canonical preprocessing result layer. This is the default output users should inspect first.
R customization layer: an optional styling and publication layer that reads figure_data/ and does not recompute QC, clustering, or embeddings.

The standard gallery is declared as a recipe instead of hard-coded plotting branches. It reuses the shared skills/spatial/_lib/viz feature-map layer for spatial and UMAP projections, while skill-local renderers handle preprocessing- specific summaries such as PCA variance guidance and QC histograms.

Current gallery roles include:

overview: primary Leiden clusters on tissue and UMAP
diagnostic: QC metrics projected onto tissue coordinates
supporting: cluster-size summary, PCA variance guidance, and optional resolution sweep
uncertainty: QC metric distributions with effective threshold overlays

CLI Reference

# Standard usage
oc run spatial-preprocessing --input <data.h5ad> --output <report_dir>

# Tissue-aware QC
oc run spatial-preprocessing \
  --input <data.h5ad> --output <report_dir> --tissue brain --species human

# Explicit QC thresholds
oc run spatial-preprocessing \
  --input <data.h5ad> --output <report_dir> \
  --min-genes 200 --max-mt-pct 15 --max-genes 5000

# Graph tuning
oc run spatial-preprocessing \
  --input <data.h5ad> --output <report_dir> \
  --n-top-hvg 3000 --n-pcs 30 --n-neighbors 15 --leiden-resolution 0.6

# Multi-resolution Leiden exploration
oc run spatial-preprocessing \
  --input <data.h5ad> --output <report_dir> --resolutions 0.4,0.6,0.8,1.0

# Explicit platform hint
oc run spatial-preprocessing \
  --input <xenium_sample.zarr> --output <report_dir> --data-type xenium

# Demo
oc run spatial-preprocessing --demo --output /tmp/spatial_preprocess_demo

# Direct script entrypoint
python skills/spatial/spatial-preprocess/spatial_preprocess.py \
  --input <data.h5ad> --output <report_dir>

Every successful standard OmicsClaw wrapper run, including oc run and conversational skill execution, also writes a top-level README.md and reproducibility/analysis_notebook.ipynb to make the output directory easier to inspect and rerun. Direct script execution primarily produces the skill-native outputs plus reproducibility/commands.sh.

Example Queries

"Preprocess my Visium dataset with standard QC and tell me which thresholds you will use first."
"Load this Xenium data and generate a clustered h5ad."
"Run spatial QC, preserve raw counts, and do a Leiden resolution sweep."

Algorithm / Methodology

`scanpy_standard`

Tissue presets: optionally fill QC defaults from the tissue preset table.
QC metrics: compute n_genes_by_counts, total_counts, and mitochondrial percentage using qc_vars=["mt"].
Filtering: apply min_genes, min_cells, max_mt_pct, and optional max_genes.
Counts preservation: save raw counts into adata.layers["counts"] and adata.raw.
Normalization: run sc.pp.normalize_total(target_sum=1e4) followed by sc.pp.log1p().
HVG selection: run sc.pp.highly_variable_genes(..., flavor="seurat_v3", layer="counts").
Scaling and PCA: scale only the HVG subset and compute PCA.
PC guidance: suggest an informative PC count from cumulative explained variance, clamped to [15, 30].
Neighbors and UMAP: build the neighbor graph and compute UMAP.
Leiden clustering: run the primary Leiden clustering plus optional extra resolutions.

Core tuning flags:

data_type: wrapper-level loader hint, not a Scanpy science parameter.
species: wrapper-level mitochondrial prefix selector.
tissue: wrapper-level QC preset selector.
min_genes, min_cells, max_mt_pct, max_genes: main QC thresholds.
n_top_hvg: public Scanpy HVG budget.
n_pcs: requested PCA dimensions before internal clipping by data shape.
n_neighbors: public neighbor-graph size.
leiden_resolution: public Leiden clustering resolution.
resolutions: optional wrapper-level sweep over multiple Leiden resolutions.

Current OmicsClaw behavior: reports include both the user-requested params and the actual effective params after preset application, plus requested / computed / used / suggested PC counts.

Output Structure

output_dir/
├── README.md
├── report.md
├── result.json
├── processed.h5ad
├── figures/
│   ├── spatial_leiden.png
│   ├── umap_leiden.png
│   ├── qc_metrics_spatial.png
│   ├── cluster_size_barplot.png
│   ├── pca_variance_curve.png
│   ├── leiden_resolution_sweep.png        # if --resolutions was used
│   ├── qc_metric_distributions.png
│   └── manifest.json
├── figure_data/
│   ├── cluster_summary.csv
│   ├── qc_metric_distributions.csv
│   ├── preprocess_run_summary.csv
│   ├── pca_variance_ratio.csv
│   ├── multi_resolution_summary.csv       # if --resolutions was used
│   ├── preprocess_spatial_points.csv      # when spatial coordinates are available
│   ├── preprocess_umap_points.csv
│   └── manifest.json
├── tables/
│   ├── cluster_summary.csv
│   ├── qc_summary.csv
│   ├── pca_variance_ratio.csv
│   └── multi_resolution_summary.csv       # if --resolutions was used
└── reproducibility/
    ├── analysis_notebook.ipynb
    ├── commands.sh
    ├── environment.txt
    └── r_visualization.sh

The bundled optional R templates live under:

skills/spatial/spatial-preprocess/r_visualization/
├── README.md
└── preprocess_publication_template.R

Safety

Local-first: all data processing stays local.
Raw preservation: raw counts are preserved in both adata.layers["counts"] and adata.raw.
Audit trail: reports, result.json, figures/manifest.json, and figure_data/manifest.json record effective QC and graph parameters plus visualization outputs.
Method-aware output: the Python gallery is canonical; optional R styling consumes exported figure data and must not silently rerun preprocessing.
Platform-aware loading: native vendor loaders are only used where the current wrapper actually supports them.

Dependencies

Required:

scanpy
anndata
numpy, pandas, matplotlib

Common runtime companions:

igraph, leidenalg
squidpy

Optional (R):

ggplot2

Integration with Orchestrator

Trigger conditions:

preprocess
spatial preprocessing
spatial QC
normalization
Leiden
UMAP

Chaining partners:

Often provides processed.h5ad to spatial-domains, spatial-annotate, spatial-de, spatial-register, and other downstream spatial skills

spatial-preprocess

SKILL.mdSkills/spatial/spatial-preprocess/SKILL.md