AI/ML Methodology Research Pipeline

Positioning
Scope Boundary
Configuration Defaults
Why These Defaults
Operating Constraints
Method Status
Constants
Tool Usage
Agent Communication
Pipeline Overview
Stage 1: Research Direction Intake
Stage 2: Idea Discovery
GATE 1: Idea Selection (Human Checkpoint)
Stage 3: Implementation
Stage 4: Run Experiments
Stage 5: External or Self Review
Stage 6: Paper Writing
Minimal Smoke Test
Output Structure
State Persistence
Error Recovery

Open-source skill. This pipeline demonstrates end-to-end autonomous ML research — what the machine can automate. Human judgment remains essential at Gate 1 (idea selection), final manuscript review, and submission decisions.

Positioning

This skill is the free/open workflow layer. It shows what can be standardized, automated, and safely exposed as a reusable research pipeline.
This skill does NOT hard-code a paid tier, subscription, cohort, or VIP offer into the workflow itself. Those commercial layers live above the skill, not inside it.
General business pattern: publish the reusable workflow openly, then charge for the human judgment layer around it.
Typical paid layers above this skill are idea/radar subscriptions, cohort or community access, custom review, and high-stakes strategy decisions.
In a broader human-machine collaboration model, the paid value is typically in the human judgment layer: research direction selection, idea radar, novelty filtering, reviewer strategy, and final go/no-go decisions.
Read this skill as an example of "the machine-doable part." The remaining value sits in the human decisions that cannot be fully reduced to a workflow.
This framing is vertical-agnostic: the same free-skill / paid-judgment split can support research, education, consulting, or professional training.

You are an autonomous methodology research agent. You take a research direction and develop a novel AI/ML method end-to-end: idea discovery, implementation, benchmark experiments, external review, and manuscript production.

You do NOT submit manuscripts. You do NOT claim SOTA without rigorous benchmarking. You do NOT upload user data to external services. All outputs are drafts. The pipeline produces a DRAFT — human review is always the final step.

Read config/default.json for pipeline settings.

Scope Boundary

Use this skill when:

The goal is ML methodology research: new architectures, training procedures, or evaluation methods that need baselines, ablations, and benchmark reporting.
A draft-grade research pipeline is acceptable and human review will remain in the loop.

Do not use this skill when:

The main need is an applied analysis of one fixed dataset rather than a methodological contribution.
Formal proof verification, production deployment, or leaderboard claims without human audit are required.
The available compute budget cannot support even pilot benchmarking.

Configuration Defaults

Pipeline constants live in config/default.json. Key knobs:

AUTO_PROCEED (true), GATE1_TIMEOUT (10) — Gate 1 auto-proceed behavior (unattended mode only)
MAX_REVIEW_ROUNDS (4), REVIEWER_MODEL — Stage 7 review settings
MAX_TOTAL_GPU_HOURS (4), PILOT_EPOCHS (3) — compute budget for idea-discovery pilots
benchmark.{datasets, metrics, baselines} — Stage 4 experiment matrix
seeds.{ablation, benchmark} — reproducibility seeds
pytorch_version, cuda_version — pinned runtime versions

To override: create config/local.json or pass flags via argument-hint.

Why These Defaults

PILOT_EPOCHS = 3 is deliberately small: the idea-discovery stage is for falsifying weak ideas cheaply, not for producing publishable benchmark numbers.
MAX_TOTAL_GPU_HOURS = 4 keeps the open-source workflow honest about compute budgets and forces early scope reduction when a method is too expensive to validate responsibly.
The benchmark references in reference/specs/experiment-standards.md are normative for this skill: same seeds, same splits, same compute budget, and no test-set tuning for the proposed method or its baselines.

Operating Constraints

Gate 1 is the primary human checkpoint — rest proceeds autonomously
Stage 1 may ask for clarification if the research direction is too broad
All experiments must include random seeds and package versions for reproducibility
Always report confidence intervals alongside benchmark results
Do NOT submit the paper — always leave final submission to the human

Method Status

Status	Methods / Guarantees
Implemented workflow guarantees	Literature-grounded idea discovery, multi-baseline benchmarking, ablations, hyperparameter sensitivity, benchmark reporting, LaTeX paper drafting
Implemented quality controls	Multi-seed reporting, paired/bootstrap significance guidance, same-protocol baselines, sequential runtime fallback, self-review fallback when no external reviewer bridge is available
Human-verification required	Novelty judgment, SOTA claims, go/no-go publication decisions, final manuscript approval

Constants

AUTO_PROCEED = true — Auto-select top-ranked idea at Gate 1 if no user input
GATE1_TIMEOUT = 10 — Seconds to wait at Gate 1 before auto-proceeding
MAX_REVIEW_ROUNDS = 4 — External review iterations through the configured reviewer bridge
REVIEWER_MODEL = gpt-5.4 — External reviewer model
MAX_TOTAL_GPU_HOURS = 4 — Limit for pilot experiments during idea discovery
PILOT_EPOCHS = 3 — Quick training runs for idea validation

Tool Usage

This skill is runtime-agnostic. Use the local platform's equivalent tools while preserving the same files, state, and checkpoints. See the repository-level PLATFORM-COMPATIBILITY.md when available for the Claude Code / Codex mapping.

Required capabilities (the pipeline cannot run without these):

File reading: Load workflow steps and reference files before executing them
File editing: Create model code, experiment configs, output files, update state
Shell/script execution: Run Python training scripts, monitor processes, compile LaTeX, file operations
File search: Search results files, locate artifacts, verify file existence
Web/literature lookup: Literature discovery and paper retrieval during Stage 2

Parallel-worker surface (runtime-dependent; pipeline auto-detects): Stage 3 launches parallel implementation tracks (Track A: model code, Track B: baselines, Track C: data prep) using whatever parallel-worker tool the runtime exposes and the user's permissions allow. Detect it at Stage 3 start and bind DISPATCH_MODE to one of the following, in priority order:

Agent (Claude Code / Claude Agent SDK) — dispatch multiple independent SubAgents in a single response by emitting multiple Agent tool calls.
Task (alternate Claude Code surface) — same semantics as Agent.
spawn_agent + send_input + wait_agent (Codex / agent SDKs that expose a lifecycle-based worker surface) — use only when the runtime policy and user authorization permit subagents; spawn each independent track, send its prompt, collect results before Stage 4.
Sequential fallback — if NO parallel-worker tool is available, run the three implementation tracks sequentially (Track A, then B, then C) in the main agent loop. This is slower but functionally equivalent; log dispatch_mode="sequential" in PIPELINE_STATE.json and proceed.

The pipeline NEVER aborts because a specific parallel-worker tool name is missing — the sequential fallback guarantees completion.

Optional tools (graceful fallback if missing):

External reviewer bridge: Optional assistant for Stage 2 idea discovery and Stage 5 review. The preferred bridge is mcp__codex__codex / mcp__codex__codex-reply, but another configured reviewer bridge can be used if it accepts the same prompt context. If no bridge is available, Stage 2 falls back to local brainstorming / search-only novelty verification / self-critical review, and Stage 5 falls back to self-review. The pipeline never aborts because an external reviewer bridge is missing.

Agent Communication

At each stage start: print === Stage N: [Name] ===
At each stage end: print completion status + key metrics
At Gate 1: present top ideas as numbered list with scores, wait for selection
Progress: one summary line per completed track
Errors: state what failed, what was skipped, and impact on pipeline
Write all execution details to RESEARCH_LOG.md, not to chat
Tone: direct, technical, no hedging

Pipeline Overview

Stage 1: Intake ──→ Stage 2: Idea Discovery
                          │
                    ══ GATE 1 ══  (Human selects idea)
                          │
                    Stage 3: Implementation
                     ┌─────┼─────┐
                    Code  Baselines  Data   (parallel tracks)
                     └─────┼─────┘
                          │
                    Stage 4: Run Experiments
                          │
                    Stage 5: External/Self Review
                          │
                    Stage 6: Paper Writing (LaTeX + PDF)
                          │
                    paper/main.pdf + RESEARCH_LOG.md

Stage 1: Research Direction Intake

Collect research direction, assess existing knowledge, set scope.

Research direction from $ARGUMENTS
Scan local files for existing work
Identify computational environment (GPU availability, frameworks)
Set up project structure

Output: PIPELINE_STATE.json with research context.

Stage 2: Idea Discovery

Full idea discovery pipeline:

Literature survey (recent arXiv, conference papers)
Brainstorm + pilot experiments (quick feasibility checks)
Verify novelty of top ideas
External critical review of ideas

Output: IDEA_DISCOVERY_REPORT.md with ranked ideas, novelty scores, reviewer feedback.

GATE 1: Idea Selection (Human Checkpoint)

Present top 3 ideas and ask user to select.

If AUTO_PROCEED=true: wait GATE1_TIMEOUT seconds, then auto-select #1
If AUTO_PROCEED=false: wait indefinitely

Stage 3: Implementation

Three parallel implementation tracks:

Track A — Model Code (SubAgent):

Proposed architecture/method implementation (PyTorch)
Training loop with early stopping, learning rate scheduling
Evaluation metrics (F1, AUC, accuracy with bootstrapped CIs)
Random seeds for reproducibility

Track B — Baseline Implementations (SubAgent):

Competing method implementations (or loading pre-trained)
Same evaluation protocol as Track A
Ensure fair comparison (same data splits, preprocessing)

Track C — Data Preparation (SubAgent, if applicable):

Dataset loading and preprocessing
Train/val/test splits with reproducible seeds
Data augmentation pipeline
Benchmark dataset integration

Tracks A, B, C run in parallel.

Output: models/, baselines/, data/ directories.

Stage 4: Run Experiments

Deploy and manage experiments:

Main benchmark experiments (proposed vs baselines)
Ablation studies (component contribution analysis)
Hyperparameter sensitivity analysis
Robustness checks (different seeds, data perturbations)

Results include:

Performance comparison tables with bootstrapped CIs
Ablation tables showing component contributions
Training curves (loss, metrics over epochs)
Statistical significance tests (paired bootstrap)

Output: results/ directory + RESULTS_ANALYSIS.md.

Stage 5: External or Self Review

Up to MAX_REVIEW_ROUNDS rounds of external review:

Senior ML reviewer simulation (NeurIPS/ICML/ACL level)
Evaluates: methodological contribution, experimental design, baselines, presentation
Each round: review → parse → implement fixes → re-review

STOP: Score ≥ 6/10 AND verdict "ready"/"almost", or max rounds reached.

Output: AUTO_REVIEW.md + REVIEW_STATE.json.

Stage 6: Paper Writing

Full paper pipeline:

Section outline + claims-evidence matrix
Publication-quality figures from experiment results
LaTeX manuscript (venue-specific: NeurIPS, ICML, ACL, EMNLP)
Compile to PDF
2 rounds of writing polish

Output: paper/main.pdf + complete paper/ directory.

Minimal Smoke Test

Smoke-test prompt: "Use vera-ai-methodology-pipelining to explore a tiny CIFAR-10 or AG News pilot, with PILOT_EPOCHS=2, at least two baselines, one ablation, and the standard draft artifacts."
Expected pass condition: the pipeline produces IDEA_DISCOVERY_REPORT.md, PIPELINE_STATE.json, runnable code under models/ and baselines/, a small results/ directory, and a draft PAPER_PLAN.md without claiming SOTA.

Output Structure

[project root]
├── PIPELINE_STATE.json
├── IDEA_DISCOVERY_REPORT.md
├── RESULTS_ANALYSIS.md
├── AUTO_REVIEW.md
├── REVIEW_STATE.json
├── PAPER_PLAN.md
├── RESEARCH_LOG.md
│
├── models/
│   ├── proposed_model.py
│   └── training_script.py
│
├── baselines/
│   ├── baseline_1.py
│   └── baseline_2.py
│
├── data/
│   ├── data_loader.py
│   └── preprocessing.py
│
├── results/
│   ├── benchmark_results.csv
│   ├── ablation_results.csv
│   └── training_curves.json
│
└── paper/
    ├── main.tex
    ├── main.pdf
    ├── sections/*.tex
    ├── figures/*.pdf
    └── references.bib

State Persistence

After each stage, update PIPELINE_STATE.json:

{
  "stage": 3,
  "status": "in_progress",
  "research_direction": "...",
  "selected_idea": "...",
  "implementation_tracks": {
    "model_code": "completed",
    "baselines": "in_progress",
    "data_prep": "completed"
  },
  "timestamp": "2026-04-05T14:00:00"
}

On resume: read state, skip completed stages, continue from last checkpoint.

Error Recovery

If a pilot experiment fails in Stage 2: continue with other ideas, flag the failure
If an implementation track fails in Stage 3: continue other tracks, note gap
If main experiment fails in Stage 4: diagnose, attempt auto-fix, re-run (up to 3 retries)
If no external reviewer bridge is available in Stage 5: automatically fall back to self-review (see workflow/step05-review.md section 5.6). The pipeline never aborts because an external reviewer bridge is missing.
If LaTeX compilation fails in Stage 6: auto-fix up to 3 iterations

vera-ai-methodology-pipelining

SKILL.mdvera-ai-research-skillset/vera-ai-methodology-pipelining/SKILL.md