Super Factory

Autonomous Agent development platform with 4-stage pipeline: Research → Planning → Build → Eval.

When to Use

• Running E2E tests for the Super Factory pipeline
• Executing real LLM-driven pipeline runs
• Debugging agent contract precondition failures
• Adding new agent roles or modifying pipeline YAML
• Setting up API keys for the project

Project Location

~/ai_github/super_factory/

Quick Commands

# Run E2E test suite (all mocked)
cd ~/ai_github/super_factory && python3 -m pytest tests/test_e2e_pipeline.py -v

# Run full test suite
cd ~/ai_github/super_factory && python3 -m pytest tests/ -v

# Run real pipeline (requires valid API keys)
cd ~/ai_github/super_factory && python3 scripts/run_real_pipeline.py

# Run CLI pipeline
cd ~/ai_github/super_factory && python3 scripts/run_pipeline.py --pipeline default --context "query=topic"

E2E Testing Pattern

The E2E test suite (tests/test_e2e_pipeline.py) validates the full pipeline through the real Orchestrator + PipelineExecutor stack with mocked LLM responses.

Key Test Classes

Class	Coverage
TestFullPipelineEndToEnd	4-agent happy path, contract precondition blocking, postcondition failure
TestPipelineExecutorIntegration	Full DAG execution, context passing, failure propagation
TestTraceCollectionEndToEnd	TraceCollector record/read round-trip, disk persistence
TestHookLifecycleE2E	PIPELINE_START, STEP_START, AGENT_POST_EXECUTE, PIPELINE_END hooks

Mock Response Requirements

Each agent role has specific contract preconditions that mock contexts MUST satisfy:

# Research: query (non-empty) + available_sources + knowledge_sources_consulted
research_context = {"query": "topic", "available_sources": ["url"], "knowledge_sources_consulted": [".wiki/"]}

# Planning: vision (non-empty string) + (research_refs (list) OR knowledge_sources (list))
planning_context = {"vision": "Build something", "research_refs": ["ref.md"]}

# Build: plan (non-empty dict) + task_id (non-empty string)
build_context = {"plan": {"tasks": ["T-1"]}, "task_id": "T-1"}

# Eval: spec_path (non-empty string) + target_output (not None)
eval_context = {"spec_path": "specs/agents/build.yaml", "target_output": build_result}

Contract Debugging Flow

When a test hits "blocked" status:

1. Check which agent's precondition failed (look at log: Pre-condition failed for <role>)
2. Read contracts/<role>.py to find required fields
3. Ensure the context dict provides all required fields with correct types
4. Common issue: research_refs must be a list, not a string; plan must be a dict, not a string

Real Pipeline Execution

API Key Setup

# 1. Extract API key from hermes config or use your own
# 2. Write to .env
cat > ~/ai_github/super_factory/.env << 'EOF'
DASHSCOPE_API_KEY=your-key-here
# or KIMI_API_KEY=...
EOF

# 3. Update agents/config.yaml to use the provider with the valid key
#    default_provider: dashscope  (or kimi)
#    fallback_chain: [dashscope]  (or [kimi])

Pitfalls

• API keys in .hermes/config.yaml may be truncated or expired — always test with a simple call first
• fallback_chain must only include providers with valid API keys — remove zhipu/kimi/openai if no keys set
• dotenv load_dotenv() fails in REPL mode — use manual env loading in scripts
• PipelineExecutor marks pipeline as "completed" even when ALL steps fail — check state.step_results for individual status, not just state.status
• claude CLI auth is broken (HTTP 403) — ALL models return 403 — kimi-k2.5, qwen3-coder-plus, glm-5, glm-5.1 all fail with Failed to authenticate. API Error: 403 Request not allowed. Do NOT use claude CLI for this project. Use opencode --model kimi-k2.5 --print 'prompt' as fallback for code review and patches instead.
• Critical runtime bugs (as of May 2026): _make_trace() returns list[Message] instead of list[dict] causing JSON serialization crash; _build_messages() is abstract but unimplemented in domain agents causing NotImplementedError; ModelRouter lacks exponential backoff/retry. These must be patched before any pipeline run will succeed.

Execution Script

Use scripts/run_real_pipeline.py for real LLM execution. It:

1. Loads .env manually (avoids dotenv REPL issues)
2. Initializes real Orchestrator with live LLM routing
3. Runs default 4-step pipeline with proper context
4. Prints step-by-step results with scores
5. Saves summary to outputs/pipeline-summary.json

Knowledge Base Module

Located at knowledge/. Implements Repository Pattern with SQLite backend.

File	Purpose
models.py	Entity / Relation / Source data models
repository/base.py	Abstract KnowledgeRepository interface (CRUD + search + spec)
repository/sqlite.py	SQLite backend, default .super_factory/data/knowledge.db
service.py	KnowledgeService facade: add/get/search/link/promote_tier
validator.py	EntityValidator with spec-based validation (K003-K011)
adapter.py	KnowledgeService → Hook bridge layer

Key patterns:

• Three-tier entity system: research → finding → knowledge (promote-only, no downgrade)
• Source tracking for citation provenance
• Spec validation before write (K003: must have source; K005: must match type spec)
• Currently keyword-only search (no vector/semantic search yet)
• External kg.db (~/wiki/kg.db) not yet integrated

Memory System Module

Located at memory/. File-first JSON persistent storage.

File	Purpose
types.py	MemoryEntry frozen dataclass (id, pipeline_id, agent_role, category, content, confidence, tags, created_at, expires_at)
store.py	MemoryStore: per-entry JSON files + _index.json, metadata query, keyword relevance, prune
blocked/	Blocked pipeline records
skills/	Agent skill cache

Key patterns:

• Individual JSON files per entry with index for fast querying
• Thread-safe via threading.Lock
• query() filters by pipeline_id/agent_role/category/tags, sorted by created_at desc
• relevant() keyword matching with confidence-weighted scoring
• prune(max_age_days=30) removes expired + old entries
• No vector/semantic search, no layered memory architecture yet

Pipeline Architecture

Research Agent → Planning Agent → Build Agent → Eval Agent
     ↓                ↓               ↓            ↓
  Contract         Contract        Contract     Contract
  (pre+post)       (pre+post)      (pre+post)   (pre+post)
     ↓                ↓               ↓            ↓
  Scorer          Scorer          Scorer       Scorer
     ↓                ↓               ↓            ↓
  Trace           Trace             Trace        Trace

Context flows via context_map in pipeline YAML:

• research.summary → planning.research_refs
• planning.phases → build.plan
• build → eval.target_output

Support Files

• references/contract-reference.md — Full contract precondition/postcondition specs for all 4 agents
• references/audit-log-20260509.md — Code quality audit findings (2026-05-09): critical runtime bugs, claude CLI auth failure, opencode fallback
• references/storage-audit-20260512.md — Knowledge/Memory storage current-state audit (2026-05-12)
• scripts/run_real_pipeline.py — Ready-to-run real pipeline execution script