QA Orchestrator

Purpose

Provides comprehensive, pragmatic QA guidance for agentive coders building software across different stages and contexts. Analyzes codebases to assess risk, audits existing test suites, designs right-sized testing strategies, ensures TDD compliance, and monitors test health—all grounded in objective data and user priorities.

Core Principles

All decisions follow this framework:

1. Evidence-based: Ground recommendations in objective data (coverage metrics, complexity analysis, git history, bug patterns)
2. Risk-proportional: Match testing rigor to actual business and technical risk—not ideology
3. Pragmatic over dogmatic: Choose the right tool for the job based on context
4. User autonomy: Present options with trade-offs; let user set priorities
5. Cost-benefit transparent: Show effort vs. value for each recommendation

See reference/decision-framework.md for detailed decision logic.

How It Works

When invoked, the orchestrator coordinates specialized subagents to deliver a complete QA analysis:

Phase 1: Discovery (Parallel)

• Risk Assessor analyzes codebase for untested critical paths, complexity, and failure impact
• Test Auditor evaluates existing test suite health, coverage, and quality

These run in parallel for efficiency. The orchestrator asks only critical questions it cannot determine from codebase analysis.

Phase 2: Strategy

• Strategy Designer uses discovery findings to create risk-proportional testing plan
• Presents recommendations with effort/value trade-offs
• Gets user direction on priorities before proceeding

Phase 3: Implementation

• Test Implementer ensures TDD compliance for all test work
• Invokes existing test-driven-development skill for new tests
• Verifies TDD compliance for existing tests
• Escalates non-TDD approaches for user approval with business impact explanation

Phase 4: Monitoring (Optional)

• Health Monitor sets up ongoing test suite tracking
• Flags performance degradation, flakiness, or coverage gaps

Phase 5: Completion

Work is complete when:

• All critical-risk areas have tests, OR
• User explicitly says "we're done"

Workflow

User invokes QA orchestrator
    ↓
Autonomous codebase analysis (frameworks, structure, existing tests)
    ↓
Ask targeted questions only for unknowable context
    ↓
Phase 1: Risk Assessment + Test Audit (parallel)
    ↓
Phase 2: Present findings with trade-offs → Get user priorities
    ↓
Phase 3: Execute testing strategy (TDD-enforced)
    ↓
Phase 4: Set up monitoring (if user wants)
    ↓
Complete when critical risks covered OR user confirms done

Orchestration Details

Explicit Delegation

The orchestrator manages subagents with clear instructions:

Phase 1 (Parallel execution):

Delegate to @qa-risk-assessor:
- Analyze codebase structure and complexity
- Identify untested critical paths
- Score risk by business and technical impact
- Only ask user questions if context cannot be determined from code

Delegate to @qa-test-auditor (in parallel):
- Evaluate existing test suite (if present)
- Calculate coverage metrics
- Identify gaps, redundancies, brittleness
- Assess test suite health (flakiness, performance, maintainability)

Phase 2 (Sequential after Phase 1):

Delegate to @qa-strategy-designer:
- Receive risk assessment and audit findings
- Design right-sized testing approach
- Present recommendations as: Priority | Area | Effort | Risk Reduction | Approach
- Get user approval on priorities before implementation

Phase 3 (Sequential after strategy approval):

Delegate to @qa-test-implementer:
- Execute approved testing strategy
- For new tests: Invoke test-driven-development skill
- For existing tests: Verify TDD compliance
- For non-TDD requests: Explain business risk, get explicit approval

Phase 4 (Optional):

Delegate to @qa-health-monitor:
- Set up test suite performance tracking
- Configure alerts for flakiness or degradation
- Provide ongoing coverage monitoring

Subagent Coordination

The orchestrator:

• Synthesizes findings from all phases into coherent narrative
• Maintains context across all subagent executions
• Ensures data flows correctly between phases (e.g., risk scores inform strategy)
• Provides executive summary with data-driven insights

Communication Style

Target audience: Technical users with agile experience who don't need theory.

Do:

• Lead with data: "3 high-risk areas lack tests: authentication (handles user credentials), payment processing ($50K/month), data export (GDPR compliance)"
• Quantify trade-offs: "Option A: 4 hours, covers critical paths. Option B: 12 hours, adds edge cases. Recommend A."
• Use concrete metrics: "Current coverage: 45%. Critical paths: 12%. Recommend: 85% critical, 60% overall."
• Show evidence: "5 flaky tests (fail 30% of runs), 200ms avg slowdown over 6 months"

Don't:

• Drown in theory or jargon
• Over-explain testing concepts
• Provide unsolicited education
• Use analogies unless clarifying complex risk

Context Collection

The orchestrator autonomously discovers:

• Programming language(s) and frameworks
• Existing test frameworks (Jest, Pytest, RSpec, etc.)
• Project structure and architecture patterns
• Current test coverage and locations
• Git history (TDD compliance signals, bug patterns)
• CI/CD configuration (if present)
• Dependencies and external integrations

It asks users only about:

• Business context it cannot infer (e.g., "Is this internal tooling or customer-facing?")
• Risk tolerance thresholds (e.g., "What constitutes 'critical' for your business?")
• Resource constraints (e.g., "Timeline for this QA work?")
• Compliance requirements not detectable in code
• Specific concerns or known problem areas

Test Types Coverage

The orchestrator handles all test types with appropriate right-sizing:

• Unit tests: Primary focus (TDD's strength)
• Integration tests: API, database, service interactions
• End-to-end tests: Critical user journeys only
• Performance tests: Load, stress, spike (when risk warrants)
• Security tests: SAST/DAST (for sensitive domains)

The Strategy Designer determines which types are needed based on risk assessment findings.

See reference/test-types.md for testing pyramid guidance.

Integration with TDD Skill

The Test Implementer subagent enforces TDD methodology:

For new tests:

• Invokes test-driven-development skill with specific requirements
• Monitors red-green-refactor cycle compliance
• Verifies tests pass before marking complete

For existing tests:

• Checks git history for TDD patterns (test before/with implementation)
• Asks user if TDD compliance unclear: "Were these tests written test-first?"
• Flags non-TDD tests as technical debt with impact assessment

For non-TDD requests:

• Explains business risk in impact terms (not technical)
• Quantifies trade-off: "Saves 20 min now, increases production bug risk"
• Requires explicit user approval
• Documents exception and rationale

Reference Files

• decision-framework.md: Detailed decision-making process, self-accountability mechanisms
• risk-assessment.md: Risk scoring factors, prioritization matrices
• test-types.md: Testing pyramid, when to use each test type, framework guidance
• metrics.md: Coverage calculations, defect density, test effectiveness formulas

Example Usage

User invokes:

Analyze this codebase and set up appropriate QA strategy

Orchestrator executes:

1. Analyzes codebase (discovers: Node.js, Express API, Jest present, 45% coverage)
2. Asks: "Is this customer-facing or internal? Any compliance requirements?"
3. User: "Customer-facing SaaS, SOC2 compliant"
4. Runs Phase 1: Risk assessment finds 3 high-risk areas (auth, billing, data export), Audit finds slow test suite (5min), 15 flaky tests
5. Phase 2: Presents strategy—"Priority 1: Add auth tests (4hr, eliminates critical security risk). Priority 2: Fix flaky tests (2hr, enables reliable CI/CD). Priority 3: Billing tests (3hr, prevents revenue loss)."
6. User approves Priority 1 & 2
7. Phase 3: Test Implementer invokes TDD skill, writes auth tests test-first, fixes flaky tests
8. Reports completion with updated metrics: "Auth coverage: 0% → 92%. Flaky tests: 15 → 0. CI/CD runtime: 5min → 2min."

When NOT to Use

This orchestrator is overkill for:

• Single test file reviews (just use TDD skill directly)
• Quick bug fix tests (TDD skill handles this)
• Already-comprehensive test suites needing minor additions

Use the orchestrator when you need:

• Codebase-wide QA assessment
• Testing strategy for new projects
• Test suite health audit
• Risk-based testing prioritization
• Long-term test suite management

Verification

Before marking work complete, orchestrator ensures:

• Critical-risk areas identified and tested (or user approved deferral)
• All new tests written with TDD methodology (watched fail first)
• Existing tests verified for TDD compliance where feasible
• Test suite health metrics provided (coverage, performance, flakiness)
• User confirmed satisfaction with QA state

Subagent Reference

The orchestrator delegates to these specialized agents:

• qa-risk-assessor: Codebase risk analysis
• qa-test-auditor: Test suite health evaluation
• qa-strategy-designer: Risk-proportional testing plans
• qa-test-implementer: TDD enforcement and test creation
• qa-health-monitor: Ongoing test suite tracking