Rootly Incident Management

Overview

Incidents are the primary resource in Rootly — the central record for any production issue, outage, or service degradation. Rootly is used by SRE and platform engineering teams to coordinate response in real time and drive continuous improvement via postmortems. In MSP environments, Rootly typically manages internal infrastructure incidents rather than per-client ticketing, but can be configured with team-based routing to support multi-customer workflows.

The incident system supports:

• Automated Detection - Incidents created from monitoring alert integrations (Datadog, PagerDuty, Prometheus, Grafana, Sentry, etc.)
• Manual Creation - On-call engineers and responders create incidents via Slack, web UI, or API
• AI-Assisted Analysis - find_related_incidents and suggest_solutions surface past patterns and recommendations
• Real-Time Coordination - Slack channel auto-creation, Zoom bridges, timeline tracking
• Post-Incident Learning - Automated postmortem generation and action item tracking

All read and write operations on incidents are available through the Rootly MCP tools.

Anti-triggers

"Incident" names three different objects across this marketplace, and the routing test is what the operator does next: coordinate a response and write it up (this skill), approve a remediation on a compromised endpoint, or answer a customer under an SLA clock.

• A confirmed security incident — malware, intrusion, a host to contain. That object carries a SOC-recommended remediation to approve, not a response team to assemble; use huntress-incidents, or sentinelone-alerts for raw EDR detections.
• A customer ticket typed "Incident" — ITIL ticket classification inside a PSA or helpdesk, where the deliverable is an SLA-timed response and a billable time entry rather than a war room; use freshdesk-ticketing, halopsa-tickets, connectwise-psa-tickets, or autotask-tickets.
• A PagerDuty incident — a different vendor with a different lifecycle (triggered → acknowledged → resolved, not detected → in_triage → mitigated → resolved → closed); use pagerduty-incidents.
• The retrospective written after resolution — postmortem content and its action items are rootly-postmortems; this skill covers action items only as live response tasks.
• What fired and who it routed to — alert sources, routing rules, and escalation policies are rootly-alerts.
• A monitoring or uptime incident — an incident opened by a failing health check, whose deliverable is a status-page update rather than a response team, is betterstack-incidents.

MCP Tools

Core Incident Tools

Tool	Description	Key Parameters
list_incidents / search_incidents	List and search incidents	filter[status], filter[severity], page[number], page[size]
get_incident	Get a single incident	Incident ID
create_incident	Create a new incident	title, severity_id, team_ids, service_ids
update_incident	Update an incident's fields	Incident ID
attach_alert	Attach an alert to an incident	incident_id, alert data
list_incident_alerts	List alerts attached to an incident	incident_id
create_incident_action_item	Create an action item on an incident	incident_id, summary, assigned_to
list_incident_action_items	List action items on an incident	incident_id

AI Analysis Tools

Tool	Description	Key Parameters
find_related_incidents	Find historically similar incidents using TF-IDF	query or incident_id
suggest_solutions	Suggest remediation steps based on past resolutions	incident_id or description

Supporting Lookups

Tool	Description
list_severities	List configured severity levels (ID, slug, color)
list_services	List services to scope incident to the right owner
list_teams	List teams for incident assignment and routing
list_incident_types	List incident types (bug, outage, performance, etc.)
list_environments	List environments (production, staging, etc.)
list_users	List users for assignment

Discover Available Tools

Call list_endpoints to get the current list of all available API endpoints and tool names from the Rootly OpenAPI specification.

Key Concepts

Incident Lifecycle

┌───────────┐   Triage starts   ┌────────────┐   Contained   ┌───────────┐
│ detected  │ ─────────────────> │  in_triage │ ────────────> │ mitigated │
└───────────┘                   └────────────┘               └───────────┘
                                                                     │
                                                               Full fix done
                                                                     ▼
                                                               ┌──────────┐
                                                               │ resolved │
                                                               └──────────┘
                                                                     │
                                                             PIR complete
                                                                     ▼
                                                               ┌────────┐
                                                               │ closed │
                                                               └────────┘

Key lifecycle timestamps on the incident record:

Timestamp	Meaning
detected_at	Alert fired or issue first observed
acknowledged_at	Responder acknowledged the page
in_triage_at	Active investigation started
started_at	Response team coordinating
mitigated_at	Immediate impact contained
resolved_at	Issue fully resolved
closed_at	Post-incident review complete
cancelled_at	False alarm; incident cancelled

Severity Levels

Severities are configurable per Rootly organization. Common conventions:

Severity	Typical Name	Description	SLA Target
SEV-1 / Critical	P1	Complete outage or data loss; business-critical impact	Immediate (15 min)
SEV-2 / High	P2	Major feature degraded; significant user impact	30 minutes
SEV-3 / Medium	P3	Partial degradation; workaround available	2 hours
SEV-4 / Low	P4	Minor issue; minimal user impact	Next business day

Note: Severity IDs are UUIDs in Rootly. Always call list_severities to map severity slugs to IDs before creating an incident.

Incident Status Values

Rootly uses free-text status descriptors alongside timestamps. Common values:

• detected — Newly created, not yet acknowledged
• in_triage — Actively being investigated
• mitigated — Impact contained, monitoring for recurrence
• resolved — Issue fixed, normal operation restored
• cancelled — False alarm or invalid incident

Field Reference

Field	Type	Description
id	string	UUID of the incident
sequential_id	integer	Human-readable incident number (e.g., INC-342)
title	string	Short summary of the incident
summary	string	Detailed description of impact and current status
status	string	Current lifecycle status
severity	object	Severity record with slug, color, description
services	array	Affected services
environments	array	Affected environments (production, staging)
teams	array	Teams assigned to respond
labels	array	Custom tags for filtering and reporting
started_at	datetime	When the incident started
resolved_at	datetime	When the incident was resolved (null if open)
slack_channel_id	string	Auto-created Slack channel for coordination
url	string	Web UI link to the incident

Common Workflows

Triage New Incidents

1. Call list_incidents with filter[status]=in_triage or filter[status]=detected, sorted by severity
2. For each high-severity incident, review title and summary
3. Call find_related_incidents with the incident_id to surface similar past incidents
4. Call suggest_solutions with the incident_id to get AI-generated remediation suggestions
5. Create action items via create_incident_action_item for each remediation step
6. Update status as the incident progresses via update_incident

Create an Incident

1. Call list_severities to find the correct severity ID for the impact level
2. Call list_services to identify which service is affected
3. Call list_teams to find the on-call team to assign
4. Call create_incident with title, severity_id, service_ids, team_ids
5. Attach any triggering alerts via attach_alert

Investigate a Specific Incident

1. Find the incident via get_incident or list_incidents filtered by sequential_id
2. Call find_related_incidents — look for recurring patterns (same service, same time of day, same symptoms)
3. Call suggest_solutions — surface past resolutions that worked for similar incidents
4. Review action items via list_incident_action_items
5. Review attached alerts via list_incident_alerts

Daily Incident Review

1. Call list_incidents with a 24-hour window and filter[status]=resolved
2. Count by severity for a daily health summary
3. Identify incidents that exceeded SLA targets (compare detected_at vs resolved_at)
4. Flag any incidents without postmortem action items for follow-up

Cross-Vendor PSA Ticket Correlation

Rootly incidents often correspond to PSA service tickets for MSP billing:

1. When an incident is created or resolved, create a matching ticket in your PSA (ConnectWise, HaloPSA, Autotask, etc.)
2. Include the Rootly sequential_id (e.g., INC-342) and url in the PSA ticket body
3. Map Rootly severity → PSA priority: SEV-1 → Critical, SEV-2 → High, SEV-3 → Medium, SEV-4 → Low
4. When the Rootly incident resolves, update the PSA ticket with resolution summary from summary
5. Use action items as sub-tasks in the PSA ticket for follow-up work

On-Call Handoff Summary

Before handing off to the next on-call engineer:

1. Call get_oncall_handoff_summary to see current and next on-call status plus open incidents
2. Call list_incidents with filter[status]=in_triage to review any actively open incidents
3. Add handoff notes as action items on open incidents
4. See the oncall skill for full on-call workflows

Error Handling

Common Errors

Error	HTTP Code	Resolution
Invalid API token	401	Regenerate at Account > Manage API Keys
Insufficient permissions	403	Token may be Team-scoped; use an Account or Global token
Incident not found	404	Verify incident_id; use list_incidents to list active incidents
Severity ID invalid	422	Call list_severities to get valid IDs before creating
Rate limited	429	Back off 30 seconds; retry with exponential backoff

Authentication Error

401 Unauthorized

Verify your Rootly API token:
- Generate at Account > Manage API Keys
- Token type: Global (full access) or Team (limited to team resources)
- Pass as: Authorization: Bearer <token>

Best Practices

1. Use AI analysis first — Always call find_related_incidents and suggest_solutions before manual investigation
2. Track action items — Every resolved incident should have at least one follow-up action item
3. Attach alerts — Link the triggering alert to the incident for audit trail completeness
4. Scope to the right team — Include team_ids when creating incidents so on-call routing works correctly
5. Map severity consistently — Use list_severities to confirm slug-to-ID mapping rather than hardcoding
6. Use sequential IDs in external tools — Reference INC-{sequential_id} in PSA tickets and Slack
7. Check on-call health — Use check_oncall_health_risk before major deployments or planned maintenance

Related Skills

• On-Call Management — On-call schedules, handoffs, shift metrics
• API Patterns — Auth, pagination, all available tools