sre-pro

Overview

SRE applies engineering discipline to operations: define reliability in measurable terms, automate the toil away, and make deliberate trade-offs between shipping speed and stability. This skill covers the core SRE framework — SLIs, SLOs, error budgets — and the practices that sustain reliability as systems grow.

When to use

• Defining SLIs (indicators) and SLOs (objectives) for a service
• Setting up error budgets and the policies they drive
• Reducing operational toil through automation
• Designing for reliability: redundancy, graceful degradation, capacity
• Building an SRE practice or reliability review process on a team

Core concepts

SLIs measure, SLOs promise, SLAs contract. A Service Level Indicator is a quantified metric (99th-percentile latency, successful-request ratio). The Objective is the target (99.9% of requests succeed over 30 days). The Agreement adds business consequences. Most teams need SLIs + SLOs; SLAs only where contracts demand them.

Error budgets make reliability a resource. 99.9% SLO = 0.1% budget for failure (~43 min/month). Spend it on releases and experiments; when it's exhausted, freeze non-essential changes until reliability recovers. This turns "move fast vs be stable" from an argument into arithmetic.

Toil is the enemy. Toil = manual, repetitive, automatable operational work with no enduring value. Track it (aim for <50% of on-call time, trending down); every toil task is a candidate for automation or elimination. Toil that grows with traffic is a scaling bug.

Design for failure. Redundancy across zones, graceful degradation (serve stale cache rather than error), timeouts and retries with backoff and jitter, bulkheads and circuit breakers to contain cascading failures, and load shedding under extreme pressure. Reliability is a property of the architecture, not of hope.

Blameless culture. Postmortems focus on systemic causes, never individual fault. People don't cause outages; systems that allow human error to become outages do. Psychological safety is a reliability control.

Practical workflow

1. Choose SLIs from the user's perspective: availability (successful responses / valid requests), latency (p50/p99 of successful requests), and domain-specific ones (freshness, correctness, durability) — 2–4 per service, not 20.
2. Set SLOs deliberately: start slightly below current performance (achievable, not aspirational), with a 28–30 day rolling window; multi-window burn-rate alerting (fast-burn page, slow-burn ticket).
3. Implement the error budget policy: define what happens at 50%, 100%, and sustained exhaustion (feature freeze, reliability sprint); get leadership sign-off so it's enforceable.
4. Instrument everything: the "four golden signals" (latency, traffic, errors, saturation) per service, plus business-level indicators; dashboards that a new on-call can read at 3am.
5. Attack toil systematically: list recurring manual tasks, estimate hours/quarter, automate the highest-cost ones first; measure toil percentage over time.
6. Run blameless postmortems for every significant incident: timeline, contributing factors, action items with owners and deadlines — and track action-item completion as a reliability metric.

Common pitfalls

• Too many SLIs/SLOs — 15 objectives nobody looks at; a few meaningful ones beat comprehensive-but-ignored.
• SLOs nobody enforces — an error budget without a freeze policy is theater; get the policy agreed before the first exhaustion.
• Alerting on symptoms instead of burn rate — pages on CPU spikes while users are fine (or silence while the budget burns); alert on SLI burn.
• Aspirational SLOs (five nines on a best-effort service) — unachievable targets teach the team to ignore SLOs entirely.
• Toil normalized as "just ops work" — manual deploys, ticket-driven routine changes, and hand-run runbooks that never get automated compound until the team drowns.
• Skipping postmortems for "small" incidents — patterns hide in the small ones; the discipline matters more than the severity threshold.