Set Up Alerting

An alert rule is code: one rule set per environment, applied by the pipeline that applies the service, promoted by the promotion that promotes the service. Signals, resource attributes, SLO burn-rate: observability; log fields: logging-contract; telemetry path: deploy-otel-collector; the why: alerting-patterns.md.

1. Audit current state (change nothing)

cat .claude/stack-profile.md 2>/dev/null   # observability.backend, observability.otel, iac, ci
ls alerts/ 2>/dev/null; ls terraform/alerts.tf 2>/dev/null
grep -rn --include='*.tf' -iE 'signoz|grafana|monitor_metric_alert|monitor_scheduled_query|cloudwatch_metric_alarm' . | head
grep -rn --include='*.tfvars' -iE 'alert|threshold' terraform/envs/ 2>/dev/null
gh workflow list 2>/dev/null               # an observability workflow beside the infra pipeline?

Then list the rules in the backend itself (read-only API or UI): name, query, threshold, window, route, runbook, owner. Rules that exist only in the UI are unmanaged drift; record them for step 2.

2. Decide what to do

• No rules in code or backend → full setup (steps 3 to 7).
• Rules only in the backend UI → codify: write the catalogue from what the API returns, import the rules into state or recreate them — one of the two, never both.
• Rules in code, but a separate observability workflow applies them only from main → move them into the same per-environment pipeline (step 6).
• Catalogue, code, per-env pipeline and read-back all present → "already in place"; run step 7 only.

3. Detect the track

observability.backend	Rules	As code
signoz (default)	alert rules, notification channels	Terraform provider SigNoz/signoz — unverified: source address and current version must be re-verified on registry.terraform.io — or the SigNoz REST API
grafana	rule groups, contact points	grafana provider (grafana_rule, grafana_contact_point) — unverified, same registry check
Azure Monitor	metric alerts, log alerts	azurerm provider: azurerm_monitor_metric_alert, azurerm_monitor_scheduled_query_rules_alert
CloudWatch	metric alarms, composite alarms	aws provider: aws_cloudwatch_metric_alarm; log-based alerts via metric filters

The backend is an exporter setting (observability); if the profile names none, ask. Queries below use SigNoz/OTel names; other tracks translate.

4. Install only what is missing

• The Terraform toolchain is already there (set-up-opentofu); the alerts root adds one provider, pinned and locked like the others.
• One SigNoz API key per environment, as a CI environment secret (manage-secrets); never in tfvars.
• Notification channels (Slack, Teams, email, webhook) need webhook URLs. A provider-managed channel writes the URL into state: one dedicated, revocable webhook per channel, rotated — or channels by hand, referenced by id.

5. Generate

The alert catalogue first — alerts/catalogue.md in the target repo, or the same table as Terraform locals: the single source the rules are generated from, the artifact the monthly noise review reads. Name format [P<n>][<env>] <service> — <symptom>:

Name	Query	Threshold (dev/stg/prd)	Window	Route	Runbook	Owner
[P0][<env>] <service> — fatal log lines	level = 'fatal' on the service's logs	any / any / any	5 m	page	<runbook-url>	<owner>
[P1][<env>] <service> — error rate	5xx ratio from http.server.request.duration	5% / 1% / 0.5%	10 m	chat channel	<runbook-url>	<owner>
[P1][<env>] <service> — latency burn	SLO burn rate, two windows (observability)	14.4× over 1 h, 6× over 6 h	1 h + 6 h	chat channel	<runbook-url>	<owner>
[P2][<env>] <service> — job failed	event = 'job.failed'	≥ 1 / ≥ 1 / ≥ 1	15 m	ticket	<runbook-url>	<owner>
[P2][<env>] <service> — queue age	oldest message age	10 m / 5 m / 2 m	5 m	ticket	<runbook-url>	<owner>
[P1][<env>] <service> — heartbeat missing	count of event = 'job.completed'	< 1 in 25 h	25 h	chat channel	<runbook-url>	<owner>

Per-environment thresholds via tfvars — one number variable per threshold, one value per environment file:

# terraform/envs/dev.tfvars
alert_error_rate_threshold = 0.05   # stg 0.01, prd 0.005 — one file per env

The rules. Every HCL snippet below is unverified — written from the provider's documented shape, never run; confirm source, version and attribute names on registry.terraform.io first.

# terraform/alerts.tf — provider block once per root (*unverified*)
terraform {
  required_providers {
    signoz = { # *unverified* — re-verify source address and version on registry.terraform.io
      source = "SigNoz/signoz", version = "0.x.y" # placeholder — pin the current registry version
    }
  }
}

provider "signoz" {
  api_url = "https://<signoz-host>" # self-hosted host, or the SigNoz Cloud URL of <org>
  api_key = var.signoz_api_key      # CI environment secret for this env
}

locals {
  # the value the exporter writes — a query that spells it differently never matches
  env_attr = { dev = "development", stg = "staging", prd = "production" }[var.env]
}

Log-based alert:

resource "signoz_alert" "fatal_log" { # *unverified* resource and attribute names
  name        = "[P0][${var.env}] ${var.service} — fatal log lines"
  description = "Any fatal line in the service logs. Runbook <runbook-url>, owner <owner>."
  rule_type   = "log"
  query       = "level = 'fatal' AND service.name = '${var.service}' AND deployment.environment.name = '${local.env_attr}'"
  labels      = { severity = "P0", service = var.service, env = var.env }
}

Metric / error-rate alert:

resource "signoz_alert" "error_rate" { # *unverified*
  name        = "[P1][${var.env}] ${var.service} — error rate"
  description = "Failed-request ratio above the env threshold. Runbook <runbook-url>, owner <owner>."
  rule_type   = "metric"
  query       = "sum(rate(http_server_request_duration_count{status_code=~'5..'}[10m])) / sum(rate(http_server_request_duration_count[10m]))" # shape only — *unverified*
  threshold   = var.alert_error_rate_threshold
  labels      = { severity = "P1", service = var.service, env = var.env }
}

Heartbeat / absence alert (cron jobs, pipelines):

resource "signoz_alert" "heartbeat" { # *unverified*
  name        = "[P1][${var.env}] <service> — heartbeat missing"
  description = "No job.completed event in 25 h. Runbook <runbook-url>, owner <owner>."
  rule_type   = "log"
  query       = "event = 'job.completed' AND service.name = '${var.service}' AND deployment.environment.name = '${local.env_attr}'"
  threshold   = 1 # fires when the count in the window is below this
  window      = "25h" # job period plus slack
  labels      = { severity = "P1", service = var.service, env = var.env }
}

Notification channel:

resource "signoz_notification_channel" "oncall_chat" { # *unverified*
  name = "<team>-oncall-chat"
  type = "slack" # slack | msteams | email | webhook
  # endpoint or recipients per the provider docs; webhook URL handling per step 4
}

6. Wire

• Same pipeline, same promotion. The alert resources live in the same Terraform root per environment as everything else: every rule reaches dev with its PR, stg only after promote-stg, prd only after promote-prd (set-up-terraform-pipeline, branch-promotion). An observability workflow that applies only from main never reaches stg — the failure mode this wiring prevents.
• Post-apply read-back. After every apply, list the rules through the API and assert each catalogue name exists exactly once and every channel reference resolves (commands in step 7).

7. Verify

# read-back — *unverified* endpoint; confirm the path in the SigNoz API docs
curl -fsS -H "Authorization: Bearer $SIGNOZ_API_KEY" "https://<signoz-host>/api/v1/rules" > rules.json
jq -r '.data[].name' rules.json | sort | uniq -d          # expect: no output — nothing exists twice
jq -r '.data[].name' rules.json | grep -cxF '[P0][dev] <service> — fatal log lines'   # expect: 1

• Test-fire every new alert in dev before it promotes further: trigger one real matching event (a request that returns 500, one forced fatal log line, skip one heartbeat emission), then check the route: one card, once, on the right channel — not zero (query or route broken), not two (duplicate routes). Resolve it and confirm it clears; an alert that cannot be cleared in dev cannot be cleared in prd at 03:00.

References

• alerting-patterns.md: symptom choice, severity routing, per-env thresholds, the stuck-gauge and negative-filter traps, heartbeats, CI/CD health, noise review.
• Signals, SLOs, burn-rate windows: observability; log fields: logging-contract; telemetry path: deploy-otel-collector.
• The pipeline alerts ride on: set-up-terraform-pipeline, branch-promotion.