Webhooks — the inbound front door that survives retries and forgeries

You are building the side of a webhook that receives. Some external system pushes an HTTP POST at your endpoint; your job is to prove it is real, refuse to act on it twice, and answer fast. That is the whole mandate.

This skill is provider-agnostic — it teaches the mechanics every webhook source shares, not any one vendor's event catalog — and it ends at "enqueued." What happens to the event afterward is a different job:

• The Stripe-specific scheme (Stripe-Signature, t=,v1=, stripe listen, the event model) → ../stripe/SKILL.md.
• Email bounce/complaint webhooks where the point is suppression state → ../email-connector/SKILL.md.
• The outbound client that calls someone else's API → api-connector-builder.
• Multi-step orchestration after the event lands → automation-flows.
• Broker/queue tuning (BullMQ concurrency, DLQ ops) → redis.
• Constant-time compare, secret handling, supply-chain hygiene → ../secure-coding/SKILL.md.

The pipeline

These five stages run in exactly this order. Each one exists to protect the one after it; reorder them and you either trust forged data or waste work.

#	Stage	Why it precedes the next
1	Read the raw body	Parsing first destroys the bytes the signature was computed over. Capture the raw buffer before any JSON parse.
2	Verify the signature	Until this passes, every field in the body is attacker-controlled. Verify before you read anything.
3	Check the timestamp window	A cheap reject of replayed-but-valid payloads, before you touch a datastore.
4	Dedupe on the event id	At-least-once delivery means duplicates are normal. Mark it seen before doing work, not after.
5	Persist/enqueue → 2xx	Hand the event to durable storage, then ack. The ack means "I own this now."

Delivery is at-least-once, never exactly-once. Every major provider retries on non-2xx or timeout, so duplicate deliveries are normal traffic and idempotency is required, not a nice-to-have.

Verify the signature

Verify over the raw bytes, never over re-serialized JSON. Re-serializing a parsed object reorders keys and normalizes whitespace, so the HMAC you compute no longer matches the one the sender computed. This is the single most common cause of "signature fails even though the secret is right."

// Bad — body was parsed, so the bytes are gone. HMAC will never match.
app.post("/webhooks", express.json(), (req, res) => {
  const expected = sign(JSON.stringify(req.body)); // reordered, re-spaced
});

// Good — keep the raw buffer for the HMAC; parse only after verifying.
app.post("/webhooks", express.raw({ type: "*/*" }), (req, res) => {
  const raw = req.body; // a Buffer, the exact bytes received
  if (!verify(raw, req.headers)) return res.sendStatus(400);
  const event = JSON.parse(raw.toString("utf8"));
});

Compare signatures in constant time. A normal ===/== short-circuits on the first differing byte, which leaks how much of the signature an attacker guessed correctly. Use crypto.timingSafeEqual (Node) or hmac.compare_digest (Python). The buffers must be equal length, so guard that first.

The cross-vendor Standard Webhooks spec (adopted by Svix, Clerk, Resend, Brex and others) defines the scheme this skill defaults to: headers webhook-id, webhook-timestamp, webhook-signature; the signature is base64(HMAC-SHA256(secret, "{id}.{timestamp}.{body}")), version-prefixed as v1,<sig>, and the header may carry space-separated signatures so a secret can be rotated without dropped deliveries — accept the request if any match.

import crypto from "node:crypto";

function verify(raw, headers) {
  const id = headers["webhook-id"];
  const ts = headers["webhook-timestamp"];
  const secret = process.env.WEBHOOK_SECRET; // never a literal
  const signed = `${id}.${ts}.${raw.toString("utf8")}`;
  const expected = crypto
    .createHmac("sha256", Buffer.from(secret, "base64"))
    .update(signed)
    .digest("base64");
  // header is "v1,<sig> v1,<sig2>"; accept if any matches (secret rotation)
  return String(headers["webhook-signature"] || "")
    .split(" ")
    .map((part) => part.split(",")[1])
    .some((sig) => sameLength(sig, expected) &&
      crypto.timingSafeEqual(Buffer.from(sig), Buffer.from(expected)));
}
const sameLength = (a = "", b = "") => a.length === b.length;

import hmac, hashlib, base64, os

def verify(raw: bytes, headers) -> bool:
    secret = base64.b64decode(os.environ["WEBHOOK_SECRET"])  # never a literal
    signed = f'{headers["webhook-id"]}.{headers["webhook-timestamp"]}.'.encode() + raw
    expected = base64.b64encode(hmac.new(secret, signed, hashlib.sha256).digest()).decode()
    sent = headers.get("webhook-signature", "")
    for part in sent.split(" "):          # space-separated for rotation
        sig = part.split(",", 1)[-1]
        if hmac.compare_digest(sig, expected):  # constant time
            return True
    return False

Per-provider header formats (Stripe t=,v1=, GitHub X-Hub-Signature-256, Shopify base64 HMAC, Slack v0=, Svix) all map onto the same primitive — mapping table and per-language verify primitives in references/signature-schemes.md.

Timestamp & replay

A valid signature does not stop a replay: an attacker who captures one delivery can re-send the exact bytes, signature intact. Bound it with a timestamp.

Reject any event whose webhook-timestamp is more than ~5 minutes from now. That five-minute window is the de-facto industry value (Benchling and others recommend it). The Standard Webhooks spec mandates that you perform a tolerance check but does not fix the number — pick a small one.

const skewSeconds = Math.abs(Date.now() / 1000 - Number(ts));
if (skewSeconds > 300) return res.sendStatus(400); // outside the 5-min window

The timestamp window bounds how long a captured payload is replayable; the idempotency check (next section) stops exact replays that arrive inside the window. You need both — neither alone is enough.

Idempotency

The idempotency key is the provider's stable event id (webhook-id, or for Stripe event.id) — never a hash of the body or your own generated id. Mark it seen before doing the work, so a retry that arrives mid-processing is caught.

Two stores, same idea:

Store	Mechanism	TTL / lifetime
Relational DB	INSERT the event id into a table with a UNIQUE constraint; a duplicate insert throws → skip	Keep the row at least as long as the retry window
Redis	SET key val NX EX <ttl> — SETNX returns false if already present → skip	TTL ≥ the provider's retry window (hours to days)

The dedupe TTL must outlive the provider's entire retry schedule. Retries are exponential and capped — a provider may redeliver hours or days later. If your key expires first, that late retry looks brand-new and you process twice.

// Insert-then-process: the UNIQUE insert is the gate. Do work only if we won it.
try {
  await db.query("INSERT INTO seen_events (id) VALUES ($1)", [id]);
} catch (e) {
  if (e.code === "23505") return res.sendStatus(200); // already seen → ack, skip
  throw e;
}
await queue.add("process", { id, event });
res.sendStatus(200);

Prefer insert-then-process over process-then-mark. Marking after the work leaves a window where two concurrent deliveries both pass the "have I seen it?" check and both run. Let the UNIQUE constraint (or NX) be the lock.

Fast-ack & queue

Heavy synchronous work inside the handler causes provider timeouts and a storm of retries. Return 2xx as soon as the event is durably stored or enqueued, then do the real work in a background worker. The status code is a contract with the sender:

You return	Meaning to the provider	Use it when
2xx	"I own this now"	After enqueue/persist — even if downstream processing later fails. Recover failures from your own queue, not by asking for a redelivery.
4xx	Permanent reject, do not retry	Bad/forged signature (400/401), or a payload you will never accept. A retry of a forged request is still forged.
5xx / timeout	"Try again"	A genuine transient failure before you took ownership (e.g. the queue itself is down).

Once work is in the queue, handle failures there: bounded retries with backoff, then route exhausted jobs to a dead-letter queue / poison shelf for manual inspection. Do not let a poison event loop forever or block the queue head. Queue/broker tuning itself (concurrency, DLQ wiring) is redis's job.

Framework raw-body recipes

Capturing the raw body is framework-specific and is the usual culprit behind "mismatch though the secret is correct." One line each; full recipes in references/framework-raw-body.md:

• Express — mount express.raw({ type: "*/*" }) on the webhook route only; do not let a global express.json() consume it first.
• Next.js App Router — read await req.text() (or req.arrayBuffer()) in the route handler; the App Router does not auto-parse, but await req.json() discards the raw bytes, so verify off the text.
• FastAPI / Starlette — await request.body() for the raw bytes; do not type the handler param as a Pydantic model (that parses for you).
• Hono — await c.req.text() before any c.req.json().
• Serverless (Lambda/Vercel/Cloudflare) — disable body parsing / read the raw stream; base64-decode if the platform wraps the body.

Anti-patterns

Anti-pattern	Why it bites	Do instead
HMAC over parsed/re-serialized JSON	Key reorder + whitespace break the signature ("works with the right secret but still fails")	Verify over the raw bytes captured before parsing
=== / == on the signature	Timing side-channel leaks the secret byte by byte	crypto.timingSafeEqual / hmac.compare_digest
Heavy synchronous work in the handler	Provider times out → retry storm → duplicate processing	Fast-ack: verify → dedupe → enqueue → 2xx, work in a worker
Ack before persisting/enqueueing	2xx means "I own it"; if you crash now the event is lost forever (provider won't retry a 2xx)	Persist or enqueue first, then return 2xx
No dedupe / dedupe on a body hash	At-least-once delivery double-processes; body hashes drift across redeliveries	UNIQUE-constraint or SETNX on the provider's stable event id
Dedupe TTL shorter than the retry window	A late retry looks new and runs again	TTL ≥ the provider's full retry schedule (hours–days)
Signing secret hard-coded in source	Leaks in git history; can't rotate	Read from env; accept space-separated sigs for rotation
Returning 200 on a bad signature	Silently swallows forged traffic and masks misconfig	400/401 on signature failure
Skipping the timestamp check	Valid-but-replayed payloads sail through	Reject outside the ~5-min window

Verify your handler

scripts/verify.sh is a read-only heuristic linter. Run it from the root of the project that contains your handler; it scans candidate files and prints PASS/WARN/FAIL per invariant (raw-body verify, constant-time compare, timestamp check, secret-from-env, dedupe-on-event-id). A WARN means "I could not find evidence," which on a clean tree is expected and exits 0.