NATS & JetStream Distributed Messaging AI Skill Guide

Overview & Engine Architecture

Core NATS is fire-and-forget pub/sub (at-most-once on lossy paths). JetStream adds persistence, replay, consumer acks, KV/Object stores, and Raft-replicated metadata. Publish deduplication (Nats-Msg-Id + stream duplicate_window) prevents duplicate records in the stream; consumer delivery remains at-least-once unless handlers are idempotent and ack timing is correct.

Publishers (TLS, creds/JWT)
    -> NATS cluster (4222 client / 6222 route)
        -> JetStream streams (limits/workqueue/interest retention)
        -> consumers (pull preferred in prod)
Subscribers ack/nak/in-progress

When to use / when not to

Use when

Lightweight cloud-native messaging, request-reply, edge telemetry fanout
Work queues with JetStream WorkQueue retention (one consumer per filter subject)
KV/Object config with TTL and history limits

Do not use when

You need Kafka-grade log retention analytics ecosystem without ops appetite
Global ordering across all messages without partition/key design
Marketing “exactly-once delivery” without idempotent consumers—correct to at-least-once + dedupe

Operational Capabilities & Agent Directives

Subject design: Hierarchical app.region.entity.action; scope stream subjects narrowly—avoid catch-all > on high-throughput streams without filters.
Publish dedupe: Set Nats-Msg-Id to stable business IDs; size duplicate_window to retry horizon—not a substitute for consumer idempotency (streams docs).
Pull consumers (prod): Durable pull with explicit batch/max_bytes; keep MaxAckPending ≥ batch size; avoid huge batches if processing is sequential (pull consumers).
Ack timing: AckWait must exceed p99 processing or send in-progress heartbeats; short AckWait causes double delivery while work still runs (ack docs). duplicate_window ≠ AckWait (nats-server #6628).
Backpressure: Core NATS slow consumer drops—move to JetStream pull; tune fetch batch vs memory; scale consumer workers.
Auth/TLS: Operator mode/JWT or NKeys; TLS everywhere; never commit creds files to git.
Backup/restore: JetStream file store snapshots / server backup procedures; restore drills on staging; WorkQueue streams delete messages on ack—design retention accordingly.

Production examples

Stream + deduped publish (TypeScript)

import { connect, JSONCodec, headers } from "nats";

const nc = await connect({ servers: ["nats://127.0.0.1:4222"], tls: { /* caFile */ } });
const jsm = await nc.jetstreamManager();
await jsm.streams.add({
  name: "ORDERS",
  subjects: ["orders.*"],
  retention: "limits",
  max_bytes: 1024 ** 3,
  duplicate_window: 120 * 1e9, // 2m publish dedupe window
});

const hdrs = headers();
hdrs.set("Nats-Msg-Id", `order-${order.orderId}`);
await nc.jetstream().publish("orders.created", codec.encode(order), { headers: hdrs });

Durable pull consumer

const consumer = await js.consumers.get("ORDERS", "order-worker");
const iter = await consumer.fetch({ max_messages: 10, expires: 30_000 });
for await (const msg of iter) {
  try {
    await processIdempotent(msg);
    msg.ack();
  } catch (e) {
    msg.nak(millis(2000));
  }
}

Work queue constraints

WorkQueue retention: one durable consumer per overlapping subject filter; message deleted after ack.

Technical troubleshooting matrix

Failure signature	Root cause	Diagnostic & fix
Slow consumer (core)	Push faster than handler	JetStream pull; scale workers
Duplicate deliveries	AckWait too short / no ack	Raise AckWait; in-progress; idempotent handler
Duplicate publishes retried	Missing `Nats-Msg-Id`	Stable msg id + duplicate_window
Stalled pull	`MaxAckPending` too low	Raise pending ≥ batch; ack faster
`No storage` / disk full	Stream limits / volume	`nats stream info`; expand PVC
Raft leader issues	Even-sized cluster / partition	Odd node count; check route mesh
Lost ack redelivery	Network drop after process	Double ack mode for critical ops (delivery doc)

Best practices

Use interest or limits retention for fanout analytics; workqueue for job processing.
Terminate poison messages (term()) after N deliveries with alert.
Monitor stream bytes, consumer ack floor lag, and redelivery counts.
Supercluster/disaster recovery: document RPO/RTO separately for meta vs file store.

Limitations

Cross-region active/active is non-trivial; expect eventual consistency between sites.
JetStream performance depends on disk fsync and batch sizes.
Not a drop-in replacement for Kafka log compaction semantics in all cases.

Related skills

@kafka — heavier log-oriented streaming
@rabbitmq — classic broker queues
@temporal — durable orchestration over activities

Agent Operational Directive

MANDATORY: Do not equate duplicate_window with exactly-once processing. Use pull consumers with explicit ack after side effects. Set AckWait above processing p99 or use in-progress. Financial flows require idempotent handlers and consider double ack.

nats

SKILL.mdskills/nats/SKILL.md