Envoy troubleshooting

Find out why Envoy does not pass a request on, and propose a fix. Read first, ask before changing. Run the investigation commands freely. Run a command under an "Ask first" line only after the user approves that exact command; if your credentials do not allow it, give it to the user to run. Never run a "Proposed fix"; hand it to the user.

Envoy takes a request on a listener, matches a route by the Host header (domains) and the path, and sends it to a cluster, whose hosts come from DNS (STRICT_DNS, LOGICAL_DNS), a static list or a control plane (EDS). When it cannot, it answers itself and records why: in the access log as a response flag (UH, UF, NR ...) and %RESPONSE_CODE_DETAILS%, and in its stats. A standalone Envoy reads its bootstrap once at start, so a config error stops it and an edited ConfigMap needs a restart.

The commands use placeholders: my-namespace, my-envoy (the Envoy workload and its ConfigMap), my-envoy-admin (a Service in front of the admin port, if there is one), my-app (the backend) and my-web (a client). Replace them with the real names. Never read Secrets. The bootstrap can hold TLS keys inline (inline_string): never repeat them. Never fetch a full /config_dump, any *_secrets resource, or /certs; use config_dump with ?resource= for clusters, listeners and routes only.

When to use

• Clients get 503, 504 or 404 from Envoy, or connections reset.
• Envoy pods crash at start or are not Ready.
• One backend gets no traffic through Envoy, or all of it.

Hand-offs, by skill name. When the evidence points elsewhere, continue with that skill; if it is not installed, keep going with the steps here and tell the user which skill would help:

• Envoy runs as an Istio sidecar or gateway: k8s-istio-troubleshooting; as Envoy Gateway's proxy: k8s-envoy-gateway-troubleshooting. Their control planes write the config, so start there.
• The backend Service has no endpoints, or a NetworkPolicy blocks Envoy: k8s-network-troubleshooting.
• The backend pods themselves fail: k8s-workload-troubleshooting.
• Contour: kubectl get httpproxy -A shows each HTTPProxy's status (valid, or invalid with the reason); invalid ones are ignored. Emissary and Gloo report errors in their resources' status and their controller logs.

Investigate

Work through the steps in order. Stop when you have the cause and its evidence.

1. See Envoy's pods and the client's result.

kubectl get pods -n my-namespace -o wide
kubectl logs -n my-namespace deployment/my-web --tail=10

2. Read Envoy's log: the access log (if one is configured) with the response flag and details for each request, and Envoy's own messages. After a crash, read the previous container.

kubectl logs -n my-namespace deployment/my-envoy --tail=50
kubectl logs -n my-namespace deployment/my-envoy --previous --tail=20

3. Read the bootstrap: listeners, routes (domains, match, cluster) and clusters (type, addresses, ports, timeouts). Do not repeat keys you see.

kubectl get deployment my-envoy -n my-namespace -o jsonpath='{.spec.template.spec.containers[0].args}{"\n"}{range .spec.template.spec.volumes[*]}{.name}{" "}{.configMap.name}{.secret.secretName}{"\n"}{end}'
kubectl get configmap my-envoy -n my-namespace -o yaml
kubectl get services,endpointslices -n my-namespace

4. Ask Envoy's admin interface, if it listens on a pod address and your credentials allow get on services/proxy or pods/proxy (the built-in view role does not). These are read-only: Envoy answers 405 to a GET on any endpoint that changes state. If admin listens on 127.0.0.1 only (the docs' recommendation, Istio, Envoy Gateway, Contour), there is no kubectl path to it; say so and work from the log.

kubectl get --raw /api/v1/namespaces/my-namespace/services/my-envoy-admin:admin/proxy/clusters
kubectl get --raw "/api/v1/namespaces/my-namespace/services/my-envoy-admin:admin/proxy/stats?filter=cluster.my-app."
kubectl get --raw "/api/v1/namespaces/my-namespace/services/my-envoy-admin:admin/proxy/config_dump?resource=static_route_configs"
kubectl get --raw /api/v1/namespaces/my-namespace/services/my-envoy-admin:admin/proxy/server_info

5. Open the matching reference and follow its steps:

What you see	Reference
A response flag, an error body, /clusters and stats, DNS clusters, health checks, timeouts, circuit breakers	references/responses-and-upstreams.md
A crash at start, bootstrap errors, the admin interface and what it may show, images	references/bootstrap-and-admin.md

Read the results

What the common findings mean (Envoy 1.40):

• 503 no healthy upstream, flag UH, details no_healthy_upstream: the cluster has no healthy host. With STRICT_DNS, a name that does not exist counts as a successful lookup with no addresses: /clusters shows no host lines for the cluster, membership_total is 0, and update_success keeps rising, so the DNS name is the first thing to check against the Services.
• 503 upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, flag UF: Envoy reached a host but the connection was refused (a wrong port, nothing listening). The access log's %UPSTREAM_TRANSPORT_FAILURE_REASON% adds delayed connect error: Connection refused; since 1.39 it is no longer in the response body. reset reason: connection timeout means no answer within connect_timeout (a NetworkPolicy, a dropped port).
• 404 with an empty body, flag NR, details route_not_found: no virtual host's domains match the request's Host header, or no route matches the path.
• 504 upstream request timeout, flag UT: the backend did not answer within the route timeout (15s by default).
• Envoy exits at start with Protobuf message (type envoy.config.bootstrap.v3.Bootstrap reason INVALID_ARGUMENT: ... no such field: '...') has unknown fields: a misspelled or misplaced field; the @ <path> names where. route: unknown cluster '...' means a static route names a cluster that is not defined.

State the cause in one sentence, then the evidence you saw (the client's status and body, the access-log flag and details, the bootstrap field, /clusters), then the proposed fix.

Proposed fix

Show each fix to the user before running it: what it changes and how to undo it. Envoy's bootstrap serves every route and backend behind it: never change it yourself; hand it to the user as a proposed fix. Prefer fixing the source (the Helm values, the manifest in Git, or the control plane's resources). Never use the admin endpoints that change state (/quitquitquit, /healthcheck/fail, /drain_listeners, /logging, /reset_counters, /runtime_modify).

Fix the bootstrap in its source, apply it and restart Envoy so it reads it:

Proposed fix: review before running. The agent does not run this.

kubectl apply -f my-envoy-configmap.yaml
kubectl rollout restart deployment/my-envoy -n my-namespace
# Undo: kubectl apply -f my-envoy-configmap-before.yaml, then restart my-envoy again

If an edited ConfigMap is already in place but Envoy still runs the old bootstrap, restart it:

Ask first: the agent runs this only after the user approves it.

kubectl rollout restart deployment/my-envoy -n my-namespace
# Undo: none needed; the restart only recreates the pods with the same template

Sources

Checked against the Envoy 1.40 documentation and source (v1.40.1):

• Admin interface and command line options.
• Access logging and response flags.
• Service discovery, circuit breaking, outlier detection, cluster statistics and timeouts.
• Installing Envoy: image variants.
• Messages from the source: router.cc, admin.cc and the 1.39 changelog (transport failure reason out of the body). The docs say a GET to a mutating admin endpoint gets 400; the source returns 405.
• Contour fundamentals, Emissary debugging and Gloo debugging.
• The responses and log lines quoted here were seen on a Kubernetes 1.37 kind cluster with Envoy 1.40.1.