Service Unreachable

KUBERNETES RUNBOOK

Service Unreachable

Trace the request hop by hop so you know whether DNS, the Service, endpoints, ports, policy or the application is failing.

What does “Service unreachable” mean?

It can mean several different things: the name does not resolve, the Service IP cannot be reached, the Service has no ready backends, traffic is blocked, the target port is wrong, or the application responds incorrectly. First locate the broken hop.

The traffic path

Client↓DNS name (if used)↓Service + port↓EndpointSlice / ready backend↓targetPort↓Application

Step 1 — Inspect the Service

kubectl get svc <service> -n <namespace>
kubectl describe svc <service> -n <namespace>

Check the Service type, ClusterIP, selector, client-facing port and backend targetPort.

port vs targetPort: clients connect to the Service port. Kubernetes forwards that traffic to the backend targetPort. If the application listens on 8080 but targetPort points to 8000, the Service can exist normally while requests fail.

Step 2 — Does the Service have endpoints?

kubectl get endpointslice -n <namespace> -l kubernetes.io/service-name=<service>
NAME              ADDRESSTYPE   PORTS   ENDPOINTS        AGE
payments-7fd2     IPv4          8080    10.244.2.17      4h

What to check: an endpoint address and expected backend port should be present. No suitable endpoint is a strong reason to compare Service selectors, pod labels and readiness before debugging DNS or HTTP. If no suitable backend endpoints appear, compare Service selectors with pod labels:

kubectl get svc <service> -n <namespace> -o yaml
kubectl get pods -n <namespace> --show-labels

Also confirm backend pods are Ready. Endpoint readiness matters because an unhealthy backend should not receive normal Service traffic.

Step 3 — Test from the client side

Use an existing authorized diagnostic/client pod where possible and test the same destination the failing workload uses. A typical HTTP test might be:

curl -v http://<service>:<port>/health

Interpret the failure: “could not resolve host” points toward DNS; “connection refused” suggests something reachable is not accepting the connection; timeout suggests routing/policy/network path; an HTTP 4xx/5xx means connectivity exists and the application/proxy layer is responding.

Step 4 — If DNS fails

Try the Kubernetes service FQDN:

<service>.<namespace>.svc.cluster.local

Then inspect cluster DNS components and configuration according to your distribution. If direct Service IP access works while name resolution fails, DNS is a much stronger lead than Service selectors.

Step 5 — If endpoints exist but traffic fails

Verify the application actually listens on the expected port and an interface reachable through the pod network—not only 127.0.0.1. Check application logs and readiness behavior.

Common HTTP status codes

2xx — tested endpoint returned success; compare the original client path, headers and auth.

301/302/307/308 — redirect; check scheme, hostname, path and proxy/Ingress behavior.

400 — invalid request; inspect body, headers and rewriting.

401 / 403 — authentication or authorization path.

404 — route/resource not found; check URL, app routing and proxy/Ingress rules.

405 — HTTP method not allowed.

429 — rate limiting/throttling.

500 — responder hit an internal error.

502 — proxy/gateway could not get a valid upstream response.

503 — unavailable/no healthy upstream depending on the responder; check readiness, endpoints and backend availability.

504 — gateway timed out waiting for upstream.

Important: Do not diagnose from the status code alone. Identify whether the application, Ingress proxy or gateway generated it.

Compare Service and backend

curl -v http://<service>:<port>/health
curl -v http://<pod-ip>:<target-port>/health
Pod IP works; Service fails — Service selector/ports, policy or Service path.
Both fail — application listener, readiness, policy or pod network.
Service works; original client fails — client DNS/config/auth/network path.

Direct Pod-IP access is diagnostic only; pod addresses are ephemeral.

Cross-namespace DNS

payments
payments.backend
payments.backend.svc.cluster.local

Step 6 — Check NetworkPolicies

kubectl get networkpolicy -A
kubectl describe networkpolicy <policy> -n <namespace>

Check both ingress to the backend and egress from the client where policies apply. Verify pod/namespace selectors and allowed ports.

Do not delete production NetworkPolicies broadly just to “see if it works.” Prefer a controlled policy change consistent with your source of truth and security model.

Know when to leave the Service layer

If the Service works internally but an external hostname fails, continue with Ingress/Gateway routes, load balancer, TLS, external DNS and firewall/security controls.

Verify

Retest from the original client path. Confirm DNS resolution if used, Service connection, backend response and application-level success.

Common mistakes

  • Debugging DNS when the Service has zero endpoints.
  • Confusing Service port with targetPort.
  • Testing only from a laptop instead of the failing workload path.
  • Removing all NetworkPolicies as the first test.
  • Assuming a TCP connection proves the application is healthy.