Triaging 5xx Errors in Kubernetes: A Practical Guide

Varshith Peddi
April 17, 2025
0 comments

Introduction

In a Kubernetes environment, 5xx server errors (e.g., 500, 502, 503, 504) indicate issues with the server-side application or infrastructure. These errors can disrupt user experience and signal underlying problems in your cluster, such as misconfigurations, resource constraints, or application bugs. Triaging 5xx errors effectively requires a structured approach to identify the root cause and implement fixes. This guide outlines the steps to diagnose and resolve 5xx errors in Kubernetes, along with common reasons for these issues.

Common Reasons for 5xx Errors in Kubernetes

500 Internal Server Error: Application-level issues, such as unhandled exceptions, database failures, or logic errors in the code.
502 Bad Gateway: Issues with the ingress controller, proxy, or upstream services failing to respond correctly.
503 Service Unavailable: Overloaded services, pod crashes, or scaling issues, often due to resource limits or liveness probe failures.
504 Gateway Timeout: Slow or unresponsive upstream services, often caused by network latency, timeouts, or overloaded backend pods.

Steps to Triage 5xx Errors

1. Identify the Scope and Pattern

Check Logs: Use monitoring tools to identify the frequency, endpoints, and services affected by 5xx errors.
Inspect Ingress Logs: If using an ingress controller (e.g., NGINX, HA-Proxy), check its logs for clues about failed requests.
Reproduce the Issue: If possible, replicate the error in a non-production environment to understand the conditions triggering it.

2. Examine Kubernetes Resources

Pod Status: Run kubectl get pods -n <namespace> to check for crashing or unhealthy pods. Use kubectl describe pod <pod-name> to inspect events and conditions.
Resource Limits: Verify CPU and memory limits (spec.containers[].resources) in the pod spec. Overloaded pods may return 503 errors.
Liveness/Readiness Probes: Ensure probes are correctly configured. Misconfigured probes can cause pods to be marked as unhealthy, leading to 503 errors.

3. Investigate Application Logs

Access Logs: Use kubectl logs <pod-name> -n <namespace> to review application logs for stack traces or error messages.
Correlate Timestamps: Match the timing of 5xx errors with log entries to pinpoint the issue.
Check Dependencies: Look for errors related to external services (e.g., databases, APIs) that might cause 500 or 504 errors.

4. Use `curl` to Diagnose Timeouts and Dependency Failures

Test Endpoints: Use curl -v <endpoint-url> to send requests to the affected endpoint and inspect the HTTP response headers, status codes, and timing.
- Look for timeouts 504 Gateway Timeout indicating slow upstream or downstream dependencies.
- Check for 502 Bad Gateway errors that may point to issues with external load balancers or misconfigured proxies.
Verify Authentication: Include authentication headers e.g., curl -H "Authorization: Bearer <token>" <endpoint-url> to rule out 5xx errors caused by auth failures.
Check Dependency Connectivity: Test connectivity to external dependencies (e.g., databases, APIs) using curl <dependency-url> from within a pod kubectl exec -it <pod-name> -- curl <dependency-url>. Look for connection refusals or timeouts.
Validate Request Routing: Use curl -H "Host: <domain>" <load-balancer-ip> to confirm that requests are being forwarded to the correct cluster or load balancer hosting the application. Check response headers or errors to detect misrouting.
Enable Debugging: Add --trace-ascii - to curl commands to log detailed request/response data for deeper analysis of failures.

5. Analyze Networking

Ingress Configuration: Verify ingress rules and annotations for misconfigurations causing 502 or 504 errors.
Service Connectivity: Run kubectl describe service <service-name> to ensure the service is routing traffic to healthy pods.
Network Policies: Check if network policies are blocking traffic to or from pods.
DNS Issues: Confirm that DNS resolution is working correctly within the cluster.

6. Monitor Cluster Health

Node Resources: Use kubectl top nodes to check for node-level resource exhaustion, which can cause pod evictions and 503 errors.
Cluster Autoscaler: Ensure the autoscaler is functioning if 5xx errors correlate with traffic spikes.
Control Plane: Verify that the Kubernetes control plane components (API server, scheduler) are operational.

7. Apply Fixes

Scale Resources: Increase replicas or resource limits for overloaded services kubectl scale deployment <name> --replicas=<n>
Fix Application Bugs: Address code-level issues identified in logs.
Tune Timeouts: Adjust ingress or service timeouts to prevent 504 errors.
Update Configurations: Correct misconfigured probes, ingress rules, or network policies.
Resolve Dependency Issues: Fix authentication errors or adjust external service configurations based on curlfindings.

8. Validate and Monitor

Test Fixes: Verify that 5xx errors are resolved after applying changes.
Set Up Alerts: Configure alerts in your monitoring system to catch future 5xx errors early.
Document Findings: Record the root cause and resolution steps for future reference.

Best Practices to Prevent 5xx Errors

Implement robust liveness and readiness probes to ensure only healthy pods receive traffic.
Use horizontal pod autoscaling (HPA) to handle traffic spikes.
Monitor resource usage and set appropriate CPU/memory limits.
Regularly audit ingress and service configurations.
Maintain detailed logging and monitoring for quick diagnostics.
Test external dependencies and load balancer routing periodically using curl or similar tools.

Conclusion

Triaging 5xx errors in Kubernetes involves a systematic approach to identify whether the issue lies in the application, networking, or cluster infrastructure. By incorporating tools like curl to diagnose timeouts, dependency failures, and routing issues, you can pinpoint problems more effectively. Proactive monitoring, proper configuration, and regular testing are key to minimizing 5xx errors in production environments.