AWS 502 Bad Gateway

The symptom

After every deploy to the evomedia.net production stack, the site would return a 502 Bad Gateway for anywhere from 30 seconds to several minutes — even after the deploy script confirmed the origin container was healthy. The only fix was logging into the AWS CloudFront console and creating a manual cache invalidation. This had been happening consistently enough that it became a routine step after every deployment.

The deploy warning that triggered the investigation:

WARNING: Evomedia version did not match v0.0.0.40 within 45s
(upload or Docker build may have failed).

The deploy itself was succeeding — the Docker build completed, the container started, the origin was responding correctly when hit directly. CloudFront was the one serving stale errors to real visitors.

What's actually happening — three stacked problems

The full picture turned out to be three independent issues compounding each other. Understanding all three is what makes the fix durable rather than just treating symptoms.

Problem #1 creates the 502. Problem #2 freezes it. Problem #3 means even after the outage clears, visitors still see old HTML. All three need to be fixed to stop the invalidation cycle.

Fix 1 — Lower the nginx DNS cache TTL

The edge proxy is an nginx container on the same Docker network as the app. nginx resolves container names (like evomedia) via Docker's internal DNS at 127.0.0.11. The valid= parameter controls how long nginx caches each resolved IP.

The default config had valid=10s. When the evomedia container was recreated during a deploy it got a new internal IP, but nginx kept using the old cached address for up to 10 seconds — returning 502s for every request in that window.

The fix: drop it to valid=2s in nginx.conf:

# Before
resolver 127.0.0.11 8.8.8.8 valid=10s ipv6=off;

# After — picks up new container IP within ~2s after a deploy
resolver 127.0.0.11 8.8.8.8 valid=2s ipv6=off;

This shrinks the outage window from ~10 seconds to ~2 seconds. The additional DNS lookups per second are negligible since Docker's internal resolver is in-process. After editing, hot-reload the running proxy with no downtime:

docker exec  nginx -s reload

This is a live hot-reload — it does not restart the proxy container or drop any in-flight connections. nginx finishes existing requests with the old config and worker processes gracefully hand off to the new ones.

Fix 2 — Stop CloudFront from caching error responses

Even with the nginx TTL fixed, the ~2 second blip can still get frozen by CloudFront. By default, CloudFront caches 5xx responses it receives from the origin. A single 502 during that 2-second window gets served to every visitor hitting that CloudFront edge node until the error TTL expires — which by default can be up to 5 minutes.

The fix is to set the error caching minimum TTL to 0 for all 5xx status codes. This tells CloudFront to never freeze an error response — the next request always goes back to the origin to check if it has recovered.

Steps in the CloudFront console

1. Go to CloudFront → Distributions → click your distribution.
2. Open the Error pages tab.
3. Click Create custom error response.
4. Repeat for each of 500, 502, 503, 504:
   • HTTP error code: 502 (etc.)
   • Customize error response: No
   • Error caching minimum TTL: 0
5. Save each one.

Setting the TTL to 0 does not disable CloudFront's error detection — it just means CloudFront will re-check the origin on the very next request instead of serving a cached error. The origin still only receives one request per user during the blip, not a flood.

Fix 3 — Tell CloudFront to respect your origin's Cache-Control headers

The third problem is why deploys don't propagate: the default CachingOptimized policy ignores the Cache-Control headers your origin sends and caches everything on its own schedule. So even if the origin serves fresh HTML immediately after a deploy, CloudFront keeps serving the cached old version until it decides to revalidate — or until you manually invalidate.

The evomedia.net nginx already sends the right headers:

# HTML files — revalidate on every request
Cache-Control: no-cache, must-revalidate

# Hashed JS/CSS assets — cache forever (new filename per build)
Cache-Control: max-age=31536000, immutable

The fix is to switch the CloudFront behavior's cache policy from CachingOptimized to UseOriginCacheControlHeaders — a managed AWS policy that passes your origin's Cache-Control through instead of overriding it.

Steps in the CloudFront console

1. Go to CloudFront → Distributions → click your distribution.
2. Open the Behaviors tab → select the Default (*) behavior → Edit.
3. Scroll to Cache key and origin requests.
4. Set Cache policy to UseOriginCacheControlHeaders.
   Listed as "Recommended for custom origins" in the dropdown. If your plan doesn't allow custom policies, use this managed one — it does the same job.
5. Save changes. Propagation takes 3–5 minutes.

With this policy in place, HTML is revalidated by CloudFront on every request (because the origin says no-cache), and hashed JS/CSS assets are cached indefinitely (because they have new filenames after every Vite build anyway). No invalidation is ever needed — new content appears as soon as the origin serves it.

End result

The three fixes are independent and each one improves things on its own — but all three together is what makes deploys truly silent. Fix 1 shortens the origin outage window. Fix 2 stops CloudFront from amplifying it. Fix 3 ensures the rest of your deploy (new HTML) reaches visitors without any manual steps.

← Back to Blog · Contact