NGINX reverse proxy upstream investigation cheat sheet
Connect configuration tests, service logs, listener state and direct upstream requests during a reverse-proxy outage.
NGINX on a systemd-based Linux host; log paths, executable paths and service names vary by installation.
Updated · NGINX · 6 min read
| Command | What it does |
|---|---|
nginx -t | Test the selected configuration files |
systemctl status nginx.service --no-pager --full | Check the proxy service state |
journalctl -u nginx.service --since "30 minutes ago" -n 100 --no-pager | Inspect recent service-level events |
tail -n 100 /var/log/nginx/error.log | Read a bounded proxy error sample |
ss -ltnp | Compare proxy and upstream TCP listeners |
curl --silent --show-error --output /dev/null --write-out "%{http_code}\n" --connect-timeout 5 --max-time 15 -H "Host: app.example.com" http://127.0.0.1:8080/health | Test the configured local upstream directly |
Walkthrough
Use authorized read access and replace the log paths, app.example.com and 127.0.0.1:8080 with your configured values. nginx -t validates and opens referenced files; it does not reload running workers.
Step 1: Test the selected configuration files
nginx -t
Read the tested configuration path and any filename/line error. File permissions can make the test fail under a different account from the service. A successful test does not show that the running workers have loaded the same on-disk configuration.
Step 2: Check the proxy service state
systemctl status nginx.service --no-pager --full
Record active state, PID and recent events. A running service may still return upstream errors or serve an older configuration. Compare the incident time with restarts and deployment history rather than restarting solely to change the status.
Step 3: Inspect recent service-level events
journalctl -u nginx.service --since "30 minutes ago" -n 100 --no-pager
Look for startup, reload and service errors around the incident. Application request details may be written to separate files or another logging destination, so an empty journal excerpt does not rule out proxy errors.
Step 4: Read a bounded proxy error sample
tail -n 100 /var/log/nginx/error.log
Correlate the request time and upstream address with the incident. Connection refused, timeout and certificate verification errors describe different failure paths. Use the configured destination if errors go to stderr, syslog or another file.
Step 5: Compare proxy and upstream TCP listeners
ss -ltnp
Check the expected proxy port and local upstream port, if the upstream is on this host. A missing local listener supports a service or binding issue. For a remote upstream, a local socket listing cannot establish whether it is listening remotely.
Step 6: Test the configured local upstream directly
curl --silent --show-error --output /dev/null --write-out "%{http_code}\n" --connect-timeout 5 --max-time 15 -H "Host: app.example.com" http://127.0.0.1:8080/health
Use the intended protocol, path and host header; do not substitute HTTP for an HTTPS upstream. This bypasses the proxy and therefore cannot validate the external TLS listener, proxy routing or authentication behavior.
WORKED INCIDENT
Practical case: The public site returns 502 after an application restart
Fictional teaching scenario; observations are examples, not captured production output.
Situation
NGINX remains active but requests return Bad Gateway after the backend deployment.
Investigation sequence
1. Test the selected configuration files
nginx -tCheck that the on-disk configuration can be parsed; distinguish this from whether it is loaded.
2. Read a bounded proxy error sample
tail -n 100 /var/log/nginx/error.logRead the exact upstream error. The example reports connection refused at 127.0.0.1:8080.
3. Compare proxy and upstream TCP listeners
ss -ltnpCompare local listeners. No process is listening on the configured upstream port in this scenario.
4. Test the configured local upstream directly
curl --silent --show-error --output /dev/null --write-out "%{http_code}\n" --connect-timeout 5 --max-time 15 -H "Host: app.example.com" http://127.0.0.1:8080/healthTest the same upstream endpoint directly. It also refuses the connection, independently of public DNS and client TLS.
Example findings
Illustrative findings: nginx configuration test succeeds; workers remain active; error log reports upstream connection refused; TCP 8080 has no listener.
Interpret the findings
The evidence points to the backend listener or its configured address. Changing proxy timeouts will not create the missing listener. Confirm whether deployment changed the backend port or stopped the service.
Verify the outcome
After the application owner restores the intended listener or the proxy owner approves the corrected upstream, verify direct health response and a normal request through the public hostname. Compare fresh error-log entries.
Escalation handoff
Provide the upstream address, request time, log error and listener result to the backend/proxy owners. Include only the relevant configuration excerpt, with credentials removed.