Linux systemd service failure triage cheat sheet
Connect service state, exit codes, unit configuration and journal events without hiding evidence behind a restart.
Linux distributions running systemd; examples use nginx.service, whose unit name varies by distribution.
Updated · Linux · 6 min read
| Command | What it does |
|---|---|
systemctl status nginx.service --no-pager --full | Capture service state and recent messages |
systemctl show nginx.service -p ActiveState -p SubState -p Result -p ExecMainStatus -p NRestarts | Read machine-friendly failure fields |
journalctl -u nginx.service --since "30 minutes ago" -n 100 --no-pager -o short-iso | Read a bounded incident window |
systemctl cat nginx.service | Inspect the unit and drop-in files |
systemctl list-dependencies nginx.service --no-pager | Inspect declared dependencies |
ss -ltnp | Compare expected TCP listeners with processes |
Walkthrough
Replace nginx.service with the affected unit. Use an account permitted to read its journal and configuration. status exits nonzero for a failed unit; that is useful evidence, not a reason to restart it.
Step 1: Capture service state and recent messages
systemctl status nginx.service --no-pager --full
Record Loaded, Active, Main PID and the failure timestamp. The short log excerpt can omit the original cause, so use it as a starting point. A failed unit can still have unrelated processes listening on the same application port.
Step 2: Read machine-friendly failure fields
systemctl show nginx.service -p ActiveState -p SubState -p Result -p ExecMainStatus -p NRestarts
Compare Result and ExecMainStatus with the service documentation. A nonzero application exit code is different from a timeout or signal. NRestarts records automatic restarts tracked by the manager, not every historical manual start.
Step 3: Read a bounded incident window
journalctl -u nginx.service --since "30 minutes ago" -n 100 --no-pager -o short-iso
Look for the first error before repeated restart messages. Correlate ISO timestamps with the reported interruption. Empty results can reflect permissions, retention or a wrong unit name; they do not prove that the service produced no errors.
Step 4: Inspect the unit and drop-in files
systemctl cat nginx.service
Read ExecStart, User, dependencies and any override files. This shows files on disk; if they changed without a daemon reload, the manager may still use older settings. Compare the configured executable and account with the journal error.
Step 5: Inspect declared dependencies
systemctl list-dependencies nginx.service --no-pager
Check required units and their state, then investigate any dependency named in the failure log. This is the unit dependency graph, not a full map of DNS, databases or other external services the application needs.
Step 6: Compare expected TCP listeners with processes
ss -ltnp
Find the expected port and local binding. Loopback-only and all-interface listeners serve different paths. Process information can require elevated privileges; an occupied port may explain a bind failure but does not identify the correct owner by itself.
WORKED INCIDENT
Practical case: A web service fails immediately after a deployment
Fictional teaching scenario; observations are examples, not captured production output.
Situation
The nginx unit fails at startup after an application deployment. Capture evidence before requesting a restart.
Investigation sequence
1. Capture service state and recent messages
systemctl status nginx.service --no-pager --fullRecord the failed state and the first reported failure time.
2. Read a bounded incident window
journalctl -u nginx.service --since "30 minutes ago" -n 100 --no-pager -o short-isoRead the incident window. In this fictional example the journal reports that binding to port 80 failed because the address is already in use.
3. Inspect the unit and drop-in files
systemctl cat nginx.serviceCheck the intended executable and drop-ins so that the unit being examined is actually the expected web server.
4. Compare expected TCP listeners with processes
ss -ltnpIdentify the listener on port 80. The example has a separate process holding the port; record its PID for the service owner.
Example findings
Illustrative findings: Active = failed; journal contains an address-in-use bind error; ss shows another process listening on TCP 80.
Interpret the findings
A port conflict is supported. Do not kill the listener until its owner and purpose are established. The deployment may have introduced a second service or changed a binding.
Verify the outcome
After the owner resolves the conflicting configuration and performs an approved start, repeat status, journal and listener checks, then test the application through its normal URL.
Escalation handoff
Provide unit/drop-in paths, the first failure message, timestamps and listener PID to the application owner. Remove credentials from configuration excerpts.