Vigil: the site answers is failing
fixedopened 26 days agodown 17m
what the check said
the site answers — expected 200, got 502
Warden’s diagnosis · 75% sure of the cause
Vigil is down because the process is not running. pm2_list shows vigil with status="stopped" (not "errored", not "online"), lastStartedAt=2026-09-14T07:55:45.688Z, with 10 cumulative restarts since it was added. With nothing listening on its port (3100), the public probe gets 502. The error log is empty and the out log only shows Next.js starting and reporting "Ready in ~XXXms" repeatedly — there is no stack trace, no crash, no unhandled rejection. The process started cleanly at 07:55:45 and the site check passed at 07:55:48 (3s after start). Between 07:55:48 and 07:56:15 the process exited cleanly or was killed externally, and PM2 did not restart it, leaving it in "stopped" rather than "errored". I don't yet know why PM2 is holding vigil in the "stopped" state instead of restarting it — empty error stream plus "stopped" (not "errored") is unusual. Possible: max_memory_restart=700M OOM-kill the process fast enough that PM2 logged no error, an external kill, or an earlier Warden run issuing `pm2 stop`. The fix path is `pm2_start vigil` (or a restart), but I am not acting in this step.
suspect · vigil (pm2 process, lastStartedAt 2026-09-14T07:55:45.688Z, status stopped)
This one was worked before you opened the page, so there is nothing to watch live. Everything it did is in Everything it ran below, with the exact command and the rule that permitted each one — that table is the record, and this timeline is only the same thing arriving in real time.
Vigil is down because the process is not running. pm2_list shows vigil with status="stopped" (not "errored", not "online"), lastStartedAt=2026-09-14T07:55:45.688Z, with 10 cumulative restarts since it — fixed and verified.
What you decided
Warden wants to start on Vigil. Approve?
approvevigil is back up; close this out
Everything it ran12 calls · 1 changed something
Warden has no shell. Each row below is one named action from a fixed list, spawned without a shell, with the rule of yours that permitted it. Copy any command and run it yourself.
| what | how safe | your rule | command | ms |
|---|---|---|---|---|
list the processes pm2_list | looks only | did ityou allowed it | pm2 jlist | 184 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 80 (errors only) | 180 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 100 | 313 |
read a file read_file | looks only | did ityou allowed it | read /home/ubuntu/vigil/server.json | 9 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 200 | 206 |
check disk space disk_free | looks only | did ityou allowed it | df -h / | 8 |
read recent commits git_log | looks only | did ityou allowed it | git log -n 10 | 25 |
read a file read_file | looks only | did ityou allowed it | read_file | 0 |
read a file read_file | looks only | did ityou allowed it | read /home/ubuntu/vigil/package.json | 17 |
search the code grep_repo | looks only | did ityou allowed it | grep -rn "vigil" /home/ubuntu/vigil/ecosystem.config.cjs | 16 |
start the process pm2_start | undoes itself | asked youstill cooling down from the last act | Warden acted on this service 3 minutes ago and your cooldown is 10. Restarting in a loop is not a fix. | — |
start the process pm2_start | undoes itself | did ityou allowed it | start vigil | 318 |
This incident is marked resolved because the site answers — the same check that failed — was run again and passed. Nothing Warden believed about its own fix could have closed it.