Vigil: the site answers is failing
fixedopened 26 days agodown 10m
what the check said
the site answers — expected 200, got 502
Warden’s diagnosis · 75% sure of the cause
502 because vigil is not running. `pm2 jlist` shows vigil with status "stopped" (not "errored"), lastStartedAt 2026-09-14T09:12:06.078Z, cwd /home/ubuntu/vigil. So the reverse proxy has nothing on port 3100. The output log shows Next.js reaching "Ready in ~1s" each time, and the error stream is empty — it isn't crashing on startup, it has been stopped. lastPassedAt 09:12:09 is ~3s after lastStartedAt, lining up with a brief window where the site answered 200 before the process exited, after which the proxy returned 502. "stopped" (not "errored") is the key clue: either pm2 was told not to auto-restart, or someone ran `pm2 stop vigil`. Standing rules show this happened twice before on this incident — the pattern is vigil being stopped and the owner approving a start. Restarting vigil (pm2_start, allowed without asking) should clear the 502.
suspect · vigil process (pm2 status: stopped)
This one was worked before you opened the page, so there is nothing to watch live. Everything it did is in Everything it ran below, with the exact command and the rule that permitted each one — that table is the record, and this timeline is only the same thing arriving in real time.
502 because vigil is not running. `pm2 jlist` shows vigil with status "stopped" (not "errored"), lastStartedAt 2026-09-14T09:12:06.078Z, cwd /home/ubuntu/vigil. So the reverse proxy has nothing on por — fixed and verified.
What you decided
Warden wants to start vigil on Vigil. Approve?
approverestoring the fleet after a shoot
Warden wants to run the test suite on Vigil. Approve?
novigil is already back up
Everything it ran13 calls · 1 changed something
Warden has no shell. Each row below is one named action from a fixed list, spawned without a shell, with the rule of yours that permitted it. Copy any command and run it yourself.
| what | how safe | your rule | command | ms |
|---|---|---|---|---|
list the processes pm2_list | looks only | did ityou allowed it | pm2 jlist | 185 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 100 (errors only) | 261 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 200 | 200 |
search the code grep_repo | looks only | did ityou allowed it | grep -rn "crash|exit|SIGTERM|fatal|killed|EADDR|throw|ENOENT|TypeError" /home/ubuntu/vigil | 24 |
read recent commits git_log | looks only | did ityou allowed it | git log -n 10 | 9 |
read recent commits git_log | looks only | did ityou allowed it | git log -n 25 | 11 |
read one commit git_show | looks only | did ityou allowed it | git show 6fad866 | 15 |
read one commit git_show | looks only | did ityou allowed it | git show 503accd | 37 |
read recent commits git_log | looks only | did ityou allowed it | git log -n 30 | 10 |
read a file read_file | looks only | did ityou allowed it | read /home/ubuntu/vigil/ecosystem.config.cjs | 23 |
start the process pm2_start | undoes itself | asked youstill cooling down from the last act | Warden acted on this service 8 minutes ago and your cooldown is 10. Restarting in a loop is not a fix. | — |
start the process pm2_start | undoes itself | did ityou allowed it | start vigil | 319 |
run the tests run_tests | undoes itself | asked youstill cooling down from the last act | Warden acted on this service 0 minutes ago and your cooldown is 10. Restarting in a loop is not a fix. | — |
This incident is marked resolved because the site answers — the same check that failed — was run again and passed. Nothing Warden believed about its own fix could have closed it.