Vigil: the site answers is failing
fixedopened 26 days agodown 5m
what the check said
the site answers — expected 200, got 502
Warden’s diagnosis · 55% sure of the cause
The vigil process is stopped (pm2 status: "stopped", restartsSinceAdded: 8, lastStartedAt: 2026-09-14T07:48:57Z — i.e. it exited about 4 minutes ago, which matches minutesDown: 4). The upstream 502 is the consequence, not the cause: the gateway is proxying to a dead listener. The stdout shows the app reaches "✓ Ready in ~1000ms" on every boot, then dies with no error logged and the error stream is empty. That pattern — clean startup, no exception, sudden death — is what an OOM kill (pm2 max_memory_restart, or the host OOM killer) leaves behind, because SIGKILL produces no log line. Repository is on the correct disk (/home/ubuntu/vigil), 20G free, so disk is not the cause. The check started failing at 07:20:01Z; pm2's restart count climbing in that window is consistent with the process being repeatedly killed shortly after start. I did not find a stack trace pointing at a specific code path, and I deliberately did not read the application source — the process is not running, so no code path is the answer right now.
suspect · pm2 process vigil (post-startup kill, most likely OOM)
This one was worked before you opened the page, so there is nothing to watch live. Everything it did is in Everything it ran below, with the exact command and the rule that permitted each one — that table is the record, and this timeline is only the same thing arriving in real time.
The vigil process is stopped (pm2 status: "stopped", restartsSinceAdded: 8, lastStartedAt: 2026-09-14T07:48:57Z — i.e. it exited about 4 minutes ago, which matches minutesDown: 4). The upstream 502 is — fixed and verified.
Everything it ran10 calls · 1 changed something
Warden has no shell. Each row below is one named action from a fixed list, spawned without a shell, with the rule of yours that permitted it. Copy any command and run it yourself.
| what | how safe | your rule | command | ms |
|---|---|---|---|---|
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 80 | 189 |
load the page http_probe | looks only | did ityou allowed it | GET https://vigil.80.225.209.190.sslip.io/ | 561 |
list the processes pm2_list | looks only | did ityou allowed it | pm2 jlist | 270 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 80 (errors only) | 193 |
read recent commits git_log | looks only | did ityou allowed it | git log -n 10 | 9 |
read one commit git_show | looks only | did ityou allowed it | git show 6fad866 | 15 |
read a file read_file | looks only | did ityou allowed it | read /home/ubuntu/vigil/package.json | 9 |
check disk space disk_free | looks only | did ityou allowed it | df -h / | 9 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 200 | 192 |
start the process pm2_start | undoes itself | did ityou allowed it | start vigil | 353 |
This incident is marked resolved because the site answers — the same check that failed — was run again and passed. Nothing Warden believed about its own fix could have closed it.