Vigil: the process is up is failing
fixedopened 26 days agodown 26m
what the check said
the process is up — pm2 says "stopped"
Warden’s diagnosis · 45% sure of the cause
pm2 says vigil `status: "stopped", restarts: 4`. Error stream is EMPTY (120-line tail returned no content). Out log shows four clean Next.js 15.5.19 boots, each reaching "Ready in ~1s" on :3100, then stopping with no stack trace and no Next.js error. A Next.js process that boots fine and exits with empty stderr is being stopped by an EXTERNAL signal (pm2 stop, kernel OOM, or a deploy script that stopped the old process and didn't start a new one) — not an app crash. Sibling `vigil-sweep` is also `stopped`; its last line is a clean "sweep done" at 06:00:01 UTC — also a clean shutdown, pointing to external stop. Latest commit (36a05c0, docs) landed 05:35:51 UTC, incident opened 07:20:01 UTC — ~95 min gap, does NOT line up cleanly with a deploy regression. Disk fine (80%). Could not get the very tail of the out log (the "Killed"/"stopped" reason line) — the log tool capped at 120 lines and refused more. Recommend: before restarting, owner should check who/what stopped vigil and vigil-sweep around 07:00–07:20 UTC, then pm2 start both.
suspect · vigil + vigil-sweep stopped externally (pm2 stop / OOM / deploy script) — empty stderr, four clean Ready→stop cycles, no app error
This one was worked before you opened the page, so there is nothing to watch live. Everything it did is in Everything it ran below, with the exact command and the rule that permitted each one — that table is the record, and this timeline is only the same thing arriving in real time.
The probe that failed came back clean.
Everything it ran10 calls · 0 changed something
Warden has no shell. Each row below is one named action from a fixed list, spawned without a shell, with the rule of yours that permitted it. Copy any command and run it yourself.
| what | how safe | your rule | command | ms |
|---|---|---|---|---|
list the processes pm2_list | looks only | did ityou allowed it | pm2 jlist | 175 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 100 (errors only) | 183 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 200 | 187 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil-sweep --lines 50 | 185 |
read a file read_file | looks only | did ityou allowed it | read_file | 0 |
check disk space disk_free | looks only | did ityou allowed it | df -h / | 9 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 120 | 178 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 120 (errors only) | 181 |
read the log pm2_logs | looks only | did ityou allowed it | pm2 logs vigil --lines 200 (errors only) | 185 |
read recent commits git_log | looks only | did ityou allowed it | git log -n 10 | 19 |
This incident is marked resolved because the process is up — the same check that failed — was run again and passed. Nothing Warden believed about its own fix could have closed it.