When the dashboard didn’t come back: making a service always-on without admin

A restart, and the control panel was simply gone. The server on
http://127.0.0.1:8765/ — our Mission Control — refused every connection. It had
worked the night before. Nothing in the app had changed. So why did a reboot kill
it, and why didn’t it heal itself?

This is the postmortem of that failure and the fix, because the root cause is one
almost every self-hosted setup hits eventually: the service was fine; the thing
that was supposed to keep it alive was not.

The service was never the problem

The first instinct is to blame the server. It was innocent. Launched by hand, the
Python server bound the port and answered HTTP 200 immediately. The code was
healthy. What was missing was everything around it: the process that boots the
whole stack at login and keeps it up.

That process is a small resident supervisor. It’s meant to run from two places:

1. A Scheduled Task — fire at logon, then repeat on a short interval. The repeat
   is the important part: if the supervisor ever dies, the next tick brings it back.
   This is the self-healing layer.
2. A one-shot Startup shortcut as a fallback — launch the supervisor once when
   you sign in, and that’s it.

On this machine, layer 1 did not exist. And that changes everything.

The real root cause: a wall you can’t see until you hit it

Creating that Scheduled Task failed. Not intermittently — deterministically, with
Access is denied. Both the classic schtasks /Create and the PowerShell
Register-ScheduledTask cmdlet returned the same thing, even for a least-privilege
task in the user’s own space. Group Policy on the box blocks non-admin task
creation outright.

So the “robust” layer had silently never installed. All that remained was the
one-shot shortcut: it starts the supervisor once at sign-in and never again. If the
supervisor stops for any reason — a crash, a stray kill, a resume from sleep that
the logon trigger never sees — nothing restarts it. The stack stays dark until a
human notices. That is exactly what “it didn’t work after restart” looks like.

The lesson that generalizes: a fallback that removes self-healing isn’t a
fallback, it’s a single point of failure wearing a fallback’s clothes.

Designing the fix — and asking two reviewers first

The constraint was hard: no admin, so no scheduled tasks, ever. The fix had to
be a pure user-space, self-healing mechanism. Before writing it, the design went to
two independent reviewers for a critique. Their guidance shaped the final shape more
than the first draft did, and three points stuck:

• Don’t build mutual guardians. The tempting design is two processes that each
  restart the other. Both reviewers flagged it: mutual guarding invites split-brain,
  ambiguous ownership, and restart storms. Use a one-way hierarchy instead — a
  keeper that owns the supervisor, and ownership that flows in a single direction.
• A TCP port is the wrong lock. The original single-instance guard was “bind a
  loopback port.” That conflates three different questions — is the lock held, is
  the holder really the right process, is it making progress — and it leaves stale
  state after a hard crash. The right primitive on Windows is a kernel named
  mutex: the moment the owning process dies, the kernel releases it. No stale
  state, no time-of-check/time-of-use race.
• Ship the launcher as a shortcut, not a script. A box locked down enough to
  block non-admin scheduled tasks often disables Windows Script Host too. A native
  .lnk pointing straight at the interpreter removes the dependency on WSH that a
  .vbs carries. Worth being precise, though — and a reviewer pushed back on the
  first draft here: a shortcut does not inherently bypass application-control like
  AppLocker or WDAC, which still evaluate the target executable and its path. The
  .lnk is more reliable than a script, not universally permitted.

What shipped

The supervisor grew a small companion — a keeper. The keeper’s only job is to
own a supervisor child, watch its process handle, and relaunch it with bounded
backoff if it ever exits. Single direction, one owner. Both processes take a kernel
named mutex on startup, so no matter how many times the launcher fires — shortcut,
fallback, a manual run — exactly one of each survives; the rest exit cleanly.

Two more details earned their place the hard way:

• A heartbeat that proves progress. The supervisor rewrites a tiny record every
  loop with a monotonically increasing sequence number, written atomically. Now
  “is it alive?” isn’t “does it hold a socket?” but “is the sequence advancing?” —
  the difference between a running loop and a hung one that still holds its port.
• A boot that never blocks the loop. The first version re-launched the stack
  synchronously and captured its output. During testing the heartbeat froze — the
  loop had blocked inside that boot. The reviewers had predicted exactly this: a
  live supervisor can still be stuck. The fix was to fire the (idempotent) launcher
  and return immediately, with no inherited pipes to deadlock on, and let the next
  probe observe recovery.

Proving it, not claiming it

A fix to a reliability mechanism is worthless unless you break it on purpose and
watch it recover. So we did:

• Kill the supervisor. A new one was resident within a probe interval, heartbeat
  fresh.
• Kill the dashboard itself. :8765 dropped to a refused connection, then came
  back to HTTP 200 in about ten seconds — and the heartbeat sequence kept
  advancing the whole time, proving the loop stayed live through the recovery.
• Launch a second keeper. It printed “already running” and exited. One owner.

Honest boundaries remain, and they’re worth stating plainly rather than discovering
later:

• Nothing in user space runs before you sign in. This design guarantees fast
  recovery after logon, resume, or a crash — not before a human has logged on.
  That’s the real ceiling without admin.
• The single-instance lock is per-session. Two simultaneous interactive sessions
  (fast user switching, remote desktop) could each start a keeper; a machine-wide
  lock is the stronger answer, but it can itself require elevation, so the honest
  non-admin choice is a per-session lock plus this caveat.
• A healthy-looking port isn’t proof of identity. “The port answers 200” is a
  weaker claim than “the right service answers with its own version marker.” The
  latter is the correct bar for declaring a foreign squatter.

Pretending any of these away would just be the next silent failure waiting to happen.

The takeaway

The bug wasn’t in the server. It was in an assumption — that the self-healing layer
was there — that had quietly been false since its very first install attempt. The
fixes that mattered were boring on purpose: one owner, a kernel lock, a launcher
chosen for the fewest policy dependencies, a heartbeat that can tell alive from
stuck, and a boot that can’t wedge the thing meant to keep you alive. Reliability is
rarely one clever trick. It’s removing the places where “should be running” and “is
running” are allowed to disagree.

— Skynet