health

---
name: health
description: Read-only health sweep of the whole Coolify estate — infrastructure overview, unhealthy resources, tailnet spot-checks, and drift detection between live compose and the stacks/ reference copies. Use when the user asks "how is everything", "health check", "status", "anything broken?", or at the start of an ops session before touching anything. Never mutates anything.
license: MIT
compatibility: Run from a deployment repo created by setup-coolify-devops with the Coolify MCP configured. Read-only; needs network access to the Coolify API and, optionally, the Tailscale and Hetzner APIs.
metadata:
  author: KasperHonore
  version: "0.8.2"
---

# Estate health sweep

**Read-only by contract.** No deploys, restarts, env writes, or storage changes — if the
sweep finds something that needs fixing, report it and stop; fixing is `/change-service`
territory and a separate decision.

## 1. Coolify layer

- `get_infrastructure_overview` and `find_issues`.
- `find_issues` flags every healthcheck-less service as a `running:unknown` warning on
  every run — cross-check its list against the expected set before reporting any of
  them as a finding. A warning about an **available Traefik update** on the server is
  real signal, not noise — report it (fixing is a `/change-service` decision).
- `get_version` and `get_mcp_version`: if either differs from what
  `docs/infrastructure.md`'s platform table records, report the drift (the MCP's tool
  names have changed between majors before — see "Tool surface" there).
- **Which `running:unknown` statuses are expected (images without healthchecks) and
  which are real signal is documented in the inventory section of
  `docs/infrastructure.md`** — read it and do not re-litigate the expected ones.

## 1b. Disk space

A full disk is the failure that takes everything down at once (builds, databases,
Docker itself), and it is cheap to see coming.

- **On the host** (`operator.on_host` true in `instance.yaml`): `df -P / /var/lib/docker`
  in the shell. Read-only, and the one shell command this sweep runs.
- **Elsewhere:** `scheduled_tasks` `run_once` with `df -P /` in the canary's container
  (`canary` in `instance.yaml`). A container's `/` is Docker's overlay, so it reports
  the host filesystem Docker writes to, which is the one that fills. `run_once` deletes its
  task when it finishes, so the sweep stays read-only in effect.

Report use as a percentage. Under 80% is one line in the table; 80–90% is a finding
with the cleanup to consider (Coolify's *Server → Settings* cleanup threshold, old
images); over 90% leads the report, ahead of anything but an open port. Compare with
what "Housekeeping in place" in `docs/infrastructure.md` records, and report a change
in the cleanup settings as drift. Never delete anything to make room from here.

## 2. Reachability

- **Public lane** resolves from this host: `curl -sI` each public resource at
  `https://<name>.<domains.public_suffix>` (suffix from `instance.yaml`, resources from
  the inventory) and expect 200/30x. A Traefik `no available server` 503 often means an *unhealthy
  container was dropped from the load balancer*, not a network fault — check the
  healthcheck first.
- **Internal lane does not resolve from this host** — a failed curl proves nothing.
  Verify via the two paths that work from anywhere (`docs/internal-services.md`):
  the registrar's logs through the MCP (`plumbing.tailnet_registrar` in
  `instance.yaml`; the `logs` rough edge may 500 — skip, don't fight it), and the
  Tailscale control-plane API using the registrar's own OAuth credentials:
  every service should read `ports=['tcp:443']` and its `/devices` should show
  `approved:auto` + `configured: ready`. Never echo those credentials.

## 2b. Exposure — is the perimeter what `instance.yaml` says?

Two checks, both read-only, both cheap; skip neither.

- **Read the firewall through the Hetzner API when `HCLOUD_TOKEN` is in the shell**
  (`host.provider` hetzner): `GET /v1/servers` → the server by public IP → its
  firewalls → rules. Compare with the set `docs/provisioning.md` §4 renders for this
  instance; a missing firewall, an extra rule (22 to the world, 8000 to `0.0.0.0/0` in
  `github` mode) or a missing one is the lead finding. This is read-only and needs
  only a read token. Without a token, fall back to the probe:
- **Probe the public IP from outside.** `get_server` for the address, then the loop in
  `/setup-coolify-devops` step 2 (`</dev/tcp/<ip>/<port>` with a 3 s timeout) over 22, 80, 443,
  3000, 8000, 6001, 6002 — run here only if this machine is neither the host nor on
  the tailnet (`operator.on_host` false and `domains.operator_tailnet` empty in
  `instance.yaml`); otherwise the loop is a prepared step for a human on a machine
  off the tailnet, and a sweep that could not get the answer says so rather than
  reporting the perimeter as checked. Expected: 80/443 open only with a public lane
  (`domains.public_suffix` set), 8000 open only when `exposure.coolify_ui` is
  `internet` (in `github` mode it must read closed from here), all else closed. A
  port that answers when it should not is the lead finding of the whole sweep — it
  means the cloud firewall is missing or wrong (`docs/provisioning.md`), not that
  some container is misconfigured.
- **Scan the internal lane for published ports.** For every resource in
  `projects.internal` (and `projects.infrastructure`), read the compose
  (`get_service` / `get_application`; `ports_exposes` on applications is Coolify's
  own field, not a published port) and flag any `ports:` entry or `ports_mappings`.
  Internal tools are reached by the registrar over the container network; a
  published port is a leak waiting for the firewall to be relaxed. Report it as
  drift from the lane model even though the firewall currently masks it.

## 3. Drift — live vs. reference copies

For each folder in `stacks/` with a compose copy: fetch the live compose
(`get_service`) and diff it against `stacks/<name>/docker-compose.yml`; read live
file-mount contents (`storages` `action: list`) and diff against the broken-out files.
Report drift with the differing lines — **do not reconcile in either direction** during
a sweep; which side is right is a decision for the user or a `/change-service` session.

## 4. Report

One table — resource, Coolify status, reachability check, drift — leading with anything
actually broken (an unexpected open port first of all, then a disk over 90%), then
disk use and drift, then the
expected-unknowns marked as such. If nothing is
wrong, say so in one line; do not pad. Note anything the sweep could not check and why
(e.g. internal browser-level reachability needs a human on the tailnet).