Self-host ntfy on pebble for private off-site alerts (ntfy.e.lyte.dev) #690

Merged

lytedev merged 1 commit from pebble-ntfy into main

2026-07-01 15:04:41 -05:00

lytedev commented

2026-07-01 15:03:10 -05:00

Owner

What & why

The Tier 0 off-site uptime watcher (from #683, lib/doc/alerting.md) needs a push channel that does not depend on beefcake. Hosted ntfy.sh works, but locking/reserving a topic is a paid feature — so only an unauthenticated public topic is free there. Self-hosting ntfy gives topic auth for free, and pebble is the ideal host: external (Hetzner, static IP), already in the fleet, independent of beefcake, and a tiny footprint (~20 MB RSS measured against pebble's 3.2 GB free / 3.8 GB total).

Also introduces the .e.lyte.dev convention = external / not-home hosts (mirrors .h for home), as static records to a public IP (unlike the dynamic .h records written by dns-updater).

Changes

dns-zones.nix: ntfy.e A → pebble (204.168.181.230).
packages/hosts/pebble/ntfy.nix (new):
- services.ntfy-sh on 127.0.0.1:2586, auth-default-access = deny-all.
- Caddy TLS edge (Let's Encrypt HTTP-01 — pebble has a public IP and ntfy.e.lyte.dev resolves to it, so no DNS-01/TSIG like beefcake) + open :80/:443.
- Idempotent ExecStartPost provisioning a single alerts user (read-write on the infra-alerts topic) from a sops password (--ignore-exists, so it's a no-op on restart).
secrets/pebble/secrets.yml: seed ntfy-env (ALERTS_PASSWORD=…, a generated 40-char random password) so the build-time sops-manifest check passes and the service is secure on first deploy (no CHANGEME placeholder).

Testing

nix build …#pebble…toplevel ✅
nix build …#beefcake…toplevel ✅ (beefcake is the zone primary, so the DNS change builds there too)
Not deployed.

Exposure note

This adds Caddy + public :80/:443 to pebble, which was previously DNS/mail-only. Surface is small: ntfy binds loopback, Caddy fronts it, and deny-all means only the alerts user can touch any topic.

Deploy + post-deploy (yours)

Deploy beefcake (serves the new ntfy.e record as zone primary) and pebble.
Then I (or you) flip the val via the val.town API: NTFY_URL → https://ntfy.e.lyte.dev/infra-alerts, add NTFY_USER=alerts + NTFY_PASSWORD (same value now in pebble sops), and subscribe the phone's ntfy app to infra-alerts with those creds.

Follow-up to #683 (which documents the Tier 0 watcher and keeps working on the public ntfy.sh topic until this lands).

## What & why The Tier 0 off-site uptime watcher (from #683, `lib/doc/alerting.md`) needs a push channel that does **not** depend on beefcake. Hosted ntfy.sh works, but locking/reserving a topic is a **paid** feature — so only an *unauthenticated public topic* is free there. Self-hosting ntfy gives topic auth for free, and **pebble** is the ideal host: external (Hetzner, static IP), already in the fleet, independent of beefcake, and a tiny footprint (**~20 MB RSS** measured against pebble's 3.2 GB free / 3.8 GB total). Also introduces the **`.e.lyte.dev`** convention = external / not-home hosts (mirrors `.h` for home), as **static** records to a public IP (unlike the dynamic `.h` records written by dns-updater). ## Changes - **`dns-zones.nix`**: `ntfy.e` A → pebble (`204.168.181.230`). - **`packages/hosts/pebble/ntfy.nix`** (new): - `services.ntfy-sh` on `127.0.0.1:2586`, `auth-default-access = deny-all`. - **Caddy** TLS edge (Let's Encrypt **HTTP-01** — pebble has a public IP and `ntfy.e.lyte.dev` resolves to it, so no DNS-01/TSIG like beefcake) + open `:80/:443`. - Idempotent `ExecStartPost` provisioning a single **`alerts`** user (read-write on the `infra-alerts` topic) from a sops password (`--ignore-exists`, so it's a no-op on restart). - **`secrets/pebble/secrets.yml`**: seed `ntfy-env` (`ALERTS_PASSWORD=…`, a generated 40-char random password) so the build-time sops-manifest check passes **and** the service is secure on first deploy (no CHANGEME placeholder). ## Testing - `nix build …#pebble…toplevel` ✅ - `nix build …#beefcake…toplevel` ✅ (beefcake is the zone primary, so the DNS change builds there too) - **Not deployed.** ## Exposure note This adds **Caddy + public :80/:443** to pebble, which was previously DNS/mail-only. Surface is small: ntfy binds loopback, Caddy fronts it, and `deny-all` means only the `alerts` user can touch any topic. ## Deploy + post-deploy (yours) 1. Deploy **beefcake** (serves the new `ntfy.e` record as zone primary) and **pebble**. 2. Then I (or you) flip the val via the val.town API: `NTFY_URL → https://ntfy.e.lyte.dev/infra-alerts`, add `NTFY_USER=alerts` + `NTFY_PASSWORD` (same value now in pebble sops), and subscribe the phone's ntfy app to `infra-alerts` with those creds. Follow-up to #683 (which documents the Tier 0 watcher and keeps working on the public ntfy.sh topic until this lands).

lytedev added 1 commit

2026-07-01 15:03:10 -05:00

feat(pebble): self-host ntfy for private off-site push alerts (ntfy.e.lyte.dev)

/ check-format (push) Successful in 7s

Details

/ build (push) Successful in 5m49s

Details

316b5bdfb3

The Tier 0 uptime watcher (lib/doc/alerting.md) needs a push channel that does
NOT depend on beefcake. ntfy.sh (hosted) works but locking a topic is a paid
feature, so only an unauthenticated public topic is free there. Self-hosting
ntfy gives topic auth for free, and pebble is the ideal host: external (Hetzner,
static IP), already in the fleet, independent of beefcake, and tiny footprint
(~20 MB RSS against 3.2 GB free).

Introduces the `.e.lyte.dev` convention = external/not-home hosts (mirrors `.h`
for home), as STATIC records to a public IP (unlike the dynamic `.h` records).

- dns-zones.nix: add `ntfy.e` A -> pebble (204.168.181.230).
- packages/hosts/pebble/ntfy.nix: services.ntfy-sh on loopback + Caddy TLS edge
  (Let's Encrypt HTTP-01 — pebble has a public IP, no DNS-01/TSIG needed) +
  open :80/:443; auth-default-access = deny-all with an idempotent ExecStartPost
  that provisions a single `alerts` user (read-write on the `infra-alerts`
  topic) from a sops-provided password.
- secrets/pebble/secrets.yml: seed `ntfy-env` (ALERTS_PASSWORD=…, a generated
  40-char random password) so the build-time sops manifest check passes and the
  service is secure on first deploy.

Build-checked: pebble + beefcake (zone primary) toplevels both build.

NOT deployed. Post-deploy steps (own follow-up, done via the val.town API):
point the val's NTFY_URL at https://ntfy.e.lyte.dev/infra-alerts and set
NTFY_USER=alerts + NTFY_PASSWORD (same sops value); subscribe the phone.
Deploying the DNS record requires a beefcake (primary) deploy too.

lytedev merged commit 6a8eb43646 into main

2026-07-01 15:04:41 -05:00

lytedev deleted branch pebble-ntfy

2026-07-01 15:04:41 -05:00