Sync both server snapshots with what the live machines run, and keep their backup tooling

This commit is contained in:
Gabriel Brown committed 2026-10-30 13:44:29 -04:00
1 parent 95fa480469
commit cb089b39ea
42 files changed
+3099 -21

No files matched your search

+40 -2
View File
@@ -8,7 +8,9 @@ screen, plus rootless podman compose services managed as systemd user units.
server/
vps/ The catalog panama-server installs from, one directory
per service: compose.yml, the podman-<name>.service user
unit, and an .env.example naming what the service needs told
unit, and an .env.example naming what the service needs told.
Loose files beside the directories are the VPS's own
host tooling (see "Host tooling" below)
home/ The home server's services in the same shape (see below)
scripts/ update-containers, the nightly image updater
systemd/ The units behind it, linked by setup-server
@@ -25,6 +27,8 @@ installable: panama-server reads `server/vps/` only, so a service moves into
the catalog when it actually moves to podman.
Paths keep their nesting (`server/home/Agents/Gitea` is `~/Server/Agents/Gitea`).
`Media/compact-tv/` is the nightly TV re-encode job that runs beside the Media
stack, with its own README.
Three services are checkouts of upstream repositories. For those, only the
local files are kept, and an `UPSTREAM` file names the repository and the exact
commit the checkout sits at: `PostHog`, `Sentry` and `Plausible`.
@@ -38,7 +42,8 @@ compose and its copy here can differ on those lines alone.
The repository carries the definitions; the machine carries the state.
`panama server enable <Name>` creates `~/Server/<Name>/` as a **real
directory** and symlinks only `compose.yml` into it; the unit symlinks into
directory** and symlinks the service's visible top-level files into it:
`compose.yml`, plus any helper script or README. The units symlink into
`~/.config/systemd/user/`. The `.env` (seeded once from `.env.example`, then
yours) and the bind-mounted `data/` live in `~/Server/<Name>/` and are never
inside the checkout — so nothing git can do, `clean -fdx` included, can reach
@@ -99,6 +104,39 @@ existing service. The unit is the standard oneshot `podman compose up -d`
shape (`WorkingDirectory=%h/Server/<Name>`); the shape contract pins the
details. Secrets go in `.env.example` as `KEY=CHANGE_ME` lines.
A service that needs a scheduled job of its own carries it beside the
compose unit: a helper `<job>.service` and a `<job>.timer` with the same
name, plus the script it runs. Enable links all three and enables the timer.
Disable stops the timer and unlinks the units. `UseSend` is the example: its
`usesend-cert-refresh` timer keeps the SMTP proxy's certificate in step with
Nginx Proxy Manager's.
## Host tooling
Each server's own scripts and runbooks sit as loose files at the top of its
tree. They are reference copies for rebuilding a machine. Nothing installs
them.
- `vps/backup`, `vps/server-backup.service`, `vps/server-backup.timer`: the
VPS's nightly backup, run as a user unit. It dumps every database into
`~/Server/_backup_state/`, then rsyncs `~/Server` to the home server's ZFS
pool with `--fake-super`, so subuid ownership survives the trip.
- `vps/verify-rootless`: the post-reboot health check for the rootless
stack. It covers linger, the unprivileged port floor, unit state and boot
enablement, postgres, endpoints, and admin ports bound to the WireGuard
address.
- `vps/AGENTS.md`: the VPS runbook, with both public IPs redacted.
- `home/RESTORE.md`: how to restore the home server's `~/Server` from the
ZFS backup.
- `home/backup` and `home/systemd/`: the home server's backup trigger and its
root system units (`server-backup` nightly, `media-backup` weekly). The
scripts those units run live in `/usr/local/sbin` on that machine. They are
readable by root alone, so this tree has no copy of them.
The VPS units carry `RequiresMountsFor=/home/gib/Media` because that host
keeps its podman graphroot on an attached volume mounted there. The setting
is harmless on a machine without the mount.
## Nightly updates
`server/scripts/update-containers` pulls images and restarts changed