1790 lines
90 KiB
Markdown
1790 lines
90 KiB
Markdown
# AGENTS.md — `~/Server` container environment
|
||
|
||
> Snapshot of `junior.gib:~/Server/AGENTS.md`. The two public addresses read
|
||
> `<HOME_PUBLIC_IP>` and `<VPS_PUBLIC_IP>` here. The live copy has the real ones.
|
||
|
||
Operating guide for AI agents and humans working on this VPS. Read this **before** adding a
|
||
container, editing a compose file, or touching the firewall. Much of what follows is
|
||
non-obvious and was learned the hard way; deviating will break things.
|
||
|
||
**Host:** Hetzner VPS, Fedora 44, kernel 7.1.7-200, `podman 5.8.4`, `podman-compose 1.6.0`,
|
||
SELinux **Enforcing** (targeted), firewalld active.
|
||
**User:** `gib` (uid 1000), subuid/subgid range `524288:65536`.
|
||
|
||
> **This file is the document. `CLAUDE.md` is a symlink to it.** That way Claude Code loads
|
||
> it automatically, while the 20+ compose files and scripts that cite `AGENTS.md §N` keep
|
||
> resolving. Edit `AGENTS.md`; never replace either name with a second real copy, or the two
|
||
> will drift apart silently.
|
||
>
|
||
> The symlink previously ran the other way and a markdown formatter dissolved it on
|
||
> 2026-09-22 — saving through a symlink replaced it with a regular file and orphaned the
|
||
> original. If `CLAUDE.md` ever turns back into a real file, that is what happened.
|
||
|
||
## Docs that deliberately live elsewhere
|
||
|
||
Kept next to the thing they describe, because that is where you will be standing when you
|
||
need them:
|
||
|
||
| Path | Covers |
|
||
| ------------------------------------- | ------------------------------------------------------- |
|
||
| `UseSend/certs/README.md` | Why the SMTP proxy needs a real cert, and refreshing it |
|
||
| `Authentik/blueprints/REPRODUCE.md` | Rebuilding the password-recovery flow from scratch |
|
||
| `Next_Apps/Completeuphoria/README.md` | That app — **its own git repo**, do not fold in |
|
||
| `Search/Bang/README.md` | That app — **its own git repo**, do not fold in |
|
||
|
||
Completed history (migration plan, session reports, one-off agent prompts) moved to
|
||
`archive/` on 2026-09-22. Nothing there is current; consult it for _why_, never for _how_.
|
||
|
||
---
|
||
|
||
## 1. The one rule that governs everything
|
||
|
||
> **Everything is ROOTLESS podman. There is no root podman on this box and there must
|
||
> never be again.**
|
||
|
||
`/var/lib/containers` is empty, all root systemd units are deleted, and root's store was
|
||
wiped with `podman system reset`. If you catch yourself typing `sudo podman`, stop — you are
|
||
about to create a second, invisible container store that duplicates disk and silently
|
||
diverges from the running stack.
|
||
|
||
| Do | Never |
|
||
| ---------------------------------- | ----------------------------------- |
|
||
| `podman ...` as `gib` | `sudo podman ...` |
|
||
| `systemctl --user ...` | `sudo systemctl ...` for containers |
|
||
| units in `~/.config/systemd/user/` | units in `/etc/systemd/system/` |
|
||
| `WantedBy=default.target` | `WantedBy=multi-user.target` |
|
||
|
||
---
|
||
|
||
## 2. Layout and conventions
|
||
|
||
```
|
||
~/Server/
|
||
├── AGENTS.md <- this file
|
||
├── archive/ <- completed history; see archive/README.md
|
||
├── verify-rootless.sh <- run after ANY change; exits non-zero on failure
|
||
├── backup <- rsync to remote; runs unprivileged
|
||
├── podman-example.service <- user-unit template for new services
|
||
├── compose.example.yml <- compose template
|
||
└── <Service>/
|
||
├── compose.yml
|
||
├── .env <- secrets; never commit, never echo in full
|
||
├── podman-<name>.service <- copy of the installed user unit (reference)
|
||
└── volumes/ | data/ | <vol>/ <- bind-mounted data, gib-owned
|
||
```
|
||
|
||
### Service inventory (as of 2026-08-12, end of migration)
|
||
|
||
**40 containers across 22 active user units.** `podman ps` is the source of truth; this is
|
||
orientation. Verify enablement with `verify-rootless.sh` (§13.3), not by eye.
|
||
|
||
| Unit | Directory | Notes |
|
||
| ------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------- |
|
||
| postgresql | PostgreSQL | **shared, 14 databases.** pgvector, `-trixie` base only (§13.4) |
|
||
| nginx-proxy-manager | Nginx_Proxy_Manager | the front door; admin on `nginx.gibbyb.com` |
|
||
| authentik | Authentik | SSO. Port 9000 on the WireGuard IP only. **Protects many hosts by forward auth (§13.9)** |
|
||
| gitea | Gitea | git + container registry. No `userns_mode` (s6). Packages on the volume |
|
||
| watchtower | Watchtower | auto-update; API on WireGuard only; excludes postgres + authentik |
|
||
| adminer | Adminer | DB admin UI; no auth of its own |
|
||
| vaultwarden | Vaultwarden | |
|
||
| infisical | Infisical | secrets; DB on the shared postgres |
|
||
| tools | Tools | convertx, it-tools, pdf-tools, bento-pdf |
|
||
| rustdesk | Rustdesk | **split DNS — see §7.** Relay needs raw TCP; key must never be regenerated |
|
||
| beszel | Beszel | hub + this host's agent |
|
||
| uptime | Uptime | Uptime Kuma; unconfined (socket) |
|
||
| portainer | Portainer | unconfined (socket). Public hostname is `port.gbrown.org` |
|
||
| bang | Search/Bang/docker | built locally; defaults to Google since searxng was removed |
|
||
| isponsorblocktv | ISponsorBlockTV | host network, `keep-id` |
|
||
| completeuphoria | Next_Apps/Completeuphoria/docker | built locally |
|
||
| sierraandtyler | Next_Apps/sierraandtyler/docker | built locally |
|
||
| convexmonorepo | ConvexMonorepo | Next + Convex backend + dashboard |
|
||
| spoon | Spoon | Next + Convex + agent-worker (unconfined, needs the socket) |
|
||
| stpeteit | StPeteIT | Next + Convex + dashboard |
|
||
| lashaddict | LashAddict | Next + Convex + dashboard. **lashaddictstudio.com** (own Cloudflare zone); see §13.17 for the DNS pin |
|
||
| usesend | UseSend | app + redis + smtp-server + RustFS storage (`storage.usesend.gbrown.org`, pinned tag; replaced MinIO 2026-10-27) |
|
||
| **n8n** | N8n | **intentionally stopped** — unit disabled, data kept. `systemctl --user enable --now podman-n8n.service` |
|
||
|
||
**Convex stacks use two ports** — `api.*` → 3210 (client API), `convex.*` → 3211 (HTTP
|
||
actions). Getting these backwards produces a working-looking site that 500s.
|
||
|
||
**Compose files present but NEVER STARTED** (no unit enabled): `Documenso`, `CalCom`,
|
||
`Penpot`. Each needs its role and database created first; Documenso additionally needs a
|
||
signing certificate. Instructions are in each compose file's header.
|
||
|
||
Never migrated / stays home: nextcloud, immich, plex, jellyfin, \*arr, sentry, openwebui,
|
||
frigate, romm, obsidian-livesync, the KVMs. **Home runs Docker, not podman** (§13.8b).
|
||
|
||
Service directory names are `TitleCase`; unit and container names are `lowercase`.
|
||
Some services nest under a category (`Search/Bang/docker`, `Next_Apps/Completeuphoria/docker`) —
|
||
the unit's `WorkingDirectory` is what matters, not depth.
|
||
|
||
---
|
||
|
||
## 3. Adding a new container — the checklist
|
||
|
||
### 3.1 compose.yml
|
||
|
||
```yaml
|
||
networks:
|
||
nginx-bridge:
|
||
external: true # ALWAYS external; see §5
|
||
|
||
services:
|
||
myservice:
|
||
image: docker.io/vendor/image:tag
|
||
container_name: myservice
|
||
hostname: myservice
|
||
networks: ["nginx-bridge"]
|
||
environment:
|
||
- TZ=America/New_York # see §8
|
||
volumes:
|
||
- ./data:/var/lib/whatever:Z # :Z is MANDATORY; see §6
|
||
restart: unless-stopped
|
||
```
|
||
|
||
Do **not** publish ports unless the service must be reachable off-box. Everything on
|
||
`nginx-bridge` resolves by container name, so NPM proxies to `http://myservice:8080`
|
||
internally with no host port at all. Fewer published ports = less firewall surface.
|
||
|
||
### 3.2 Does it need `userns_mode`?
|
||
|
||
This is the subtlest part of the setup. Decide by how the image starts:
|
||
|
||
```bash
|
||
podman inspect <image> --format '{{.Config.User}}' # empty => starts as root
|
||
```
|
||
|
||
| Image behaviour | What to do | Host file ownership |
|
||
| ------------------------------------------------------------------------------------------------------------- | -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------- |
|
||
| **Starts as root**, drops privileges internally (most images: nginx, postgres, uptime-kuma, influxdb, beszel) | Nothing. Default mapping puts container-root at host uid 1000. | `gib` for root-owned paths; the container's own chown pushes service-user files into subuid space (e.g. `525264`) |
|
||
| **Runs directly as a fixed non-root uid**, or you want host files to stay `gib`-owned | `userns_mode: "keep-id:uid=<N>,gid=<N>"` | `gib` |
|
||
|
||
**Never use `user: 1000:1000`.** Under a rootless userns that resolves to subuid `525287`,
|
||
not to `gib`, and the container loses access to its own gib-owned data. This exact mistake
|
||
is why n8n was broken. Use `keep-id` instead.
|
||
|
||
Live examples:
|
||
|
||
- `PostgreSQL` — `keep-id:uid=999,gid=999` (postgres runs as 999; keeps `pg_data` gib-owned)
|
||
- `N8n` — `keep-id:uid=1000,gid=1000`
|
||
- everything else — default mapping
|
||
|
||
**uid mapping cheat sheet** (subuid base 524288):
|
||
|
||
```
|
||
container uid 0 -> host 1000 (gib)
|
||
container uid 999 -> host 525286
|
||
container uid 1000 -> host 525287
|
||
container uid N -> host 524288 + (N-1)
|
||
```
|
||
|
||
### 3.3 Host directory ownership
|
||
|
||
Create data dirs **owned by gib** before first start:
|
||
|
||
```bash
|
||
mkdir -p ~/Server/MyService/data # already gib-owned
|
||
```
|
||
|
||
If you ever import data written by another uid, `sudo chown -R gib:gib` it first. A uid
|
||
outside gib's subuid range appears as `nobody` inside the container and cannot be fixed by
|
||
the container itself.
|
||
|
||
### 3.4 systemd user unit
|
||
|
||
Copy `podman-example.service`, adjust `Description` and `WorkingDirectory`:
|
||
|
||
```bash
|
||
cp ~/Server/podman-example.service ~/.config/systemd/user/podman-myservice.service
|
||
# edit WorkingDirectory=%h/Server/MyService
|
||
systemctl --user daemon-reload
|
||
systemctl --user enable --now podman-myservice.service
|
||
cp ~/.config/systemd/user/podman-myservice.service ~/Server/MyService/ # keep a copy
|
||
```
|
||
|
||
If it depends on Postgres, add to `[Unit]`:
|
||
|
||
```ini
|
||
After=network-online.target podman-postgresql.service
|
||
Wants=podman-postgresql.service
|
||
```
|
||
|
||
> ⚠️ Ordering ≠ readiness. `podman compose up -d` returns when the container is _created_,
|
||
> not when Postgres accepts connections. Dependents must tolerate a brief DB-unavailable
|
||
> window (NPM and n8n both retry). If one doesn't, add an `ExecStartPre` polling
|
||
> `pg_isready`.
|
||
|
||
### 3.5 Verify
|
||
|
||
```bash
|
||
~/Server/verify-rootless.sh # add your service to the lists inside it
|
||
```
|
||
|
||
---
|
||
|
||
## 4. Boot behaviour (linger)
|
||
|
||
User services start at boot **without anyone logging in** because lingering is enabled:
|
||
|
||
```bash
|
||
loginctl show-user gib --property=Linger # must be Linger=yes
|
||
```
|
||
|
||
If linger is ever off, nothing starts until someone SSHes in — and logging in _masks_ the
|
||
failure. When testing boot behaviour, check from **outside** the box first, before SSHing in.
|
||
|
||
Proven working: after reboot, `[email protected]` starts ~9 s after boot and all containers
|
||
~11–13 s after, well before any login.
|
||
|
||
---
|
||
|
||
## 5. Networking
|
||
|
||
All services share the external network **`nginx-bridge` (172.18.0.0/24, gw 172.18.0.1)**.
|
||
|
||
Created out-of-band so no stack races to create it at boot:
|
||
|
||
```bash
|
||
podman network create --subnet 172.18.0.0/24 nginx-bridge
|
||
```
|
||
|
||
Every compose file declares it `external: true` — including NPM, which used to define it
|
||
inline. Containers resolve each other by `container_name` over this network.
|
||
|
||
> **Rootless and root networks are entirely separate namespaces.** This is why the migration
|
||
> had to be a big-bang cutover. Irrelevant now that root podman is gone, but it explains why
|
||
> you cannot "test one service" against a differently-privileged stack.
|
||
|
||
### Privileged ports
|
||
|
||
NPM binds 80/443 without root because of `/etc/sysctl.d/99-rootless-ports.conf`:
|
||
|
||
```
|
||
net.ipv4.ip_unprivileged_port_start=80
|
||
```
|
||
|
||
Host-wide: **any** unprivileged process can bind ≥80. Acceptable on a single-user VPS. If
|
||
this file is lost, NPM silently fails to bind and every site goes down.
|
||
|
||
### ⚠️ Published ports lose the client IP
|
||
|
||
Rootless publishes ports through the **`rootlessport` userspace proxy**, so the peer address
|
||
nginx sees is `172.18.0.x`, not the real client. **No header trick recovers this when there
|
||
is no proxy in front** — nothing upstream sets one.
|
||
|
||
The only real fix is to put the hostname behind Cloudflare, so `CF-Connecting-IP` exists.
|
||
See §7.
|
||
|
||
---
|
||
|
||
## 6. SELinux — Enforcing
|
||
|
||
Every bind-mounted volume **must** carry `:Z` (or `:rw,Z` / `:ro,Z`):
|
||
|
||
```yaml
|
||
volumes:
|
||
- ./data:/var/lib/whatever:Z
|
||
```
|
||
|
||
`:Z` makes podman relabel the host path to `container_file_t` with a **private MCS category**
|
||
unique to that container, re-derived on every start. Without it, Enforcing denies all access
|
||
and the container fails.
|
||
|
||
### ⚠️ Shared mounts need `:z` (lowercase), not `:Z`
|
||
|
||
`:Z` assigns a **private** MCS category unique to one container. If two containers mount the
|
||
**same host path**, the second to start relabels it and locks the first out. Use lowercase
|
||
`:z` for shared paths:
|
||
|
||
```yaml
|
||
server:
|
||
volumes:
|
||
- ./volumes/media:/data/media:z # shared with worker -> lowercase
|
||
- ./volumes/server/custom-templates:/templates:Z # exclusive -> uppercase
|
||
worker:
|
||
volumes:
|
||
- ./volumes/media:/data/media:z # same path, same lowercase z
|
||
- ./volumes/worker/certs:/certs:Z # exclusive -> uppercase
|
||
```
|
||
|
||
Verify: a shared dir shows `container_file_t:s0` with **no** category; an exclusive dir's
|
||
category must match its container's `ProcessLabel`.
|
||
|
||
```bash
|
||
ls -dZ volumes/media # want: container_file_t:s0
|
||
podman inspect authentik-server --format '{{.ProcessLabel}}'
|
||
```
|
||
|
||
### ⚠️ Watchtower recreates DROP the :Z/:z bind flag (found 2026-08-13)
|
||
|
||
When watchtower recreates a container after pulling a new image, the copied
|
||
bind loses its relabel option (`Binds` shows only `rprivate,rbind`). With `:Z`
|
||
the volume keeps the OLD container’s MCS category while the new container gets
|
||
a fresh pair — the service is locked out of its own data. agentchat hit this
|
||
as SQLITE_CANTOPEN in a crashloop; fixed by switching its bind to `:z`.
|
||
|
||
**Any watchtower-updated container with a bind mount must use lowercase `:z`.**
|
||
After a recreate the files stay `container_file_t:s0` (no category), which a
|
||
confined container can still access — the only labeling that survives.
|
||
The Convex stacks (lashaddict / stpeteit / convexmonorepo / spoon) were
|
||
switched to `:z` and restarted 2026-08-13; all backends verified healthy.
|
||
|
||
Two deliberate exceptions:
|
||
|
||
1. **The podman socket mount gets NO `:Z`.**
|
||
```yaml
|
||
- /run/user/1000/podman/podman.sock:/var/run/docker.sock # no :Z, ever
|
||
```
|
||
Relabeling the live socket would break podman itself.
|
||
2. **Containers mounting that socket need SELinux separation off**, because systemd
|
||
recreates the socket each boot as `user_tmp_t`:
|
||
```yaml
|
||
security_opt: ["label:disable"]
|
||
```
|
||
Currently only **portainer** and **uptime**. They run unconfined (empty `ProcessLabel`);
|
||
the other nine are confined as `container_t:s0:c…`. Accept this only for
|
||
container-management tooling.
|
||
|
||
Check confinement:
|
||
|
||
```bash
|
||
podman inspect <name> --format '{{.ProcessLabel}}' # empty = unconfined
|
||
```
|
||
|
||
> ### ⚠️ NEVER run `restorecon -F` against `~/Server`
|
||
>
|
||
> Plain `restorecon -r` is **safe** — it reports the `:Z` labels as "customized by admin"
|
||
> and leaves them alone. `-F` forces them back to `user_home_t` and every container loses
|
||
> access to its data simultaneously.
|
||
|
||
Going Permissive→Enforcing needed no relabel here: `restorecon -nr` on `/etc /usr/sbin
|
||
/usr/bin /usr/lib/systemd /var/lib /root /boot` reported 0 mislabeled files. Re-check that
|
||
before flipping SELinux on any rebuild.
|
||
|
||
---
|
||
|
||
## 7. Firewall and Cloudflare
|
||
|
||
firewalld zones — **rules apply by arriving interface**:
|
||
|
||
| Interface | Zone | Policy |
|
||
| ------------------------------- | --------- | ------------------------------------ |
|
||
| `eth0` (<VPS_PUBLIC_IP>, public) | `public` | `ssh` + explicitly opened ports only |
|
||
| `Home` (192.168.2.2, WireGuard) | `trusted` | target ACCEPT — full LAN/VPN access |
|
||
|
||
Binding `Home` to `trusted` is what keeps LAN devices working when `public` is locked down.
|
||
Before that it had **no zone** and silently inherited `public`.
|
||
|
||
Deliberately closed on the public interface: **81** (NPM admin), 3001, 9443, **5432**.
|
||
Reach NPM admin via `https://nginx.gibbyb.com` or over WireGuard at `192.168.2.2:81`.
|
||
Never open 5432 to the internet.
|
||
|
||
### ⚠️ Every VPS hostname must be Proxied (orange cloud)
|
||
|
||
This is the single most important DNS rule here, and it has bitten repeatedly. Because
|
||
80/443 accept only Cloudflare ranges, **a DNS-only record pointing at <VPS_PUBLIC_IP> is
|
||
unreachable** — it fails at connect, not with an error page. The only exception is
|
||
rustdesk, below, which uses ports the firewall opens explicitly.
|
||
|
||
### Rustdesk — the one service that cannot be proxied
|
||
|
||
RustDesk's ID/relay protocol is raw TCP/UDP on 21115-21119. Cloudflare's proxy carries only
|
||
HTTP/HTTPS, so it uses **two hostnames on purpose**:
|
||
|
||
| Hostname | Cloud | Serves |
|
||
| --------------------- | ------------------------------------------- | ----------------------------- |
|
||
| `relay.gbrown.org` | **grey / DNS-only** → <VPS_PUBLIC_IP> | ID + relay, ports 21115-21119 |
|
||
| `rustdesk.gbrown.org` | **proxied** → NPM → `rustdesk-server:21114` | web console + API, over TLS |
|
||
|
||
Do not "simplify" these into one name — the API would then have to travel unencrypted, or
|
||
the relay would break. `relay.gbrown.org` publishes the VPS IP; that is accepted, because
|
||
80/443 remain Cloudflare-only so the IP grants no access to any web service.
|
||
|
||
Firewall ports opened for it (public zone): `21114-21119/tcp` plus **`21116/udp`** — the UDP
|
||
one is easy to miss and heartbeats fail silently without it.
|
||
|
||
The server keypair at `Rustdesk/volumes/server/id_ed25519` was carried over from home. Its
|
||
public half must keep matching `RUSTDESK_API_RUSTDESK_KEY` in the compose, or **every client
|
||
needs re-pairing**. Never regenerate it.
|
||
|
||
### Split-horizon DNS — LAN clients need their own records
|
||
|
||
The UniFi gateway answers `*.gbrown.org` → `192.168.1.2` (home) for every hostname. So after
|
||
migrating a service, LAN clients keep silently reaching the OLD copy — no error, just stale
|
||
data. Public DNS being correct does not help them.
|
||
|
||
**For each migrated service, add a UniFi Host (A) record → `192.168.2.2`** (the VPS's
|
||
WireGuard address). UniFi honours exact matches over the wildcard, so the wildcard stays put
|
||
for everything still at home. Verified working: LAN clients reach `192.168.2.2:443`, the
|
||
certificate validates, and the firewall's `trusted` zone permits it.
|
||
|
||
Do **not** point LAN records at `<VPS_PUBLIC_IP>` — the firewall drops non-Cloudflare traffic.
|
||
Do **not** point them at Cloudflare edge IPs — those are not stable.
|
||
|
||
Currently carrying UniFi overrides: `vault`, `tools`, `infisical`, `auth`.
|
||
|
||
> **Never add `nginx.gbrown.org` → VPS.** That hostname reaches _home's_ proxy manager admin
|
||
> UI. Pointing it at the VPS locks you out of it. The VPS admin lives on `nginx.gibbyb.com`.
|
||
|
||
### Advanced Certificate Manager
|
||
|
||
**ACM is active on `gbrown.org` only** (~$10/mo, confirmed billed to that zone). Universal
|
||
SSL covers one subdomain level; ACM covers the multi-level wildcards. Active certificate
|
||
(expires 2026-11-09) carries seven SANs:
|
||
|
||
```
|
||
gbrown.org, *.gbrown.org, *.tools.gbrown.org, *.spoon.gbrown.org,
|
||
*.lashaddict.gbrown.org, *.convexmonorepo.gbrown.org, *.convex.gbrown.org
|
||
```
|
||
|
||
A **second** advanced certificate (ordered 2026-10-27) carries the two-label patterns added
|
||
later. Advanced certificates cannot be edited, so a new pattern means ordering another one, never
|
||
deleting the first (that would drop every name only it covers):
|
||
|
||
```
|
||
*.usesend.gbrown.org, usesend.gbrown.org, *.billtracker.gbrown.org, billtracker.gbrown.org
|
||
```
|
||
|
||
Without it, `convert.tools.gbrown.org` and the 12 other two-label hostnames cannot be
|
||
proxied at all. If a new two-label pattern appears, it needs adding to this certificate.
|
||
|
||
**Ports 80/443 on `eth0` are restricted to Cloudflare only** (active since 2026-08-11), via
|
||
firewalld ipsets `cloudflare4` (15 ranges) and `cloudflare6` (7 ranges):
|
||
|
||
```
|
||
tcp dport 22 accept <- ssh, never remove
|
||
ip saddr @cloudflare4 tcp dport 80,443 accept
|
||
ip6 saddr @cloudflare6 tcp dport 80,443 accept
|
||
<- everything else: zone default reject
|
||
```
|
||
|
||
**Refresh the ranges periodically** — Cloudflare does change them, and a stale set silently
|
||
blocks legitimate traffic:
|
||
|
||
```bash
|
||
curl -s https://www.cloudflare.com/ips-v4 -o /tmp/cf-v4.txt
|
||
curl -s https://www.cloudflare.com/ips-v6 -o /tmp/cf-v6.txt
|
||
sudo firewall-cmd --permanent --ipset=cloudflare4 --add-entries-from-file=/tmp/cf-v4.txt
|
||
sudo firewall-cmd --permanent --ipset=cloudflare6 --add-entries-from-file=/tmp/cf-v6.txt
|
||
sudo firewall-cmd --reload
|
||
```
|
||
|
||
> **You cannot test this restriction from the VPS itself.** Traffic from the host to its own
|
||
> public IP routes over loopback and never traverses the `eth0` INPUT path, so it always
|
||
> succeeds and proves nothing. Verify from an external machine:
|
||
>
|
||
> ```bash
|
||
> curl -k --max-time 10 --resolve any.gibbyb.com:443:<VPS_PUBLIC_IP> https://any.gibbyb.com/
|
||
> ```
|
||
>
|
||
> **Verified 2026-08-11:** `curl: (7) Failed to connect ... after 46 ms: Could not connect`.
|
||
> A fast _rejection_ (tens of ms) is the correct signal — the zone's `default` target sends
|
||
> an ICMP reject. A 10 s hang would mean a DROP instead; a `200`/`302` would mean the rule
|
||
> is not taking effect. Meanwhile `https://any.gibbyb.com/` via Cloudflare returns normally.
|
||
|
||
> ### ⚠️ Verify DNS is actually proxied before restricting 80/443
|
||
>
|
||
> A Cloudflare-only rule black-holes any hostname whose DNS points straight at the origin.
|
||
> **This happened during setup**: `*.gibbyb.com` was DNS-only, and applying the rule took
|
||
> every one of those sites down until the ports were reopened. Check each hostname; do not
|
||
> trust an assumption that "everything is behind Cloudflare":
|
||
>
|
||
> ```bash
|
||
> resolvectl query uptime.gibbyb.com # <VPS_PUBLIC_IP> = DIRECT (would be blocked)
|
||
> # 104.21.x / 172.67.x = Cloudflare (safe)
|
||
> ```
|
||
>
|
||
> **Any new hostname pointed at this VPS must be Proxied (orange cloud) in Cloudflare
|
||
> before it will work at all.** DNS-only records are blocked by the firewall.
|
||
>
|
||
> Current DNS: `gibbyb.com` A → `<VPS_PUBLIC_IP>` **Proxied**, `*.gibbyb.com` CNAME →
|
||
> `gibbyb.com` **Proxied**, so new subdomains inherit proxying automatically.
|
||
> Cloudflare SSL/TLS mode must be **Full (strict)** — NPM serves valid Let's Encrypt certs;
|
||
> "Flexible" causes redirect loops.
|
||
>
|
||
> Note `*.gbrown.org` resolves to `<HOME_PUBLIC_IP>` — a **different machine** (home network).
|
||
> NPM here holds configs for those names but traffic does not arrive here.
|
||
|
||
### Real client IPs
|
||
|
||
`volumes/data/nginx/custom/server_proxy.conf` (included in every proxy host's **server**
|
||
block) contains:
|
||
|
||
```nginx
|
||
real_ip_header CF-Connecting-IP;
|
||
```
|
||
|
||
It must be in _server_ context — `nginx.conf:81` already declares `real_ip_header` at http
|
||
level, and redeclaring there is a fatal duplicate-directive error. NPM's `nginx.conf` already
|
||
trusts the bridge via `set_real_ip_from 172.16.0.0/12`, so no `set_real_ip_from` is needed.
|
||
|
||
This is now trustworthy **because** the firewall permits 80/443 only from Cloudflare ranges:
|
||
nothing else can reach nginx to forge the header. The two controls depend on each other —
|
||
if you ever reopen 80/443 to the internet, `CF-Connecting-IP` becomes forgeable again and
|
||
logged IPs can no longer be trusted for access lists or fail2ban.
|
||
|
||
---
|
||
|
||
## 8. Timezone
|
||
|
||
Host and all containers are **`America/New_York`**. New services should set
|
||
`TZ=America/New_York` in `environment:`, or their logs stamp UTC and correlating across
|
||
services becomes painful.
|
||
|
||
---
|
||
|
||
## 9. Backups
|
||
|
||
`~/Server/backup` rsyncs to a remote host and **runs unprivileged** — do not reintroduce
|
||
`sudo`. As of 2026-08-11 every file under `~/Server` is gib-owned and readable (verified:
|
||
4,307 entries traversed, 0 permission errors), so nothing is being silently skipped.
|
||
|
||
It still carries `--exclude=dump.rdb` defensively. That was for valkey, which chowned its
|
||
own data to subuid 525286 at mode 600 inside the userns — unreadable to gib. Valkey went
|
||
with searxng, but any future service that writes as a non-root uid with restrictive modes
|
||
will hit the same thing. **If you add one, re-run the check above**; a `Permission denied`
|
||
in rsync means data is silently missing from your backups:
|
||
|
||
```bash
|
||
rsync -avn --exclude=node_modules --exclude=.git ~/Server/ /tmp/probe/ 2>&1 | grep -i denied
|
||
```
|
||
|
||
Database backups are **not** covered by rsync in any useful form — take a logical dump:
|
||
|
||
```bash
|
||
podman exec postgresql pg_dumpall -U npm > ~/pg_dumpall_$(date +%F).sql
|
||
```
|
||
|
||
Do this before any change touching `pg_data` ownership, permissions or SELinux labels.
|
||
|
||
Pre-migration snapshot kept at `~/premigration_backup_2026-08-10/` (volume tarball +
|
||
4,043-entry ownership manifest + pg dump).
|
||
|
||
---
|
||
|
||
## 10. Operational gotchas
|
||
|
||
### ⚠️ Encrypted ClientHello breaks split-horizon DNS — keep `ech` OFF
|
||
|
||
**This cost more time than everything else combined, and nothing about it is guessable.**
|
||
|
||
Cloudflare publishes an HTTPS/SVCB record for proxied hostnames containing an `ech=`
|
||
parameter whose public name decodes to `cloudflare-ech.com`. Browsers read that record,
|
||
enable Encrypted ClientHello, and send **`cloudflare-ech.com` as the outer SNI**. With a
|
||
UniFi override sending that connection to our own nginx instead of Cloudflare, nginx has
|
||
never heard of `cloudflare-ech.com` and replies with **TLS alert 112, unrecognized_name**.
|
||
Browsers do not retry.
|
||
|
||
Symptoms, all of which we chased down blind alleys:
|
||
|
||
- Browser shows `ERR_SSL_UNRECOGNIZED_NAME_ALERT`; **`curl` and `openssl` return 200**.
|
||
They never query HTTPS RRs, so they send plaintext SNI and always succeed.
|
||
- Public access works fine — ECH terminates at Cloudflare, which understands it.
|
||
- Only LAN clients with a UniFi override are affected.
|
||
- A hostname with **no** HTTPS record (grey cloud, e.g. `sonarr`) works normally. That
|
||
contrast is the diagnostic.
|
||
|
||
The zone setting is `ech`. It is **hard-disabled in the dashboard UI on the Free plan but
|
||
editable via the zone settings API**. It must stay `off` for as long as split-horizon DNS
|
||
points any proxied hostname at a local origin.
|
||
|
||
Reproduce in one command:
|
||
|
||
```bash
|
||
openssl s_client -connect 192.168.2.2:443 -servername cloudflare-ech.com
|
||
# -> tlsv1 unrecognized name ... SSL alert number 112
|
||
dig vault.gbrown.org HTTPS # must show NO ech= parameter
|
||
```
|
||
|
||
> **Do not test browser problems with curl.** curl bypasses HTTPS RRs, ECH, HSTS, service
|
||
> workers and the HTTP cache. A green curl next to a failing browser means the difference
|
||
> between them _is_ the bug. Reproduce with the actual browser engine — headless works:
|
||
> `chrome --headless=new --dump-dom <url>` and read stderr for `net_error` codes.
|
||
|
||
**nginx sends no SNI on `proxy_pass` by default.** Proxying to an HTTPS upstream that serves
|
||
multiple vhosts — e.g. home NPM → `https://192.168.2.2:443` — fails the TLS handshake with
|
||
no useful error, surfacing as a 502. Cost us hours. Always add:
|
||
|
||
```nginx
|
||
proxy_ssl_server_name on;
|
||
proxy_ssl_name $host;
|
||
```
|
||
|
||
Symptom to recognise: `curl` to the bare IP gives `000`, but `curl --resolve host:443:IP`
|
||
gives a real status. That difference _is_ the diagnosis.
|
||
|
||
**Scheme/port mismatches produce misleading errors.** We hit this three separate times:
|
||
`http` → port 9443, `https` → port 9000, `http` → port 443. All produced 400/502/000 rather
|
||
than anything naming the cause. When a proxy host misbehaves, check scheme against port
|
||
_first_. Authentik serves HTTP on 9000 and HTTPS on 9443; NPM serves HTTPS on 443.
|
||
|
||
**NPM reloads nginx on every save, and requests during the reload get a 503 with an empty
|
||
body.** Wait ~10 s and retry before treating a 503 as a real failure. Multiple "outages"
|
||
this session were this.
|
||
|
||
**A 401 is not a failure for CouchDB or for authentik's outpost.** CouchDB demands
|
||
credentials on every endpoint; browsers render the resulting auth prompt as an error page.
|
||
Check the nginx access log for the status nginx actually returned rather than trusting what
|
||
a browser displays.
|
||
|
||
**Don't test forward-auth with bare curl.** The authentik outpost needs `X-Forwarded-*`
|
||
headers to reconstruct the original URL; without them it returns 500 and logs "failed to
|
||
detect a forward URL from nginx". That 500 is a bad test, not a broken outpost.
|
||
|
||
**DNS negative caching outlives record changes.** After deleting and recreating a record,
|
||
`systemd-resolved` will keep serving NXDOMAIN. `sudo resolvectl flush-caches`. A hostname
|
||
that resolves with `dig @1.1.1.1` but fails with `curl` is this.
|
||
|
||
**`nc -z` inside the NPM container is not a reliable upstream probe** — it reports false
|
||
negatives for containers that are demonstrably serving. Test the real path with `curl
|
||
--resolve host:443:127.0.0.1` instead.
|
||
|
||
### Older gotchas
|
||
|
||
**Check `systemctl --failed` and capture logs BEFORE vacuuming journals.** A
|
||
`journalctl --vacuum-size` during cleanup destroyed the logs explaining an n8n failure that
|
||
had been running for two months. Diagnose first, reclaim space second.
|
||
|
||
**`podman image prune -a` is dangerous while the stack is stopped.** With no containers
|
||
running, `-a` considers every image unused and deletes images the stack depends on. Use plain
|
||
`podman image prune` (dangling only).
|
||
|
||
**Env var names must match what the image expects.** `PostgreSQL/.env` defined
|
||
`POSTGRES_NAME` while compose passed `POSTGRES_DB`, so the healthcheck expanded to
|
||
`pg_isready -d -U npm` and failed 178,503 consecutive times while Postgres itself was fine.
|
||
Both keys are present now; keep them in sync.
|
||
|
||
**Every service in a compose file needs its `networks:` key.** n8n's `task-runners` omitted
|
||
it, landed on the compose default network, and could never resolve
|
||
`N8N_RUNNERS_TASK_BROKER_URI=http://n8n:5679`. That single missing line kept n8n down for
|
||
two months.
|
||
|
||
**Restart the full stack after a podman upgrade**, before rebooting, so failures are
|
||
attributable. Re-verify `keep-id` behaviour after a `podman-compose` version bump — it is
|
||
the mechanism postgres and n8n depend on:
|
||
|
||
```bash
|
||
podman run --rm --userns keep-id:uid=999,gid=999 -v /tmp/t:/t:rw \
|
||
docker.io/library/postgres:17 touch /t/probe # host file must be owned by uid 1000
|
||
```
|
||
|
||
---
|
||
|
||
## 11. Hardware access — what rootless genuinely cannot do
|
||
|
||
Some tools want raw hardware. Rootless cannot give it to them, and this is a real boundary,
|
||
not a configuration puzzle. Verified on this host 2026-08-11:
|
||
|
||
```bash
|
||
podman run --rm --device /dev/sda:/dev/sda alpine ls -l /dev/sda
|
||
# ls: /dev/sda: Permission denied <- gib is not in the `disk` group
|
||
```
|
||
|
||
`--cap-add SYS_RAWIO` / `SYS_ADMIN` **appear** to work — the container starts and the bit
|
||
shows in `CapEff` — but the capability is granted only inside the user namespace. It confers
|
||
nothing over host devices. Do not read a successful start as proof of access; test the actual
|
||
operation.
|
||
|
||
**Do not "fix" this** by adding `gib` to `disk`, running the container as root, or
|
||
reintroducing root podman. If a workload truly needs raw devices, run its collector on a host
|
||
that has them and keep only the UI here.
|
||
|
||
### The split-deployment pattern (use this for monitoring tools)
|
||
|
||
Most monitoring tools separate a **hub/web** component (no privileges) from an **agent/
|
||
collector** (needs hardware or host access). Put the hub on the VPS; run agents wherever the
|
||
thing being monitored lives.
|
||
|
||
| Service | On this VPS | Elsewhere |
|
||
| ------------ | ------------------------------------------------ | ------------------------------------- |
|
||
| **Beszel** | `beszel` hub + `beszel-agent` (monitors the VPS) | `beszel-agent` on each other host |
|
||
| **Scrutiny** | ❌ nothing — see below | full omnibus stays on the home server |
|
||
|
||
Beszel's hub lives here (`beszel.gibbyb.com`) and agents report in. The home server's agent
|
||
reaches this hub over WireGuard; add it in the UI as host `192.168.1.2` port `45876`. The
|
||
VPS's own agent is host `host.docker.internal` port `45876`. Both agents carry the hub's
|
||
public key, derived with `ssh-keygen -y -f Beszel/volumes/hub/id_ed25519`. **If the hub is
|
||
ever rebuilt its keypair changes and every agent's `KEY` must be updated.**
|
||
|
||
### ❌ Do not host Scrutiny here (decided 2026-08-11, twice over)
|
||
|
||
It was built here, evaluated, and deliberately removed. Two independent reasons:
|
||
|
||
1. **Scrutiny has no authentication at all.** Verified: `/web/dashboard` → 200 and
|
||
`/api/summary` → 200 with no credentials. Publishing it would expose disk models, serial
|
||
numbers and health to anyone who found the hostname. Beszel has real accounts; Scrutiny
|
||
does not. Do not put it behind Cloudflare without Cloudflare Access or NPM auth in front.
|
||
2. **There is no SMART data on this host.** `/dev/sda` (QEMU HARDDISK) and `/dev/sdb`
|
||
(Hetzner Volume) both report `SMART support is: Unavailable`, and rootless cannot reach
|
||
raw devices anyway (§11 above).
|
||
|
||
The collector must run where the disks are, so the web half gains nothing by moving. It runs
|
||
as the full omnibus on the home server, LAN/VPN-only. Revisit only if a second disk-bearing
|
||
host appears and one pane of glass becomes worth adding auth for.
|
||
|
||
### Authentik — migrated copy, NOT yet authoritative
|
||
|
||
`~/Server/Authentik` is a 1:1 copy of the home server's `~/Server/auth`, built 2026-08-11.
|
||
**The home instance is still authoritative and still serving `auth.gbrown.org`.** This copy
|
||
has its own database restored from a point-in-time dump, so the two diverge from that moment
|
||
— it is a cutover target, not a hot standby. Changes made in one do not appear in the other.
|
||
|
||
- DB: role + database `authentik` in the shared VPS `postgresql`, same `PG_PASS` as home,
|
||
so `.env` needed no edits. Restored 212 tables; verified 11 users / 37 apps / 15 flows /
|
||
37 providers / 11 groups on both sides.
|
||
- `AUTHENTIK_SECRET_KEY` in `.env` (mode 600) **must** match the key the database was
|
||
encrypted with. Never regenerate it on a migrated database.
|
||
- No Redis — authentik 2026.x is Postgres-backed. Home has none either.
|
||
- Version pinned via `AUTHENTIK_TAG` in `.env` (2026.2.2). Keep both sides on the same tag
|
||
through a cutover; authentik runs DB migrations on start and they are not reversible.
|
||
|
||
**Cutover, when you want it:** create an explicit `auth.gbrown.org` record pointing at the
|
||
VPS (an explicit record overrides the `*.gbrown.org` wildcard), add an NPM proxy host to
|
||
`http://authentik-server:9000`, and re-dump/restore the database immediately beforehand so
|
||
nothing is lost in the gap. Roll back by deleting the DNS record.
|
||
|
||
### Adapting a Docker/root compose file to rootless
|
||
|
||
Checklist when porting from the home server (which runs Docker as root):
|
||
|
||
| Docker/root | Rootless podman here |
|
||
| -------------------------------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
|
||
| `/var/run/docker.sock` | `/run/user/1000/podman/podman.sock` (rootless containers only), **no `:Z`**, plus `security_opt: ['label:disable']` |
|
||
| `privileged: true`, `devices:`, `cap_add: SYS_RAWIO/SYS_ADMIN` | Not available. Split the workload — see above. |
|
||
| `network_mode: host` | Works. Ports ≥80 bind fine; the agent is reachable from bridge containers via `extra_hosts: host.docker.internal:host-gateway` (resolves to `169.254.1.2`). |
|
||
| bind mounts | add `:Z` (§6) |
|
||
| `/etc/localtime:/etc/localtime:ro` | drop it; set `TZ=America/New_York` instead (§8) |
|
||
| host-specific env (e.g. `EXTRA_FILESYSTEMS=/boot`) | re-check against _this_ host — the VPS has no separate `/boot` partition |
|
||
|
||
Verify a ported service is reachable the way NPM will reach it, before wiring a proxy host:
|
||
|
||
```bash
|
||
podman run --rm --network nginx-bridge alpine \
|
||
sh -c 'apk add -q curl && curl -s -o /dev/null -w "%{http_code}\n" http://<container>:<port>/'
|
||
```
|
||
|
||
---
|
||
|
||
## 12. Quick reference
|
||
|
||
```bash
|
||
# state
|
||
podman ps
|
||
systemctl --user list-units 'podman-*'
|
||
~/Server/verify-rootless.sh
|
||
|
||
# lifecycle (NEVER sudo)
|
||
systemctl --user restart podman-<name>.service
|
||
systemctl --user stop podman-<name>.service
|
||
podman logs <container>
|
||
|
||
# after editing a compose file
|
||
systemctl --user restart podman-<name>.service
|
||
|
||
# after editing a unit file
|
||
systemctl --user daemon-reload && systemctl --user restart podman-<name>.service
|
||
|
||
# SELinux
|
||
getenforce
|
||
podman inspect <name> --format '{{.ProcessLabel}}'
|
||
sudo ausearch -m avc -ts recent # denials
|
||
|
||
# firewall
|
||
sudo firewall-cmd --zone=public --list-all
|
||
sudo firewall-cmd --zone=trusted --list-all
|
||
|
||
# nginx / NPM
|
||
podman exec nginx-proxy-manager nginx -t
|
||
podman exec nginx-proxy-manager nginx -s reload
|
||
```
|
||
|
||
**Temporary root for an agent** (revoke when done):
|
||
|
||
```bash
|
||
echo 'gib ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/99-claude-temp >/dev/null
|
||
sudo chmod 0440 /etc/sudoers.d/99-claude-temp
|
||
# revoke:
|
||
sudo rm /etc/sudoers.d/99-claude-temp
|
||
```
|
||
|
||
---
|
||
|
||
## §13. Findings from the 2026-08-12 post-migration audit
|
||
|
||
### 13.1 Upstream Hetzner Cloud Firewall (not visible from this machine)
|
||
|
||
There is a **cloud firewall in front of the VM** that firewalld cannot see and that
|
||
`firewall-cmd` will happily contradict. Verified 2026-08-12:
|
||
|
||
| port | firewalld | nftables | service healthy | reachable from internet |
|
||
| ---------------------------- | --------- | -------- | --------------- | ----------------------- |
|
||
| 22 | allow | allow | yes | **yes** |
|
||
| 2222 (gitea SSH) | allow | allow | yes | **no** |
|
||
| 21115-21119 (rustdesk relay) | allow | allow | yes | **no** |
|
||
|
||
Confirmed from an unrelated external IP and independently by portchecker.io.
|
||
|
||
**Consequence:** `git clone ssh://git@git.gbrown.org:2222/...` works over WireGuard but
|
||
fails from any outside network, and RustDesk cannot relay for clients that are not on
|
||
the LAN or tunnel.
|
||
|
||
⚠️ **When a port is open in firewalld but still unreachable, check the Hetzner console
|
||
before debugging anything on this box.** Two hours were spent proving the local stack
|
||
was correct. See `archive/BROWSER_AUDIT_PROMPT_3.md` §A1.
|
||
|
||
**RESOLVED 2026-08-12** — the rules below were added and verified open from an external
|
||
IP and by portchecker.io. `ssh -p 2222 git@<VPS_PUBLIC_IP>` now completes a handshake and
|
||
returns `Permission denied (publickey)`, which is the correct key-only response.
|
||
|
||
⚠️ The console names this server **`Junior`** (id 110193938), while the OS hostname is
|
||
`LittleMan`. Same machine — do not go looking for a second server.
|
||
|
||
**The rules** (Hetzner Cloud Console → Firewalls → `Default_Firewall`, id 10059209).
|
||
All **INBOUND**; source `0.0.0.0/0` + `::/0`:
|
||
|
||
| protocol | ports | purpose |
|
||
| -------- | ----------- | ---------------------------------------------------------------- |
|
||
| TCP | 21115-21117 | RustDesk: NAT test (21115), ID/hole-punch (21116), relay (21117) |
|
||
| **UDP** | 21116 | RustDesk ID registration — without it clients never come online |
|
||
| TCP | 2222 | Gitea SSH (verified key-only: password auth is refused) |
|
||
|
||
Verified listeners: `hbbs` on 21115/21116/21118 tcp + 21116 udp, `hbbr` on 21117/21119,
|
||
`apimain` on 21114.
|
||
|
||
**Deliberately NOT opened:**
|
||
|
||
- `21114/tcp` — the RustDesk web API. Already served via Cloudflare on 443
|
||
(`rustdesk.gbrown.org`). Opening it publishes an admin API that currently sits behind
|
||
Cloudflare. **This is why the rule is 21115-21117 and not 21114-21119.**
|
||
- `21118-21119/tcp` — websocket variants, needed only by the browser client.
|
||
|
||
⚠️ **Add no OUTBOUND rules.** Hetzner's firewall is stateful and permits all egress while
|
||
zero outbound rules exist; adding even one flips outbound to default-deny and every
|
||
destination must then be enumerated.
|
||
|
||
⚠️ **Never add a broad inbound range.** Ports 81, 3001 and 9443 are bound to 0.0.0.0 on
|
||
this host (§13.2) and are safe only because this firewall permits nothing but 22/80/443
|
||
plus the rows above.
|
||
|
||
### 13.2 Publish admin ports to the WireGuard address, not 0.0.0.0
|
||
|
||
`ports: ['5432:5432']` bound all 14 databases to the public interface, with only
|
||
firewalld plus the Hetzner firewall in between. Fixed to `192.168.2.2:5432:5432`.
|
||
|
||
**RESOLVED 2026-08-12.** All now bound to the WireGuard address:
|
||
|
||
| container | port | binding |
|
||
| ------------------- | ----------- | ------------------ |
|
||
| postgresql | 5432 | `192.168.2.2:5432` |
|
||
| portainer | 9443 | `192.168.2.2:9443` |
|
||
| nginx-proxy-manager | **81 only** | `192.168.2.2:81` |
|
||
| uptime | 3001 | `192.168.2.2:3001` |
|
||
|
||
⚠️ **NPM's 80 and 443 deliberately stay on `0.0.0.0`** — they are the public front door,
|
||
and firewalld already restricts them to Cloudflare ranges. Only the admin port moved.
|
||
|
||
Verified after: public sites unaffected (git/vault/port 200), all admin UIs reachable over
|
||
WireGuard (npm:81 200, uptime:3001 302, portainer:9443 200).
|
||
|
||
**Keep this in proportion.** None of these are reachable from the internet: the Hetzner
|
||
cloud firewall (§13.1) permits only 22/80/443, and firewalld does not open them either.
|
||
All three also have their own login. So this is a defense-in-depth observation, not an
|
||
exposure — the binding is wider than it needs to be, behind two firewalls and an
|
||
application login.
|
||
|
||
It matters in exactly one scenario: **if the Hetzner firewall rules are ever broadened**,
|
||
these ports become internet-facing immediately and silently, with no further change on
|
||
this box. That is the reason to prefer `192.168.2.2:` bindings — it removes the dependency
|
||
on a rule set that lives in a web console someone else may edit.
|
||
|
||
⚠️ **When adding Hetzner firewall rules, open only the specific ports intended**
|
||
(2222 for gitea SSH, 21115-21119 for the rustdesk relay). Do not open a broad range —
|
||
that would expose all three of the above.
|
||
|
||
Postgres was the one genuine finding here and is fixed. The remaining three are a
|
||
one-line change each (`- "192.168.2.2:9443:9443"`), deliberately left alone: changing
|
||
NPM's 81 alters how the proxy itself is administered, and a mistake there locks you out
|
||
of the tool you would use to repair it.
|
||
|
||
**Rule for new services: never publish an admin port as `NNNN:NNNN`.** Either bind it to
|
||
`192.168.2.2` or do not publish it at all and let NPM reach it over nginx-bridge.
|
||
|
||
### 13.3 `systemctl --user enable` is a separate step from starting
|
||
|
||
Five stacks (convexmonorepo, lashaddict, spoon, stpeteit, usesend) were **running but
|
||
`disabled`** — they would have vanished on the next reboot, silently, because everything
|
||
looked healthy in `podman ps`. `podman-compose up` starting the containers does not
|
||
enable the unit.
|
||
|
||
**Always verify after adding a service:**
|
||
|
||
systemctl --user is-enabled podman-<name>.service # must print "enabled"
|
||
|
||
`verify-rootless.sh` checks this now. Note `podman-auto-update`, `podman-restart`,
|
||
`podman-clean-transient` and `podman-kube@` are podman's own units and are correctly
|
||
disabled — do not enable them.
|
||
|
||
### 13.4 Postgres image must stay on a -trixie base
|
||
|
||
See the comment block in PostgreSQL/compose.yml. Switching to a bookworm-based image
|
||
changes glibc from 2.41 to 2.36, which changes text collation and can make existing
|
||
indexes return **silently wrong results**. Any base change requires
|
||
`REINDEX DATABASE` + `ALTER DATABASE ... REFRESH COLLATION VERSION` on all 14 databases.
|
||
|
||
### 13.5 The backup exits non-zero for benign reasons
|
||
|
||
Gitea runs without `userns_mode`, so `sessions/`, `issues.bleve` and `Gitea/ssh` are
|
||
owned by subuid 525288 and unreadable to gib. rsync therefore exited 23 on every run.
|
||
Now excluded — none are needed to restore. **The gitea SSH host keys are NOT among them**
|
||
(they are gib-owned in `Gitea/volume/ssh` and are backed up); losing those would give
|
||
every client a host-key mismatch.
|
||
|
||
### 13.6 Cloudflare Load Balancers — DELETED 2026-08-13 (were: two unproxied LBs shadowing the tunnel)
|
||
|
||
> **RESOLVED 2026-08-13: both load balancers were deleted.** The proxied `*.gbrown.org`
|
||
> and `gbrown.org` CNAMEs into the Cloudflare tunnel now carry all gbrown.org traffic,
|
||
> which was the "delete" option from §4.3 of `archive/SESSION_2026-08-12.md`. The pools (home-pool,
|
||
> vps-pool) still exist at the account level and are re-attachable if LBs are ever
|
||
> recreated. Full incident report: `archive/SESSION_2026-08-13.md`.
|
||
>
|
||
> What forced the decision: the LBs had been flipped to **proxied**, and home-pool's
|
||
> endpoint carried a **`Host: gbrown.org` header override**. A proxied LB applies that
|
||
> override to the SNI and Host of _every_ request — first causing 525s on the whole
|
||
> namespace (no `gbrown.org` server block existed on home NPM), then, once an apex host
|
||
> was added, routing every subdomain to the apex site. Unproxied LBs never applied it,
|
||
> which is why the override sat harmless for months. See §13.13 for the apex host that
|
||
> is now load-bearing.
|
||
|
||
Historical analysis from 2026-08-12, kept because the shadowing/diagnostic lessons stand:
|
||
|
||
> **Correction:** an earlier revision of this section blamed a "DNS-only wildcard record".
|
||
> That was wrong. There is no such record, and no `portainer.gbrown.org` record either —
|
||
> a browser audit enumerated all 53 gbrown.org records and found neither. The cause is
|
||
> **two Cloudflare Load Balancers**, which shadow DNS records of the same name.
|
||
|
||
Measured 2026-08-12 against three public resolvers and Cloudflare's own authoritative
|
||
nameservers (`junade.ns.cloudflare.com`, `megan.ns.cloudflare.com`):
|
||
|
||
portainer.gbrown.org. 30 IN A <HOME_PUBLIC_IP>
|
||
definitelynotarealhost.gbrown.org. -> A <HOME_PUBLIC_IP>
|
||
|
||
A randomly invented name resolves to the same address. The mechanism:
|
||
|
||
| LB (zone gbrown.org) | enabled | **proxied** | TTL | pools | fallback |
|
||
| -------------------- | ------- | ----------- | --- | ------------------- | -------- |
|
||
| `*.gbrown.org` | true | **false** | 30 | home-pool, vps-pool | vps-pool |
|
||
| `gbrown.org` | true | **false** | 30 | home-pool, vps-pool | vps-pool |
|
||
|
||
| pool | origin | monitor |
|
||
| --------- | -------------- | ----------------- |
|
||
| home-pool | <HOME_PUBLIC_IP> | **none attached** |
|
||
| vps-pool | <VPS_PUBLIC_IP> | **none attached** |
|
||
|
||
**Three consequences:**
|
||
|
||
1. A load balancer takes precedence over a same-name DNS record, so the _proxied_
|
||
`*.gbrown.org` and `gbrown.org` CNAMEs to `cfargotunnel.com` are **shadowed and never
|
||
take effect** — the tunnel carries no gbrown.org traffic today.
|
||
2. Because the LBs are `proxied: false`, they answer with the pool origin address
|
||
directly, publishing the home IP for the whole namespace including the apex and names
|
||
that do not exist. Cloudflare's own DNS page flags _"Your origin IP address is
|
||
partially exposed."_
|
||
3. With **no monitor attached to either pool**, both are permanently "healthy", steering
|
||
always lands on the first default pool (home-pool), and **failover to the VPS can
|
||
never occur.** This is paid-for load balancing that structurally cannot fail over.
|
||
|
||
Explicit hostnames (`port`, `git`, `auth`, `vault`, …) have their own proxied A records
|
||
at <VPS_PUBLIC_IP> and win over the wildcard LB, so they are unaffected.
|
||
|
||
⚠️ **Diagnostic rule:** before concluding a subdomain "has a stale record", query a name
|
||
that cannot exist (`dig +short thisdoesnotexist.gbrown.org`). If it resolves, you are
|
||
seeing catch-all behaviour — a wildcard _or an LB_ — not a per-host record.
|
||
|
||
**Before changing the LBs, read §13.8** — the client-IP consequence is the part that
|
||
actually bites.
|
||
|
||
### 13.7 Gitea container registry retention
|
||
|
||
One cleanup rule (Site Administration -> Packages -> Cleanup Rules): container
|
||
packages keep the newest 10 versions, drop versions older than 7 days, and always
|
||
keep tags matching `latest|main|prod`. Gitea applies it with its built-in daily
|
||
package cleanup cron. On 2026-10-01 the registry held 136 versions across 10
|
||
packages, 9.8 GB of blobs (down from 19 GB before the rule).
|
||
|
||
Checking without a browser session:
|
||
|
||
sqlite3 -readonly ~/Server/Gitea/volume/gitea/gitea.db \
|
||
"SELECT count(*) FROM package_version; SELECT * FROM package_cleanup_rule;"
|
||
|
||
### 13.8 Home's NPM real_ip — FIXED 2026-08-12 (was: no config at all)
|
||
|
||
Verified 2026-08-12 on the home server: there are **no custom nginx snippets** under its
|
||
NPM data directory, i.e. no `set_real_ip_from` / `real_ip_header`. The VPS _does_ have
|
||
this (`Nginx_Proxy_Manager/volumes/data/nginx/custom/server_proxy.conf`, §7); home does
|
||
not.
|
||
|
||
**Why this matters.** The home cloudflared tunnel is active and enabled, and its ingress
|
||
already covers the whole namespace:
|
||
|
||
tunnel: e4f0d4d8-5e57-4bee-ab7b-56ef1ed0b7da (/etc/cloudflared/config.yml)
|
||
ingress:
|
||
- hostname: gbrown.org -> https://localhost:443
|
||
- hostname: "*.gbrown.org" -> https://localhost:443
|
||
- hostname: gibbyb.com / "*.gibbyb.com" -> https://localhost:443
|
||
- hostname: madelinemcguigan.com / "*." -> https://localhost:443
|
||
- hostname: completeuphoria.club -> https://localhost:443
|
||
- service: http_status:404 (catch-all)
|
||
|
||
So **deleting the LBs would NOT dark the home namespace** — the tunnel catch-all already
|
||
routes it. That risk, raised in the browser audit, is disproved.
|
||
|
||
**The real consequence is the client IP.** Today the LBs hand out home's raw address, so
|
||
home's NPM sees the _actual client IP_ and IP-based access lists work. Change that and:
|
||
|
||
| route | what home's NPM sees as the client |
|
||
| ------------------------------------- | ------------------------------------------------- |
|
||
| today: unproxied LB -> home IP direct | the real client IP |
|
||
| delete LBs -> tunnel | **127.0.0.1** (cloudflared dials `localhost:443`) |
|
||
| proxy the LBs | **a Cloudflare edge IP** |
|
||
|
||
Both alternatives break every IP-based access list on the home NPM, because without
|
||
`real_ip` nginx has no way to recover the true client address.
|
||
|
||
**This is also why `allow 127.0.0.1` once existed in the "private" access list, and why
|
||
removing it was correct.** With it present, _anyone on the internet_ arriving through the
|
||
tunnel would have matched a rule meant to mean "LAN only". Removing it closed that hole;
|
||
it also means tunnel-routed traffic now fails closed.
|
||
|
||
⚠️ **Prerequisite for either LB change: configure real_ip on the home NPM first**, the
|
||
same way the VPS does. For tunnel traffic that means trusting the loopback:
|
||
|
||
set_real_ip_from 127.0.0.1;
|
||
real_ip_header CF-Connecting-IP;
|
||
|
||
For a proxied LB it means trusting Cloudflare's published ranges instead. Do this
|
||
_before_ touching the load balancers, not after — otherwise the KVMs and anything else
|
||
behind an access list go dark the moment DNS changes.
|
||
|
||
**Safe to do independently, right now:** attach the existing `https /health` monitor to
|
||
both pools. It only adds health checking and moves no traffic while home-pool is up.
|
||
|
||
#### 13.8b The fix, as applied (home server, 2026-08-12)
|
||
|
||
⚠️ Home runs **Docker**, not podman. `docker exec nginx-proxy-manager ...`, not
|
||
`sudo podman`. NPM data volume: `/home/gib/Server/nginx/volumes/data` → `/data`.
|
||
|
||
Two files, because the directive must be split across contexts:
|
||
|
||
`/data/nginx/custom/http_top.conf` (http context — `set_real_ip_from` is additive):
|
||
|
||
set_real_ip_from 127.0.0.1;
|
||
set_real_ip_from ::1;
|
||
|
||
`/data/nginx/custom/server_proxy.conf` (server context — included by all 31 proxy hosts):
|
||
|
||
real_ip_header CF-Connecting-IP;
|
||
|
||
⚠️ **`real_ip_header` must NOT go in `http_top.conf`.** NPM's `nginx.conf:81` already
|
||
declares `real_ip_header X-Real-IP;` in the http context, and `http_top.conf` is included
|
||
into that same context — a second declaration is a duplicate directive and **nginx will
|
||
refuse to start**. The server block legally overrides it. The VPS is arranged the same
|
||
way for the same reason.
|
||
|
||
**Verified after reload** (against `scrutiny` and `ncmaster`, both access-listed and
|
||
enabled):
|
||
|
||
| request from loopback | result |
|
||
| -------------------------------- | -------------------------------------------- |
|
||
| no header | **403** — loopback is not in the allow list |
|
||
| `CF-Connecting-IP: 192.168.1.50` | **302** — LAN range allowed, real_ip working |
|
||
| `CF-Connecting-IP: 8.8.8.8` | **403** — public IP still denied |
|
||
|
||
Access logs now record the forwarded client instead of `127.0.0.1`. All home sites
|
||
healthy afterwards (jellyfin/sonarr/radarr 302, immich 200, obsidian 401), NPM 0 restarts.
|
||
|
||
Useful facts discovered while doing this:
|
||
|
||
- NPM already trusts Cloudflare's ranges (`104.16.0.0/13`, `172.64.0.0/13`,
|
||
`103.21.244.0/22`) via `conf.d/include/ip_ranges.conf` — 265 entries. Only loopback was
|
||
missing, and the header was wrong. So **proxying the LBs would also now resolve
|
||
correctly**, not just the tunnel path.
|
||
- Home's "Private" access list is `192.168.1.0/24`, `192.168.2.0/24`, `192.168.112.1/24`
|
||
— confirmed **no `127.0.0.1`**.
|
||
- `nano`/`macnano` proxy hosts exist in home's DB but are **disabled** (no generated
|
||
conf); they are served by the VPS NPM (hosts 42/43) forwarding to `192.168.1.10/.11`
|
||
across WireGuard.
|
||
|
||
### 13.9 Two ways hosts are protected — do NOT "fix" one into the other
|
||
|
||
**Correction.** An earlier revision of this section claimed `prowlarr.gbrown.org` and
|
||
`comfyui.gbrown.org` were "not access-controlled despite the UI saying they are". **That
|
||
was wrong, and acting on it would have broken working SSO.** Both are protected — by
|
||
**authentik forward auth**, not by an IP access list.
|
||
|
||
Evidence:
|
||
|
||
$ curl -sk -D - https://prowlarr.gbrown.org
|
||
HTTP/2 302
|
||
location: https://prowlarr.gbrown.org/outpost.goauthentik.io/start?rd=/
|
||
|
||
id=29 prowlarr -> 29 auth_request/authentik lines, 0 allow/deny
|
||
id=48 comfyui -> 21 auth_request/authentik lines, 0 allow/deny
|
||
id=116 scrutiny -> 0 authentik lines, 4 allow/deny
|
||
id=47 ncmaster -> 0 authentik lines, 4 allow/deny
|
||
|
||
The generated config carries:
|
||
|
||
auth_request /outpost.goauthentik.io/auth/nginx;
|
||
error_page 401 = @goauthentik_proxy_signin;
|
||
|
||
**This environment deliberately uses two different protection mechanisms, per host:**
|
||
|
||
| mechanism | how it looks | example hosts |
|
||
| -------------------------------- | ------------------------------------------------------------------------------------------ | --------------------------------- |
|
||
| **authentik forward auth (SSO)** | `auth_request` lines; unauthenticated request → **302** to `/outpost.goauthentik.io/start` | prowlarr, comfyui |
|
||
| **IP access list** | `allow`/`deny` lines; off-LAN request → **403** | scrutiny, ncmaster, nano, macnano |
|
||
|
||
⚠️ **A 302 to `outpost.goauthentik.io` is PROTECTION, not exposure.** Do not read it as
|
||
"the service was served". Follow the `location` header before concluding anything.
|
||
|
||
⚠️ **Public reachability here is intentional.** Services are meant to be usable from
|
||
outside the LAN; SSO is what secures them, including services that have no login of their
|
||
own. Do **not** attach IP access lists to SSO-protected hosts, and do not treat a
|
||
publicly-resolving hostname as a defect. An `access_list_id` set in the NPM database with
|
||
no `allow`/`deny` in the generated config is **not** evidence of a hole on an SSO host.
|
||
|
||
Correct way to check what protects a host:
|
||
|
||
id=<n>
|
||
docker exec nginx-proxy-manager grep -cE '^[[:space:]]*(allow|deny) ' /data/nginx/proxy_host/$id.conf
|
||
docker exec nginx-proxy-manager grep -c 'auth_request' /data/nginx/proxy_host/$id.conf
|
||
|
||
Zero for **both** would be a genuine finding. Non-zero for either is a protected host.
|
||
|
||
### 13.10 The LB health monitor cannot validate either origin — it sends no SNI
|
||
|
||
> **Moot for gbrown.org since 2026-08-13** — the LBs were deleted (§13.6) and the tunnel
|
||
> needs no health monitor. Kept because the no-SNI failure mode applies to any future
|
||
> monitor, and the pools still exist at the account level.
|
||
|
||
**Do not attach a monitor to a pool expecting it to be a safe, traffic-neutral change.**
|
||
An earlier revision of this guide claimed attaching one "moves no traffic while home-pool
|
||
is up". **That is wrong** and was disproved in practice on 2026-08-12.
|
||
|
||
What happened: monitor `Gib's Monitor` (HTTPS GET `/health`, port 443, expect 200,
|
||
`Host: health.gbrown.org`) was attached to both pools. After one check cycle **both** went
|
||
unhealthy:
|
||
|
||
home-pool healthy: false <HOME_PUBLIC_IP> "TLS unrecognized name error"
|
||
vps-pool healthy: false <VPS_PUBLIC_IP> "TLS unrecognized name error"
|
||
|
||
With every pool unhealthy the LB fell through to its **fallback pool (vps-pool)** and DNS
|
||
moved immediately — the apex, `sonarr`, and every unmatched name flipped to
|
||
<VPS_PUBLIC_IP> on a 30s TTL. It was rolled back by detaching the monitor, and DNS
|
||
returned to <HOME_PUBLIC_IP>.
|
||
|
||
**Root cause, proved from the shell — the monitor is not sending SNI:**
|
||
|
||
# WITH SNI -- both origins are fine
|
||
$ openssl s_client -connect 127.0.0.1:443 -servername health.gbrown.org
|
||
VPS: subject=CN=*.completeuphoria.club Verify return code: 0 (ok)
|
||
home: subject=CN=*.gbrown.org
|
||
|
||
# WITHOUT SNI -- both origins reject, on both machines
|
||
$ openssl s_client -connect 127.0.0.1:443
|
||
tlsv1 unrecognized name ... SSL alert number 112
|
||
no peer certificate available
|
||
|
||
So it is **not** "the `/health` path is missing" and **not** a certificate problem —
|
||
`health.gbrown.org` returns 200 and both certs verify. nginx has no default server with a
|
||
cert for a no-SNI handshake, so it aborts before HTTP is ever spoken. (Alert 112 again —
|
||
the same alert family as the ECH incident in §7.)
|
||
|
||
**Options, in order of preference:**
|
||
|
||
1. **Use an HTTP monitor on port 80** — no TLS, so no SNI to get wrong. Set expected codes
|
||
to include `301`, since NPM's Force-SSL will redirect.
|
||
2. Give both origins a default server with a valid certificate so no-SNI handshakes
|
||
succeed.
|
||
3. `allow_insecure` will **not** help — the handshake fails before certificate validation.
|
||
|
||
⚠️ **Whatever you try, attach it to `vps-pool` ALONE first** and confirm it reports
|
||
healthy before home-pool is ever added. A monitor that fails on every pool does not
|
||
disable failover — it triggers a silent flip to the fallback pool.
|
||
|
||
⚠️ **Corollary:** the earlier note that a "flapping monitor" was observed spontaneously
|
||
was wrong too. That DNS movement was this attach/detach, not an independent fault.
|
||
|
||
### 13.11 Home's NPM database — collation fixed 2026-08-12
|
||
|
||
Home's `nginx-proxy-manager-db` ran `postgres:17` (glibc 2.41) on a cluster created under
|
||
glibc 2.36 — the mirror image of §13.4, and the same silent-wrong-results risk, on the
|
||
database that stores every proxy host and access list.
|
||
|
||
Fixed properly rather than by silencing the warning:
|
||
|
||
docker exec nginx-proxy-manager-db pg_dump -U npm -d npm > ~/npm-db-backup-<date>.sql
|
||
docker exec nginx-proxy-manager-db psql -U npm -d npm -c "REINDEX DATABASE npm;"
|
||
docker exec nginx-proxy-manager-db psql -U npm -d npm -c "ALTER DATABASE npm REFRESH COLLATION VERSION;"
|
||
# repeated for the `postgres` database
|
||
|
||
`NOTICE: changing version from 2.36 to 2.41`. Data intact (117 proxy_host rows), sites
|
||
serving afterwards. A 1.3 MB dump was taken first and is on the home server.
|
||
|
||
⚠️ **Home runs Docker.** `docker exec`, not `podman`.
|
||
|
||
### 13.12 Passwordless sudo was removed 2026-08-12 — both servers
|
||
|
||
`/etc/sudoers.d/99-claude-temp` (`gib ALL=(ALL) NOPASSWD: ALL`) is **gone from the VPS and
|
||
the home server**. `gib` retains normal sudo via `%wheel ALL=(ALL) ALL`, now
|
||
password-prompted.
|
||
|
||
**This does not affect running anything here.** Every container is rootless and every unit
|
||
is a `systemctl --user` unit, so day-to-day operation needs no sudo at all. Only host-level
|
||
work does — firewalld, SELinux relabeling, reading container-owned files under
|
||
`Gitea/volume`, and `docker` on the home server.
|
||
|
||
⚠️ **Non-interactive sudo will now fail**, including over `ssh` without a TTY. An agent
|
||
running `sudo ...` in a pipeline gets `sudo: a password is required` and the surrounding
|
||
command may still report success — check for that string rather than trusting an exit code.
|
||
If host-level access is genuinely needed, ask the user rather than re-creating the drop-in.
|
||
|
||
### 13.13 The `gbrown.org` apex proxy host on home NPM is LOAD-BEARING for the tunnel
|
||
|
||
Home's cloudflared (`/etc/cloudflared/config.yml`) dials NPM with
|
||
`originServerName: gbrown.org` — that is the SNI for **every** request through the
|
||
tunnel, regardless of which site the visitor asked for (NPM then routes by the Host
|
||
header, which the tunnel preserves). Home NPM has no default TLS server, so a handshake
|
||
with an unrecognized SNI is aborted (alert 112 — same failure family as §13.10).
|
||
|
||
Consequence: **the `gbrown.org` proxy host in home NPM (currently → Home Assistant,
|
||
added 2026-08-13) must exist for any tunnel-served site to work.** Deleting or renaming
|
||
it 525s the entire gbrown.org namespace, plus gibbyb.com / madelinemcguigan.com /
|
||
completeuphoria.club traffic riding the same tunnel. If the apex host must ever go away,
|
||
first change `originServerName` to another hostname NPM serves (e.g.
|
||
`nginx.gbrown.org`) and `sudo systemctl restart cloudflared`.
|
||
|
||
### 13.14 VPS Actions runner — `cirunner` user (added 2026-08-13)
|
||
|
||
CI redundancy: the three dockerized runners + one host runner all live on the **home
|
||
server**, so a home outage used to stop CI (and with it, deploys of VPS apps). A fourth
|
||
runner now runs on the VPS under a **dedicated `cirunner` user** (uid 1001, subuids
|
||
`589824:65536`, linger enabled) — NOT under gib, so a CI escape lands in an empty
|
||
account with no access to the production containers.
|
||
|
||
Layout (all under `/home/cirunner`, systemd user unit `podman-runner.service`):
|
||
|
||
| piece | value |
|
||
| --------------------------- | ------------------------------------------------------------------------------------------------------------ |
|
||
| runner name / labels | `cirunner-vps` — `ubuntu-latest`, `ubuntu-24.04`, `ubuntu-22.04`, **`vps`** |
|
||
| compose + act_runner config | `~cirunner/runner/{compose.yml,config.yaml}` |
|
||
| container storage | `/home/gib/Media/cirunner/storage` (196 GB volume; traverse-only ACL `u:cirunner:x` on `/home/gib{,/Media}`) |
|
||
| job execution | sibling containers via cirunner's podman API socket |
|
||
|
||
Because it shares the `ubuntu-latest` label with the home runners, whichever runner
|
||
polls first takes the job — home down ⇒ the VPS keeps CI moving. `runs-on: vps`
|
||
forces a job here.
|
||
|
||
**Two traps, both hit and fixed during setup — do not regress them:**
|
||
|
||
1. **Socket path must be identical on host and in the runner container**
|
||
(`/run/user/1001/podman/podman.sock`, with `DOCKER_HOST` pointing at it). act_runner
|
||
passes its socket path to the daemon when mounting it into job containers; a
|
||
container-only path like `/var/run/docker.sock` fails with
|
||
`mkdir /var/run/docker.sock: permission denied`.
|
||
2. **SELinux blocks confined job containers from the socket** even when mounted —
|
||
`config.yaml` sets `container.options: "--security-opt label=disable"` (the §13.2/
|
||
watchtower trade, contained by the cirunner userns). Symptom if lost:
|
||
`ls: /var/run/docker.sock: Permission denied` inside jobs.
|
||
|
||
Smoke-test repo: `gib/ci-smoke` (private) — a `runs-on: vps` workflow that hard-checks
|
||
the docker socket inside a job and the watchtower hook at `192.168.2.2:8080`. Push any
|
||
commit to re-verify the runner end to end. Registration used the reusable instance
|
||
token (`gitea actions generate-runner-token`).
|
||
|
||
### 13.15 Watchtower RETIRED 2026-08-17 — it took gitea down for three days
|
||
|
||
**Do not reinstate watchtower on this host.** Updates are now handled by
|
||
`~/Server/update-containers.sh`, run nightly at 00:00 by `podman-update.timer`.
|
||
|
||
#### What happened
|
||
|
||
| time (EDT) | event |
|
||
| --------------- | ---------------------------------------------------------------------------------------------------------------- |
|
||
| Aug 14 04:30:03 | watchtower found a new `gitea:latest`, stopped the container, started a replacement, logged `failed=0 updated=1` |
|
||
| Aug 14 04:30:22 | Uptime Kuma monitor 17: **502**, gitea DOWN |
|
||
| Aug 17 12:39 | manual `systemctl --user restart podman-gitea.service` — UP |
|
||
|
||
Gitea logged nothing at all between those points. Watchtower reported success, so
|
||
nothing escalated.
|
||
|
||
#### Root cause
|
||
|
||
podman-compose puts every project in a **pod** — 39 of 41 containers here. Watchtower
|
||
does not update through compose; it builds a replacement container from the image plus
|
||
the old container's config, and **the replacement lands outside the pod**, losing
|
||
whatever the pod provided.
|
||
|
||
Proof: after the Aug 13 run, `portainer` was sitting podless, as was `agentchat`. Both
|
||
were restored by restarting their units. Check for this with:
|
||
|
||
for c in $(podman ps --format '{{.Names}}'); do
|
||
[ -z "$(podman inspect $c --format '{{.Pod}}')" ] && echo "PODLESS: $c"
|
||
done
|
||
|
||
Honest limit: `portainer` ran podless without visible breakage, so leaving the pod is
|
||
not automatically fatal, and the exact reason gitea's replacement returned 502 could not
|
||
be reconstructed — that container and its logs were removed by watchtower itself. What is
|
||
certain is the trigger, the timing, and that the replacement was not built by compose.
|
||
|
||
#### The replacement
|
||
|
||
`update-containers.sh` never touches containers. It pulls the image, then restarts the
|
||
**systemd unit**, so podman-compose rebuilds the project exactly as it does at boot.
|
||
|
||
./update-containers.sh # all services except SKIP (what the timer runs)
|
||
./update-containers.sh gitea # one service — use this from CI
|
||
./update-containers.sh --dry-run # report only
|
||
./update-containers.sh --list # services and their resolved images
|
||
|
||
Config lives in the two arrays at the top. `SKIP=(postgresql authentik watchtower)`.
|
||
Log: `~/Server/logs/update-containers.log`.
|
||
|
||
Three non-obvious things it has to handle — do not "simplify" these away:
|
||
|
||
1. **Compare the running container to the image on disk, not before-vs-pull.** A
|
||
`--dry-run` pulls the image, so a later real run would see before == after, decide
|
||
nothing changed, and never restart — leaving the service on the old image with the
|
||
new one on disk, permanently invisible. Anchoring on the container makes it
|
||
idempotent and self-healing.
|
||
2. **`podman inspect .Image`, never `podman ps --format {{.ImageID}}`.** The first is
|
||
the full 64-char ID matching `image inspect .Id`; the second is a 12-char short ID.
|
||
Comparing them is always unequal, and every service looks like it needs updating.
|
||
3. **Never restart an inactive unit.** `systemctl restart` _starts_ a stopped unit, so
|
||
without a guard a new n8n image would resurrect n8n months after it was deliberately
|
||
shut down. The script skips any unit that is not already `active`.
|
||
|
||
Also: image tags like `${BACKEND_TAG:-latest}` are expanded from each service's `.env`
|
||
in a **subshell**, so those files (database passwords, API keys) never leak into the
|
||
script's environment.
|
||
|
||
#### ⚠️ Push-to-deploy needs rewiring
|
||
|
||
CI used to POST to watchtower's API at `192.168.2.2:8080/v1/update`. **That endpoint no
|
||
longer exists.** The equivalent is:
|
||
|
||
ssh [email protected] '~/Server/update-containers.sh <service>'
|
||
|
||
which needs an SSH key for the runner. Until that is done, pushing an image updates the
|
||
registry but does not deploy it — the nightly timer will pick it up at midnight.
|
||
|
||
### 13.16 Instant deploys — SSH forced command (replaces watchtower's HTTP API)
|
||
|
||
Watchtower's `192.168.2.2:8080/v1/update` is gone (§13.15). CI now triggers a deploy by
|
||
SSHing to `gib@192.168.2.2` with a key that can do exactly one thing.
|
||
|
||
**Why SSH and not a small HTTP daemon:** the alternative was writing a network service
|
||
that executes `systemctl` based on request bodies. sshd with a forced command achieves
|
||
the same thing using software that already exists, is already exposed, and has decades of
|
||
review behind it. No new listening port either.
|
||
|
||
**Why a bridge is needed at all:** the VPS runner is user `cirunner` (uid 1001) and the
|
||
other runners are on the home server. systemd user managers are per-user, so neither can
|
||
restart `gib`'s units. This crosses that boundary — and only that boundary.
|
||
|
||
Pieces:
|
||
|
||
| file | role |
|
||
| ----------------------------- | ------------------------------------------------------- |
|
||
| `~/Server/ci-deploy.sh` | the forced command; validates and dispatches |
|
||
| `~/Server/ci/ci_deploy_key` | private key — goes in the Gitea secret `DEPLOY_SSH_KEY` |
|
||
| `~/.ssh/authorized_keys` | pins the key to the forced command |
|
||
| §15 of this file | the workflow snippet to paste |
|
||
| `~/Server/logs/ci-deploy.log` | every request, allowed or denied |
|
||
|
||
The authorized_keys entry:
|
||
|
||
command="/home/gib/Server/ci-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc ssh-ed25519 AAAA...
|
||
|
||
⚠️ **`$SSH_ORIGINAL_COMMAND` is attacker-controlled if the key leaks. Never eval it,
|
||
never word-split it, never interpolate it into a shell string.** `ci-deploy.sh` matches
|
||
it against `^[a-z0-9][a-z0-9-]{0,40}$`, confirms a matching unit is _already active_,
|
||
rejects the frozen set, and passes it as a single quoted argument.
|
||
|
||
Verified behaviour — re-run these after any change to the script or the key:
|
||
|
||
| request | result |
|
||
| -------------------------- | ------------------- |
|
||
| `adminer` | deploys |
|
||
| _(bare shell)_ | denied — malformed |
|
||
| `cat .../PostgreSQL/.env` | denied — malformed |
|
||
| `adminer; cat /etc/passwd` | denied — malformed |
|
||
| `adminer$(id)` | denied — malformed |
|
||
| `postgresql` | denied — frozen |
|
||
| `n8n` | denied — not active |
|
||
|
||
The "must already be active" check is doing double duty: it keeps the allowlist
|
||
self-maintaining as services come and go, and it means the key cannot start something
|
||
that was deliberately stopped.
|
||
|
||
**Only two workflows ever called watchtower** — `gib/agentchat` and `gib/ci-smoke`. The
|
||
app stacks (`spoon`, `lashaddict`, `convex-monorepo`, `stpeteit-payload`,
|
||
`convex-monorepo-payload`) push images and wait for the nightly timer. Add the step from
|
||
the §15 step to any of them that should deploy immediately.
|
||
|
||
⚠️ `~/.ssh/authorized_keys` was backed up to `authorized_keys.bak-<timestamp>` before the
|
||
key was appended. Four pre-existing keys were preserved — check `wc -l` is 5, not 1, if
|
||
you ever edit it.
|
||
|
||
#### 13.16b Verified end-to-end 2026-08-19
|
||
|
||
First real push through the new path, `gib/agentchat` run 129:
|
||
|
||
10:55:01 ci-deploy.log: deploy requested from 192.168.1.2: agentchat
|
||
10:55:02 update-containers.log: agentchat on 5d796ab4f297 -> new image 343d961d96ff
|
||
10:55:09 update-containers.log: agentchat: RESTARTED ok
|
||
10:55:09 update-containers.log: pruned: 5d796ab4f297...
|
||
10:55:09 ci-deploy.log: deploy finished: agentchat (exit 0)
|
||
|
||
Container came back **inside its pod** (`271d52b578ba`) on the new image, `chat.gbrown.org` 200. Roughly 8 seconds from trigger to live. `192.168.1.2` is the home server, so the
|
||
runner crossed the WireGuard tunnel to reach the VPS.
|
||
|
||
**Pushing from the VPS.** Only `~/.ssh/id_rsa.pub` is registered with Gitea as
|
||
"Gib's VPS (Junior)". The desktop and home keys are passphrase-protected and unusable
|
||
non-interactively, and the key registered as "Gib's Server" matches neither machine —
|
||
it predates a rebuild. To push from here:
|
||
|
||
GIT_SSH_COMMAND="ssh -p 2222 -i $HOME/.ssh/id_rsa" git clone ssh://[email protected]:2222/gib/<repo>.git
|
||
|
||
⚠️ `scp` to `desktopw.gib` hangs, though plain `ssh` works. Pipe instead:
|
||
`ssh desktopw.gib 'cat > /tmp/f' < localfile`.
|
||
|
||
⚠️ Do not build shell snippets for a remote host through nested SSH quoting — an earlier
|
||
attempt mangled `printf '%s\n'` into a literal newline and silently corrupted the
|
||
workflow. Write the file locally, pipe it over, and apply it with a script on the far end.
|
||
|
||
**Remaining repos still without a deploy trigger** (they push an image and wait for the
|
||
midnight timer): `spoon`, `lashaddict`, `convex-monorepo`, `convex-monorepo-payload`,
|
||
`stpeteit-payload`, and `ci-smoke` (which still references the dead watchtower endpoint).
|
||
Add the step from §15 below, changing the service-name argument.
|
||
|
||
---
|
||
|
||
### 13.17 LashAddict containers pin DNS to Cloudflare (2026-09-30)
|
||
|
||
lashaddict moved from `*.lashaddict.gbrown.org` to `lashaddictstudio.com`
|
||
(`api.` → backend:3210, `convex.` → backend:3211, `dashboard.` → 6791, apex → next:3000,
|
||
`www` redirect host; cert NPM #44). Hetzner's resolvers (185.12.64.1/.2) kept a
|
||
cached NXDOMAIN for the brand-new hostnames for hours, and the Convex backend fetches
|
||
its own auth discovery URL (`https://convex.lashaddictstudio.com/.well-known/...`), so
|
||
every session failed (`AuthProviderDiscoveryFailed`). All three services in
|
||
`LashAddict/compose.yml` therefore set `dns: [1.1.1.1, 1.0.0.1]`; container names still
|
||
resolve through aardvark-dns. Do not remove the pin. The same trap applies to any new
|
||
hostname a container must reach soon after it is created: `resolvectl flush-caches`
|
||
fixes the host, not the Hetzner upstream.
|
||
|
||
## 14. Restoring ~/Server from the home-server backup
|
||
|
||
_Folded in from RESTORE.md, 2026-09-22. Read it before you need it._
|
||
|
||
Written 2026-09-17. Read this _before_ you need it.
|
||
|
||
### Layout
|
||
|
||
[email protected]:/home/gib/Public/Server_Backup/Containers/lilman.gib/current/
|
||
|
||
**One directory, overwritten daily.** History does not live in dated folders — it lives in
|
||
ZFS. `Media/Server_Backup` is its own dataset and `zfs-auto-snapshot` keeps **14 dailies**,
|
||
taken by `cron.daily` between roughly 03:22 and 03:52. The backup timer runs at **01:00** so
|
||
every snapshot captures a finished backup, never a half-written one.
|
||
|
||
Older copies are reachable through the hidden snapshot directory (`snapdir=hidden`, so it
|
||
will not show in `ls`, but you can still `cd` into it):
|
||
|
||
ls ~/Public/Server_Backup/.zfs/snapshot/
|
||
ls ~/Public/Server_Backup/.zfs/snapshot/zfs-auto-snap_daily-2026-09-17-0722/Containers/lilman.gib/current/
|
||
|
||
ZFS snapshots are block-level, so a 1.7 GB database that changes a few pages costs a few
|
||
pages — not a new 1.7 GB copy. That is why this is a single mirror plus snapshots rather
|
||
than dated directories or `rsync --link-dest`, both of which copy whole files.
|
||
|
||
The `03-17-2026`, `08-12-2026` and `09-11-2026` directories are leftovers from the old
|
||
scheme. They predate the ownership fix below and are **not reliable restore sources**.
|
||
|
||
### What a backup contains
|
||
|
||
| Path | What it is |
|
||
| ---------------------------------------- | -------------------------------------------------------- |
|
||
| `Adminer/`, `Gitea/`, `PostgreSQL/`, ... | Raw bind-mount data for each service |
|
||
| `_backup_state/postgres/all.sql.gz` | `pg_dumpall` of the live cluster — **authoritative** |
|
||
| `_backup_state/sqlite/<original path>` | Online `.backup` of each SQLite DB — **authoritative** |
|
||
| `_backup_state/files/` | Files gib cannot read off disk, pulled via `podman exec` |
|
||
|
||
> **Use `_backup_state/` for databases, not the raw data dirs.** The raw dirs are copied
|
||
> while containers run, so they are crash-consistent at best. The dumps use the proper
|
||
> online-backup APIs and are what will actually restore cleanly.
|
||
|
||
### Ownership, and why `--fake-super` is everywhere
|
||
|
||
Most services run with `userns=private`, which maps the container's user back to `gib` —
|
||
their data is plain `gib`-owned on the host and needs nothing special. **Gitea is the
|
||
exception.** It runs _without_ `userns_mode` (s6 needs container-root, see AGENTS.md §3),
|
||
so its data is owned by subuid **525288:525290**, which `gib` cannot recreate on the home
|
||
server. The backup therefore stores the real mode/uid/gid in a `user.rsync.%stat` xattr
|
||
(`rsync --fake-super`), verified working on the home server's ZFS.
|
||
|
||
**Consequence: a plain `rsync`/`scp`/`cp` of the backup gives you correct file contents with
|
||
every file owned by whoever ran it, and Gitea will fail on its own data dir.** Every restore
|
||
command below passes `-M--fake-super` so the sending side decodes those xattrs, and runs
|
||
under `sudo` locally so the ownership can actually be applied.
|
||
|
||
(The old dated-directory backups were taken without this, so everything in them is uid 1000.
|
||
Restoring one means chown'ing Gitea's tree to 525288:525290 by hand afterwards.)
|
||
|
||
### Restore in place (same machine)
|
||
|
||
Stop everything first:
|
||
|
||
systemctl --user stop 'podman-*.service'
|
||
|
||
Pull it back. `sudo` has no SSH key of its own, so point it at gib's:
|
||
|
||
sudo rsync -a --numeric-ids -M--fake-super --delete \
|
||
-e "ssh -i /home/gib/.ssh/id_rsa -o UserKnownHostsFile=/home/gib/.ssh/known_hosts" \
|
||
[email protected]:/home/gib/Public/Server_Backup/Containers/lilman.gib/current/ \
|
||
/home/gib/Server/
|
||
|
||
To restore from an older day, swap the source for a snapshot path:
|
||
|
||
.../.zfs/snapshot/zfs-auto-snap_daily-2026-09-15-0751/Containers/lilman.gib/current/
|
||
|
||
Verify Gitea's ownership came back — this must say `525288`, not `1000`:
|
||
|
||
ls -ln /home/gib/Server/Gitea/volume/gitea/ | head
|
||
|
||
Then restore databases (below) and start services:
|
||
|
||
systemctl --user start 'podman-*.service'
|
||
|
||
### Restoring the databases
|
||
|
||
**Postgres** — start only the postgres container, then:
|
||
|
||
gunzip -c /home/gib/Server/_backup_state/postgres/all.sql.gz \
|
||
| podman exec -i postgresql psql -U npm -d postgres
|
||
|
||
`pg_dumpall` includes roles and every database (npm, authentik, usesend, ...), so this
|
||
restores the whole cluster in one shot.
|
||
|
||
**SQLite** — each dump sits at its original relative path. Stop the owning container, copy
|
||
the dump over the live file, restart. Gitea, which is the one needing a chown:
|
||
|
||
systemctl --user stop podman-gitea.service
|
||
cp /home/gib/Server/_backup_state/sqlite/Gitea/volume/gitea/gitea.db \
|
||
/home/gib/Server/Gitea/volume/gitea/gitea.db
|
||
sudo chown 525288:525290 /home/gib/Server/Gitea/volume/gitea/gitea.db
|
||
systemctl --user start podman-gitea.service
|
||
|
||
For the others (uptime-kuma, vaultwarden, beszel, portainer, ...) the copy is enough — no
|
||
chown, since they are gib-owned already.
|
||
|
||
**bridge.json** — `_backup_state/files/BillTracker--data--mcp--bridge.json` goes back to
|
||
`BillTracker/data/mcp/bridge.json`, owned `525288`, mode 0600. It holds the MCP bridge's
|
||
registered OAuth clients; without it those clients must re-register.
|
||
|
||
### Migrating to a new server
|
||
|
||
1. Install podman, `loginctl enable-linger gib`, and create `gib` with the **same uid (1000)**
|
||
and the **same `/etc/subuid` range**. If the subuid base differs, Gitea's restored files
|
||
land on the wrong owner and it will not start.
|
||
2. Restore as above, to the new box.
|
||
3. Restore the databases from `_backup_state/`.
|
||
4. Re-create the systemd units from each service's `podman-*.service` file — they are in the
|
||
backup, next to each `compose.yml`.
|
||
5. Re-point DNS, then re-issue certificates through Nginx Proxy Manager.
|
||
|
||
> **If the new server is ARM (Hetzner CAX):** the data restores fine, it is
|
||
> architecture-independent, but every image must be rebuilt or re-pulled for `arm64`.
|
||
> Checked 2026-09-17: all third-party images used here publish arm64 except `minio`, which
|
||
> needs verifying. The `git.gbrown.org/gib/*` images rebuild natively via CI.
|
||
|
||
### What is deliberately NOT backed up
|
||
|
||
- `sessions/`, `issues.bleve` — ephemeral Gitea logins and a rebuildable search index.
|
||
- `dump.rdb` — redis/valkey cache dumps, regenerated on boot.
|
||
- `Gitea/ssh` — a 25-byte shell env file. **The Gitea SSH _host keys_ ARE backed up**
|
||
(`Gitea/volume/ssh`); losing those would give every client a host-key mismatch.
|
||
- `node_modules`, build output, `*.log`, media files.
|
||
- The podman image store and Gitea package registry — they live on the Hetzner volume at
|
||
`~/Media`, not `~/Server`. All rebuildable or re-pullable.
|
||
|
||
### Checking it is actually working
|
||
|
||
systemctl --user list-timers server-backup.timer
|
||
tail -40 ~/Server/logs/backup.log
|
||
ssh [email protected] 'ls -la ~/Public/Server_Backup/Containers/lilman.gib/current/ | head'
|
||
|
||
A healthy run ends with `=== backup complete ===`. Things to watch for in the log:
|
||
|
||
- `ZFS history OK` — the run confirmed snapshots are still being taken. If this ever says
|
||
**WARNING**, versioning is broken and `current/` is just a mirror: a mistake made here
|
||
would propagate with nothing to roll back to. Fix `zfs-auto-snapshot` before anything else.
|
||
- `rsync exit 24` is logged as a NOTE, not a failure. It only means a file changed
|
||
mid-transfer, which is normal on a live system.
|
||
|
||
Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful
|
||
runs will ping it, so a silent failure shows up as a down monitor.
|
||
|
||
---
|
||
|
||
## 15. Instant deploys — the CI step
|
||
|
||
_Folded in from ci/DEPLOY_STEP.md, 2026-09-22._
|
||
|
||
Watchtower was removed 2026-08-17 (AGENTS.md §13.15), so
|
||
`http://192.168.2.2:8080/v1/update` no longer answers. Deploys still happen, just at
|
||
midnight via `podman-update.timer`. To get them instant again, do the two steps below.
|
||
|
||
---
|
||
|
||
### Step 1 — add the deploy key as a Gitea secret (once)
|
||
|
||
The private key is on the VPS at:
|
||
|
||
/home/gib/Server/ci/ci_deploy_key
|
||
|
||
In Gitea → **Site Administration → Secrets** (or per-repo → Settings → Secrets), add:
|
||
|
||
| name | value |
|
||
| ---------------- | --------------------------------------------------------------- |
|
||
| `DEPLOY_SSH_KEY` | the entire contents of that file, including the BEGIN/END lines |
|
||
|
||
An org- or admin-level secret means every repo gets it without repeating yourself.
|
||
|
||
**You can delete `secrets.WATCHTOWER_TOKEN` afterwards** — nothing consumes it now.
|
||
|
||
### Step 2 — swap the deploy step in each workflow
|
||
|
||
Replace this:
|
||
|
||
```yaml
|
||
- name: Trigger immediate deploy via Watchtower API
|
||
run: |
|
||
curl -fsS -X POST -H "Authorization: Bearer ${{ secrets.WATCHTOWER_TOKEN }}" \
|
||
http://192.168.2.2:8080/v1/update \
|
||
|| echo "::warning::watchtower trigger failed; nightly sweep will deploy instead"
|
||
```
|
||
|
||
with this — changing **`agentchat`** on the last line to the service being deployed:
|
||
|
||
```yaml
|
||
- name: Trigger immediate deploy
|
||
run: |
|
||
install -m600 /dev/null "$HOME/.deploy_key"
|
||
printf '%s\n' "${{ secrets.DEPLOY_SSH_KEY }}" > "$HOME/.deploy_key"
|
||
ssh -i "$HOME/.deploy_key" \
|
||
-o BatchMode=yes \
|
||
-o StrictHostKeyChecking=accept-new \
|
||
-o ConnectTimeout=15 \
|
||
[email protected] agentchat \
|
||
|| echo "::warning::deploy trigger failed; nightly sweep will deploy instead"
|
||
rm -f "$HOME/.deploy_key"
|
||
```
|
||
|
||
Kept deliberately from the old step:
|
||
|
||
- **`|| echo "::warning::"`** — a deploy hiccup warns instead of failing a build whose
|
||
image already pushed fine. The midnight timer is the backstop.
|
||
- **`192.168.2.2`** — the VPS's WireGuard address, so this works from the home runners
|
||
and the VPS runner alike, and never crosses the public internet.
|
||
|
||
### Repo → service name
|
||
|
||
The argument after `gib@192.168.2.2` is the **systemd service name**, not the repo name:
|
||
|
||
| repo | argument |
|
||
| ------------------ | ---------------- |
|
||
| `agentchat` | `agentchat` |
|
||
| `spoon` | `spoon` |
|
||
| `lashaddict` | `lashaddict` |
|
||
| `convex-monorepo` | `convexmonorepo` |
|
||
| `stpeteit-payload` | `stpeteit` |
|
||
|
||
Confirm any name with `~/Server/update-containers.sh --list`.
|
||
|
||
---
|
||
|
||
### What the key can and cannot do
|
||
|
||
It is pinned to a forced command in `~/.ssh/authorized_keys`:
|
||
|
||
command="/home/gib/Server/ci-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc ssh-ed25519 AAAA...
|
||
|
||
sshd runs `ci-deploy.sh` no matter what the client asks for. Verified by testing:
|
||
|
||
| attempt | result |
|
||
| ------------------------------------ | ---------------------------------------- |
|
||
| `ssh ... adminer` | deploys adminer |
|
||
| `ssh ...` (bare shell) | `denied: expected a single service name` |
|
||
| `ssh ... 'cat .../PostgreSQL/.env'` | `denied` |
|
||
| `ssh ... 'adminer; cat /etc/passwd'` | `denied` |
|
||
| `ssh ... 'adminer$(id)'` | `denied` |
|
||
| `ssh ... postgresql` | `denied: frozen` |
|
||
| `ssh ... n8n` | `denied: not an active service` |
|
||
|
||
So a leaked key redeploys a service that is already running — it cannot open a shell,
|
||
read files, forward ports, start a stopped service, or touch postgres/authentik.
|
||
|
||
### Optional hardening
|
||
|
||
`StrictHostKeyChecking=accept-new` trusts the host key on first connection. To pin it
|
||
instead, add the VPS's key to a secret and write it to `known_hosts` in the job. The
|
||
current fingerprint is in `~/Server/ci/vps_host_key.pub` on the VPS.
|
||
|
||
---
|
||
|
||
## 16. Adding a database and user to the shared Postgres
|
||
|
||
_Folded in from PostgreSQL/add_db_and_user.md, 2026-09-22._
|
||
|
||
Here is how you can add a database & a user to postgres so you don't have 100 postgres containers for every little app that requires postgres.
|
||
|
||
You can edit these values to quickly get started & copy this.
|
||
Database name: n8n
|
||
Database user: n8n_user
|
||
|
||
```sql
|
||
-- 1. Create user
|
||
CREATE ROLE n8n_user
|
||
LOGIN
|
||
PASSWORD 'Overvalue-Unstuffed-Coconut-Gutless7-Prevent-Sharpie';
|
||
|
||
-- 2. Create database
|
||
CREATE DATABASE n8n
|
||
ENCODING 'UTF8';
|
||
|
||
-- 3. Allow connection
|
||
GRANT CONNECT ON DATABASE n8n TO n8n_user;
|
||
|
||
-- 4. Switch to the database
|
||
\c n8n
|
||
|
||
-- 5. Schema access
|
||
GRANT ALL PRIVILEGES ON SCHEMA public TO n8n_user;
|
||
|
||
-- 6. Existing objects
|
||
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO n8n_user;
|
||
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO n8n_user;
|
||
GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO n8n_user;
|
||
|
||
-- 7. Future objects
|
||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||
GRANT ALL PRIVILEGES ON TABLES TO n8n_user;
|
||
|
||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||
GRANT ALL PRIVILEGES ON SEQUENCES TO n8n_user;
|
||
|
||
ALTER DEFAULT PRIVILEGES IN SCHEMA public
|
||
GRANT ALL PRIVILEGES ON FUNCTIONS TO n8n_user;
|
||
```
|