gib/Panama

0 Fork 0

Files

T

Gabriel Brown cb089b39ea Sync both server snapshots with what the live machines run, and keep their backup tooling

2026-10-30 13:44:29 -04:00

90 KiB

Raw Blame History

AGENTS.md — `~/Server` container environment

Snapshot of junior.gib:~/Server/AGENTS.md. The two public addresses read <HOME_PUBLIC_IP> and <VPS_PUBLIC_IP> here. The live copy has the real ones.

Operating guide for AI agents and humans working on this VPS. Read this before adding a container, editing a compose file, or touching the firewall. Much of what follows is non-obvious and was learned the hard way; deviating will break things.

Host: Hetzner VPS, Fedora 44, kernel 7.1.7-200, podman 5.8.4, podman-compose 1.6.0, SELinux Enforcing (targeted), firewalld active. User: gib (uid 1000), subuid/subgid range 524288:65536.

This file is the document. CLAUDE.md is a symlink to it. That way Claude Code loads it automatically, while the 20+ compose files and scripts that cite AGENTS.md §N keep resolving. Edit AGENTS.md; never replace either name with a second real copy, or the two will drift apart silently.

The symlink previously ran the other way and a markdown formatter dissolved it on 2026-09-22 — saving through a symlink replaced it with a regular file and orphaned the original. If CLAUDE.md ever turns back into a real file, that is what happened.

Docs that deliberately live elsewhere

Kept next to the thing they describe, because that is where you will be standing when you need them:

Path	Covers
`UseSend/certs/README.md`	Why the SMTP proxy needs a real cert, and refreshing it
`Authentik/blueprints/REPRODUCE.md`	Rebuilding the password-recovery flow from scratch
`Next_Apps/Completeuphoria/README.md`	That app — its own git repo, do not fold in
`Search/Bang/README.md`	That app — its own git repo, do not fold in

Completed history (migration plan, session reports, one-off agent prompts) moved to archive/ on 2026-09-22. Nothing there is current; consult it for why, never for how.

1. The one rule that governs everything

Everything is ROOTLESS podman. There is no root podman on this box and there must never be again.

/var/lib/containers is empty, all root systemd units are deleted, and root's store was wiped with podman system reset. If you catch yourself typing sudo podman, stop — you are about to create a second, invisible container store that duplicates disk and silently diverges from the running stack.

Do	Never
`podman ...` as `gib`	`sudo podman ...`
`systemctl --user ...`	`sudo systemctl ...` for containers
units in `~/.config/systemd/user/`	units in `/etc/systemd/system/`
`WantedBy=default.target`	`WantedBy=multi-user.target`

2. Layout and conventions

~/Server/
├── AGENTS.md                    <- this file
├── archive/                     <- completed history; see archive/README.md
├── verify-rootless.sh           <- run after ANY change; exits non-zero on failure
├── backup                       <- rsync to remote; runs unprivileged
├── podman-example.service       <- user-unit template for new services
├── compose.example.yml          <- compose template
└── <Service>/
    ├── compose.yml
    ├── .env                     <- secrets; never commit, never echo in full
    ├── podman-<name>.service    <- copy of the installed user unit (reference)
    └── volumes/ | data/ | <vol>/  <- bind-mounted data, gib-owned

Service inventory (as of 2026-08-12, end of migration)

40 containers across 22 active user units. podman ps is the source of truth; this is orientation. Verify enablement with verify-rootless.sh (§13.3), not by eye.

Unit	Directory	Notes
postgresql	PostgreSQL	shared, 14 databases. pgvector, `-trixie` base only (§13.4)
nginx-proxy-manager	Nginx_Proxy_Manager	the front door; admin on `nginx.gibbyb.com`
authentik	Authentik	SSO. Port 9000 on the WireGuard IP only. Protects many hosts by forward auth (§13.9)
gitea	Gitea	git + container registry. No `userns_mode` (s6). Packages on the volume
watchtower	Watchtower	auto-update; API on WireGuard only; excludes postgres + authentik
adminer	Adminer	DB admin UI; no auth of its own
vaultwarden	Vaultwarden
infisical	Infisical	secrets; DB on the shared postgres
tools	Tools	convertx, it-tools, pdf-tools, bento-pdf
rustdesk	Rustdesk	split DNS — see §7. Relay needs raw TCP; key must never be regenerated
beszel	Beszel	hub + this host's agent
uptime	Uptime	Uptime Kuma; unconfined (socket)
portainer	Portainer	unconfined (socket). Public hostname is `port.gbrown.org`
bang	Search/Bang/docker	built locally; defaults to Google since searxng was removed
isponsorblocktv	ISponsorBlockTV	host network, `keep-id`
completeuphoria	Next_Apps/Completeuphoria/docker	built locally
sierraandtyler	Next_Apps/sierraandtyler/docker	built locally
convexmonorepo	ConvexMonorepo	Next + Convex backend + dashboard
spoon	Spoon	Next + Convex + agent-worker (unconfined, needs the socket)
stpeteit	StPeteIT	Next + Convex + dashboard
lashaddict	LashAddict	Next + Convex + dashboard. lashaddictstudio.com (own Cloudflare zone); see §13.17 for the DNS pin
usesend	UseSend	app + redis + smtp-server + RustFS storage (`storage.usesend.gbrown.org`, pinned tag; replaced MinIO 2026-10-27)
n8n	N8n	intentionally stopped — unit disabled, data kept. `systemctl --user enable --now podman-n8n.service`

Convex stacks use two ports — api.* → 3210 (client API), convex.* → 3211 (HTTP actions). Getting these backwards produces a working-looking site that 500s.

Compose files present but NEVER STARTED (no unit enabled): Documenso, CalCom, Penpot. Each needs its role and database created first; Documenso additionally needs a signing certificate. Instructions are in each compose file's header.

Never migrated / stays home: nextcloud, immich, plex, jellyfin, *arr, sentry, openwebui, frigate, romm, obsidian-livesync, the KVMs. Home runs Docker, not podman (§13.8b).

Service directory names are TitleCase; unit and container names are lowercase. Some services nest under a category (Search/Bang/docker, Next_Apps/Completeuphoria/docker) — the unit's WorkingDirectory is what matters, not depth.

3. Adding a new container — the checklist

3.1 compose.yml

networks:
  nginx-bridge:
    external: true # ALWAYS external; see §5

services:
  myservice:
    image: docker.io/vendor/image:tag
    container_name: myservice
    hostname: myservice
    networks: ["nginx-bridge"]
    environment:
      - TZ=America/New_York # see §8
    volumes:
      - ./data:/var/lib/whatever:Z # :Z is MANDATORY; see §6
    restart: unless-stopped

Do not publish ports unless the service must be reachable off-box. Everything on nginx-bridge resolves by container name, so NPM proxies to http://myservice:8080 internally with no host port at all. Fewer published ports = less firewall surface.

3.2 Does it need `userns_mode`?

This is the subtlest part of the setup. Decide by how the image starts:

podman inspect <image> --format '{{.Config.User}}'   # empty => starts as root

Image behaviour	What to do	Host file ownership
Starts as root, drops privileges internally (most images: nginx, postgres, uptime-kuma, influxdb, beszel)	Nothing. Default mapping puts container-root at host uid 1000.	`gib` for root-owned paths; the container's own chown pushes service-user files into subuid space (e.g. `525264`)
Runs directly as a fixed non-root uid, or you want host files to stay `gib`-owned	`userns_mode: "keep-id:uid=<N>,gid=<N>"`	`gib`

Never use user: 1000:1000. Under a rootless userns that resolves to subuid 525287, not to gib, and the container loses access to its own gib-owned data. This exact mistake is why n8n was broken. Use keep-id instead.

Live examples:

PostgreSQL — keep-id:uid=999,gid=999 (postgres runs as 999; keeps pg_data gib-owned)
N8n — keep-id:uid=1000,gid=1000
everything else — default mapping

uid mapping cheat sheet (subuid base 524288):

container uid 0    -> host 1000 (gib)
container uid 999  -> host 525286
container uid 1000 -> host 525287
container uid N    -> host 524288 + (N-1)

3.3 Host directory ownership

Create data dirs owned by gib before first start:

mkdir -p ~/Server/MyService/data     # already gib-owned

If you ever import data written by another uid, sudo chown -R gib:gib it first. A uid outside gib's subuid range appears as nobody inside the container and cannot be fixed by the container itself.

3.4 systemd user unit

Copy podman-example.service, adjust Description and WorkingDirectory:

cp ~/Server/podman-example.service ~/.config/systemd/user/podman-myservice.service
# edit WorkingDirectory=%h/Server/MyService
systemctl --user daemon-reload
systemctl --user enable --now podman-myservice.service
cp ~/.config/systemd/user/podman-myservice.service ~/Server/MyService/   # keep a copy

If it depends on Postgres, add to [Unit]:

After=network-online.target podman-postgresql.service
Wants=podman-postgresql.service

⚠️ Ordering ≠ readiness. podman compose up -d returns when the container is created, not when Postgres accepts connections. Dependents must tolerate a brief DB-unavailable window (NPM and n8n both retry). If one doesn't, add an ExecStartPre polling pg_isready.

3.5 Verify

~/Server/verify-rootless.sh      # add your service to the lists inside it

4. Boot behaviour (linger)

User services start at boot without anyone logging in because lingering is enabled:

loginctl show-user gib --property=Linger    # must be Linger=yes

If linger is ever off, nothing starts until someone SSHes in — and logging in masks the failure. When testing boot behaviour, check from outside the box first, before SSHing in.

Proven working: after reboot, [email protected] starts ~9 s after boot and all containers ~11–13 s after, well before any login.

5. Networking

All services share the external network nginx-bridge (172.18.0.0/24, gw 172.18.0.1).

Created out-of-band so no stack races to create it at boot:

podman network create --subnet 172.18.0.0/24 nginx-bridge

Every compose file declares it external: true — including NPM, which used to define it inline. Containers resolve each other by container_name over this network.

Rootless and root networks are entirely separate namespaces. This is why the migration had to be a big-bang cutover. Irrelevant now that root podman is gone, but it explains why you cannot "test one service" against a differently-privileged stack.

Privileged ports

NPM binds 80/443 without root because of /etc/sysctl.d/99-rootless-ports.conf:

net.ipv4.ip_unprivileged_port_start=80

Host-wide: any unprivileged process can bind ≥80. Acceptable on a single-user VPS. If this file is lost, NPM silently fails to bind and every site goes down.

⚠️ Published ports lose the client IP

Rootless publishes ports through the rootlessport userspace proxy, so the peer address nginx sees is 172.18.0.x, not the real client. No header trick recovers this when there is no proxy in front — nothing upstream sets one.

The only real fix is to put the hostname behind Cloudflare, so CF-Connecting-IP exists. See §7.

6. SELinux — Enforcing

Every bind-mounted volume must carry :Z (or :rw,Z / :ro,Z):

volumes:
  - ./data:/var/lib/whatever:Z

:Z makes podman relabel the host path to container_file_t with a private MCS category unique to that container, re-derived on every start. Without it, Enforcing denies all access and the container fails.

⚠️ Shared mounts need `:z` (lowercase), not `:Z`

:Z assigns a private MCS category unique to one container. If two containers mount the same host path, the second to start relabels it and locks the first out. Use lowercase :z for shared paths:

server:
  volumes:
    - ./volumes/media:/data/media:z # shared with worker -> lowercase
    - ./volumes/server/custom-templates:/templates:Z # exclusive -> uppercase
worker:
  volumes:
    - ./volumes/media:/data/media:z # same path, same lowercase z
    - ./volumes/worker/certs:/certs:Z # exclusive -> uppercase

Verify: a shared dir shows container_file_t:s0 with no category; an exclusive dir's category must match its container's ProcessLabel.

ls -dZ volumes/media                                    # want: container_file_t:s0
podman inspect authentik-server --format '{{.ProcessLabel}}'

⚠️ Watchtower recreates DROP the :Z/:z bind flag (found 2026-08-13)

When watchtower recreates a container after pulling a new image, the copied bind loses its relabel option (Binds shows only rprivate,rbind). With :Z the volume keeps the OLD container’s MCS category while the new container gets a fresh pair — the service is locked out of its own data. agentchat hit this as SQLITE_CANTOPEN in a crashloop; fixed by switching its bind to :z.

Any watchtower-updated container with a bind mount must use lowercase :z. After a recreate the files stay container_file_t:s0 (no category), which a confined container can still access — the only labeling that survives. The Convex stacks (lashaddict / stpeteit / convexmonorepo / spoon) were switched to :z and restarted 2026-08-13; all backends verified healthy.

Two deliberate exceptions:

The podman socket mount gets NO :Z.
```
- /run/user/1000/podman/podman.sock:/var/run/docker.sock # no :Z, ever
```
Relabeling the live socket would break podman itself.
Containers mounting that socket need SELinux separation off, because systemd recreates the socket each boot as user_tmp_t:
```
security_opt: ["label:disable"]
```
Currently only portainer and uptime. They run unconfined (empty ProcessLabel); the other nine are confined as container_t:s0:c…. Accept this only for container-management tooling.

Check confinement:

podman inspect <name> --format '{{.ProcessLabel}}'    # empty = unconfined

⚠️ NEVER run restorecon -F against ~/Server

Plain restorecon -r is safe — it reports the :Z labels as "customized by admin" and leaves them alone. -F forces them back to user_home_t and every container loses access to its data simultaneously.

Going Permissive→Enforcing needed no relabel here: restorecon -nr on /etc /usr/sbin /usr/bin /usr/lib/systemd /var/lib /root /boot reported 0 mislabeled files. Re-check that before flipping SELinux on any rebuild.

7. Firewall and Cloudflare

firewalld zones — rules apply by arriving interface:

Interface	Zone	Policy
`eth0` (<VPS_PUBLIC_IP>, public)	`public`	`ssh` + explicitly opened ports only
`Home` (192.168.2.2, WireGuard)	`trusted`	target ACCEPT — full LAN/VPN access

Binding Home to trusted is what keeps LAN devices working when public is locked down. Before that it had no zone and silently inherited public.

Deliberately closed on the public interface: 81 (NPM admin), 3001, 9443, 5432. Reach NPM admin via https://nginx.gibbyb.com or over WireGuard at 192.168.2.2:81. Never open 5432 to the internet.

⚠️ Every VPS hostname must be Proxied (orange cloud)

This is the single most important DNS rule here, and it has bitten repeatedly. Because 80/443 accept only Cloudflare ranges, a DNS-only record pointing at <VPS_PUBLIC_IP> is unreachable — it fails at connect, not with an error page. The only exception is rustdesk, below, which uses ports the firewall opens explicitly.

Rustdesk — the one service that cannot be proxied

RustDesk's ID/relay protocol is raw TCP/UDP on 21115-21119. Cloudflare's proxy carries only HTTP/HTTPS, so it uses two hostnames on purpose:

Hostname	Cloud	Serves
`relay.gbrown.org`	grey / DNS-only → <VPS_PUBLIC_IP>	ID + relay, ports 21115-21119
`rustdesk.gbrown.org`	proxied → NPM → `rustdesk-server:21114`	web console + API, over TLS

Do not "simplify" these into one name — the API would then have to travel unencrypted, or the relay would break. relay.gbrown.org publishes the VPS IP; that is accepted, because 80/443 remain Cloudflare-only so the IP grants no access to any web service.

Firewall ports opened for it (public zone): 21114-21119/tcp plus 21116/udp — the UDP one is easy to miss and heartbeats fail silently without it.

The server keypair at Rustdesk/volumes/server/id_ed25519 was carried over from home. Its public half must keep matching RUSTDESK_API_RUSTDESK_KEY in the compose, or every client needs re-pairing. Never regenerate it.

Split-horizon DNS — LAN clients need their own records

The UniFi gateway answers *.gbrown.org → 192.168.1.2 (home) for every hostname. So after migrating a service, LAN clients keep silently reaching the OLD copy — no error, just stale data. Public DNS being correct does not help them.

For each migrated service, add a UniFi Host (A) record → 192.168.2.2 (the VPS's WireGuard address). UniFi honours exact matches over the wildcard, so the wildcard stays put for everything still at home. Verified working: LAN clients reach 192.168.2.2:443, the certificate validates, and the firewall's trusted zone permits it.

Do not point LAN records at <VPS_PUBLIC_IP> — the firewall drops non-Cloudflare traffic. Do not point them at Cloudflare edge IPs — those are not stable.

Currently carrying UniFi overrides: vault, tools, infisical, auth.

Never add nginx.gbrown.org → VPS. That hostname reaches home's proxy manager admin UI. Pointing it at the VPS locks you out of it. The VPS admin lives on nginx.gibbyb.com.

Advanced Certificate Manager

ACM is active on gbrown.org only (~$10/mo, confirmed billed to that zone). Universal SSL covers one subdomain level; ACM covers the multi-level wildcards. Active certificate (expires 2026-11-09) carries seven SANs:

gbrown.org, *.gbrown.org, *.tools.gbrown.org, *.spoon.gbrown.org,
*.lashaddict.gbrown.org, *.convexmonorepo.gbrown.org, *.convex.gbrown.org

A second advanced certificate (ordered 2026-10-27) carries the two-label patterns added later. Advanced certificates cannot be edited, so a new pattern means ordering another one, never deleting the first (that would drop every name only it covers):

*.usesend.gbrown.org, usesend.gbrown.org, *.billtracker.gbrown.org, billtracker.gbrown.org

Without it, convert.tools.gbrown.org and the 12 other two-label hostnames cannot be proxied at all. If a new two-label pattern appears, it needs adding to this certificate.

Ports 80/443 on eth0 are restricted to Cloudflare only (active since 2026-08-11), via firewalld ipsets cloudflare4 (15 ranges) and cloudflare6 (7 ranges):

tcp dport 22 accept                            <- ssh, never remove
ip  saddr @cloudflare4 tcp dport 80,443 accept
ip6 saddr @cloudflare6 tcp dport 80,443 accept
                                               <- everything else: zone default reject

Refresh the ranges periodically — Cloudflare does change them, and a stale set silently blocks legitimate traffic:

curl -s https://www.cloudflare.com/ips-v4 -o /tmp/cf-v4.txt
curl -s https://www.cloudflare.com/ips-v6 -o /tmp/cf-v6.txt
sudo firewall-cmd --permanent --ipset=cloudflare4 --add-entries-from-file=/tmp/cf-v4.txt
sudo firewall-cmd --permanent --ipset=cloudflare6 --add-entries-from-file=/tmp/cf-v6.txt
sudo firewall-cmd --reload

You cannot test this restriction from the VPS itself. Traffic from the host to its own public IP routes over loopback and never traverses the eth0 INPUT path, so it always succeeds and proves nothing. Verify from an external machine:
curl -k --max-time 10 --resolve any.gibbyb.com:443:<VPS_PUBLIC_IP> https://any.gibbyb.com/
Verified 2026-08-11: curl: (7) Failed to connect ... after 46 ms: Could not connect. A fast rejection (tens of ms) is the correct signal — the zone's default target sends an ICMP reject. A 10 s hang would mean a DROP instead; a 200/302 would mean the rule is not taking effect. Meanwhile https://any.gibbyb.com/ via Cloudflare returns normally.

⚠️ Verify DNS is actually proxied before restricting 80/443

A Cloudflare-only rule black-holes any hostname whose DNS points straight at the origin. This happened during setup: *.gibbyb.com was DNS-only, and applying the rule took every one of those sites down until the ports were reopened. Check each hostname; do not trust an assumption that "everything is behind Cloudflare":
resolvectl query uptime.gibbyb.com     # <VPS_PUBLIC_IP> = DIRECT (would be blocked)
                                       # 104.21.x / 172.67.x = Cloudflare (safe)
Any new hostname pointed at this VPS must be Proxied (orange cloud) in Cloudflare before it will work at all. DNS-only records are blocked by the firewall.

Current DNS: gibbyb.com A → <VPS_PUBLIC_IP> Proxied, *.gibbyb.com CNAME → gibbyb.com Proxied, so new subdomains inherit proxying automatically. Cloudflare SSL/TLS mode must be Full (strict) — NPM serves valid Let's Encrypt certs; "Flexible" causes redirect loops.

Note *.gbrown.org resolves to <HOME_PUBLIC_IP> — a different machine (home network). NPM here holds configs for those names but traffic does not arrive here.

Real client IPs

volumes/data/nginx/custom/server_proxy.conf (included in every proxy host's server block) contains:

real_ip_header CF-Connecting-IP;

It must be in server context — nginx.conf:81 already declares real_ip_header at http level, and redeclaring there is a fatal duplicate-directive error. NPM's nginx.conf already trusts the bridge via set_real_ip_from 172.16.0.0/12, so no set_real_ip_from is needed.

This is now trustworthy because the firewall permits 80/443 only from Cloudflare ranges: nothing else can reach nginx to forge the header. The two controls depend on each other — if you ever reopen 80/443 to the internet, CF-Connecting-IP becomes forgeable again and logged IPs can no longer be trusted for access lists or fail2ban.

8. Timezone

Host and all containers are America/New_York. New services should set TZ=America/New_York in environment:, or their logs stamp UTC and correlating across services becomes painful.

9. Backups

~/Server/backup rsyncs to a remote host and runs unprivileged — do not reintroduce sudo. As of 2026-08-11 every file under ~/Server is gib-owned and readable (verified: 4,307 entries traversed, 0 permission errors), so nothing is being silently skipped.

It still carries --exclude=dump.rdb defensively. That was for valkey, which chowned its own data to subuid 525286 at mode 600 inside the userns — unreadable to gib. Valkey went with searxng, but any future service that writes as a non-root uid with restrictive modes will hit the same thing. If you add one, re-run the check above; a Permission denied in rsync means data is silently missing from your backups:

rsync -avn --exclude=node_modules --exclude=.git ~/Server/ /tmp/probe/ 2>&1 | grep -i denied

Database backups are not covered by rsync in any useful form — take a logical dump:

podman exec postgresql pg_dumpall -U npm > ~/pg_dumpall_$(date +%F).sql

Do this before any change touching pg_data ownership, permissions or SELinux labels.

Pre-migration snapshot kept at ~/premigration_backup_2026-08-10/ (volume tarball + 4,043-entry ownership manifest + pg dump).

10. Operational gotchas

⚠️ Encrypted ClientHello breaks split-horizon DNS — keep `ech` OFF

This cost more time than everything else combined, and nothing about it is guessable.

Cloudflare publishes an HTTPS/SVCB record for proxied hostnames containing an ech= parameter whose public name decodes to cloudflare-ech.com. Browsers read that record, enable Encrypted ClientHello, and send cloudflare-ech.com as the outer SNI. With a UniFi override sending that connection to our own nginx instead of Cloudflare, nginx has never heard of cloudflare-ech.com and replies with TLS alert 112, unrecognized_name. Browsers do not retry.

Symptoms, all of which we chased down blind alleys:

Browser shows ERR_SSL_UNRECOGNIZED_NAME_ALERT; curl and openssl return 200. They never query HTTPS RRs, so they send plaintext SNI and always succeed.
Public access works fine — ECH terminates at Cloudflare, which understands it.
Only LAN clients with a UniFi override are affected.
A hostname with no HTTPS record (grey cloud, e.g. sonarr) works normally. That contrast is the diagnostic.

The zone setting is ech. It is hard-disabled in the dashboard UI on the Free plan but editable via the zone settings API. It must stay off for as long as split-horizon DNS points any proxied hostname at a local origin.

Reproduce in one command:

openssl s_client -connect 192.168.2.2:443 -servername cloudflare-ech.com
# -> tlsv1 unrecognized name ... SSL alert number 112
dig vault.gbrown.org HTTPS      # must show NO ech= parameter

Do not test browser problems with curl. curl bypasses HTTPS RRs, ECH, HSTS, service workers and the HTTP cache. A green curl next to a failing browser means the difference between them is the bug. Reproduce with the actual browser engine — headless works: chrome --headless=new --dump-dom <url> and read stderr for net_error codes.

nginx sends no SNI on proxy_pass by default. Proxying to an HTTPS upstream that serves multiple vhosts — e.g. home NPM → https://192.168.2.2:443 — fails the TLS handshake with no useful error, surfacing as a 502. Cost us hours. Always add:

proxy_ssl_server_name on;
proxy_ssl_name $host;

Symptom to recognise: curl to the bare IP gives 000, but curl --resolve host:443:IP gives a real status. That difference is the diagnosis.

Scheme/port mismatches produce misleading errors. We hit this three separate times: http → port 9443, https → port 9000, http → port 443. All produced 400/502/000 rather than anything naming the cause. When a proxy host misbehaves, check scheme against port first. Authentik serves HTTP on 9000 and HTTPS on 9443; NPM serves HTTPS on 443.

NPM reloads nginx on every save, and requests during the reload get a 503 with an empty body. Wait ~10 s and retry before treating a 503 as a real failure. Multiple "outages" this session were this.

A 401 is not a failure for CouchDB or for authentik's outpost. CouchDB demands credentials on every endpoint; browsers render the resulting auth prompt as an error page. Check the nginx access log for the status nginx actually returned rather than trusting what a browser displays.

Don't test forward-auth with bare curl. The authentik outpost needs X-Forwarded-* headers to reconstruct the original URL; without them it returns 500 and logs "failed to detect a forward URL from nginx". That 500 is a bad test, not a broken outpost.

DNS negative caching outlives record changes. After deleting and recreating a record, systemd-resolved will keep serving NXDOMAIN. sudo resolvectl flush-caches. A hostname that resolves with dig @1.1.1.1 but fails with curl is this.

nc -z inside the NPM container is not a reliable upstream probe — it reports false negatives for containers that are demonstrably serving. Test the real path with curl --resolve host:443:127.0.0.1 instead.

Older gotchas

Check systemctl --failed and capture logs BEFORE vacuuming journals. A journalctl --vacuum-size during cleanup destroyed the logs explaining an n8n failure that had been running for two months. Diagnose first, reclaim space second.

podman image prune -a is dangerous while the stack is stopped. With no containers running, -a considers every image unused and deletes images the stack depends on. Use plain podman image prune (dangling only).

Env var names must match what the image expects. PostgreSQL/.env defined POSTGRES_NAME while compose passed POSTGRES_DB, so the healthcheck expanded to pg_isready -d -U npm and failed 178,503 consecutive times while Postgres itself was fine. Both keys are present now; keep them in sync.

Every service in a compose file needs its networks: key. n8n's task-runners omitted it, landed on the compose default network, and could never resolve N8N_RUNNERS_TASK_BROKER_URI=http://n8n:5679. That single missing line kept n8n down for two months.

Restart the full stack after a podman upgrade, before rebooting, so failures are attributable. Re-verify keep-id behaviour after a podman-compose version bump — it is the mechanism postgres and n8n depend on:

podman run --rm --userns keep-id:uid=999,gid=999 -v /tmp/t:/t:rw \
  docker.io/library/postgres:17 touch /t/probe   # host file must be owned by uid 1000

11. Hardware access — what rootless genuinely cannot do

Some tools want raw hardware. Rootless cannot give it to them, and this is a real boundary, not a configuration puzzle. Verified on this host 2026-08-11:

podman run --rm --device /dev/sda:/dev/sda alpine ls -l /dev/sda
# ls: /dev/sda: Permission denied        <- gib is not in the `disk` group

--cap-add SYS_RAWIO / SYS_ADMIN appear to work — the container starts and the bit shows in CapEff — but the capability is granted only inside the user namespace. It confers nothing over host devices. Do not read a successful start as proof of access; test the actual operation.

Do not "fix" this by adding gib to disk, running the container as root, or reintroducing root podman. If a workload truly needs raw devices, run its collector on a host that has them and keep only the UI here.

The split-deployment pattern (use this for monitoring tools)

Most monitoring tools separate a hub/web component (no privileges) from an agent/ collector (needs hardware or host access). Put the hub on the VPS; run agents wherever the thing being monitored lives.

Service	On this VPS	Elsewhere
Beszel	`beszel` hub + `beszel-agent` (monitors the VPS)	`beszel-agent` on each other host
Scrutiny	❌ nothing — see below	full omnibus stays on the home server

Beszel's hub lives here (beszel.gibbyb.com) and agents report in. The home server's agent reaches this hub over WireGuard; add it in the UI as host 192.168.1.2 port 45876. The VPS's own agent is host host.docker.internal port 45876. Both agents carry the hub's public key, derived with ssh-keygen -y -f Beszel/volumes/hub/id_ed25519. If the hub is ever rebuilt its keypair changes and every agent's KEY must be updated.

❌ Do not host Scrutiny here (decided 2026-08-11, twice over)

It was built here, evaluated, and deliberately removed. Two independent reasons:

Scrutiny has no authentication at all. Verified: /web/dashboard → 200 and /api/summary → 200 with no credentials. Publishing it would expose disk models, serial numbers and health to anyone who found the hostname. Beszel has real accounts; Scrutiny does not. Do not put it behind Cloudflare without Cloudflare Access or NPM auth in front.
There is no SMART data on this host. /dev/sda (QEMU HARDDISK) and /dev/sdb (Hetzner Volume) both report SMART support is: Unavailable, and rootless cannot reach raw devices anyway (§11 above).

The collector must run where the disks are, so the web half gains nothing by moving. It runs as the full omnibus on the home server, LAN/VPN-only. Revisit only if a second disk-bearing host appears and one pane of glass becomes worth adding auth for.

Authentik — migrated copy, NOT yet authoritative

~/Server/Authentik is a 1:1 copy of the home server's ~/Server/auth, built 2026-08-11. The home instance is still authoritative and still serving auth.gbrown.org. This copy has its own database restored from a point-in-time dump, so the two diverge from that moment — it is a cutover target, not a hot standby. Changes made in one do not appear in the other.

DB: role + database authentik in the shared VPS postgresql, same PG_PASS as home, so .env needed no edits. Restored 212 tables; verified 11 users / 37 apps / 15 flows / 37 providers / 11 groups on both sides.
AUTHENTIK_SECRET_KEY in .env (mode 600) must match the key the database was encrypted with. Never regenerate it on a migrated database.
No Redis — authentik 2026.x is Postgres-backed. Home has none either.
Version pinned via AUTHENTIK_TAG in .env (2026.2.2). Keep both sides on the same tag through a cutover; authentik runs DB migrations on start and they are not reversible.

Cutover, when you want it: create an explicit auth.gbrown.org record pointing at the VPS (an explicit record overrides the *.gbrown.org wildcard), add an NPM proxy host to http://authentik-server:9000, and re-dump/restore the database immediately beforehand so nothing is lost in the gap. Roll back by deleting the DNS record.

Adapting a Docker/root compose file to rootless

Checklist when porting from the home server (which runs Docker as root):

Docker/root	Rootless podman here
`/var/run/docker.sock`	`/run/user/1000/podman/podman.sock` (rootless containers only), no `:Z`, plus `security_opt: ['label:disable']`
`privileged: true`, `devices:`, `cap_add: SYS_RAWIO/SYS_ADMIN`	Not available. Split the workload — see above.
`network_mode: host`	Works. Ports ≥80 bind fine; the agent is reachable from bridge containers via `extra_hosts: host.docker.internal:host-gateway` (resolves to `169.254.1.2`).
bind mounts	add `:Z` (§6)
`/etc/localtime:/etc/localtime:ro`	drop it; set `TZ=America/New_York` instead (§8)
host-specific env (e.g. `EXTRA_FILESYSTEMS=/boot`)	re-check against this host — the VPS has no separate `/boot` partition

Verify a ported service is reachable the way NPM will reach it, before wiring a proxy host:

podman run --rm --network nginx-bridge alpine \
  sh -c 'apk add -q curl && curl -s -o /dev/null -w "%{http_code}\n" http://<container>:<port>/'

12. Quick reference

# state
podman ps
systemctl --user list-units 'podman-*'
~/Server/verify-rootless.sh

# lifecycle (NEVER sudo)
systemctl --user restart podman-<name>.service
systemctl --user stop    podman-<name>.service
podman logs <container>

# after editing a compose file
systemctl --user restart podman-<name>.service

# after editing a unit file
systemctl --user daemon-reload && systemctl --user restart podman-<name>.service

# SELinux
getenforce
podman inspect <name> --format '{{.ProcessLabel}}'
sudo ausearch -m avc -ts recent          # denials

# firewall
sudo firewall-cmd --zone=public  --list-all
sudo firewall-cmd --zone=trusted --list-all

# nginx / NPM
podman exec nginx-proxy-manager nginx -t
podman exec nginx-proxy-manager nginx -s reload

Temporary root for an agent (revoke when done):

echo 'gib ALL=(ALL) NOPASSWD: ALL' | sudo tee /etc/sudoers.d/99-claude-temp >/dev/null
sudo chmod 0440 /etc/sudoers.d/99-claude-temp
# revoke:
sudo rm /etc/sudoers.d/99-claude-temp

§13. Findings from the 2026-08-12 post-migration audit

13.1 Upstream Hetzner Cloud Firewall (not visible from this machine)

There is a cloud firewall in front of the VM that firewalld cannot see and that firewall-cmd will happily contradict. Verified 2026-08-12:

port	firewalld	nftables	service healthy	reachable from internet
22	allow	allow	yes	yes
2222 (gitea SSH)	allow	allow	yes	no
21115-21119 (rustdesk relay)	allow	allow	yes	no

Confirmed from an unrelated external IP and independently by portchecker.io.

Consequence: git clone ssh://[email protected]:2222/... works over WireGuard but fails from any outside network, and RustDesk cannot relay for clients that are not on the LAN or tunnel.

⚠️ When a port is open in firewalld but still unreachable, check the Hetzner console before debugging anything on this box. Two hours were spent proving the local stack was correct. See archive/BROWSER_AUDIT_PROMPT_3.md §A1.

RESOLVED 2026-08-12 — the rules below were added and verified open from an external IP and by portchecker.io. ssh -p 2222 git@<VPS_PUBLIC_IP> now completes a handshake and returns Permission denied (publickey), which is the correct key-only response.

⚠️ The console names this server Junior (id 110193938), while the OS hostname is LittleMan. Same machine — do not go looking for a second server.

The rules (Hetzner Cloud Console → Firewalls → Default_Firewall, id 10059209). All INBOUND; source 0.0.0.0/0 + ::/0:

protocol	ports	purpose
TCP	21115-21117	RustDesk: NAT test (21115), ID/hole-punch (21116), relay (21117)
UDP	21116	RustDesk ID registration — without it clients never come online
TCP	2222	Gitea SSH (verified key-only: password auth is refused)

Verified listeners: hbbs on 21115/21116/21118 tcp + 21116 udp, hbbr on 21117/21119, apimain on 21114.

Deliberately NOT opened:

21114/tcp — the RustDesk web API. Already served via Cloudflare on 443 (rustdesk.gbrown.org). Opening it publishes an admin API that currently sits behind Cloudflare. This is why the rule is 21115-21117 and not 21114-21119.
21118-21119/tcp — websocket variants, needed only by the browser client.

⚠️ Add no OUTBOUND rules. Hetzner's firewall is stateful and permits all egress while zero outbound rules exist; adding even one flips outbound to default-deny and every destination must then be enumerated.

⚠️ Never add a broad inbound range. Ports 81, 3001 and 9443 are bound to 0.0.0.0 on this host (§13.2) and are safe only because this firewall permits nothing but 22/80/443 plus the rows above.

13.2 Publish admin ports to the WireGuard address, not 0.0.0.0

ports: ['5432:5432'] bound all 14 databases to the public interface, with only firewalld plus the Hetzner firewall in between. Fixed to 192.168.2.2:5432:5432.

RESOLVED 2026-08-12. All now bound to the WireGuard address:

container	port	binding
postgresql	5432	`192.168.2.2:5432`
portainer	9443	`192.168.2.2:9443`
nginx-proxy-manager	81 only	`192.168.2.2:81`
uptime	3001	`192.168.2.2:3001`

⚠️ NPM's 80 and 443 deliberately stay on 0.0.0.0 — they are the public front door, and firewalld already restricts them to Cloudflare ranges. Only the admin port moved.

Verified after: public sites unaffected (git/vault/port 200), all admin UIs reachable over WireGuard (npm:81 200, uptime:3001 302, portainer:9443 200).

Keep this in proportion. None of these are reachable from the internet: the Hetzner cloud firewall (§13.1) permits only 22/80/443, and firewalld does not open them either. All three also have their own login. So this is a defense-in-depth observation, not an exposure — the binding is wider than it needs to be, behind two firewalls and an application login.

It matters in exactly one scenario: if the Hetzner firewall rules are ever broadened, these ports become internet-facing immediately and silently, with no further change on this box. That is the reason to prefer 192.168.2.2: bindings — it removes the dependency on a rule set that lives in a web console someone else may edit.

⚠️ When adding Hetzner firewall rules, open only the specific ports intended (2222 for gitea SSH, 21115-21119 for the rustdesk relay). Do not open a broad range — that would expose all three of the above.

Postgres was the one genuine finding here and is fixed. The remaining three are a one-line change each (- "192.168.2.2:9443:9443"), deliberately left alone: changing NPM's 81 alters how the proxy itself is administered, and a mistake there locks you out of the tool you would use to repair it.

Rule for new services: never publish an admin port as NNNN:NNNN. Either bind it to 192.168.2.2 or do not publish it at all and let NPM reach it over nginx-bridge.

13.3 `systemctl --user enable` is a separate step from starting

Five stacks (convexmonorepo, lashaddict, spoon, stpeteit, usesend) were running but disabled — they would have vanished on the next reboot, silently, because everything looked healthy in podman ps. podman-compose up starting the containers does not enable the unit.

Always verify after adding a service:

systemctl --user is-enabled podman-<name>.service   # must print "enabled"

verify-rootless.sh checks this now. Note podman-auto-update, podman-restart, podman-clean-transient and podman-kube@ are podman's own units and are correctly disabled — do not enable them.

13.4 Postgres image must stay on a -trixie base

See the comment block in PostgreSQL/compose.yml. Switching to a bookworm-based image changes glibc from 2.41 to 2.36, which changes text collation and can make existing indexes return silently wrong results. Any base change requires REINDEX DATABASE + ALTER DATABASE ... REFRESH COLLATION VERSION on all 14 databases.

13.5 The backup exits non-zero for benign reasons

Gitea runs without userns_mode, so sessions/, issues.bleve and Gitea/ssh are owned by subuid 525288 and unreadable to gib. rsync therefore exited 23 on every run. Now excluded — none are needed to restore. The gitea SSH host keys are NOT among them (they are gib-owned in Gitea/volume/ssh and are backed up); losing those would give every client a host-key mismatch.

13.6 Cloudflare Load Balancers — DELETED 2026-08-13 (were: two unproxied LBs shadowing the tunnel)

RESOLVED 2026-08-13: both load balancers were deleted. The proxied *.gbrown.org and gbrown.org CNAMEs into the Cloudflare tunnel now carry all gbrown.org traffic, which was the "delete" option from §4.3 of archive/SESSION_2026-08-12.md. The pools (home-pool, vps-pool) still exist at the account level and are re-attachable if LBs are ever recreated. Full incident report: archive/SESSION_2026-08-13.md.

What forced the decision: the LBs had been flipped to proxied, and home-pool's endpoint carried a Host: gbrown.org header override. A proxied LB applies that override to the SNI and Host of every request — first causing 525s on the whole namespace (no gbrown.org server block existed on home NPM), then, once an apex host was added, routing every subdomain to the apex site. Unproxied LBs never applied it, which is why the override sat harmless for months. See §13.13 for the apex host that is now load-bearing.

Historical analysis from 2026-08-12, kept because the shadowing/diagnostic lessons stand:

Correction: an earlier revision of this section blamed a "DNS-only wildcard record". That was wrong. There is no such record, and no portainer.gbrown.org record either — a browser audit enumerated all 53 gbrown.org records and found neither. The cause is two Cloudflare Load Balancers, which shadow DNS records of the same name.

Measured 2026-08-12 against three public resolvers and Cloudflare's own authoritative nameservers (junade.ns.cloudflare.com, megan.ns.cloudflare.com):

portainer.gbrown.org.               30  IN  A  <HOME_PUBLIC_IP>
definitelynotarealhost.gbrown.org.  ->      A  <HOME_PUBLIC_IP>

A randomly invented name resolves to the same address. The mechanism:

LB (zone gbrown.org)	enabled	proxied	TTL	pools	fallback
`*.gbrown.org`	true	false	30	home-pool, vps-pool	vps-pool
`gbrown.org`	true	false	30	home-pool, vps-pool	vps-pool

pool	origin	monitor
home-pool	<HOME_PUBLIC_IP>	none attached
vps-pool	<VPS_PUBLIC_IP>	none attached

Three consequences:

A load balancer takes precedence over a same-name DNS record, so the proxied *.gbrown.org and gbrown.org CNAMEs to cfargotunnel.com are shadowed and never take effect — the tunnel carries no gbrown.org traffic today.
Because the LBs are proxied: false, they answer with the pool origin address directly, publishing the home IP for the whole namespace including the apex and names that do not exist. Cloudflare's own DNS page flags "Your origin IP address is partially exposed."
With no monitor attached to either pool, both are permanently "healthy", steering always lands on the first default pool (home-pool), and failover to the VPS can never occur. This is paid-for load balancing that structurally cannot fail over.

Explicit hostnames (port, git, auth, vault, …) have their own proxied A records at <VPS_PUBLIC_IP> and win over the wildcard LB, so they are unaffected.

⚠️ Diagnostic rule: before concluding a subdomain "has a stale record", query a name that cannot exist (dig +short thisdoesnotexist.gbrown.org). If it resolves, you are seeing catch-all behaviour — a wildcard or an LB — not a per-host record.

Before changing the LBs, read §13.8 — the client-IP consequence is the part that actually bites.

13.7 Gitea container registry retention

One cleanup rule (Site Administration -> Packages -> Cleanup Rules): container packages keep the newest 10 versions, drop versions older than 7 days, and always keep tags matching latest|main|prod. Gitea applies it with its built-in daily package cleanup cron. On 2026-10-01 the registry held 136 versions across 10 packages, 9.8 GB of blobs (down from 19 GB before the rule).

Checking without a browser session:

sqlite3 -readonly ~/Server/Gitea/volume/gitea/gitea.db \
  "SELECT count(*) FROM package_version; SELECT * FROM package_cleanup_rule;"

13.8 Home's NPM real_ip — FIXED 2026-08-12 (was: no config at all)

Verified 2026-08-12 on the home server: there are no custom nginx snippets under its NPM data directory, i.e. no set_real_ip_from / real_ip_header. The VPS does have this (Nginx_Proxy_Manager/volumes/data/nginx/custom/server_proxy.conf, §7); home does not.

Why this matters. The home cloudflared tunnel is active and enabled, and its ingress already covers the whole namespace:

tunnel: e4f0d4d8-5e57-4bee-ab7b-56ef1ed0b7da     (/etc/cloudflared/config.yml)
ingress:
  - hostname: gbrown.org        -> https://localhost:443
  - hostname: "*.gbrown.org"    -> https://localhost:443
  - hostname: gibbyb.com / "*.gibbyb.com"            -> https://localhost:443
  - hostname: madelinemcguigan.com / "*."            -> https://localhost:443
  - hostname: completeuphoria.club                   -> https://localhost:443
  - service: http_status:404                          (catch-all)

So deleting the LBs would NOT dark the home namespace — the tunnel catch-all already routes it. That risk, raised in the browser audit, is disproved.

The real consequence is the client IP. Today the LBs hand out home's raw address, so home's NPM sees the actual client IP and IP-based access lists work. Change that and:

route	what home's NPM sees as the client
today: unproxied LB -> home IP direct	the real client IP
delete LBs -> tunnel	127.0.0.1 (cloudflared dials `localhost:443`)
proxy the LBs	a Cloudflare edge IP

Both alternatives break every IP-based access list on the home NPM, because without real_ip nginx has no way to recover the true client address.

This is also why allow 127.0.0.1 once existed in the "private" access list, and why removing it was correct. With it present, anyone on the internet arriving through the tunnel would have matched a rule meant to mean "LAN only". Removing it closed that hole; it also means tunnel-routed traffic now fails closed.

⚠️ Prerequisite for either LB change: configure real_ip on the home NPM first, the same way the VPS does. For tunnel traffic that means trusting the loopback:

set_real_ip_from 127.0.0.1;
real_ip_header CF-Connecting-IP;

For a proxied LB it means trusting Cloudflare's published ranges instead. Do this before touching the load balancers, not after — otherwise the KVMs and anything else behind an access list go dark the moment DNS changes.

Safe to do independently, right now: attach the existing https /health monitor to both pools. It only adds health checking and moves no traffic while home-pool is up.

13.8b The fix, as applied (home server, 2026-08-12)

⚠️ Home runs Docker, not podman. docker exec nginx-proxy-manager ..., not sudo podman. NPM data volume: /home/gib/Server/nginx/volumes/data → /data.

Two files, because the directive must be split across contexts:

/data/nginx/custom/http_top.conf (http context — set_real_ip_from is additive):

set_real_ip_from 127.0.0.1;
set_real_ip_from ::1;

/data/nginx/custom/server_proxy.conf (server context — included by all 31 proxy hosts):

real_ip_header CF-Connecting-IP;

⚠️ real_ip_header must NOT go in http_top.conf. NPM's nginx.conf:81 already declares real_ip_header X-Real-IP; in the http context, and http_top.conf is included into that same context — a second declaration is a duplicate directive and nginx will refuse to start. The server block legally overrides it. The VPS is arranged the same way for the same reason.

Verified after reload (against scrutiny and ncmaster, both access-listed and enabled):

request from loopback	result
no header	403 — loopback is not in the allow list
`CF-Connecting-IP: 192.168.1.50`	302 — LAN range allowed, real_ip working
`CF-Connecting-IP: 8.8.8.8`	403 — public IP still denied

Access logs now record the forwarded client instead of 127.0.0.1. All home sites healthy afterwards (jellyfin/sonarr/radarr 302, immich 200, obsidian 401), NPM 0 restarts.

Useful facts discovered while doing this:

NPM already trusts Cloudflare's ranges (104.16.0.0/13, 172.64.0.0/13, 103.21.244.0/22) via conf.d/include/ip_ranges.conf — 265 entries. Only loopback was missing, and the header was wrong. So proxying the LBs would also now resolve correctly, not just the tunnel path.
Home's "Private" access list is 192.168.1.0/24, 192.168.2.0/24, 192.168.112.1/24 — confirmed no 127.0.0.1.
nano/macnano proxy hosts exist in home's DB but are disabled (no generated conf); they are served by the VPS NPM (hosts 42/43) forwarding to 192.168.1.10/.11 across WireGuard.

13.9 Two ways hosts are protected — do NOT "fix" one into the other

Correction. An earlier revision of this section claimed prowlarr.gbrown.org and comfyui.gbrown.org were "not access-controlled despite the UI saying they are". That was wrong, and acting on it would have broken working SSO. Both are protected — by authentik forward auth, not by an IP access list.

Evidence:

$ curl -sk -D - https://prowlarr.gbrown.org
HTTP/2 302
location: https://prowlarr.gbrown.org/outpost.goauthentik.io/start?rd=/

id=29 prowlarr -> 29 auth_request/authentik lines,  0 allow/deny
id=48 comfyui  -> 21 auth_request/authentik lines,  0 allow/deny
id=116 scrutiny -> 0 authentik lines,               4 allow/deny
id=47 ncmaster  -> 0 authentik lines,               4 allow/deny

The generated config carries:

auth_request  /outpost.goauthentik.io/auth/nginx;
error_page    401 = @goauthentik_proxy_signin;

This environment deliberately uses two different protection mechanisms, per host:

mechanism	how it looks	example hosts
authentik forward auth (SSO)	`auth_request` lines; unauthenticated request → 302 to `/outpost.goauthentik.io/start`	prowlarr, comfyui
IP access list	`allow`/`deny` lines; off-LAN request → 403	scrutiny, ncmaster, nano, macnano

⚠️ A 302 to outpost.goauthentik.io is PROTECTION, not exposure. Do not read it as "the service was served". Follow the location header before concluding anything.

⚠️ Public reachability here is intentional. Services are meant to be usable from outside the LAN; SSO is what secures them, including services that have no login of their own. Do not attach IP access lists to SSO-protected hosts, and do not treat a publicly-resolving hostname as a defect. An access_list_id set in the NPM database with no allow/deny in the generated config is not evidence of a hole on an SSO host.

Correct way to check what protects a host:

id=<n>
docker exec nginx-proxy-manager grep -cE '^[[:space:]]*(allow|deny) ' /data/nginx/proxy_host/$id.conf
docker exec nginx-proxy-manager grep -c 'auth_request'                /data/nginx/proxy_host/$id.conf

Zero for both would be a genuine finding. Non-zero for either is a protected host.

13.10 The LB health monitor cannot validate either origin — it sends no SNI

Moot for gbrown.org since 2026-08-13 — the LBs were deleted (§13.6) and the tunnel needs no health monitor. Kept because the no-SNI failure mode applies to any future monitor, and the pools still exist at the account level.

Do not attach a monitor to a pool expecting it to be a safe, traffic-neutral change. An earlier revision of this guide claimed attaching one "moves no traffic while home-pool is up". That is wrong and was disproved in practice on 2026-08-12.

What happened: monitor Gib's Monitor (HTTPS GET /health, port 443, expect 200, Host: health.gbrown.org) was attached to both pools. After one check cycle both went unhealthy:

home-pool  healthy: false  <HOME_PUBLIC_IP>  "TLS unrecognized name error"
vps-pool   healthy: false  <VPS_PUBLIC_IP>  "TLS unrecognized name error"

With every pool unhealthy the LB fell through to its fallback pool (vps-pool) and DNS moved immediately — the apex, sonarr, and every unmatched name flipped to <VPS_PUBLIC_IP> on a 30s TTL. It was rolled back by detaching the monitor, and DNS returned to <HOME_PUBLIC_IP>.

Root cause, proved from the shell — the monitor is not sending SNI:

# WITH SNI -- both origins are fine
$ openssl s_client -connect 127.0.0.1:443 -servername health.gbrown.org
VPS:  subject=CN=*.completeuphoria.club   Verify return code: 0 (ok)
home: subject=CN=*.gbrown.org

# WITHOUT SNI -- both origins reject, on both machines
$ openssl s_client -connect 127.0.0.1:443
tlsv1 unrecognized name ... SSL alert number 112
no peer certificate available

So it is not "the /health path is missing" and not a certificate problem — health.gbrown.org returns 200 and both certs verify. nginx has no default server with a cert for a no-SNI handshake, so it aborts before HTTP is ever spoken. (Alert 112 again — the same alert family as the ECH incident in §7.)

Options, in order of preference:

Use an HTTP monitor on port 80 — no TLS, so no SNI to get wrong. Set expected codes to include 301, since NPM's Force-SSL will redirect.
Give both origins a default server with a valid certificate so no-SNI handshakes succeed.
allow_insecure will not help — the handshake fails before certificate validation.

⚠️ Whatever you try, attach it to vps-pool ALONE first and confirm it reports healthy before home-pool is ever added. A monitor that fails on every pool does not disable failover — it triggers a silent flip to the fallback pool.

⚠️ Corollary: the earlier note that a "flapping monitor" was observed spontaneously was wrong too. That DNS movement was this attach/detach, not an independent fault.

13.11 Home's NPM database — collation fixed 2026-08-12

Home's nginx-proxy-manager-db ran postgres:17 (glibc 2.41) on a cluster created under glibc 2.36 — the mirror image of §13.4, and the same silent-wrong-results risk, on the database that stores every proxy host and access list.

Fixed properly rather than by silencing the warning:

docker exec nginx-proxy-manager-db pg_dump -U npm -d npm > ~/npm-db-backup-<date>.sql
docker exec nginx-proxy-manager-db psql -U npm -d npm -c "REINDEX DATABASE npm;"
docker exec nginx-proxy-manager-db psql -U npm -d npm -c "ALTER DATABASE npm REFRESH COLLATION VERSION;"
# repeated for the `postgres` database

NOTICE: changing version from 2.36 to 2.41. Data intact (117 proxy_host rows), sites serving afterwards. A 1.3 MB dump was taken first and is on the home server.

⚠️ Home runs Docker. docker exec, not podman.

13.12 Passwordless sudo was removed 2026-08-12 — both servers

/etc/sudoers.d/99-claude-temp (gib ALL=(ALL) NOPASSWD: ALL) is gone from the VPS and the home server. gib retains normal sudo via %wheel ALL=(ALL) ALL, now password-prompted.

This does not affect running anything here. Every container is rootless and every unit is a systemctl --user unit, so day-to-day operation needs no sudo at all. Only host-level work does — firewalld, SELinux relabeling, reading container-owned files under Gitea/volume, and docker on the home server.

⚠️ Non-interactive sudo will now fail, including over ssh without a TTY. An agent running sudo ... in a pipeline gets sudo: a password is required and the surrounding command may still report success — check for that string rather than trusting an exit code. If host-level access is genuinely needed, ask the user rather than re-creating the drop-in.

13.13 The `gbrown.org` apex proxy host on home NPM is LOAD-BEARING for the tunnel

Home's cloudflared (/etc/cloudflared/config.yml) dials NPM with originServerName: gbrown.org — that is the SNI for every request through the tunnel, regardless of which site the visitor asked for (NPM then routes by the Host header, which the tunnel preserves). Home NPM has no default TLS server, so a handshake with an unrecognized SNI is aborted (alert 112 — same failure family as §13.10).

Consequence: the gbrown.org proxy host in home NPM (currently → Home Assistant, added 2026-08-13) must exist for any tunnel-served site to work. Deleting or renaming it 525s the entire gbrown.org namespace, plus gibbyb.com / madelinemcguigan.com / completeuphoria.club traffic riding the same tunnel. If the apex host must ever go away, first change originServerName to another hostname NPM serves (e.g. nginx.gbrown.org) and sudo systemctl restart cloudflared.

13.14 VPS Actions runner — `cirunner` user (added 2026-08-13)

CI redundancy: the three dockerized runners + one host runner all live on the home server, so a home outage used to stop CI (and with it, deploys of VPS apps). A fourth runner now runs on the VPS under a dedicated cirunner user (uid 1001, subuids 589824:65536, linger enabled) — NOT under gib, so a CI escape lands in an empty account with no access to the production containers.

Layout (all under /home/cirunner, systemd user unit podman-runner.service):

piece	value
runner name / labels	`cirunner-vps` — `ubuntu-latest`, `ubuntu-24.04`, `ubuntu-22.04`, `vps`
compose + act_runner config	`~cirunner/runner/{compose.yml,config.yaml}`
container storage	`/home/gib/Media/cirunner/storage` (196 GB volume; traverse-only ACL `u:cirunner:x` on `/home/gib{,/Media}`)
job execution	sibling containers via cirunner's podman API socket

Because it shares the ubuntu-latest label with the home runners, whichever runner polls first takes the job — home down ⇒ the VPS keeps CI moving. runs-on: vps forces a job here.

Two traps, both hit and fixed during setup — do not regress them:

Socket path must be identical on host and in the runner container (/run/user/1001/podman/podman.sock, with DOCKER_HOST pointing at it). act_runner passes its socket path to the daemon when mounting it into job containers; a container-only path like /var/run/docker.sock fails with mkdir /var/run/docker.sock: permission denied.
SELinux blocks confined job containers from the socket even when mounted — config.yaml sets container.options: "--security-opt label=disable" (the §13.2/ watchtower trade, contained by the cirunner userns). Symptom if lost: ls: /var/run/docker.sock: Permission denied inside jobs.

Smoke-test repo: gib/ci-smoke (private) — a runs-on: vps workflow that hard-checks the docker socket inside a job and the watchtower hook at 192.168.2.2:8080. Push any commit to re-verify the runner end to end. Registration used the reusable instance token (gitea actions generate-runner-token).

13.15 Watchtower RETIRED 2026-08-17 — it took gitea down for three days

Do not reinstate watchtower on this host. Updates are now handled by ~/Server/update-containers.sh, run nightly at 00:00 by podman-update.timer.

What happened

time (EDT)	event
Aug 14 04:30:03	watchtower found a new `gitea:latest`, stopped the container, started a replacement, logged `failed=0 updated=1`
Aug 14 04:30:22	Uptime Kuma monitor 17: 502, gitea DOWN
Aug 17 12:39	manual `systemctl --user restart podman-gitea.service` — UP

Gitea logged nothing at all between those points. Watchtower reported success, so nothing escalated.

Root cause

podman-compose puts every project in a pod — 39 of 41 containers here. Watchtower does not update through compose; it builds a replacement container from the image plus the old container's config, and the replacement lands outside the pod, losing whatever the pod provided.

Proof: after the Aug 13 run, portainer was sitting podless, as was agentchat. Both were restored by restarting their units. Check for this with:

for c in $(podman ps --format '{{.Names}}'); do
  [ -z "$(podman inspect $c --format '{{.Pod}}')" ] && echo "PODLESS: $c"
done

Honest limit: portainer ran podless without visible breakage, so leaving the pod is not automatically fatal, and the exact reason gitea's replacement returned 502 could not be reconstructed — that container and its logs were removed by watchtower itself. What is certain is the trigger, the timing, and that the replacement was not built by compose.

The replacement

update-containers.sh never touches containers. It pulls the image, then restarts the systemd unit, so podman-compose rebuilds the project exactly as it does at boot.

./update-containers.sh              # all services except SKIP (what the timer runs)
./update-containers.sh gitea        # one service — use this from CI
./update-containers.sh --dry-run    # report only
./update-containers.sh --list       # services and their resolved images

Config lives in the two arrays at the top. SKIP=(postgresql authentik watchtower). Log: ~/Server/logs/update-containers.log.

Three non-obvious things it has to handle — do not "simplify" these away:

Compare the running container to the image on disk, not before-vs-pull. A --dry-run pulls the image, so a later real run would see before == after, decide nothing changed, and never restart — leaving the service on the old image with the new one on disk, permanently invisible. Anchoring on the container makes it idempotent and self-healing.
podman inspect .Image, never podman ps --format {{.ImageID}}. The first is the full 64-char ID matching image inspect .Id; the second is a 12-char short ID. Comparing them is always unequal, and every service looks like it needs updating.
Never restart an inactive unit. systemctl restart starts a stopped unit, so without a guard a new n8n image would resurrect n8n months after it was deliberately shut down. The script skips any unit that is not already active.

Also: image tags like ${BACKEND_TAG:-latest} are expanded from each service's .env in a subshell, so those files (database passwords, API keys) never leak into the script's environment.

⚠️ Push-to-deploy needs rewiring

CI used to POST to watchtower's API at 192.168.2.2:8080/v1/update. That endpoint no longer exists. The equivalent is:

ssh [email protected] '~/Server/update-containers.sh <service>'

which needs an SSH key for the runner. Until that is done, pushing an image updates the registry but does not deploy it — the nightly timer will pick it up at midnight.

13.16 Instant deploys — SSH forced command (replaces watchtower's HTTP API)

Watchtower's 192.168.2.2:8080/v1/update is gone (§13.15). CI now triggers a deploy by SSHing to [email protected] with a key that can do exactly one thing.

Why SSH and not a small HTTP daemon: the alternative was writing a network service that executes systemctl based on request bodies. sshd with a forced command achieves the same thing using software that already exists, is already exposed, and has decades of review behind it. No new listening port either.

Why a bridge is needed at all: the VPS runner is user cirunner (uid 1001) and the other runners are on the home server. systemd user managers are per-user, so neither can restart gib's units. This crosses that boundary — and only that boundary.

Pieces:

file	role
`~/Server/ci-deploy.sh`	the forced command; validates and dispatches
`~/Server/ci/ci_deploy_key`	private key — goes in the Gitea secret `DEPLOY_SSH_KEY`
`~/.ssh/authorized_keys`	pins the key to the forced command
§15 of this file	the workflow snippet to paste
`~/Server/logs/ci-deploy.log`	every request, allowed or denied

The authorized_keys entry:

command="/home/gib/Server/ci-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc ssh-ed25519 AAAA...

⚠️ $SSH_ORIGINAL_COMMAND is attacker-controlled if the key leaks. Never eval it, never word-split it, never interpolate it into a shell string. ci-deploy.sh matches it against ^[a-z0-9][a-z0-9-]{0,40}$, confirms a matching unit is already active, rejects the frozen set, and passes it as a single quoted argument.

Verified behaviour — re-run these after any change to the script or the key:

request	result
`adminer`	deploys
(bare shell)	denied — malformed
`cat .../PostgreSQL/.env`	denied — malformed
`adminer; cat /etc/passwd`	denied — malformed
`adminer$(id)`	denied — malformed
`postgresql`	denied — frozen
`n8n`	denied — not active

The "must already be active" check is doing double duty: it keeps the allowlist self-maintaining as services come and go, and it means the key cannot start something that was deliberately stopped.

Only two workflows ever called watchtower — gib/agentchat and gib/ci-smoke. The app stacks (spoon, lashaddict, convex-monorepo, stpeteit-payload, convex-monorepo-payload) push images and wait for the nightly timer. Add the step from the §15 step to any of them that should deploy immediately.

⚠️ ~/.ssh/authorized_keys was backed up to authorized_keys.bak-<timestamp> before the key was appended. Four pre-existing keys were preserved — check wc -l is 5, not 1, if you ever edit it.

13.16b Verified end-to-end 2026-08-19

First real push through the new path, gib/agentchat run 129:

10:55:01  ci-deploy.log:        deploy requested from 192.168.1.2: agentchat
10:55:02  update-containers.log: agentchat on 5d796ab4f297 -> new image 343d961d96ff
10:55:09  update-containers.log: agentchat: RESTARTED ok
10:55:09  update-containers.log: pruned: 5d796ab4f297...
10:55:09  ci-deploy.log:        deploy finished: agentchat (exit 0)

Container came back inside its pod (271d52b578ba) on the new image, chat.gbrown.org 200. Roughly 8 seconds from trigger to live. 192.168.1.2 is the home server, so the runner crossed the WireGuard tunnel to reach the VPS.

Pushing from the VPS. Only ~/.ssh/id_rsa.pub is registered with Gitea as "Gib's VPS (Junior)". The desktop and home keys are passphrase-protected and unusable non-interactively, and the key registered as "Gib's Server" matches neither machine — it predates a rebuild. To push from here:

GIT_SSH_COMMAND="ssh -p 2222 -i $HOME/.ssh/id_rsa" git clone ssh://[email protected]:2222/gib/<repo>.git

⚠️ scp to desktopw.gib hangs, though plain ssh works. Pipe instead: ssh desktopw.gib 'cat > /tmp/f' < localfile.

⚠️ Do not build shell snippets for a remote host through nested SSH quoting — an earlier attempt mangled printf '%s\n' into a literal newline and silently corrupted the workflow. Write the file locally, pipe it over, and apply it with a script on the far end.

Remaining repos still without a deploy trigger (they push an image and wait for the midnight timer): spoon, lashaddict, convex-monorepo, convex-monorepo-payload, stpeteit-payload, and ci-smoke (which still references the dead watchtower endpoint). Add the step from §15 below, changing the service-name argument.

13.17 LashAddict containers pin DNS to Cloudflare (2026-09-30)

lashaddict moved from *.lashaddict.gbrown.org to lashaddictstudio.com (api. → backend:3210, convex. → backend:3211, dashboard. → 6791, apex → next:3000, www redirect host; cert NPM #44). Hetzner's resolvers (185.12.64.1/.2) kept a cached NXDOMAIN for the brand-new hostnames for hours, and the Convex backend fetches its own auth discovery URL (https://convex.lashaddictstudio.com/.well-known/...), so every session failed (AuthProviderDiscoveryFailed). All three services in LashAddict/compose.yml therefore set dns: [1.1.1.1, 1.0.0.1]; container names still resolve through aardvark-dns. Do not remove the pin. The same trap applies to any new hostname a container must reach soon after it is created: resolvectl flush-caches fixes the host, not the Hetzner upstream.

14. Restoring ~/Server from the home-server backup

Folded in from RESTORE.md, 2026-09-22. Read it before you need it.

Written 2026-09-17. Read this before you need it.

Layout

[email protected]:/home/gib/Public/Server_Backup/Containers/lilman.gib/current/

One directory, overwritten daily. History does not live in dated folders — it lives in ZFS. Media/Server_Backup is its own dataset and zfs-auto-snapshot keeps 14 dailies, taken by cron.daily between roughly 03:22 and 03:52. The backup timer runs at 01:00 so every snapshot captures a finished backup, never a half-written one.

Older copies are reachable through the hidden snapshot directory (snapdir=hidden, so it will not show in ls, but you can still cd into it):

ls ~/Public/Server_Backup/.zfs/snapshot/
ls ~/Public/Server_Backup/.zfs/snapshot/zfs-auto-snap_daily-2026-09-17-0722/Containers/lilman.gib/current/

ZFS snapshots are block-level, so a 1.7 GB database that changes a few pages costs a few pages — not a new 1.7 GB copy. That is why this is a single mirror plus snapshots rather than dated directories or rsync --link-dest, both of which copy whole files.

The 03-17-2026, 08-12-2026 and 09-11-2026 directories are leftovers from the old scheme. They predate the ownership fix below and are not reliable restore sources.

What a backup contains

Path	What it is
`Adminer/`, `Gitea/`, `PostgreSQL/`, ...	Raw bind-mount data for each service
`_backup_state/postgres/all.sql.gz`	`pg_dumpall` of the live cluster — authoritative
`_backup_state/sqlite/<original path>`	Online `.backup` of each SQLite DB — authoritative
`_backup_state/files/`	Files gib cannot read off disk, pulled via `podman exec`

Use _backup_state/ for databases, not the raw data dirs. The raw dirs are copied while containers run, so they are crash-consistent at best. The dumps use the proper online-backup APIs and are what will actually restore cleanly.

Ownership, and why `--fake-super` is everywhere

Most services run with userns=private, which maps the container's user back to gib — their data is plain gib-owned on the host and needs nothing special. Gitea is the exception. It runs without userns_mode (s6 needs container-root, see AGENTS.md §3), so its data is owned by subuid 525288:525290, which gib cannot recreate on the home server. The backup therefore stores the real mode/uid/gid in a user.rsync.%stat xattr (rsync --fake-super), verified working on the home server's ZFS.

Consequence: a plain rsync/scp/cp of the backup gives you correct file contents with every file owned by whoever ran it, and Gitea will fail on its own data dir. Every restore command below passes -M--fake-super so the sending side decodes those xattrs, and runs under sudo locally so the ownership can actually be applied.

(The old dated-directory backups were taken without this, so everything in them is uid 1000. Restoring one means chown'ing Gitea's tree to 525288:525290 by hand afterwards.)

Restore in place (same machine)

Stop everything first:

systemctl --user stop 'podman-*.service'

Pull it back. sudo has no SSH key of its own, so point it at gib's:

sudo rsync -a --numeric-ids -M--fake-super --delete \
  -e "ssh -i /home/gib/.ssh/id_rsa -o UserKnownHostsFile=/home/gib/.ssh/known_hosts" \
  [email protected]:/home/gib/Public/Server_Backup/Containers/lilman.gib/current/ \
  /home/gib/Server/

To restore from an older day, swap the source for a snapshot path:

.../.zfs/snapshot/zfs-auto-snap_daily-2026-09-15-0751/Containers/lilman.gib/current/

Verify Gitea's ownership came back — this must say 525288, not 1000:

ls -ln /home/gib/Server/Gitea/volume/gitea/ | head

Then restore databases (below) and start services:

systemctl --user start 'podman-*.service'

Restoring the databases

Postgres — start only the postgres container, then:

gunzip -c /home/gib/Server/_backup_state/postgres/all.sql.gz \
  | podman exec -i postgresql psql -U npm -d postgres

pg_dumpall includes roles and every database (npm, authentik, usesend, ...), so this restores the whole cluster in one shot.

SQLite — each dump sits at its original relative path. Stop the owning container, copy the dump over the live file, restart. Gitea, which is the one needing a chown:

systemctl --user stop podman-gitea.service
cp /home/gib/Server/_backup_state/sqlite/Gitea/volume/gitea/gitea.db \
   /home/gib/Server/Gitea/volume/gitea/gitea.db
sudo chown 525288:525290 /home/gib/Server/Gitea/volume/gitea/gitea.db
systemctl --user start podman-gitea.service

For the others (uptime-kuma, vaultwarden, beszel, portainer, ...) the copy is enough — no chown, since they are gib-owned already.

bridge.json — _backup_state/files/BillTracker--data--mcp--bridge.json goes back to BillTracker/data/mcp/bridge.json, owned 525288, mode 0600. It holds the MCP bridge's registered OAuth clients; without it those clients must re-register.

Migrating to a new server

Install podman, loginctl enable-linger gib, and create gib with the same uid (1000) and the same /etc/subuid range. If the subuid base differs, Gitea's restored files land on the wrong owner and it will not start.
Restore as above, to the new box.
Restore the databases from _backup_state/.
Re-create the systemd units from each service's podman-*.service file — they are in the backup, next to each compose.yml.
Re-point DNS, then re-issue certificates through Nginx Proxy Manager.

If the new server is ARM (Hetzner CAX): the data restores fine, it is architecture-independent, but every image must be rebuilt or re-pulled for arm64. Checked 2026-09-17: all third-party images used here publish arm64 except minio, which needs verifying. The git.gbrown.org/gib/* images rebuild natively via CI.

What is deliberately NOT backed up

sessions/, issues.bleve — ephemeral Gitea logins and a rebuildable search index.
dump.rdb — redis/valkey cache dumps, regenerated on boot.
Gitea/ssh — a 25-byte shell env file. The Gitea SSH host keys ARE backed up (Gitea/volume/ssh); losing those would give every client a host-key mismatch.
node_modules, build output, *.log, media files.
The podman image store and Gitea package registry — they live on the Hetzner volume at ~/Media, not ~/Server. All rebuildable or re-pullable.

Checking it is actually working

systemctl --user list-timers server-backup.timer
tail -40 ~/Server/logs/backup.log
ssh [email protected] 'ls -la ~/Public/Server_Backup/Containers/lilman.gib/current/ | head'

A healthy run ends with === backup complete ===. Things to watch for in the log:

ZFS history OK — the run confirmed snapshots are still being taken. If this ever says WARNING, versioning is broken and current/ is just a mirror: a mistake made here would propagate with nothing to roll back to. Fix zfs-auto-snapshot before anything else.
rsync exit 24 is logged as a NOTE, not a failure. It only means a file changed mid-transfer, which is normal on a live system.

Optional: put an uptime-kuma push URL in ~/Server/.backup-heartbeat-url and successful runs will ping it, so a silent failure shows up as a down monitor.

15. Instant deploys — the CI step

Folded in from ci/DEPLOY_STEP.md, 2026-09-22.

Watchtower was removed 2026-08-17 (AGENTS.md §13.15), so http://192.168.2.2:8080/v1/update no longer answers. Deploys still happen, just at midnight via podman-update.timer. To get them instant again, do the two steps below.

Step 1 — add the deploy key as a Gitea secret (once)

The private key is on the VPS at:

/home/gib/Server/ci/ci_deploy_key

In Gitea → Site Administration → Secrets (or per-repo → Settings → Secrets), add:

name	value
`DEPLOY_SSH_KEY`	the entire contents of that file, including the BEGIN/END lines

An org- or admin-level secret means every repo gets it without repeating yourself.

You can delete secrets.WATCHTOWER_TOKEN afterwards — nothing consumes it now.

Step 2 — swap the deploy step in each workflow

Replace this:

- name: Trigger immediate deploy via Watchtower API
  run: |
    curl -fsS -X POST -H "Authorization: Bearer ${{ secrets.WATCHTOWER_TOKEN }}" \
      http://192.168.2.2:8080/v1/update \
      || echo "::warning::watchtower trigger failed; nightly sweep will deploy instead"

with this — changing agentchat on the last line to the service being deployed:

- name: Trigger immediate deploy
  run: |
    install -m600 /dev/null "$HOME/.deploy_key"
    printf '%s\n' "${{ secrets.DEPLOY_SSH_KEY }}" > "$HOME/.deploy_key"
    ssh -i "$HOME/.deploy_key" \
        -o BatchMode=yes \
        -o StrictHostKeyChecking=accept-new \
        -o ConnectTimeout=15 \
        [email protected] agentchat \
      || echo "::warning::deploy trigger failed; nightly sweep will deploy instead"
    rm -f "$HOME/.deploy_key"

Kept deliberately from the old step:

|| echo "::warning::" — a deploy hiccup warns instead of failing a build whose image already pushed fine. The midnight timer is the backstop.
192.168.2.2 — the VPS's WireGuard address, so this works from the home runners and the VPS runner alike, and never crosses the public internet.

Repo → service name

The argument after [email protected] is the systemd service name, not the repo name:

repo	argument
`agentchat`	`agentchat`
`spoon`	`spoon`
`lashaddict`	`lashaddict`
`convex-monorepo`	`convexmonorepo`
`stpeteit-payload`	`stpeteit`

Confirm any name with ~/Server/update-containers.sh --list.

What the key can and cannot do

It is pinned to a forced command in ~/.ssh/authorized_keys:

command="/home/gib/Server/ci-deploy.sh",no-port-forwarding,no-agent-forwarding,no-X11-forwarding,no-pty,no-user-rc ssh-ed25519 AAAA...

sshd runs ci-deploy.sh no matter what the client asks for. Verified by testing:

attempt	result
`ssh ... adminer`	deploys adminer
`ssh ...` (bare shell)	`denied: expected a single service name`
`ssh ... 'cat .../PostgreSQL/.env'`	`denied`
`ssh ... 'adminer; cat /etc/passwd'`	`denied`
`ssh ... 'adminer$(id)'`	`denied`
`ssh ... postgresql`	`denied: frozen`
`ssh ... n8n`	`denied: not an active service`

So a leaked key redeploys a service that is already running — it cannot open a shell, read files, forward ports, start a stopped service, or touch postgres/authentik.

Optional hardening

StrictHostKeyChecking=accept-new trusts the host key on first connection. To pin it instead, add the VPS's key to a secret and write it to known_hosts in the job. The current fingerprint is in ~/Server/ci/vps_host_key.pub on the VPS.

16. Adding a database and user to the shared Postgres

Folded in from PostgreSQL/add_db_and_user.md, 2026-09-22.

Here is how you can add a database & a user to postgres so you don't have 100 postgres containers for every little app that requires postgres.

You can edit these values to quickly get started & copy this. Database name: n8n Database user: n8n_user

-- 1. Create user
CREATE ROLE n8n_user
  LOGIN
  PASSWORD 'Overvalue-Unstuffed-Coconut-Gutless7-Prevent-Sharpie';

-- 2. Create database
CREATE DATABASE n8n
  ENCODING 'UTF8';

-- 3. Allow connection
GRANT CONNECT ON DATABASE n8n TO n8n_user;

-- 4. Switch to the database
\c n8n

-- 5. Schema access
GRANT ALL PRIVILEGES ON SCHEMA public TO n8n_user;

-- 6. Existing objects
GRANT ALL PRIVILEGES ON ALL TABLES IN SCHEMA public TO n8n_user;
GRANT ALL PRIVILEGES ON ALL SEQUENCES IN SCHEMA public TO n8n_user;
GRANT ALL PRIVILEGES ON ALL FUNCTIONS IN SCHEMA public TO n8n_user;

-- 7. Future objects
ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT ALL PRIVILEGES ON TABLES TO n8n_user;

ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT ALL PRIVILEGES ON SEQUENCES TO n8n_user;

ALTER DEFAULT PRIVILEGES IN SCHEMA public
GRANT ALL PRIVILEGES ON FUNCTIONS TO n8n_user;

90 KiB Raw Blame History Unescape Escape

AGENTS.md — ~/Server container environment