gib/Panama
1
0 Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

Snapshot both servers: the VPS catalog moves to server/vps, the home server joins as server/home

Browse Source 
The catalog in server/containers had drifted from the VPS. Eleven compose
files and twelve units no longer matched what runs, and two running
services, Gbrown_Agent and BillTracker, were missing. server/vps is now a
copy of the live files, and panama-server reads from it.

server/home holds the home server's services in the same shape. They still
run under Docker, so the podman units there are drafts for a later move, and
nothing installs from that tree. PostHog, Sentry and Plausible are checkouts
of upstream repositories. Only their local files are kept, next to an
UPSTREAM file naming the repository and the exact commit.

Every secret in both trees is a ${VAR} reference named in .env.example, or
CHANGE_ME in files that are not compose files. The live servers still hold
some of those values inline, so a live file can differ from its copy on
those lines alone. Existing .env.example files keep their comments, with new
keys appended at the end.

The ignore rules and the secrets contract now cover both trees, plus the
volume/ and volumes/ directories the home server uses.
This commit is contained in:
Gabriel Brown committed 2026-10-06 10:52:47 -04:00
1 parent a664cf68f5
commit 913a57f8a6
146 files changed
+5258 -354
Download Patch File Download Diff File Expand all files Collapse all files

No files matched your search

server/home/PostHog/README.md
+391
View File
@@ -0,0 +1,391 @@
# PostHog (self-hosted) on Fedora
PostHog's official installer (`bin/deploy-hobby`) assumes Ubuntu/Debian and a
bare host. This directory is that install done by hand, adapted to this server.
## Why the official installer doesn't work here
| What `deploy-hobby` does | Why it fails / what we did instead |
|---|---|
| `apt update`, `apt install git brotli curl` | Not Debian. All three are already installed via dnf. |
| Installs Docker from `download.docker.com/linux/ubuntu` | Docker 29.8 is already installed. |
| Downloads the v1 `docker-compose` binary to `/usr/local/bin` | We use the Compose v5 plugin (`docker compose`). |
| Runs Caddy on host ports `80:80` / `443:443` and gets its own Let's Encrypt certs | nginx-proxy-manager owns 80/443. Caddy now serves plain HTTP on `127.0.0.1:8002` and NPM terminates TLS in front of it. |
| Publishes Temporal on `0.0.0.0:7233` and its UI on `0.0.0.0:8081` | Rebound to `127.0.0.1`. |
| Phones home to PostHog Cloud with your domain | Skipped. |
| Generates `compose/start` calling `./compose/wait` | Upstream bug, fixed here. WORKDIR is `/code` but the mount is at `/compose`, so the relative path never resolves. Harmless (`bin/migrate` retries 10x) but it meant the readiness check never ran. Ours calls `/compose/wait`. |
Note: Docker here runs *without* SELinux labelling (`docker info` lists only
seccomp and cgroupns), so the bind mounts need no `:z` / `:Z` relabelling even
though the host is SELinux-enforcing.
Caddy is kept rather than replaced by NPM because it does path-based routing
that NPM can't express in one proxy host: `/e`, `/i/v0`, `/batch`, `/capture`
go to `capture`; `/s` to `replay-capture`; `/i/v1/logs` to `capture-logs`;
`/livestream` to `livestream`; `/posthog/*` to object storage; everything else
to `web`. NPM just forwards the whole hostname to Caddy.
## Layout
compose.yml docker-compose.hobby.yml + local patches
compose.override.yml additive local settings, auto-merged by compose
docker-compose.base.yml verbatim from the repo (compose.yml `extends` it)
.env secrets + domain (chmod 600, never regenerate)
.env.services verbatim from the repo
posthog/ upstream checkout, only used for config bind mounts
share/ GeoLite2-City.mmdb
compose/ entrypoint scripts the web/worker containers run,
plus local-patches.py (see "Multiple projects" below)
bin/upgrade Fedora-native replacement for bin/upgrade-hobby
bin/apply-local-patches.py replays our compose.yml deltas after an upgrade
## nginx-proxy-manager
Add a proxy host:
- Domain: `posthog.gbrown.org`
- Scheme: `http`, Forward host: `posthog`, Forward port: `80`
- **Websockets: ON** (required — the `/livestream` endpoint)
- Request an SSL cert, force SSL
`posthog` is a network alias on `nginx-bridge` for the Caddy container.
Also worth raising NPM's `client_max_body_size` for this host (Advanced tab:
`client_max_body_size 50m;`) — session-recording batches and source-map
uploads exceed the 1 MB default.
## Two config files, two reasons
`compose.yml` holds changes that *replace* upstream values -- the proxy's ports
and networks. These have to be patched into the file directly, because Compose
merges list-valued keys like `ports` by appending: an override file saying
`8002:80` would leave you with 80, 443 *and* 8002. `bin/apply-local-patches.py`
replays them after an upgrade overwrites the file.
`compose.override.yml` holds purely *additive* settings (currently
`TRUST_ALL_PROXIES` for the three Django services). Compose merges it
automatically and no upgrade ever touches it, so nothing needs replaying.
Rule of thumb: if you're adding an env var, put it in the override file. If
you're changing a port or a network, patch compose.yml and add it to
`bin/apply-local-patches.py`. `command` is the exception: Compose replaces it
on merge rather than appending, so the worker's command lives in the override.
## Operating it
docker compose up -d # start
docker compose stop # stop (keeps volumes)
docker compose logs -f web # watch migrations / app
docker compose ps # status
bin/upgrade # upgrade (runs async migration checks first)
First boot runs Django + ClickHouse migrations and takes 5-15 minutes before
`/_health` returns 200. The first person to visit the site creates the admin
account, so log in promptly.
## Multiple projects (source patches)
Without a license, upstream PostHog allows one project per org and hides the
"New project" button. PostHog no longer sells self-hosted licenses, so
`compose/local-patches.py` patches two files in `/code` when `web`, `worker`
and `temporal-django-worker` start:
- `posthog/utils.py`: `can_create_org` follows `MULTI_ORG_ENABLED` alone, which
shows the button.
- `posthog/models/organization.py`: the org keeps the `organizations_projects`
feature, which the project API checks. Without this, the hourly
`sync_all_organization_available_product_features` task resets the org's
features to `[]` and the API refuses a second project.
Both are gated on `MULTI_ORG_ENABLED` in `compose.override.yml`. Because they run
at container start, recreating a container no longer loses them. An earlier
version patched running containers by hand and was wiped when the containers
were recreated on 2026-09-22.
If upstream changes the code, the script logs `PATTERN NOT FOUND` and the app
starts unpatched. Check after upgrades (`bin/upgrade` prints it):
docker compose logs web worker | grep LOCAL-PATCH
## Rebooting the host
A host boot is not `docker compose up`. Dockerd starts containers from each one's
restart policy alone, in no order, and never reads `depends_on`, so the dependency
ordering the stack is built around does not exist at boot time.
Upstream ships `restart: no` or `on-failure` on nearly every service, and
`on-failure` does not retry a clean exit. Several services exit *zero* on purpose:
`hypercache-server` when it loses the race to `redis7` on a cold boot, and
`capture` when it loses Kafka (on 2026-09-27 a host memory spike stalled Kafka,
capture shut down gracefully, and it stayed down for three days). So
`compose.override.yml` sets `restart: unless-stopped` on every long-running
service. A manual `docker compose stop` is still respected.
Expect `hypercache-server` to flap a few times early in boot until `redis7`
answers. That is the retry working, not a fault.
`kafka-init` sitting at `Exited (0)` is correct. It is a one-shot init job and
must keep `restart: no`.
The web UI serves fine while these four are missing, so check
`docker ps -a --filter name=posthog` for anything not `Up` before assuming a
reboot went cleanly. To bring back anything stopped, in dependency order, without
recreating running containers:
docker compose up -d --no-recreate
## Watchtower is deliberately not enabled
Every other stack here carries `com.centurylinklabs.watchtower.enable=true`.
PostHog does not, on purpose: its upgrades run schema migrations that must be
checked with `asyncmigrationscheck` before the app restarts, and a silent image
swap underneath a running ClickHouse/Kafka pipeline can corrupt data. Use
`bin/upgrade` instead.
## Resource notes
This stack is 37 containers (ClickHouse, Kafka, Zookeeper, Postgres, Redis,
Valkey, Temporal, Elasticsearch, SeaweedFS, Browserless, and ~12 Rust/Node
consumers). PostHog asks for 8 GB minimum; with Sentry already running here,
keep an eye on `free -h`.
If memory gets tight, these can be commented out of `compose.yml` with little
loss on a personal instance:
- `elasticsearch` — Temporal is configured with `ENABLE_ES=false`, so it is
only there for Temporal's advanced visibility. Saves ~1 GB.
- `temporal-ui` — debugging UI for Temporal workflows only.
- `cymbal` / `cymbal-resolution` — error-tracking stack-trace symbolication.
Skip if you're keeping errors in Sentry rather than PostHog.
## Ad blockers and the hostname
The host is `analytics.gbrown.org`. It was originally `posthog.gbrown.org`, which
uBlock Origin blocked, leaving the dashboard a blank white page.
The culprit is EasyPrivacy line 2964 (EasyPrivacy is enabled by default in uBO):
://posthog.$script,domain=~posthog.com
That is an *unanchored substring* pattern, not a domain rule, so any URL
containing `://posthog.` matches -- including `https://posthog.gbrown.org/...`.
The `$script` option is why the symptom was a white page rather than a failed
connection: the HTML document is a `document` request and loaded fine, but every
`/static/*.js` was blocked, so React never mounted.
Renaming fixes it. Checked against all five lists uBO enables by default
(EasyPrivacy, EasyList, uBO filters/privacy/badware, ~160k rules), with controls
validating in both directions:
posthog.gbrown.org BLOCKED (rule above)
analytics.gbrown.org clean
stats/insights/metrics clean
bills.gbrown.org clean (control)
Also clean on the new host: `/static/array.js`, `/static/recorder.js`,
`/static/recorder-v2.js`, `/e/`, `/i/v0/e/`, `/batch/`, `/capture/`, `/decide/`.
Generic words like "analytics" survive because the lists anchor to specific
vendor domains (`||google-analytics.com`) rather than bare words -- blocking the
word outright would break the web. The `://posthog.` rule is the unusual one.
### To change the hostname again
1. Edit `DOMAIN` in `.env`.
2. Add the proxy host in nginx-proxy-manager: http -> `posthog` -> 80,
websockets on, `client_max_body_size 50m;`.
3. `docker compose up -d`.
Caddy listens on `:80` regardless of hostname, so nothing inside the stack needs
touching -- that is the payoff of terminating TLS at NPM instead of in Caddy.
Note that `SITE_URL` feeds `CSRF_TRUSTED_ORIGINS`, so existing browser sessions
may need a fresh login after a rename. Capture/API traffic is unaffected.
Still worth proxying capture through the app's own origin (a Next.js `/ingest`
rewrite): it is same-origin, so no CORS and no preflight, and it survives future
filter-list changes regardless of what this host is called.
## Capture settings for the bill-tracker app (decided 2026-09-17)
The bill tracker (Next + Expo + Convex) sends to this instance. Gib asked for
everything on, so both clients ship with session replay, autocapture, heatmaps,
console log capture and exception capture ENABLED, on posthog-js defaults:
maskAllInputs: true (default — form inputs are masked)
rendered text: NOT masked (default)
The tradeoff, recorded deliberately rather than discovered later: this is a
finance app, so bill amounts, balances, payee names and due dates are rendered
*text*, not inputs, and therefore land in session replays verbatim. Autocapture
also stores the text of clicked elements, so a button reading "Pay $1,247.33 to
Chase" becomes an event property. Console log capture ships whatever the app
logs, which in dev is often a whole transaction object.
This is considered acceptable because the instance is self-hosted on Gib's own
hardware and serves his own household — the data never leaves the house.
To tighten later:
maskTextSelector: '*' // or scope to the money elements
class="ph-no-capture" // on balance / amount / payee nodes
One asymmetry worth remembering: tightening does NOT retroactively clean stored
replays. Anything recorded before the change stays recorded, so the cheap moment
to decide is before real household data flows, not after.
Client config lives in the bill-tracker repo, not here. The host is read from
NEXT_PUBLIC_POSTHOG_HOST / EXPO_PUBLIC_POSTHOG_HOST (Infisical), and web routes
capture through a same-origin /ingest rewrite rather than hitting this host
directly.
## Blocked asset filenames — why the client requests ext-a.js / ext-b.js
Renaming the host to `analytics.gbrown.org` fixed the dashboard, but two of
PostHog's own script filenames are blocked by rules that match the FILENAME, not
the host, so a reverse proxy cannot dodge them:
easyprivacy.txt line 1906: /posthog-recorder.js
easyprivacy.txt line 809: /dead-clicks-autocapture.js
Both are unanchored substrings with no type or domain options, so they match on
any host, any request type, first-party included. Confirmed blocked both direct
and through the app's /ingest proxy. Symptom: session replay and dead-click
capture silently never start for anyone running uBlock.
The bill-tracker app works around this with posthog-js's
`prepare_external_dependency_script` hook, serving them under inert names:
posthog-recorder.js -> ext-a.js
dead-clicks-autocapture.js -> ext-b.js
(`strict_script_versioning` off, so only the plain `/static/<name>.js?v=` form is
requested.) The Next rewrite maps the disguised names back. Verified: replay
blobs land in `posthog.session_replay_events`, which was stuck at 0 before.
Everything else PostHog serves is clean on both paths — `array.js`,
`array.full.js`, `recorder.js`, `recorder-v2.js`, `surveys.js`,
`exception-autocapture.js`, `web-vitals-with-attribution.js`,
`tracing-headers.js`, `toolbar.js`, `external-scripts-loader.js`, `/e/`,
`/flags/`. Note the legacy `recorder.js` and `recorder-v2.js` names are NOT
listed; only the newer `posthog-recorder.js` is.
One more, shape-dependent: `/s/array/<token>/config` is blocked by
ubo-privacy.txt line 3159 (`$script,xhr,1p` — first-party too), but
`/array/<token>/config.js` is clean. posthog-js 1.433.10 requests the latter, so
it does not bite today. Worth re-checking after a major posthog-js upgrade.
### Expected absences (not bugs)
- `$pageleave` does NOT fire on client-side SPA navigation. posthog-js hangs it
off the real unload/visibilitychange lifecycle, so it tracks actual page exits
only. Verified against every pageleave in the instance.
- `$web_vitals` flushes on page hide; `$dead_click` needs a click that resolves
to nothing. Both stay absent until those conditions occur.
## nginx-proxy-manager buffer sizes (required, or replay breaks)
Symptom: the Session Replay list page shows "Error while trying to load
recordings". `GET /api/environments/<id>/session_recordings` returns 502 from
openresty in ~130ms, while a SINGLE recording
(`.../session_recordings/<session_id>`) returns 200 and `/events` returns 200.
Cause is nginx, not PostHog. From `/data/logs/proxy-host-121_error.log`:
upstream sent too big header while reading response header from upstream
nginx's default `proxy_buffer_size` is 4-8KB. PostHog's session_recordings LIST
endpoint sends response headers larger than that, so nginx aborts before reading
the body and returns 502. The request never reaches Django, which is why nothing
appears in the web container logs and Caddy logs no 5xx at all. The single
recording endpoint sends normal-sized headers, hence the asymmetry.
Required in the NPM Advanced tab for this proxy host:
client_max_body_size 50m;
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;
The Sentry proxy host (1.conf) already carried the same three buffer directives —
same nginx default, same fix, just never applied to PostHog's host.
Verified in `/data/logs/proxy-host-121_access.log`: 502 at 22:46:06 before the
change, 200 at 22:46:36 after, no further "too big header" entries.
Do NOT edit the generated `.conf` files directly — NPM regenerates them from its
own database, so a direct edit reverts on the next save.
### Verifying replay from the server side
ClickHouse `session_replay_events` proves events ARRIVED; it does not prove they
can be PLAYED BACK. Those are different paths (ingestion -> Kafka -> object
storage vs. API -> proxy -> browser). To check the storage side:
docker compose exec seaweedfs sh -c \
"echo 'fs.ls -l /buckets/posthog/session_recordings/5y' | weed shell -master=localhost:9333"
Blobs live under `session_recordings/<retention>/`, named `<epoch_ms>-<hash>`.
## "No recordings found" with replay working fine
The Session Replay list ships a DEFAULT filter of `> 5 active seconds`. Any
session with 5 or fewer *active* seconds is hidden, even though it recorded
perfectly and plays back fine.
Active time is not wall-clock time. The first verified session here ran 458s
(7.6 min) but logged only 643ms of activity — one click and a little mouse
movement in an otherwise idle tab. PostHog counts interaction, not duration,
deliberately, to suppress bounces and accidental loads.
That default makes quick dev checks vanish and looks identical to "replay is
broken". To see them: open the duration chip on /replay/home and set the
condition to `> 0 active seconds`, or save a filter preset. Date range and the
"filter out internal and test users" toggle are usually NOT the culprit.
Check the real numbers server-side before assuming anything is missing:
docker compose exec clickhouse clickhouse-client -q \
"select session_id, sum(active_milliseconds) active_ms,
dateDiff('second', min(min_first_timestamp), max(max_last_timestamp)) duration_s,
sum(click_count) clicks, sum(event_count) evts
from posthog.session_replay_events group by session_id"
Verified 2026-09-17: session 01a0b177-d35c-7292-ac59-021af608d29a recorded,
listed once the threshold was lowered, and played back in the browser.
## Real client IPs through NPM -> Caddy
By default Django logged `x_forwarded_for=172.25.0.8` — nginx-proxy-manager's own
container IP — so every event's `$ip` resolved to the proxy and GeoIP failed on a
private address.
NPM was not at fault; it sends `X-Forwarded-For $proxy_add_x_forwarded_for` and
`X-Real-IP $remote_addr` from its global proxy include. The loss was in Caddy:
its global options block was empty, so `trusted_proxies` was unset, and Caddy's
default is to distrust an incoming X-Forwarded-For and OVERWRITE it with the
direct peer. Proven by sending `X-Forwarded-For: 9.9.9.9` to Caddy and watching
Django log `172.25.0.1`.
Fixed in `.env` — no compose patch, because `CADDY_TLS_BLOCK` is substituted
directly into Caddy's global options block:
CADDY_TLS_BLOCK="servers {
trusted_proxies static 172.25.0.0/24
}"
Multi-line values work in `.env`; `docker compose config` resolves it correctly
and `caddy validate` accepts it. Apply with `docker compose up -d proxy`.
Verified after: a spoofed `X-Forwarded-For: 9.9.9.9` arrives as `ip=9.9.9.9`, and
a real request through NPM arrives as the true client address.
TRADEOFF: `/24` trusts every container on nginx-bridge (~42 of them), any of
which could forge an X-Forwarded-For that Caddy now believes — compounded by
`TRUST_ALL_PROXIES=true` on Django (see compose.override.yml). Acceptable for a
LAN-only household instance; tighten to NPM's address if this is ever exposed to
the internet. Pinning `172.25.0.8/32` is narrower but breaks silently whenever
Docker reassigns NPM's IP.
Note this does NOT produce geolocation: LAN clients are 192.168.x.x, which is not
in the MaxMind database, so GeoIP still errors. What it buys is the ability to
tell devices apart.