Panama/server/home/PostHog

PostHog (self-hosted) on Fedora

PostHog's official installer (bin/deploy-hobby) assumes Ubuntu/Debian and a bare host. This directory is that install done by hand, adapted to this server.

Why the official installer doesn't work here

What deploy-hobby does	Why it fails / what we did instead
apt update, apt install git brotli curl	Not Debian. All three are already installed via dnf.
Installs Docker from download.docker.com/linux/ubuntu	Docker 29.8 is already installed.
Downloads the v1 docker-compose binary to /usr/local/bin	We use the Compose v5 plugin (docker compose).
Runs Caddy on host ports 80:80 / 443:443 and gets its own Let's Encrypt certs	nginx-proxy-manager owns 80/443. Caddy now serves plain HTTP on 127.0.0.1:8002 and NPM terminates TLS in front of it.
Publishes Temporal on 0.0.0.0:7233 and its UI on 0.0.0.0:8081	Rebound to 127.0.0.1.
Phones home to PostHog Cloud with your domain	Skipped.
Generates compose/start calling ./compose/wait	Upstream bug, fixed here. WORKDIR is /code but the mount is at /compose, so the relative path never resolves. Harmless (bin/migrate retries 10x) but it meant the readiness check never ran. Ours calls /compose/wait.

Note: Docker here runs without SELinux labelling (docker info lists only seccomp and cgroupns), so the bind mounts need no :z / :Z relabelling even though the host is SELinux-enforcing.

Caddy is kept rather than replaced by NPM because it does path-based routing that NPM can't express in one proxy host: /e, /i/v0, /batch, /capture go to capture; /s to replay-capture; /i/v1/logs to capture-logs; /livestream to livestream; /posthog/* to object storage; everything else to web. NPM just forwards the whole hostname to Caddy.

Layout

compose.yml               docker-compose.hobby.yml + local patches
compose.override.yml      additive local settings, auto-merged by compose
docker-compose.base.yml   verbatim from the repo (compose.yml `extends` it)
.env                      secrets + domain  (chmod 600, never regenerate)
.env.services             verbatim from the repo
posthog/                  upstream checkout, only used for config bind mounts
share/                    GeoLite2-City.mmdb
compose/                  entrypoint scripts the web/worker containers run,
                          plus local-patches.py (see "Multiple projects" below)
bin/upgrade               Fedora-native replacement for bin/upgrade-hobby
bin/apply-local-patches.py  replays our compose.yml deltas after an upgrade

nginx-proxy-manager

Add a proxy host:

• Domain: posthog.gbrown.org
• Scheme: http, Forward host: posthog, Forward port: 80
• Websockets: ON (required — the /livestream endpoint)
• Request an SSL cert, force SSL

posthog is a network alias on nginx-bridge for the Caddy container.

Also worth raising NPM's client_max_body_size for this host (Advanced tab: client_max_body_size 50m;) — session-recording batches and source-map uploads exceed the 1 MB default.

Two config files, two reasons

compose.yml holds changes that replace upstream values -- the proxy's ports and networks. These have to be patched into the file directly, because Compose merges list-valued keys like ports by appending: an override file saying 8002:80 would leave you with 80, 443 and 8002. bin/apply-local-patches.py replays them after an upgrade overwrites the file.

compose.override.yml holds purely additive settings (currently TRUST_ALL_PROXIES for the three Django services). Compose merges it automatically and no upgrade ever touches it, so nothing needs replaying.

Rule of thumb: if you're adding an env var, put it in the override file. If you're changing a port or a network, patch compose.yml and add it to bin/apply-local-patches.py. command is the exception: Compose replaces it on merge rather than appending, so the worker's command lives in the override.

Operating it

docker compose up -d           # start
docker compose stop            # stop (keeps volumes)
docker compose logs -f web     # watch migrations / app
docker compose ps              # status
bin/upgrade                    # upgrade (runs async migration checks first)

First boot runs Django + ClickHouse migrations and takes 5-15 minutes before /_health returns 200. The first person to visit the site creates the admin account, so log in promptly.

Multiple projects (source patches)

Without a license, upstream PostHog allows one project per org and hides the "New project" button. PostHog no longer sells self-hosted licenses, so compose/local-patches.py patches two files in /code when web, worker and temporal-django-worker start:

• posthog/utils.py: can_create_org follows MULTI_ORG_ENABLED alone, which shows the button.
• posthog/models/organization.py: the org keeps the organizations_projects feature, which the project API checks. Without this, the hourly sync_all_organization_available_product_features task resets the org's features to [] and the API refuses a second project.

Both are gated on MULTI_ORG_ENABLED in compose.override.yml. Because they run at container start, recreating a container no longer loses them. An earlier version patched running containers by hand and was wiped when the containers were recreated on 2026-09-22.

If upstream changes the code, the script logs PATTERN NOT FOUND and the app starts unpatched. Check after upgrades (bin/upgrade prints it):

docker compose logs web worker | grep LOCAL-PATCH

Rebooting the host

A host boot is not docker compose up. Dockerd starts containers from each one's restart policy alone, in no order, and never reads depends_on, so the dependency ordering the stack is built around does not exist at boot time.

Upstream ships restart: no or on-failure on nearly every service, and on-failure does not retry a clean exit. Several services exit zero on purpose: hypercache-server when it loses the race to redis7 on a cold boot, and capture when it loses Kafka (on 2026-09-27 a host memory spike stalled Kafka, capture shut down gracefully, and it stayed down for three days). So compose.override.yml sets restart: unless-stopped on every long-running service. A manual docker compose stop is still respected.

Expect hypercache-server to flap a few times early in boot until redis7 answers. That is the retry working, not a fault.

kafka-init sitting at Exited (0) is correct. It is a one-shot init job and must keep restart: no.

The web UI serves fine while these four are missing, so check docker ps -a --filter name=posthog for anything not Up before assuming a reboot went cleanly. To bring back anything stopped, in dependency order, without recreating running containers:

docker compose up -d --no-recreate

Watchtower is deliberately not enabled

Every other stack here carries com.centurylinklabs.watchtower.enable=true. PostHog does not, on purpose: its upgrades run schema migrations that must be checked with asyncmigrationscheck before the app restarts, and a silent image swap underneath a running ClickHouse/Kafka pipeline can corrupt data. Use bin/upgrade instead.

Resource notes

This stack is 37 containers (ClickHouse, Kafka, Zookeeper, Postgres, Redis, Valkey, Temporal, Elasticsearch, SeaweedFS, Browserless, and ~12 Rust/Node consumers). PostHog asks for 8 GB minimum; with Sentry already running here, keep an eye on free -h.

If memory gets tight, these can be commented out of compose.yml with little loss on a personal instance:

• elasticsearch — Temporal is configured with ENABLE_ES=false, so it is only there for Temporal's advanced visibility. Saves ~1 GB.
• temporal-ui — debugging UI for Temporal workflows only.
• cymbal / cymbal-resolution — error-tracking stack-trace symbolication. Skip if you're keeping errors in Sentry rather than PostHog.

Ad blockers and the hostname

The host is analytics.gbrown.org. It was originally posthog.gbrown.org, which uBlock Origin blocked, leaving the dashboard a blank white page.

The culprit is EasyPrivacy line 2964 (EasyPrivacy is enabled by default in uBO):

://posthog.$script,domain=~posthog.com

That is an unanchored substring pattern, not a domain rule, so any URL containing ://posthog. matches -- including https://posthog.gbrown.org/.... The $script option is why the symptom was a white page rather than a failed connection: the HTML document is a document request and loaded fine, but every /static/*.js was blocked, so React never mounted.

Renaming fixes it. Checked against all five lists uBO enables by default (EasyPrivacy, EasyList, uBO filters/privacy/badware, ~160k rules), with controls validating in both directions:

posthog.gbrown.org      BLOCKED (rule above)
analytics.gbrown.org    clean
stats/insights/metrics  clean
bills.gbrown.org        clean (control)

Also clean on the new host: /static/array.js, /static/recorder.js, /static/recorder-v2.js, /e/, /i/v0/e/, /batch/, /capture/, /decide/.

Generic words like "analytics" survive because the lists anchor to specific vendor domains (||google-analytics.com) rather than bare words -- blocking the word outright would break the web. The ://posthog. rule is the unusual one.

To change the hostname again

1. Edit DOMAIN in .env.
2. Add the proxy host in nginx-proxy-manager: http -> posthog -> 80, websockets on, client_max_body_size 50m;.
3. docker compose up -d.

Caddy listens on :80 regardless of hostname, so nothing inside the stack needs touching -- that is the payoff of terminating TLS at NPM instead of in Caddy. Note that SITE_URL feeds CSRF_TRUSTED_ORIGINS, so existing browser sessions may need a fresh login after a rename. Capture/API traffic is unaffected.

Still worth proxying capture through the app's own origin (a Next.js /ingest rewrite): it is same-origin, so no CORS and no preflight, and it survives future filter-list changes regardless of what this host is called.

Capture settings for the bill-tracker app (decided 2026-09-17)

The bill tracker (Next + Expo + Convex) sends to this instance. Gib asked for everything on, so both clients ship with session replay, autocapture, heatmaps, console log capture and exception capture ENABLED, on posthog-js defaults:

maskAllInputs: true        (default — form inputs are masked)
rendered text: NOT masked  (default)

The tradeoff, recorded deliberately rather than discovered later: this is a finance app, so bill amounts, balances, payee names and due dates are rendered text, not inputs, and therefore land in session replays verbatim. Autocapture also stores the text of clicked elements, so a button reading "Pay $1,247.33 to Chase" becomes an event property. Console log capture ships whatever the app logs, which in dev is often a whole transaction object.

This is considered acceptable because the instance is self-hosted on Gib's own hardware and serves his own household — the data never leaves the house.

To tighten later:

maskTextSelector: '*'                  // or scope to the money elements
class="ph-no-capture"                  // on balance / amount / payee nodes

One asymmetry worth remembering: tightening does NOT retroactively clean stored replays. Anything recorded before the change stays recorded, so the cheap moment to decide is before real household data flows, not after.

Client config lives in the bill-tracker repo, not here. The host is read from NEXT_PUBLIC_POSTHOG_HOST / EXPO_PUBLIC_POSTHOG_HOST (Infisical), and web routes capture through a same-origin /ingest rewrite rather than hitting this host directly.

Blocked asset filenames — why the client requests ext-a.js / ext-b.js

Renaming the host to analytics.gbrown.org fixed the dashboard, but two of PostHog's own script filenames are blocked by rules that match the FILENAME, not the host, so a reverse proxy cannot dodge them:

easyprivacy.txt line 1906:   /posthog-recorder.js
easyprivacy.txt line  809:   /dead-clicks-autocapture.js

Both are unanchored substrings with no type or domain options, so they match on any host, any request type, first-party included. Confirmed blocked both direct and through the app's /ingest proxy. Symptom: session replay and dead-click capture silently never start for anyone running uBlock.

The bill-tracker app works around this with posthog-js's prepare_external_dependency_script hook, serving them under inert names:

posthog-recorder.js        -> ext-a.js
dead-clicks-autocapture.js -> ext-b.js

(strict_script_versioning off, so only the plain /static/<name>.js?v= form is requested.) The Next rewrite maps the disguised names back. Verified: replay blobs land in posthog.session_replay_events, which was stuck at 0 before.

Everything else PostHog serves is clean on both paths — array.js, array.full.js, recorder.js, recorder-v2.js, surveys.js, exception-autocapture.js, web-vitals-with-attribution.js, tracing-headers.js, toolbar.js, external-scripts-loader.js, /e/, /flags/. Note the legacy recorder.js and recorder-v2.js names are NOT listed; only the newer posthog-recorder.js is.

One more, shape-dependent: /s/array/<token>/config is blocked by ubo-privacy.txt line 3159 ($script,xhr,1p — first-party too), but /array/<token>/config.js is clean. posthog-js 1.433.10 requests the latter, so it does not bite today. Worth re-checking after a major posthog-js upgrade.

Expected absences (not bugs)

• $pageleave does NOT fire on client-side SPA navigation. posthog-js hangs it off the real unload/visibilitychange lifecycle, so it tracks actual page exits only. Verified against every pageleave in the instance.
• $web_vitals flushes on page hide; $dead_click needs a click that resolves to nothing. Both stay absent until those conditions occur.

nginx-proxy-manager buffer sizes (required, or replay breaks)

Symptom: the Session Replay list page shows "Error while trying to load recordings". GET /api/environments/<id>/session_recordings returns 502 from openresty in ~130ms, while a SINGLE recording (.../session_recordings/<session_id>) returns 200 and /events returns 200.

Cause is nginx, not PostHog. From /data/logs/proxy-host-121_error.log:

upstream sent too big header while reading response header from upstream

nginx's default proxy_buffer_size is 4-8KB. PostHog's session_recordings LIST endpoint sends response headers larger than that, so nginx aborts before reading the body and returns 502. The request never reaches Django, which is why nothing appears in the web container logs and Caddy logs no 5xx at all. The single recording endpoint sends normal-sized headers, hence the asymmetry.

Required in the NPM Advanced tab for this proxy host:

client_max_body_size 50m;
proxy_buffering on;
proxy_buffer_size 128k;
proxy_buffers 4 256k;

The Sentry proxy host (1.conf) already carried the same three buffer directives — same nginx default, same fix, just never applied to PostHog's host.

Verified in /data/logs/proxy-host-121_access.log: 502 at 22:46:06 before the change, 200 at 22:46:36 after, no further "too big header" entries.

Do NOT edit the generated .conf files directly — NPM regenerates them from its own database, so a direct edit reverts on the next save.

Verifying replay from the server side

ClickHouse session_replay_events proves events ARRIVED; it does not prove they can be PLAYED BACK. Those are different paths (ingestion -> Kafka -> object storage vs. API -> proxy -> browser). To check the storage side:

docker compose exec seaweedfs sh -c \
  "echo 'fs.ls -l /buckets/posthog/session_recordings/5y' | weed shell -master=localhost:9333"

Blobs live under session_recordings/<retention>/, named <epoch_ms>-<hash>.

"No recordings found" with replay working fine

The Session Replay list ships a DEFAULT filter of > 5 active seconds. Any session with 5 or fewer active seconds is hidden, even though it recorded perfectly and plays back fine.

Active time is not wall-clock time. The first verified session here ran 458s (7.6 min) but logged only 643ms of activity — one click and a little mouse movement in an otherwise idle tab. PostHog counts interaction, not duration, deliberately, to suppress bounces and accidental loads.

That default makes quick dev checks vanish and looks identical to "replay is broken". To see them: open the duration chip on /replay/home and set the condition to > 0 active seconds, or save a filter preset. Date range and the "filter out internal and test users" toggle are usually NOT the culprit.

Check the real numbers server-side before assuming anything is missing:

docker compose exec clickhouse clickhouse-client -q \
  "select session_id, sum(active_milliseconds) active_ms,
          dateDiff('second', min(min_first_timestamp), max(max_last_timestamp)) duration_s,
          sum(click_count) clicks, sum(event_count) evts
   from posthog.session_replay_events group by session_id"

Verified 2026-09-17: session 01a0b177-d35c-7292-ac59-021af608d29a recorded, listed once the threshold was lowered, and played back in the browser.

Real client IPs through NPM -> Caddy

By default Django logged x_forwarded_for=172.25.0.8 — nginx-proxy-manager's own container IP — so every event's $ip resolved to the proxy and GeoIP failed on a private address.

NPM was not at fault; it sends X-Forwarded-For $proxy_add_x_forwarded_for and X-Real-IP $remote_addr from its global proxy include. The loss was in Caddy: its global options block was empty, so trusted_proxies was unset, and Caddy's default is to distrust an incoming X-Forwarded-For and OVERWRITE it with the direct peer. Proven by sending X-Forwarded-For: 9.9.9.9 to Caddy and watching Django log 172.25.0.1.

Fixed in .env — no compose patch, because CADDY_TLS_BLOCK is substituted directly into Caddy's global options block:

CADDY_TLS_BLOCK="servers {
	trusted_proxies static 172.25.0.0/24
}"

Multi-line values work in .env; docker compose config resolves it correctly and caddy validate accepts it. Apply with docker compose up -d proxy.

Verified after: a spoofed X-Forwarded-For: 9.9.9.9 arrives as ip=9.9.9.9, and a real request through NPM arrives as the true client address.

TRADEOFF: /24 trusts every container on nginx-bridge (~42 of them), any of which could forge an X-Forwarded-For that Caddy now believes — compounded by TRUST_ALL_PROXIES=true on Django (see compose.override.yml). Acceptable for a LAN-only household instance; tighten to NPM's address if this is ever exposed to the internet. Pinning 172.25.0.8/32 is narrower but breaks silently whenever Docker reassigns NPM's IP.

Note this does NOT produce geolocation: LAN clients are 192.168.x.x, which is not in the MaxMind database, so GeoIP still errors. What it buys is the ability to tell devices apart.