Sync both server snapshots with what the live machines run, and keep their backup tooling
This commit is contained in:
Gabriel Brown committed
1 parent
95fa480469
commit
cb089b39ea
42 files changed
+3099
-21
No files matched your search
@@ -19,7 +19,7 @@ x-runner-env: &runner-env
|
||||
TZ: America/New_York
|
||||
GITEA_INSTANCE_URL: https://git.gbrown.org
|
||||
GITEA_RUNNER_REGISTRATION_TOKEN: ${GITEA_RUNNER_REGISTRATION_TOKEN}
|
||||
GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04"
|
||||
GITEA_RUNNER_LABELS: "ubuntu-latest:docker://docker.gitea.com/runner-images:ubuntu-latest,ubuntu-24.04:docker://docker.gitea.com/runner-images:ubuntu-24.04,ubuntu-22.04:docker://docker.gitea.com/runner-images:ubuntu-22.04,build:docker://docker.gitea.com/runner-images:ubuntu-latest"
|
||||
DOCKER_TLS_VERIFY: "1"
|
||||
DOCKER_CERT_PATH: /certs/client
|
||||
CONFIG_FILE: /data/config.yml
|
||||
|
||||
@@ -18,7 +18,7 @@ services:
|
||||
# restart: unless-stopped
|
||||
|
||||
portainer-agent:
|
||||
image: portainer/agent:2.33.2
|
||||
image: portainer/agent:latest
|
||||
container_name: portainer-agent
|
||||
hostname: portainer-agent
|
||||
networks:
|
||||
|
||||
@@ -43,8 +43,8 @@ volumes:
|
||||
|
||||
services:
|
||||
gbrown-backend:
|
||||
# Pinned to the build the VPS Convex stacks already run. Bump on purpose.
|
||||
image: ghcr.io/get-convex/convex-backend@sha256:d715e9ec088784407ca4ba2d3db592702cd328d02c76cdca3852c0018f2a76b4
|
||||
# Pinned to the build the VPS Convex stacks run (bumped 2026-10-28). Bump on purpose.
|
||||
image: ghcr.io/convex/convex-backend@sha256:207cf95def42f01614b6e3af0af5c20d0dcaa5ff2adf258ee7bc9555576644fa
|
||||
container_name: gbrown-backend
|
||||
hostname: gbrown-backend
|
||||
networks: ['nginx-bridge']
|
||||
@@ -70,7 +70,7 @@ services:
|
||||
|
||||
gbrown-dashboard:
|
||||
# Released alongside the backend; pinned with it.
|
||||
image: ghcr.io/get-convex/convex-dashboard@sha256:f85cf0d0448b9c835ae3df5c7c1c6f0145dd4c8f0f92eb0245daf304efdd9f1a
|
||||
image: ghcr.io/convex/convex-dashboard@sha256:a55bf8abca1332f787bc4d4a79850182175027be7b70aa230be4389a1d3268e2
|
||||
container_name: gbrown-dashboard
|
||||
hostname: gbrown-dashboard
|
||||
networks: ['nginx-bridge']
|
||||
|
||||
@@ -0,0 +1,109 @@
|
||||
# compact-tv
|
||||
|
||||
Sonarr and Radarr each have exactly four quality profiles. Two of them (**Standard** and
|
||||
**Mini**) also have their files shrunk by `compact-tv.py`, which runs every night.
|
||||
|
||||
| Profile | Used for | What gets kept |
|
||||
|---|---|---|
|
||||
| **Mini** | shows I demote hard | TV: 720p at 800 kbps (~0.42 GB per hour). Movies: 720p at 1500 kbps |
|
||||
| **Standard** | **every request from other users (Seerr default)**, and anything I demote | TV: 1080p at 1200 kbps (~0.6 GB per hour). Movies: 1080p at 2500 kbps (~1.2 GB per hour) |
|
||||
| **HD** | my own library, unchanged | whatever downloads, up to 1080p |
|
||||
| **UHD** | only me, through Seerr's 4K servers | 4K |
|
||||
|
||||
SD sources keep their resolution, at 600 kbps for TV and 900 kbps for movies. Movies get
|
||||
higher rates because dark scenes, grain and action break down first at low bitrates. Nothing is ever
|
||||
upscaled. Audio becomes Opus: 96 kbps for stereo, 160 for 5.1, 192 for 7.1. Subtitles,
|
||||
chapters and attachments are copied as they are. I picked the rates from sample encodes of
|
||||
Shameless S05E06 (VMAF ~91 for Standard, ~86 for Mini).
|
||||
|
||||
## How the profiles work
|
||||
|
||||
- **Standard and Mini** are copies of HD that upgrade only toward small releases (see below),
|
||||
so Sonarr and Radarr never replace a file with a bigger one.
|
||||
- **Small downloads first.** Both profiles prefer small releases when they grab:
|
||||
- The "Small Encode" format (MeGusta, PSA, ELiTE, GalaxyTV/RG, YTS/YIFY) scores +3000.
|
||||
- LQ scores 0. In HD and UHD it scores -10000, which keeps those groups out of HD.
|
||||
- Standard merges HDTV/WEB/Bluray at each resolution, so the format score decides between
|
||||
sources rather than the source type.
|
||||
- Mini merges 720p and 1080p into one group and adds +1000 for 720p.
|
||||
- Many files therefore arrive already small, and the job skips them.
|
||||
- **Minimum sizes are shared by every profile.** For 720p I lowered them to 5 MB per minute
|
||||
and for 1080p to 6 MB per minute (8 for Bluray-1080p), so small encodes are not rejected.
|
||||
HD still never takes the LQ groups.
|
||||
- **Seerr:**
|
||||
- The "Sonarr - HD" and "Radarr" servers default to Standard. Only my admin account sees
|
||||
the profile picker, so other users always get Standard. I pick HD or UHD per request.
|
||||
- Other users no longer have "Request 4K".
|
||||
- **Configarr is opt-in** (`docker compose --profile configarr up configarr`). Its templates
|
||||
would recreate the old TRaSH profiles and size limits.
|
||||
- Backups from before all this: `backups/pre-four-profiles-20261028/`. That covers both
|
||||
apps' profiles, custom formats and size definitions, which title was on which profile,
|
||||
and Seerr's settings. `setup-small-profiles.py` reapplies the Standard and Mini settings.
|
||||
|
||||
## How the job works
|
||||
|
||||
The nightly run (00:30 to 08:00) has two phases.
|
||||
|
||||
**1. Search first.** It asks Sonarr and Radarr to search titles on Standard or Mini that it
|
||||
hasn't searched before, biggest first: 5 shows and 20 movies a night, so downloads and
|
||||
indexer use stay spread out. Standard and Mini upgrade only toward small releases:
|
||||
- Every allowed quality sits in one group, so no resolution or source upgrade can happen.
|
||||
- An upgrade needs a custom-format score jump of at least 2500, and upgrades stop at 3000.
|
||||
Only gaining "Small Encode" clears that bar.
|
||||
|
||||
So a search replaces a big file only with a small encode, never with a big HEVC release.
|
||||
After that, the apps' regular RSS checks keep picking up small releases on their own.
|
||||
|
||||
**2. Encode what's left.** A title becomes eligible for encoding 7 days after its search,
|
||||
and only its files that are still big get encoded. Files that arrive after the title's
|
||||
search work differently:
|
||||
- This covers new episodes of ongoing shows, and fresh requests.
|
||||
- The first night they're seen, they get their own search (one episode or movie at a time).
|
||||
- They're encoded only if they're still big 3 days later.
|
||||
|
||||
Downloads never wait for any of this, so new episodes and requests land in the library as
|
||||
soon as they download. Only encoding waits. For each file, the job encodes to
|
||||
`~/.cache/compact-tv/work.mkv` on the NVMe and then checks the result:
|
||||
- the duration matches the original
|
||||
- the audio and subtitle track counts match
|
||||
- the file is smaller
|
||||
|
||||
It then renames the new file over the original, keeping the same name with `.mkv`, and asks
|
||||
Sonarr or Radarr to rescan. Plex picks the change up through its folder watcher, and the
|
||||
episode keeps its Plex ID, so watch history stays. If a small release shows up later, the
|
||||
app may still replace an encoded file with it, which costs bandwidth, not GPU.
|
||||
|
||||
Other details:
|
||||
- Each output carries a `COMPACT_TIER` tag. A file is skipped if re-encoding would save less
|
||||
than 30%. Ruled-out files are cached by size in `state.json`, so they aren't probed every
|
||||
night. Search dates and failures are kept there too.
|
||||
- A downloaded HEVC file no bigger than twice the target size counts as done. That is the
|
||||
small release a search brought in (a MeGusta 1080p episode runs about 2.3 Mbps against
|
||||
Standard's 1.2), and it is neither re-encoded nor searched again. The job's own tagged
|
||||
outputs are exempt, so Standard to Mini still works.
|
||||
- **Skipped:** HDR and Dolby Vision sources (they need tone mapping), subtitle tracks ffprobe
|
||||
can't identify, and files that have failed twice.
|
||||
- Speed is about 0.19 hours of GPU per hour of 1080p video (NVENC is the bottleneck). The job
|
||||
stops starting new encodes at 08:00.
|
||||
- `cost-report.py` estimates, per title, the GPU hours the job would need against the size of
|
||||
a fresh small download.
|
||||
- A `PAUSE` file in this folder makes the nightly run do nothing.
|
||||
|
||||
## Going back up
|
||||
|
||||
Moving down works: Standard to Mini re-encodes the Standard copy. Moving up does not
|
||||
restore quality. To get quality back, I set the title to HD and search, which redownloads
|
||||
it at full size.
|
||||
|
||||
## Running it
|
||||
|
||||
./compact-tv.py --dry-run # what tonight would do
|
||||
./compact-tv.py --search-all --limit 0 # search every unsearched title now, encode nothing
|
||||
touch PAUSE # skip nightly runs until PAUSE is deleted
|
||||
./compact-tv.py --series "Shameless (US)" --limit 1
|
||||
systemctl status compact-tv.timer compact-tv.service
|
||||
tail -f compact-tv.log
|
||||
|
||||
The installed units are copies of `compact-tv.service` and `compact-tv.timer` from this
|
||||
folder. `ExecStart` goes through `/usr/bin/python3` because SELinux won't let systemd
|
||||
execute a file labeled as home content.
|
||||
Executable
+348
@@ -0,0 +1,348 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Shrink shows and movies that Sonarr/Radarr have on the "Standard" or "Mini" profile.
|
||||
|
||||
Demoting something is just picking one of those profiles. Each night this job first asks
|
||||
Sonarr/Radarr to search a few titles it hasn't searched before. Both profiles upgrade only
|
||||
toward small releases, so the search swaps whatever big files have a small release for
|
||||
bandwidth instead of GPU. A week after a title's search, the job re-encodes whatever is
|
||||
still big to HEVC on the GPU (NVENC) at the tier's bitrate, verifies the result, swaps it in
|
||||
place of the original, and has Sonarr/Radarr rescan. Plex and Jellyfin pick up changes
|
||||
through their own folder watchers.
|
||||
|
||||
Moving something back up means switching it to HD and searching, which redownloads it at
|
||||
full size. See README.md.
|
||||
|
||||
Usage:
|
||||
compact-tv.py [--until HH:MM] [--series TITLE] [--limit N] [--dry-run]
|
||||
|
||||
--series matches a show or movie title. Runs as gib (files are gib-owned) from
|
||||
compact-tv.service, nightly.
|
||||
"""
|
||||
import argparse, datetime, fcntl, json, os, pathlib, re, shutil, subprocess, sys, time, urllib.request
|
||||
|
||||
HERE = pathlib.Path(__file__).resolve().parent
|
||||
STATE = HERE / "state.json"
|
||||
LOG = HERE / "compact-tv.log"
|
||||
WORK = pathlib.Path.home() / ".cache" / "compact-tv" # NVMe, so half-done encodes never land in a ZFS snapshot
|
||||
VOLUMES = pathlib.Path.home() / "Server/Media/volumes"
|
||||
|
||||
# Per app: API port, container path -> host path, and how to list one title's files.
|
||||
APPS = {
|
||||
"sonarr": {"port": 8989, "paths": ("/tv/", "/home/gib/Public/TV/"), "titles": "/series",
|
||||
"files": lambda call, t: call(f"/episodefile?seriesId={t['id']}"),
|
||||
"rescan": lambda t: {"name": "RescanSeries", "seriesId": t["id"]}},
|
||||
"radarr": {"port": 7878, "paths": ("/movies/", "/home/gib/Public/Movies/"), "titles": "/movie",
|
||||
"files": lambda call, t: [t["movieFile"]] if t.get("movieFile") else [],
|
||||
"rescan": lambda t: {"name": "RescanMovie", "movieId": t["id"]}},
|
||||
}
|
||||
|
||||
# Video bitrate (kbps) per output class, and the widest output each tier allows.
|
||||
# TV floors come from watching samples: 1200 kbps at 1080p, 800 at 720p, 600 at SD.
|
||||
# Movies get roughly double: dark scenes, grain and action break down first at low rates,
|
||||
# and a 2-hour movie at 2500 kbps is still ~2.3 GB (a quarter of a typical 1080p file).
|
||||
# That also leaves already-small releases (YTS-style, ~1.5-2 GB) under the 30% bar.
|
||||
# Nothing is ever upscaled: a 720p or SD source keeps its size and uses that class's rate.
|
||||
TIERS = {
|
||||
"Standard": {"max_width": 1920, "kbps": {"sonarr": {"1080": 1200, "720": 1200, "sd": 600},
|
||||
"radarr": {"1080": 2500, "720": 1800, "sd": 900}}},
|
||||
"Mini": {"max_width": 1280, "kbps": {"sonarr": {"1080": 800, "720": 800, "sd": 600},
|
||||
"radarr": {"1080": 1500, "720": 1500, "sd": 900}}},
|
||||
}
|
||||
MIN_SAVING = 0.30 # leave a file alone unless re-encoding saves at least this much
|
||||
MAX_FAILURES = 2 # after this many failed attempts a file is skipped for good
|
||||
|
||||
# Search first, encode later. Standard/Mini upgrade only toward small releases (see
|
||||
# setup-small-profiles.py), so asking the *arr to search a title swaps whatever big files
|
||||
# have a small release, for bandwidth instead of GPU. Each night searches a few titles not
|
||||
# searched before, biggest first, so downloads and indexer use stay spread out. A title's
|
||||
# files are only encoded once its search is SEARCH_GRACE_DAYS old.
|
||||
#
|
||||
# Files that arrive after their title's search (new episodes, fresh requests) were never
|
||||
# searched for a small copy, so the job searches that episode or movie on its own first,
|
||||
# and encodes it only if it is still big FILE_SEARCH_GRACE_DAYS later. Downloads are never
|
||||
# delayed by any of this; only encoding waits.
|
||||
SEARCHES_PER_NIGHT = {"sonarr": 5, "radarr": 20}
|
||||
SEARCH_GRACE_DAYS = 7
|
||||
FILE_SEARCH_GRACE_DAYS = 3 # small encodes usually appear within hours to a day; 3 leaves room
|
||||
HDR_TRANSFERS = {"smpte2084", "arib-std-b67"}
|
||||
|
||||
|
||||
def log(msg):
|
||||
line = f"{datetime.datetime.now():%Y-%m-%d %H:%M:%S} {msg}"
|
||||
print(line, flush=True)
|
||||
with open(LOG, "a") as f:
|
||||
f.write(line + "\n")
|
||||
|
||||
|
||||
# ── Sonarr / Radarr ─────────────────────────────────────────────────────────
|
||||
def arr(app):
|
||||
"""Return a call(path, body=None) for the app's v3 API."""
|
||||
key = re.search(r"<ApiKey>([^<]+)", (VOLUMES / app / "config.xml").read_text()).group(1)
|
||||
ip = subprocess.run(["docker", "inspect", app, "--format",
|
||||
"{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
|
||||
capture_output=True, text=True, check=True).stdout.split()[0]
|
||||
base = f"http://{ip}:{APPS[app]['port']}/api/v3"
|
||||
|
||||
def call(path, body=None):
|
||||
req = urllib.request.Request(base + path, headers={"X-Api-Key": key, "Content-Type": "application/json"},
|
||||
data=json.dumps(body).encode() if body is not None else None)
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
return json.load(r)
|
||||
return call
|
||||
|
||||
|
||||
# ── probing and planning ────────────────────────────────────────────────────
|
||||
def probe(path):
|
||||
out = subprocess.run(["ffprobe", "-v", "error", "-print_format", "json", "-show_format", "-show_streams", str(path)],
|
||||
capture_output=True, text=True)
|
||||
return json.loads(out.stdout) if out.returncode == 0 else None
|
||||
|
||||
|
||||
def subtitles(info):
|
||||
"""Subtitle tracks ffmpeg can carry over. Tracks ffprobe can't identify are dropped."""
|
||||
return [s for s in info["streams"] if s["codec_type"] == "subtitle" and s.get("codec_name")]
|
||||
|
||||
|
||||
def audio_kbps(channels):
|
||||
return 96 if channels <= 2 else 160 if channels <= 6 else 192
|
||||
|
||||
|
||||
def plan(info, tier, app):
|
||||
"""Return (ffmpeg args, estimated output bytes) for one file, or (None, reason) to skip it."""
|
||||
t = TIERS[tier]
|
||||
video = next((s for s in info["streams"] if s["codec_type"] == "video"
|
||||
and not s.get("disposition", {}).get("attached_pic")), None)
|
||||
if not video:
|
||||
return None, "no video stream"
|
||||
if video.get("color_transfer") in HDR_TRANSFERS or any("Dolby Vision" in str(sd) for sd in video.get("side_data_list", [])):
|
||||
return None, "HDR/Dolby Vision source (not handled yet)"
|
||||
if (info["format"].get("tags") or {}).get("COMPACT_TIER") == tier:
|
||||
return None, f"already {tier}"
|
||||
|
||||
src_w = int(video["width"])
|
||||
out_w = min(src_w, t["max_width"])
|
||||
cls = "1080" if out_w >= 1700 else "720" if out_w >= 1200 else "sd"
|
||||
vkbps = t["kbps"][app][cls]
|
||||
duration = float(info["format"]["duration"])
|
||||
audios = [s for s in info["streams"] if s["codec_type"] == "audio"]
|
||||
est = int((vkbps + sum(audio_kbps(int(a.get("channels") or 2)) for a in audios)) * 1000 / 8 * duration * 1.03)
|
||||
if int(info["format"]["size"]) * (1 - MIN_SAVING) < est:
|
||||
return None, f"already small ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
|
||||
# A downloaded HEVC file within 2x the target is a small release that already replaced a
|
||||
# big one. Re-encoding it would cost GPU and quality for little space. Our own outputs
|
||||
# (tagged) still go through, so moving a title from Standard to Mini works.
|
||||
tags = info["format"].get("tags") or {}
|
||||
if video.get("codec_name") == "hevc" and "COMPACT_TIER" not in tags and int(info["format"]["size"]) <= 2 * est:
|
||||
return None, f"small HEVC release ({int(info['format']['size']) / 1e9 * 3600 / duration:.2f} GB/h)"
|
||||
|
||||
vf = []
|
||||
if video.get("field_order") not in (None, "progressive", "unknown"):
|
||||
vf.append("bwdif")
|
||||
if out_w < src_w:
|
||||
vf.append(f"scale={out_w}:-2:flags=lanczos")
|
||||
args = ["-map", f"0:{video['index']}"]
|
||||
if vf:
|
||||
args += ["-filter:v", ",".join(vf)]
|
||||
args += ["-c:v", "hevc_nvenc", "-preset", "p7", "-tune", "hq", "-profile:v", "main10", "-pix_fmt", "p010le",
|
||||
"-rc", "vbr", "-multipass", "fullres", "-spatial-aq", "1", "-temporal-aq", "1", "-rc-lookahead", "32",
|
||||
"-bf", "4", "-b_ref_mode", "middle",
|
||||
"-b:v", f"{vkbps}k", "-maxrate", f"{vkbps * 2}k", "-bufsize", f"{vkbps * 4}k"]
|
||||
for i, a in enumerate(audios):
|
||||
ch = int(a.get("channels") or 2)
|
||||
layout = {1: "mono", 2: "stereo", 6: "5.1", 8: "7.1"}.get(ch)
|
||||
args += ["-map", f"0:{a['index']}", f"-c:a:{i}", "libopus", f"-b:a:{i}", f"{audio_kbps(ch)}k",
|
||||
f"-filter:a:{i}", f"aformat=channel_layouts={layout or 'stereo'}"]
|
||||
args += ["-mapping_family", "1"]
|
||||
for i, s in enumerate(subtitles(info)):
|
||||
args += ["-map", f"0:{s['index']}", f"-c:s:{i}", "srt" if s["codec_name"] == "mov_text" else "copy"]
|
||||
args += ["-map", "0:t?", "-c:t", "copy", "-metadata", f"COMPACT_TIER={tier}"]
|
||||
return args, est
|
||||
|
||||
|
||||
# ── encode, verify, swap ────────────────────────────────────────────────────
|
||||
def encode(src, args, tier):
|
||||
WORK.mkdir(parents=True, exist_ok=True)
|
||||
tmp = WORK / "work.mkv"
|
||||
tmp.unlink(missing_ok=True)
|
||||
cmd = ["ffmpeg", "-hide_banner", "-loglevel", "error", "-nostdin", "-y", "-hwaccel", "cuda", "-i", str(src),
|
||||
*args, "-metadata", f"COMPACT_SOURCE={src.name} ({src.stat().st_size} bytes)", str(tmp)]
|
||||
r = subprocess.run(cmd, capture_output=True, text=True)
|
||||
if r.returncode != 0:
|
||||
tmp.unlink(missing_ok=True)
|
||||
raise RuntimeError("ffmpeg failed: " + r.stderr.strip()[-300:])
|
||||
return tmp
|
||||
|
||||
|
||||
def verify(src_info, tmp):
|
||||
new = probe(tmp)
|
||||
if not new:
|
||||
raise RuntimeError("output does not probe")
|
||||
a, b = float(src_info["format"]["duration"]), float(new["format"]["duration"])
|
||||
if abs(a - b) > max(2.0, a * 0.005):
|
||||
raise RuntimeError(f"duration mismatch {a:.1f}s vs {b:.1f}s")
|
||||
counts = lambda i: (sum(1 for s in i["streams"] if s["codec_type"] == "audio"), len(subtitles(i)))
|
||||
if counts(src_info) != counts(new):
|
||||
raise RuntimeError(f"audio/subtitle track counts {counts(src_info)} -> {counts(new)}")
|
||||
if int(new["format"]["size"]) >= int(src_info["format"]["size"]):
|
||||
raise RuntimeError("output is not smaller")
|
||||
return int(new["format"]["size"])
|
||||
|
||||
|
||||
def swap(src, tmp):
|
||||
"""Copy the encode next to the original, then rename it over the original in one step."""
|
||||
final = src.with_suffix(".mkv")
|
||||
staged = src.with_name(f".{final.name}.compact-tmp")
|
||||
shutil.copyfile(tmp, staged)
|
||||
os.replace(staged, final)
|
||||
if final != src:
|
||||
src.unlink()
|
||||
tmp.unlink(missing_ok=True)
|
||||
return final
|
||||
|
||||
|
||||
# ── main ────────────────────────────────────────────────────────────────────
|
||||
def main():
|
||||
ap = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
|
||||
ap.add_argument("--until", help="stop starting new encodes after this local time (HH:MM)")
|
||||
ap.add_argument("--series", help="only this series title")
|
||||
ap.add_argument("--limit", type=int, help="encode at most N files")
|
||||
ap.add_argument("--dry-run", action="store_true", help="plan only, change nothing")
|
||||
ap.add_argument("--search-all", action="store_true",
|
||||
help="search every not-yet-searched title now instead of a few per night")
|
||||
opt = ap.parse_args()
|
||||
|
||||
if (HERE / "PAUSE").exists() and not opt.dry_run:
|
||||
log("PAUSE file present, not encoding anything (delete it to resume)")
|
||||
return
|
||||
|
||||
lock = open(HERE / ".lock", "w")
|
||||
try:
|
||||
fcntl.flock(lock, fcntl.LOCK_EX | fcntl.LOCK_NB)
|
||||
except BlockingIOError:
|
||||
sys.exit("another compact-tv run is active")
|
||||
|
||||
deadline = None
|
||||
if opt.until:
|
||||
h, m = map(int, opt.until.split(":"))
|
||||
now = datetime.datetime.now()
|
||||
deadline = now.replace(hour=h, minute=m, second=0, microsecond=0)
|
||||
if deadline <= now:
|
||||
deadline += datetime.timedelta(days=1)
|
||||
|
||||
state = json.loads(STATE.read_text()) if STATE.exists() else {}
|
||||
for k in ("failures", "searched", "skip"):
|
||||
state.setdefault(k, {})
|
||||
save = lambda: STATE.write_text(json.dumps(state, indent=1))
|
||||
today = datetime.date.today()
|
||||
work = [] # (app, call, title, tier); TV first, then movies
|
||||
for app, a in APPS.items():
|
||||
call = arr(app)
|
||||
profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
|
||||
work += [(app, call, t, profiles[t["qualityProfileId"]]) for t in call(a["titles"])
|
||||
if profiles.get(t["qualityProfileId"]) in TIERS and (not opt.series or t["title"] == opt.series)]
|
||||
log(f"=== run start{' [dry run]' if opt.dry_run else ''}: {len(work)} title(s) on "
|
||||
f"{'/'.join(TIERS)}{f', until {deadline:%H:%M}' if deadline else ''} ===")
|
||||
|
||||
# ── 1. search: ask the *arr for small replacements before spending GPU ──
|
||||
size_of = lambda t: (t.get("statistics") or {}).get("sizeOnDisk") or t.get("sizeOnDisk") or 0
|
||||
for app in APPS:
|
||||
fresh = sorted((w for w in work if w[0] == app and f"{app}:{w[2]['id']}" not in state["searched"]
|
||||
and size_of(w[2]) > 0), key=lambda w: -size_of(w[2]))
|
||||
fresh = fresh if opt.search_all else fresh[:SEARCHES_PER_NIGHT[app]]
|
||||
for _, call, t, tier in fresh:
|
||||
log(f" {'would search' if opt.dry_run else 'search'} {t['title']} ({size_of(t) / 1e9:.0f} GB, {tier})")
|
||||
if fresh and not opt.dry_run:
|
||||
call = fresh[0][1]
|
||||
if app == "sonarr":
|
||||
for _, _, t, _ in fresh:
|
||||
call("/command", {"name": "SeriesSearch", "seriesId": t["id"]})
|
||||
else:
|
||||
call("/command", {"name": "MoviesSearch", "movieIds": [t["id"] for _, _, t, _ in fresh]})
|
||||
for _, _, t, _ in fresh:
|
||||
state["searched"][f"{app}:{t['id']}"] = today.isoformat()
|
||||
save()
|
||||
|
||||
# ── 2. encode what is still big after its search ────────────────────────
|
||||
age = lambda iso: (today - datetime.date.fromisoformat(iso)).days
|
||||
|
||||
def ready(app, ef, title_searched, to_search):
|
||||
"""True if this big file has had its chance to be replaced by a small download."""
|
||||
if (ef.get("dateAdded") or "")[:10] <= title_searched: # was on disk when the title was searched
|
||||
return age(title_searched) >= SEARCH_GRACE_DAYS
|
||||
key = f"{app}:file:{ef['id']}" # arrived later: search it on its own first
|
||||
if key not in state["searched"]:
|
||||
to_search.append(ef)
|
||||
return False
|
||||
return age(state["searched"][key]) >= FILE_SEARCH_GRACE_DAYS
|
||||
|
||||
def search_files(app, call, show, files):
|
||||
if opt.dry_run:
|
||||
log(f" would search {len(files)} newer file(s) of {show['title']}"); return
|
||||
if app == "sonarr":
|
||||
ids = {f["id"] for f in files}
|
||||
eps = [e["id"] for e in call(f"/episode?seriesId={show['id']}") if e.get("episodeFileId") in ids]
|
||||
call("/command", {"name": "EpisodeSearch", "episodeIds": eps})
|
||||
else:
|
||||
call("/command", {"name": "MoviesSearch", "movieIds": [show["id"]]})
|
||||
for f in files:
|
||||
state["searched"][f"{app}:file:{f['id']}"] = today.isoformat()
|
||||
log(f" search {len(files)} newer file(s) of {show['title']} before encoding"); save()
|
||||
|
||||
done = saved = 0
|
||||
for app, call, show, tier in work:
|
||||
title_searched = state["searched"].get(f"{app}:{show['id']}")
|
||||
if not title_searched:
|
||||
continue # its title search comes first, on a later night
|
||||
container, host = APPS[app]["paths"]
|
||||
files = sorted(APPS[app]["files"](call, show), key=lambda f: f["path"])
|
||||
changed, to_search = False, []
|
||||
for ef in files:
|
||||
if deadline and datetime.datetime.now() >= deadline:
|
||||
log("deadline reached, stopping"); break
|
||||
if opt.limit is not None and done >= opt.limit:
|
||||
break
|
||||
src = pathlib.Path(ef["path"].replace(container, host, 1))
|
||||
if state["failures"].get(str(src), 0) >= MAX_FAILURES or not src.exists():
|
||||
continue
|
||||
# Files already ruled out at this tier and size need no re-probe (most of them, most nights).
|
||||
seen = f"{tier}:{src.stat().st_size}"
|
||||
if state["skip"].get(str(src)) == seen:
|
||||
continue
|
||||
info = probe(src)
|
||||
if not info:
|
||||
log(f" SKIP {src.name}: does not probe"); continue
|
||||
args, est = plan(info, tier, app)
|
||||
if args is None:
|
||||
state["skip"][str(src)] = seen # reason in `est`; quiet, since most nights most files are done
|
||||
continue
|
||||
if not ready(app, ef, title_searched, to_search):
|
||||
continue
|
||||
old = int(info["format"]["size"])
|
||||
if opt.dry_run:
|
||||
log(f" would encode {src.name}: {old / 1e6:.0f} MB -> ~{est / 1e6:.0f} MB ({tier})")
|
||||
done += 1; continue
|
||||
t0 = time.monotonic()
|
||||
try:
|
||||
tmp = encode(src, args, tier)
|
||||
new = verify(info, tmp)
|
||||
final = swap(src, tmp)
|
||||
except Exception as e:
|
||||
state["failures"][str(src)] = state["failures"].get(str(src), 0) + 1
|
||||
save()
|
||||
log(f" FAIL {src.name}: {e}"); continue
|
||||
done += 1; saved += old - new; changed = True
|
||||
log(f" {final.name}: {old / 1e6:.0f} MB -> {new / 1e6:.0f} MB "
|
||||
f"(-{(1 - new / old) * 100:.0f}%, {time.monotonic() - t0:.0f}s, {tier})")
|
||||
if to_search:
|
||||
search_files(app, call, show, to_search)
|
||||
if changed:
|
||||
call("/command", APPS[app]["rescan"](show))
|
||||
if (deadline and datetime.datetime.now() >= deadline) or (opt.limit is not None and done >= opt.limit):
|
||||
break
|
||||
if not opt.dry_run:
|
||||
save()
|
||||
log(f"=== run end: {done} file(s){' planned' if opt.dry_run else ''}, saved {saved / 1e9:.1f} GB ===")
|
||||
|
||||
|
||||
if __name__ == "__main__":
|
||||
main()
|
||||
@@ -0,0 +1,17 @@
|
||||
# Installed to /etc/systemd/system/. Runs as gib because the TV files are gib-owned,
|
||||
# so the job needs no root. A system unit (rather than a user unit) because gib has
|
||||
# no lingering session on this headless box. ExecStart goes through python3 because
|
||||
# SELinux will not let systemd execute a file labeled as home content (user_home_t).
|
||||
[Unit]
|
||||
Description=Re-encode TV shows on Sonarr's Compact/Supercompact profiles
|
||||
Documentation=file:///home/gib/Server/Media/compact-tv/README.md
|
||||
After=docker.service
|
||||
Requires=docker.service
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
User=gib
|
||||
Group=gib
|
||||
ExecStart=/usr/bin/python3 /home/gib/Server/Media/compact-tv/compact-tv.py --until 08:00
|
||||
Nice=15
|
||||
IOSchedulingClass=idle
|
||||
@@ -0,0 +1,11 @@
|
||||
# Starts after the 00:00 hour and stops starting new encodes at 08:00 (see --until in
|
||||
# the service), so Plex and Frigate keep the GPU's encoder during the day.
|
||||
[Unit]
|
||||
Description=Nightly TV re-encode for Compact/Supercompact shows
|
||||
|
||||
[Timer]
|
||||
OnCalendar=*-*-* 00:30:00
|
||||
Persistent=false
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,65 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Per-title cost of shrinking everything on Standard/Mini: re-encode vs redownload.
|
||||
|
||||
For each show or movie on those profiles, probes every file with compact-tv's own planner
|
||||
and reports what the nightly job would do (encode, skip as already small, skip as HDR),
|
||||
the space it would save, and the GPU time it would take. Alongside that it estimates what a
|
||||
fresh small download would cost instead, using typical small-encode sizes.
|
||||
|
||||
GPU time is measured, not guessed: the Shameless pilot took 669 s for 57.5 minutes of
|
||||
1080p (about 0.19 h of GPU per hour of video). 4K sources decode slower; the 1.5x for them
|
||||
is an estimate until the first 4K night is logged.
|
||||
|
||||
Usage: cost-report.py > report.tsv (also prints a summary to stderr)
|
||||
"""
|
||||
import importlib.util, json, pathlib, sys
|
||||
|
||||
HERE = pathlib.Path(__file__).resolve().parent
|
||||
spec = importlib.util.spec_from_file_location("ct", HERE / "compact-tv.py")
|
||||
ct = importlib.util.module_from_spec(spec); spec.loader.exec_module(ct)
|
||||
|
||||
GPU_H_PER_H = 669 / 3450 # pilot: hours of GPU per hour of 1080p video
|
||||
UHD_FACTOR = 1.5 # estimate for 2160p sources
|
||||
SMALL_GB_PER_H = {"Standard": 0.85, "Mini": 0.57} # MeGusta-class 1080p / 720p HEVC, measured on S.W.A.T.
|
||||
|
||||
rows = []
|
||||
for app, a in ct.APPS.items():
|
||||
call = ct.arr(app)
|
||||
profiles = {p["id"]: p["name"] for p in call("/qualityprofile")}
|
||||
container, host = a["paths"]
|
||||
for t in call(a["titles"]):
|
||||
tier = profiles.get(t["qualityProfileId"])
|
||||
if tier not in ct.TIERS:
|
||||
continue
|
||||
r = {"app": app, "title": t["title"], "tier": tier, "files": 0, "hours": 0.0, "size": 0,
|
||||
"enc_files": 0, "enc_size": 0, "enc_est": 0, "gpu_h": 0.0, "hdr_size": 0, "small_size": 0}
|
||||
for f in a["files"](call, t):
|
||||
p = pathlib.Path(f["path"].replace(container, host, 1))
|
||||
info = ct.probe(p) if p.exists() else None
|
||||
if not info:
|
||||
continue
|
||||
size, dur = int(info["format"]["size"]), float(info["format"]["duration"]) / 3600
|
||||
r["files"] += 1; r["hours"] += dur; r["size"] += size
|
||||
args, why = ct.plan(info, tier, app)
|
||||
if args:
|
||||
w = max((s.get("width") or 0) for s in info["streams"] if s["codec_type"] == "video")
|
||||
r["enc_files"] += 1; r["enc_size"] += size; r["enc_est"] += why
|
||||
r["gpu_h"] += dur * GPU_H_PER_H * (UHD_FACTOR if w > 1920 else 1)
|
||||
elif "HDR" in str(why):
|
||||
r["hdr_size"] += size
|
||||
else:
|
||||
r["small_size"] += size
|
||||
if r["files"]:
|
||||
r["redownload_gb"] = round(r["hours"] * SMALL_GB_PER_H[tier], 1)
|
||||
rows.append(r)
|
||||
|
||||
rows.sort(key=lambda r: -(r["enc_size"] - r["enc_est"]))
|
||||
cols = ["app", "title", "tier", "files", "hours", "size", "enc_files", "enc_size", "enc_est", "gpu_h", "hdr_size",
|
||||
"small_size", "redownload_gb"]
|
||||
print("\t".join(cols))
|
||||
for r in rows:
|
||||
print("\t".join(str(round(r[c], 2) if isinstance(r[c], float) else r[c]) for c in cols))
|
||||
tot = lambda k: sum(r[k] for r in rows)
|
||||
print(f"{len(rows)} titles; encode {tot('enc_files')} files {tot('enc_size')/1e12:.2f} TB -> ~{tot('enc_est')/1e12:.2f} TB, "
|
||||
f"{tot('gpu_h'):.0f} GPU hours; HDR skipped {tot('hdr_size')/1e12:.2f} TB; already small {tot('small_size')/1e12:.2f} TB",
|
||||
file=sys.stderr)
|
||||
@@ -0,0 +1,99 @@
|
||||
#!/usr/bin/env python3
|
||||
"""Make Standard and Mini prefer small releases, and upgrade only toward small ones.
|
||||
|
||||
Run once per app; it is idempotent, so re-running updates everything in place.
|
||||
|
||||
- Adds "Small Encode" (release groups known for small HEVC/x264 encodes) and "720p" /
|
||||
"1080p" resolution formats. They score 0 in every profile except Standard and Mini.
|
||||
- HD and UHD are otherwise untouched: they keep LQ at -10000, so they never pick these.
|
||||
- Standard and Mini:
|
||||
- All allowed qualities share ONE group, so no "better resolution" or "better source"
|
||||
upgrade can ever download a big file. Only the custom-format score decides.
|
||||
- Scores: Small Encode +3000, x265 +2000 (so first grabs still prefer HEVC), LQ 0,
|
||||
release-tier formats 0 (one big release must never "beat" another big one).
|
||||
Standard adds +500 for 1080p and +250 for 720p; Mini adds +1000 for 720p and +500 for
|
||||
1080p. Those preferences stay below the upgrade threshold, so they never cause churn.
|
||||
- Upgrades are on, but an upgrade needs a score jump of at least 2500, and upgrading
|
||||
stops at 3000. In practice only gaining "Small Encode" clears that bar: a big H.264
|
||||
file gets replaced by a small encode, but never by a big HEVC one.
|
||||
|
||||
Usage: setup-small-profiles.py <sonarr|radarr>
|
||||
"""
|
||||
import json, re, subprocess, sys, urllib.request, pathlib
|
||||
|
||||
APP = sys.argv[1]
|
||||
cfg = pathlib.Path.home() / f"Server/Media/volumes/{APP}/config.xml"
|
||||
key = re.search(r"<ApiKey>([^<]+)", cfg.read_text()).group(1)
|
||||
ip = subprocess.run(["docker", "inspect", APP, "--format", "{{range .NetworkSettings.Networks}}{{.IPAddress}} {{end}}"],
|
||||
capture_output=True, text=True, check=True).stdout.split()[0]
|
||||
base = f"http://{ip}:{8989 if APP == 'sonarr' else 7878}/api/v3"
|
||||
|
||||
def call(path, body=None, method=None):
|
||||
req = urllib.request.Request(base + path, method=method, headers={"X-Api-Key": key, "Content-Type": "application/json"},
|
||||
data=json.dumps(body).encode() if body is not None else None)
|
||||
with urllib.request.urlopen(req, timeout=60) as r:
|
||||
return json.load(r)
|
||||
|
||||
GROUPS = r"^(MeGusta|PSA|ELiTE|GalaxyTV|GalaxyRG|GalaxyRG265|YIFY|YTS(\.(MX|LT|AG))?)$"
|
||||
resolution = lambda r: [{"name": f"{r}p", "implementation": "ResolutionSpecification",
|
||||
"negate": False, "required": True, "fields": [{"name": "value", "value": r}]}]
|
||||
WANTED = {
|
||||
"Small Encode": [{"name": "Small encode groups", "implementation": "ReleaseGroupSpecification",
|
||||
"negate": False, "required": False, "fields": [{"name": "value", "value": GROUPS}]}],
|
||||
"720p": resolution(720),
|
||||
"1080p": resolution(1080),
|
||||
}
|
||||
TIERS = {
|
||||
"Standard": {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "1080p": 500, "720p": 250},
|
||||
"Mini": {"Small Encode": 3000, "x265 (HD)": 2000, "LQ": 0, "LQ (Release Title)": 0, "720p": 1000, "1080p": 500},
|
||||
}
|
||||
RELEASE_TIER = re.compile(r"Tier \d+|WEB Scene")
|
||||
MIN_UPGRADE_JUMP, UPGRADE_UNTIL = 2500, 3000
|
||||
|
||||
cfs = {c["name"]: c for c in call("/customformat")}
|
||||
for name, specs in WANTED.items():
|
||||
body = {"name": name, "includeCustomFormatWhenRenaming": False, "specifications": specs}
|
||||
if name in cfs:
|
||||
body["id"] = cfs[name]["id"]; call(f"/customformat/{body['id']}", body, "PUT")
|
||||
else:
|
||||
call("/customformat", body)
|
||||
cfs = {c["name"]: c["id"] for c in call("/customformat")}
|
||||
|
||||
|
||||
def flatten(items):
|
||||
"""Every quality in the profile with whether it is allowed, groups expanded, lowest rank first."""
|
||||
for it in items:
|
||||
for sub in (it["items"] or [it]):
|
||||
yield sub["quality"], it["allowed"]
|
||||
|
||||
|
||||
def one_group(profile):
|
||||
"""Put every allowed quality into a single top-ranked group; leave disallowed ones as they are."""
|
||||
flat = list(flatten(profile["items"]))
|
||||
group = {"id": 1100, "name": "Any allowed", "allowed": True,
|
||||
"items": [{"quality": q, "items": [], "allowed": True} for q, ok in flat if ok]}
|
||||
profile["items"] = [{"quality": q, "items": [], "allowed": False} for q, ok in flat if not ok] + [group]
|
||||
profile["cutoff"] = group["id"]
|
||||
return profile
|
||||
|
||||
|
||||
for prof in call("/qualityprofile"):
|
||||
have = {f["format"] for f in prof["formatItems"]}
|
||||
for name in WANTED:
|
||||
if cfs[name] not in have:
|
||||
prof["formatItems"].append({"format": cfs[name], "name": name, "score": 0})
|
||||
scores = TIERS.get(prof["name"])
|
||||
if scores:
|
||||
prof = one_group(prof)
|
||||
prof.update(upgradeAllowed=True, minUpgradeFormatScore=MIN_UPGRADE_JUMP, cutoffFormatScore=UPGRADE_UNTIL)
|
||||
for f in prof["formatItems"]:
|
||||
if f["name"] in scores:
|
||||
f["score"] = scores[f["name"]]
|
||||
elif RELEASE_TIER.search(f["name"]):
|
||||
f["score"] = 0
|
||||
call(f"/qualityprofile/{prof['id']}", prof, "PUT")
|
||||
if scores:
|
||||
allowed = [q["quality"]["name"] for it in prof["items"] if it["allowed"] for q in it["items"]]
|
||||
print(f"{APP} {prof['name']}: one group of {len(allowed)} qualities ({allowed[0]} .. {allowed[-1]}); "
|
||||
f"upgrades on, jump >= {MIN_UPGRADE_JUMP}, until {UPGRADE_UNTIL}")
|
||||
print(" nonzero scores: " + ", ".join(f"{f['name']}={f['score']}" for f in prof["formatItems"] if f["score"]))
|
||||
@@ -427,3 +427,8 @@ services:
|
||||
# meant a completed sync was immediately restarted -- a hot loop of ~9
|
||||
# full syncs every 3 minutes. `no` lets it run once per `up` and stop.
|
||||
restart: "no"
|
||||
# Opt-in only (`docker compose --profile configarr up configarr`). Its templates
|
||||
# create the TRaSH profiles and size limits that were removed on 2026-10-28 in
|
||||
# favour of four hand-managed profiles (Mini/Standard/HD/UHD), so a plain `up`
|
||||
# must never run it. See Media/compact-tv/README.md.
|
||||
profiles: [configarr]
|
||||
@@ -41,8 +41,36 @@ services:
|
||||
# world is in backups/; upgrades rewrite the world and cannot be undone.
|
||||
- VERSION=26.1.2
|
||||
- DIFFICULTY=normal
|
||||
- MAX_PLAYERS=10
|
||||
- TYPE=VANILLA
|
||||
- MAX_PLAYERS=30
|
||||
# Paper since 2026-10-13. The last vanilla state is backups/gib-26.1.2-vanilla-*.
|
||||
- TYPE=PAPER
|
||||
# DonutSMP-style plugin set, pulled from Modrinth on each start. Every entry
|
||||
# needs a 26.1.2 Paper build; check Modrinth before adding or bumping VERSION.
|
||||
# Config lives in servers/gib/plugins/<Plugin>/. GrimAC only publishes alpha builds.
|
||||
- MODRINTH_DOWNLOAD_DEPENDENCIES=required
|
||||
- |
|
||||
MODRINTH_PROJECTS=
|
||||
luckperms
|
||||
vaultunlocked
|
||||
essentialsx
|
||||
essentialsx-spawn
|
||||
placeholderapi
|
||||
playerpoints
|
||||
zmenu
|
||||
zauctionhouse
|
||||
ultimateshop
|
||||
smartspawner
|
||||
wildrtp
|
||||
ultimate-teams
|
||||
asbounty
|
||||
axafkzone
|
||||
tab-was-taken
|
||||
deluxemenus
|
||||
coreprotect
|
||||
worldedit
|
||||
worldguard
|
||||
grimac:alpha
|
||||
chunky
|
||||
- MEMORY=12G
|
||||
- SIMULATION_DISTANCE=16
|
||||
- VIEW_DISTANCE=16
|
||||
|
||||
@@ -38,6 +38,7 @@ services:
|
||||
- ENABLE_OAUTH_SIGNUP=true
|
||||
- OAUTH_MERGE_ACCOUNTS_BY_EMAIL=true
|
||||
#- ENABLE_OAUTH_ROLE_MANAGEMENT=true
|
||||
#- OAUTH_ROLES_CLAIM=roles
|
||||
#- OAUTH_ADMIN_ROLES=admin,internal,superuser
|
||||
- OAUTH_CLIENT_ID=${OAUTH_CLIENT_ID}
|
||||
- OAUTH_CLIENT_SECRET=${OAUTH_CLIENT_SECRET}
|
||||
|
||||
@@ -0,0 +1,140 @@
|
||||
# Restoring ~/Server on server.gib from the ZFS-pool backup
|
||||
|
||||
Written 2026-09-17. Read this *before* you need it.
|
||||
|
||||
## What this protects against
|
||||
|
||||
The NVMe/LUKS btrfs volume (`/`, `/home`, and `/var/lib/docker`) dying. The backup lives on
|
||||
the ZFS pool in the same machine, so it does **not** protect against fire, theft, or the
|
||||
whole box failing. It is a disk-failure backup, not an offsite one.
|
||||
|
||||
## Layout
|
||||
|
||||
/home/gib/Public/Server_Backup/Containers/server.gib/current/
|
||||
|
||||
**One directory, overwritten daily at 02:00** by `server-backup.timer`. History lives in ZFS:
|
||||
`Media/Server_Backup` has `zfs-auto-snapshot` keeping **14 dailies**, taken by `cron.daily`
|
||||
between roughly 03:22 and 03:52. The 02:00 schedule is deliberate — it must finish before
|
||||
that window, or a snapshot captures a half-written backup as that day's history. It also
|
||||
staggers against the VPS backup, which pushes into this same pool at 01:00.
|
||||
|
||||
Older copies are under the hidden snapshot dir (`snapdir=hidden`, so `ls` will not show it,
|
||||
but you can `cd` in):
|
||||
|
||||
ls /home/gib/Public/Server_Backup/.zfs/snapshot/
|
||||
ls /home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
|
||||
|
||||
The `03-16-2026` and `03-31-2026` directories are leftovers from the old manual script.
|
||||
|
||||
## Why this runs as root
|
||||
|
||||
This box runs Docker, so most container data under `~/Server` is root-owned. The old script
|
||||
used `sudo rsync`, which worked but meant it could never be automated without a sudoers
|
||||
rule. The replacement runs as root via `server-backup.service`, a systemd **system** unit —
|
||||
already root, so ownership is preserved natively and **no sudoers entry is needed at
|
||||
runtime**. If you see `/etc/sudoers.d/99-claude-setup` on this machine, it was temporary
|
||||
setup scaffolding and should be deleted.
|
||||
|
||||
(This differs from the VPS, which is rootless podman and needs `rsync --fake-super` to carry
|
||||
subuid ownership through. Nothing like that is needed here.)
|
||||
|
||||
The script lives at `/usr/local/sbin/server-backup`, deliberately **not** in `~/Server` — a
|
||||
root-executed script inside a user-writable directory is a footgun. `~/Server/backup` is a
|
||||
thin wrapper that triggers the systemd unit so manual runs behave identically to nightly ones.
|
||||
|
||||
## What is in a backup
|
||||
|
||||
| Path | What it is |
|
||||
|---|---|
|
||||
| `Media/`, `Frigate/`, `Home/`, `OpenWebUI/`, ... | Raw bind-mount data per service |
|
||||
| `_backup_state/sqlite/<original path>` | Online `.backup` of every SQLite DB — **authoritative** |
|
||||
| `_backup_state/convex/<container>.zip` | `npx convex export` snapshot of each SQLite-backed Convex deployment (gbrown) — **authoritative** |
|
||||
|
||||
> **Use `_backup_state/` for databases, not the raw copies.** Containers keep running during
|
||||
> the backup, so the raw files are crash-consistent at best. The dumps use SQLite's online
|
||||
> backup API and are what will actually restore cleanly. That covers the ones that matter:
|
||||
> Plex's `com.plexapp.plugins.library.db` (watch history, ratings), Jellyfin, Frigate, Home
|
||||
> Assistant, OpenWebUI, Audiobookshelf, and every *arr config.
|
||||
>
|
||||
> **Convex is the exception.** It writes to its SQLite file constantly, so a raw `.backup`
|
||||
> never completes. Its `_backup_state/convex/` export is the restore source, and the raw
|
||||
> `db.sqlite3` is not dumped at all.
|
||||
|
||||
## What is deliberately NOT backed up
|
||||
|
||||
- **`Agents/Gitea/volumes/runner-{1,2,3}/dind` — ~203 GB.** Docker-in-Docker containerd
|
||||
image layers for the CI runners. Pure cache that churns constantly; a rebuilt runner just
|
||||
re-pulls it. The parts that matter — `runner-N/data/.runner` (the 648-byte registration
|
||||
token) and `certs/` — **are** backed up.
|
||||
> Measured as root. A plain `du` as `gib` reports these as ~13 MB because gib cannot
|
||||
> descend into them. Do not trust non-root sizes here.
|
||||
- Docker **named volumes** (`/var/lib/docker/volumes`, 47 in use, ~61 GB): Sentry (13) and
|
||||
Plausible (4) were consciously excluded — Sentry's event data is not worth restoring and
|
||||
Plausible had nothing in it. Romm looked like a gap but is not: its anonymous `/romm`
|
||||
volume is only a parent directory, with `assets/`, `config/` and `resources/` bind-mounted
|
||||
from `~/Server` and `library/` from the pool.
|
||||
- **Nextcloud AIO** — has its own borg backup at `~/Public/Nextcloud/Backups`, run by
|
||||
hand from the AIO admin UI. Last run 2026-09-17 (finished 09-18). Runs are slow and
|
||||
AIO stops the Nextcloud containers while one is going.
|
||||
- `node_modules`, build output, `*.log`, `cache`/`tmp` dirs, `*.mp4`/`*.mkv`.
|
||||
|
||||
## Restoring
|
||||
|
||||
Stop the stack first so nothing overwrites what you restore:
|
||||
|
||||
cd ~/Server/<service> && docker compose down # per service, or stop docker entirely
|
||||
|
||||
Pull it back — as root, so ownership is restored:
|
||||
|
||||
sudo rsync -a --numeric-ids --delete \
|
||||
/home/gib/Public/Server_Backup/Containers/server.gib/current/ \
|
||||
/home/gib/Server/
|
||||
|
||||
For an older day, use a snapshot path instead:
|
||||
|
||||
/home/gib/Public/Server_Backup/.zfs/snapshot/<snap>/Containers/server.gib/current/
|
||||
|
||||
Then put the databases back from the dumps. For each one, stop the owning container, copy the
|
||||
dump over the live file, and match the original ownership. Plex, for example:
|
||||
|
||||
docker compose -f ~/Server/Media/compose.yml stop plex
|
||||
D=~/Server/_backup_state/sqlite/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
|
||||
L=~/Server/Media/volumes/plex/config/Library/Application\ Support/Plex\ Media\ Server/Plug-in\ Support/Databases
|
||||
sudo cp "$D/com.plexapp.plugins.library.db" "$L/"
|
||||
sudo chown --reference="$L/com.plexapp.dlna.db" "$L/com.plexapp.plugins.library.db"
|
||||
docker compose -f ~/Server/Media/compose.yml start plex
|
||||
|
||||
`chown --reference` copies ownership from a neighbouring file, which saves looking up what
|
||||
uid each container expects.
|
||||
|
||||
For Convex, start the backend (it can be empty) and import the export through it. This
|
||||
replaces every table with the snapshot:
|
||||
|
||||
Z=~/Server/_backup_state/convex/gbrown-backend.zip
|
||||
KEY=$(docker exec gbrown-backend ./generate_admin_key.sh | tail -1)
|
||||
docker run --rm -it --network nginx-bridge -v "$Z:/backup.zip:ro" \
|
||||
-e CONVEX_SELF_HOSTED_URL=http://gbrown-backend:3210 -e CONVEX_SELF_HOSTED_ADMIN_KEY="$KEY" \
|
||||
docker.io/library/node:22-alpine sh -ec 'mkdir /w && cd /w && echo "{}" > package.json &&
|
||||
npm install --silent convex && npx convex import --replace-all /backup.zip'
|
||||
|
||||
Functions and schema are not in the export. Redeploy them from the app's repo as usual.
|
||||
|
||||
After a full restore the CI runners will have no `dind` storage. That is fine — they re-pull
|
||||
images on the next job. The registration in `runner-N/data/.runner` is intact, so they
|
||||
reconnect to Gitea without re-registering.
|
||||
|
||||
## Checking it is working
|
||||
|
||||
systemctl list-timers server-backup.timer
|
||||
systemctl status server-backup.service
|
||||
tail -40 ~/Server/logs/backup.log
|
||||
|
||||
A healthy run ends with `=== backup complete ===`. Watch for:
|
||||
|
||||
- `ZFS history OK` — confirms snapshots are still being taken. If this ever says **WARNING**,
|
||||
versioning is broken and `current/` is just a mirror: a mistake would propagate with
|
||||
nothing to roll back to. Fix `zfs-auto-snapshot` before anything else.
|
||||
- `rsync exit 24` is a NOTE, not a failure — a file changed mid-transfer, normal on a live box.
|
||||
|
||||
Optional: put an uptime-kuma push URL in `~/Server/.backup-heartbeat-url` and successful runs
|
||||
will ping it, turning a silent failure into a down monitor.
|
||||
Executable
+14
@@ -0,0 +1,14 @@
|
||||
#!/usr/bin/env bash
|
||||
# Manual trigger for the daily backup.
|
||||
#
|
||||
# The real script lives at /usr/local/sbin/server-backup and runs as ROOT via
|
||||
# server-backup.service. It is kept out of ~/Server on purpose: ~/Server is
|
||||
# user-writable, and a root-executed script sitting in a user-writable directory
|
||||
# is an obvious footgun. Run it through systemd so a manual run behaves exactly
|
||||
# like the nightly one (same environment, same logging, same lock).
|
||||
#
|
||||
# status: systemctl status server-backup.service
|
||||
# log: tail -f ~/Server/logs/backup.log
|
||||
# timer: systemctl list-timers server-backup.timer
|
||||
# restore: see ~/Server/RESTORE.md
|
||||
exec sudo systemctl start --wait server-backup.service
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Weekly copy of the Media pool to the Backup pool
|
||||
Documentation=file:///home/gib/Server/RESTORE.md
|
||||
After=zfs-mount.service
|
||||
Wants=zfs-mount.service
|
||||
RequiresMountsFor=/home/gib/Public /home/gib/.local/share/Backup
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/media-backup
|
||||
# 8.6 TB on the first run, deltas after. Generous, but bounded so a hung rsync
|
||||
# fails loudly instead of overlapping next week's fire.
|
||||
TimeoutStartSec=24h
|
||||
Nice=10
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,17 @@
|
||||
[Unit]
|
||||
Description=Weekly Media -> Backup pool copy
|
||||
|
||||
[Timer]
|
||||
# Saturday 05:00. Clear of: the cron.daily zfs-auto-snapshot window (03:22-03:52),
|
||||
# timeshift (04:00), arr-queue-reconcile (04:31), and both ~/Server backups (01:00, 02:00).
|
||||
# Saturday specifically because Monday carries the Media scrub and Sunday has
|
||||
# raid-check (01:00), docker-prune (05:33) and dind-prune (06:03).
|
||||
# gib also runs Nextcloud's borg by hand roughly monthly and is, in his words,
|
||||
# not going to be doing that at 5 AM on a Saturday -- so the borg lock should be free.
|
||||
OnCalendar=Sat *-*-* 05:00:00
|
||||
RandomizedDelaySec=15m
|
||||
Persistent=true
|
||||
Unit=media-backup.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
@@ -0,0 +1,19 @@
|
||||
[Unit]
|
||||
Description=Back up /home/gib/Server to the ZFS pool
|
||||
Documentation=file:///home/gib/Server/RESTORE.md
|
||||
# The destination is on ZFS; do not run before the pool is up.
|
||||
After=zfs-mount.service docker.service
|
||||
Wants=zfs-mount.service
|
||||
RequiresMountsFor=/home/gib/Public
|
||||
|
||||
[Service]
|
||||
Type=oneshot
|
||||
ExecStart=/usr/local/sbin/server-backup
|
||||
TimeoutStartSec=8h
|
||||
# This box is a daily driver running 145 containers. Stay out of the way.
|
||||
Nice=10
|
||||
IOSchedulingClass=best-effort
|
||||
IOSchedulingPriority=7
|
||||
|
||||
[Install]
|
||||
WantedBy=multi-user.target
|
||||
@@ -0,0 +1,14 @@
|
||||
[Unit]
|
||||
Description=Daily backup of /home/gib/Server to the ZFS pool
|
||||
|
||||
[Timer]
|
||||
# MUST finish before cron.daily fires zfs-auto-snapshot (~03:22-03:52); a snapshot that
|
||||
# catches a running rsync stores a partial backup as that day's history.
|
||||
# 02:00 also staggers against the VPS backup, which pushes into this same pool at 01:00.
|
||||
OnCalendar=*-*-* 02:00:00
|
||||
RandomizedDelaySec=10m
|
||||
Persistent=true
|
||||
Unit=server-backup.service
|
||||
|
||||
[Install]
|
||||
WantedBy=timers.target
|
||||
Reference in new issue
Block a user