From 86825e73279b87d2ce9a3f6ea9755e418290af47 Mon Sep 17 00:00:00 2001 From: Gabriel Brown Date: Fri, 21 Aug 2026 15:26:20 -0400 Subject: [PATCH] Judge the document portal by its mount, not by its service No flatpak would launch. Every one of them failed in bwrap with "Can't find source path /run/user/1000/doc/by-app/", because xdg-document-portal's fuse mount was gone -- /run/user/1000/doc was a plain empty directory. That mount is bound into every sandbox bwrap builds, so losing it takes out all 34 flatpaks at once, never a subset. It had been gone for three days. The shipped unit is Type=dbus with Restart=no, so nothing retries it on failure: after it exited 21 it came back only because a flatpak called its bus name two seconds later, and that activation landed on the dying instance's mountpoint and came up with no mount at all. systemctl reports active (running) either way -- the fusermount3 helper is still sitting there as a child, in this case for two and a half days without ever completing the mount. Nothing running notices, which is what makes it so quiet. A sandbox needs the mount only while it is being constructed, so everything already open keeps working and the symptom arrives whenever you next open a flatpak you had not opened yet. Here that was three days later, and it presented as "gearlever is missing" -- an application that was installed, healthy, and entirely innocent. Two changes, because there are two failures: it does not recover, and nothing says so. The drop-in clears the mountpoint before each start, so an activation that follows a crash lands on clean ground. ExecStartPre rather than ExecStopPost precisely because nothing restarts this unit -- the next start is whenever something next wants it, and that is the moment that has to be safe. `-` because a clean start has nothing to unmount and fusermount3 exits 1 saying so. /etc/systemd/user rather than a per-user drop-in so it covers every account, and change-settings reloads the user manager so it applies without a re-login. The check asks the mount table whether $XDG_RUNTIME_DIR/doc is mounted fuse.portal. Deliberately not a service probe, and deliberately not folded into desktop.portals: that one asks about xdg-desktop-portal, a different service which was up and healthy throughout. Service state is exactly the question that lied here, so asking it again in a new place would have been no check at all. Warning carries a restart repair, verified end to end rather than assumed. The mount table is injectable, like every other path this script reads, so the contract covers unmounted, wrong-filesystem-at-the-right-path, and unreadable against written fixtures rather than against whatever /proc happens to say -- coupling the test to this machine's live flatpak state is the same mistake in miniature. Stubbing the check to always return ok fails the contract, which was confirmed rather than hoped. What is not fixed is the crash itself: one occurrence, and restarting the service to get the desktop working destroyed the evidence. The exit was 21, it landed 21 seconds after xdg-desktop-portal restarted, and that is one sample and not a theory. What this buys is that the next one is a two-second blip the doctor names, rather than three silent days. Second time for this bug. ac231ee found the same dead mount in August while chasing "can't open Bitwarden", fixed it by hand, and recorded it as "not a config issue, so nothing to commit there". That judgement is why it was paid for twice, and it is the part most worth writing down. --- .../panama-stale-mount.conf | 22 ++++++ config/dot/quickshell/scripts/panama-doctor | 70 ++++++++++++++++++- setup/scripts/change-settings | 9 +++ tests/quickshell/panama-doctor-contract | 44 +++++++++++- 4 files changed, 140 insertions(+), 5 deletions(-) create mode 100644 config/copy/etc/systemd/user/xdg-document-portal.service.d/panama-stale-mount.conf diff --git a/config/copy/etc/systemd/user/xdg-document-portal.service.d/panama-stale-mount.conf b/config/copy/etc/systemd/user/xdg-document-portal.service.d/panama-stale-mount.conf new file mode 100644 index 0000000..cbaef50 --- /dev/null +++ b/config/copy/etc/systemd/user/xdg-document-portal.service.d/panama-stale-mount.conf @@ -0,0 +1,22 @@ +# Clear the mountpoint before starting, so a start that follows a crash lands +# on clean ground. +# +# This service provides the fuse mount at /run/user/$UID/doc, and bwrap binds +# doc/by-app/ into every flatpak sandbox it builds. No mount, no sandbox, +# no application -- all of them, not one, because they all share this. +# +# The shipped unit is Type=dbus with Restart=no, so nothing restarts it on +# failure: it comes back only when something next calls its bus name. On this +# machine that was a flatpak launching two seconds after the daemon exited +# 21, and the activation raced the dying instance's fuse state and came up +# with no mount at all. systemctl then reports active (running) -- the +# fusermount3 helper is even still there as a child -- while every new sandbox +# fails. Nothing running at the time notices, because a sandbox needs the +# mount only while it is being built, so the breakage surfaces whenever you +# next open something you had not opened yet. Here that was three days later, +# and it was the second time: the first was written off as a one-off. +# +# `-` because a clean start has nothing to unmount and fusermount3 exits 1 +# saying so, which is not a failure to start on. +[Service] +ExecStartPre=-/usr/bin/fusermount3 -u /run/user/%U/doc diff --git a/config/dot/quickshell/scripts/panama-doctor b/config/dot/quickshell/scripts/panama-doctor index 8510efb..542b6a3 100755 --- a/config/dot/quickshell/scripts/panama-doctor +++ b/config/dot/quickshell/scripts/panama-doctor @@ -72,6 +72,10 @@ class DoctorConfig: # successful repair and report it as failed even though a following # health scan would show everything recovered. See run_repair_command. repair_timeout: float = 15.0 + # The kernel's mount table, injectable for the same reason every other path + # here is: so the contract can exercise both a mounted and an unmounted + # document portal without needing root and a real fuse mount to do it. + mountinfo: Path = Path("/proc/self/mountinfo") @property def command_env(self) -> dict[str, str]: @@ -113,6 +117,7 @@ class RepairResult: CHECK_ORDER = ( "desktop.hyprland", "desktop.quickshell", "desktop.notifications", "desktop.portals", + "desktop.document-portal", "desktop.hyprpaper", "desktop.hypridle", "desktop.hyprlock", "desktop.vicinae", "input.pipewire", "input.clipboard", "input.wallpaper", "input.capture", "input.ocr", "input.brightness", "integration.nextcloud", "integration.rustdesk", "integration.kdeconnect", "integration.bluebubbles", @@ -138,6 +143,9 @@ REPAIR_COMMANDS = MappingProxyType({ "desktop.hypridle": ("systemctl", "--user", "restart", "hypridle.service"), "desktop.vicinae": ("systemctl", "--user", "restart", "vicinae.service"), "desktop.quickshell": ("panama-action", "restart-shell"), + # Restarting is the whole repair: the drop-in in config/copy clears the + # stale mountpoint on the way in, so a start is all it takes to remount. + "desktop.document-portal": ("systemctl", "--user", "restart", "xdg-document-portal.service"), }) RUNTIME_LINK_TARGETS = ( ("hypr", Path("config/dot/hypr")), @@ -173,6 +181,7 @@ CHECK_TITLES = { "desktop.quickshell": "Quickshell", "desktop.notifications": "Notifications", "desktop.portals": "Desktop portals", + "desktop.document-portal": "Document portal", "desktop.hyprpaper": "Hyprpaper", "desktop.hypridle": "Hypridle", "desktop.hyprlock": "Lock screen", @@ -215,6 +224,7 @@ def config_from_environment() -> DoctorConfig: # are invoked by resolved path instead, the same way check_hyprlock already # resolves panama-lock. scripts_dir = environment_path("PANAMA_DOCTOR_SCRIPTS_DIR", Path(__file__).resolve().parent) + mountinfo = environment_path("PANAMA_DOCTOR_MOUNTINFO", Path("/proc/self/mountinfo")) try: timeout = float(os.environ.get("PANAMA_DOCTOR_TIMEOUT", "3")) except ValueError: @@ -227,7 +237,7 @@ def config_from_environment() -> DoctorConfig: # Never shorter than the probe timeout, and bounded so a hung repair still # gives up rather than blocking the caller indefinitely. repair_timeout = max(timeout, min(repair_timeout, 30.0)) - return DoctorConfig(root, home, config_home, state_home, runtime_dir, os.environ.get("PANAMA_DOCTOR_PATH", os.environ.get("PATH", "")), timeout, scripts_dir, repair_timeout) + return DoctorConfig(root, home, config_home, state_home, runtime_dir, os.environ.get("PANAMA_DOCTOR_PATH", os.environ.get("PATH", "")), timeout, scripts_dir, repair_timeout, mountinfo) def run_command(command: tuple[str, ...], config: DoctorConfig, cwd: Path | None = None) -> CommandResult: @@ -361,6 +371,62 @@ def check_portals(config: DoctorConfig) -> Check: return Check("desktop.portals", "desktop-foundation", "Desktop portals", "warning", detail) +def check_document_portal(config: DoctorConfig) -> Check: + """Whether the flatpak document portal is actually mounted. + + Deliberately not a service probe, and deliberately separate from + check_portals -- which asks about xdg-desktop-portal, a different service + that was up and healthy throughout the outage this exists for. + + xdg-document-portal provides the fuse mount at $XDG_RUNTIME_DIR/doc, and + bwrap binds doc/by-app/ into every flatpak sandbox it builds. Without + it nothing flatpak launches -- all of them, not one, because they share it. + + The shipped unit is Type=dbus with Restart=no, so after a crash nothing + restarts it; it comes back when something next calls its bus name, and that + activation can land on the dying instance's mountpoint and come up with no + mount at all. `systemctl status` says active (running) either way, with the + fusermount3 helper still sitting there as a child. So service state is + exactly the wrong question, and asking it is how this went unnoticed for + three days and two separate debugging sessions. The only honest question is + whether the mount is in the table. + + Nothing already running notices, either: a sandbox needs the mount only + while it is being constructed. That is what makes it worth a check rather + than trusting you will spot it -- the symptom arrives whenever you next + open a flatpak you had not opened yet, with nothing to connect it to. + """ + expected = str(config.runtime_dir / "doc") + try: + table = config.mountinfo.read_text(encoding="utf-8", errors="replace") + except OSError: + return Check("desktop.document-portal", "desktop-foundation", "Document portal", "error", "Mount table probe is unavailable.") + + for line in table.splitlines(): + fields = line.split() + # mountinfo carries a variable number of optional fields, terminated by + # a lone "-": the filesystem type is the field after it, and the mount + # point is always the fifth. Splitting on "-" rather than counting from + # the end is what makes this robust to that variability. + if "-" not in fields: + continue + separator = fields.index("-") + if separator < 5 or len(fields) <= separator + 1: + continue + # Mount points are octal-escaped in this file; the space is the only + # escape that can appear in a runtime directory path in practice. + if fields[4].replace("\\040", " ") != expected: + continue + if fields[separator + 1] == "fuse.portal": + return Check("desktop.document-portal", "desktop-foundation", "Document portal", "ok", "Flatpak document portal is mounted.") + + return Check( + "desktop.document-portal", "desktop-foundation", "Document portal", "warning", + "Flatpak document portal is not mounted, so no flatpak can launch.", + Action("repair", "Restart the document portal"), + ) + + def check_hyprlock(config: DoctorConfig) -> Check: helper = config.root / "config/dot/quickshell/scripts/panama-lock" tracked_fallback = config.config_home / "hypr/hyprlock.conf" @@ -653,7 +719,7 @@ def unavailable_versions() -> list[dict[str, str]]: def collect_checks(config: DoctorConfig) -> list[Check]: probes: dict[str, Callable[[], Check]] = { - "desktop.hyprland": lambda: check_hyprland(config), "desktop.quickshell": lambda: check_quickshell(config), "desktop.notifications": lambda: check_notifications(config), "desktop.portals": lambda: check_portals(config), + "desktop.hyprland": lambda: check_hyprland(config), "desktop.quickshell": lambda: check_quickshell(config), "desktop.notifications": lambda: check_notifications(config), "desktop.portals": lambda: check_portals(config), "desktop.document-portal": lambda: check_document_portal(config), "desktop.hyprpaper": lambda: service_check("desktop.hyprpaper", "Hyprpaper", "hyprpaper", config, Action("repair", "Restart Hyprpaper")), "desktop.hypridle": lambda: service_check("desktop.hypridle", "Hypridle", "hypridle", config, Action("repair", "Restart Hypridle")), "desktop.hyprlock": lambda: check_hyprlock(config), "desktop.vicinae": lambda: service_check("desktop.vicinae", "Vicinae", "vicinae", config, Action("repair", "Restart Vicinae")), "input.pipewire": lambda: service_check("input.pipewire", "PipeWire", "pipewire", config), "input.clipboard": lambda: simple_ipc_check("input.clipboard", "Clipboard", "clipboard", config), "input.wallpaper": lambda: simple_ipc_check("input.wallpaper", "Wallpaper", "wallpaper", config), "input.capture": lambda: simple_ipc_check("input.capture", "Capture", "capture", config), "input.ocr": lambda: executable_check("input.ocr", "OCR", "tesseract", config), "input.brightness": lambda: check_brightness(config), "integration.nextcloud": lambda: check_nextcloud(config), "integration.rustdesk": lambda: check_rustdesk(config), "integration.kdeconnect": lambda: check_kdeconnect(config), "integration.bluebubbles": lambda: check_bluebubbles(config), "integration.home-assistant": lambda: check_home_assistant(config), "integration.calendar": lambda: check_calendar(config), diff --git a/setup/scripts/change-settings b/setup/scripts/change-settings index f4502f7..a8fe18e 100755 --- a/setup/scripts/change-settings +++ b/setup/scripts/change-settings @@ -19,6 +19,15 @@ if [[ -f /etc/udev/rules.d/99-panama-gpu.rules ]]; then sudo udevadm trigger --subsystem-match=drm fi +# The document-portal drop-in lands via the same copy. A user unit's drop-ins +# are read at daemon-reload, so without this the guard applies from the next +# login rather than from now -- and the failure it guards against is one you +# only find days later, when you next open a flatpak you had not opened yet. +if [[ -f /etc/systemd/user/xdg-document-portal.service.d/panama-stale-mount.conf ]]; then + log "Reloading the user manager so the document-portal drop-in applies" + systemctl --user daemon-reload || log "Could not reload the user manager" +fi + echo -e "\n--- Desktop appearance ---" # GTK and Qt apps read these directly from gsettings; there is no settings # daemon outside GNOME, so they must be correct rather than merely overridden. diff --git a/tests/quickshell/panama-doctor-contract b/tests/quickshell/panama-doctor-contract index 79f4090..9e35c3c 100755 --- a/tests/quickshell/panama-doctor-contract +++ b/tests/quickshell/panama-doctor-contract @@ -35,6 +35,19 @@ bin_dir="$fixture/bin" data_home="$home/.local/share" mkdir -p "$config_home" "$state_home" "$runtime_dir" "$bin_dir" "$data_home/vicinae/scripts" + +# A mount table with the document portal present, which is the healthy state the +# rest of this file assumes. Written rather than read from /proc so the contract +# does not pass or fail on whether this machine happens to have flatpak working +# at the moment -- which is exactly the coupling that let a three-day outage go +# unnoticed. The gvfs line is there so the check has to match on the mount point +# and filesystem type rather than on being handed a file with one entry. +mountinfo="$fixture/mountinfo" +cat >"$mountinfo" </dev/null <<<"$snapshot" || fail "invalid schema: $snapshot" [[ "$(jq -r '.checks[].id' <<<"$snapshot")" == "$expected_order" ]] \ @@ -224,6 +238,30 @@ jq -e '.checks[] | select(.id == "integration.nextcloud") | .action == {kind:"open", label:"Open Nextcloud", confirm:false}' \ >/dev/null <<<"$stopped_nextcloud" || fail 'Nextcloud action was not authored' +# The document portal is judged by the mount, never by the service. +# +# This is the check's whole reason for existing. The service can be active and +# the mount gone -- that is precisely what happened, twice -- so a fixture that +# keeps every service healthy and removes only the mount has to fail. If this +# ever passes, the check has drifted back to asking systemd how it feels. +grep -v ' fuse.portal ' "$mountinfo" >"$fixture/mountinfo-unmounted" +unmounted_portal="$(PANAMA_DOCTOR_MOUNTINFO="$fixture/mountinfo-unmounted" run_doctor --json)" +check_status "$unmounted_portal" desktop.document-portal warning +jq -e '.checks[] | select(.id == "desktop.document-portal") + | .action == {kind:"repair", label:"Restart the document portal", confirm:false}' \ + >/dev/null <<<"$unmounted_portal" || fail 'document portal repair action was not authored' + +# A mount at the right path of the wrong type is not the document portal. The +# stale mountpoint left behind by a failed activation is a plain directory, but +# matching on path alone would call any future mount there healthy. +sed 's/ fuse.portal portal / fuse.gvfsd-fuse gvfsd-fuse /' "$mountinfo" >"$fixture/mountinfo-wrongtype" +wrong_type_portal="$(PANAMA_DOCTOR_MOUNTINFO="$fixture/mountinfo-wrongtype" run_doctor --json)" +check_status "$wrong_type_portal" desktop.document-portal warning + +# An unreadable mount table is an error, not a silent pass and not a crash. +missing_mountinfo="$(PANAMA_DOCTOR_MOUNTINFO="$fixture/no-such-mountinfo" run_doctor --json)" +check_status "$missing_mountinfo" desktop.document-portal error + # DDC errors are classified without retaining connectors or bus addresses. denied_brightness="$(PANAMA_DOCTOR_FIXTURE_BRIGHTNESS=denied run_doctor --json)" check_status "$denied_brightness" input.brightness warning