Survive compositor connection loss: reconnect instead of exiting

A Wayland connection cannot be repaired once its transport breaks — one
dropped fd on a dmabuf-feedback event is enough, after which every
dispatch fails with EINVAL — and the loop's response was to exit the
process, destroying whatever the window held (a terminal's shell and
scrollback, an editor's unsaved buffer). cce-terminal died exactly this
way on 2026-08-13.

`run` is now split into a process-lifetime outer loop and `run_session`,
one connection each. Per session: the globals, surface, xdg/layer shell
objects, swapchain and renderer are built and torn down. Across
sessions: the `Application` instance, the calloop loop, and — critically
— the message channel, because worker threads hold clones of its
`Sender` (cce-terminal's pty reader is the canonical case) and a fresh
channel would orphan them into a live-but-deaf process. calloop sources
are typed on `EngineState<A>` rather than bound to one instance, so the
channel source and the app's `register_sources` (registered once,
guarded, or events would double-deliver) survive a session swap; only
the WaylandSource is removed and re-inserted via its token.

Connection loss now yields SessionEnd::ConnectionLost and reconnects
with exponential backoff (8 attempts, counter reset once a session has
lived 10s), rather than breaking the loop. Setup failures that used to
`.unwrap()`-panic — connect, registry init, initial roundtrip — return
the app intact for the next attempt, so a missing compositor is now a
logged clean exit instead of a panic. `on_exit`/process cleanup moved to
the real exit in `run`.

Renderer state belongs to the session, so `renderer_init` re-runs per
session; images uploaded elsewhere (e.g. in `Application::new`) are NOT
replayed into the new renderer, which is documented on `run`.

CCE_UI_FAULT_RECONNECT=<secs> injects a one-shot session drop, since
real connection loss can't be provoked on demand.

Verified live: with a 6s injected fault, cce-terminal kept its pid, its
pre-fault scrollback, and its shell — `echo OK-42` typed after the
reconnect ran and printed, proving keys reach the same pty and its
output still flows through the original channel. The cce-ui demo app
starts and renders (widgets, text, uploaded images) unchanged; 219
toolkit tests pass.

Co-Authored-By: Claude Fable 5 <[email protected]>

 src/backend/window_runner.rs | 216 ++++++++++++++++++++++++++++++++++++++-----
 1 file changed, 195 insertions(+), 21 deletions(-)

diff --git a/src/backend/window_runner.rs b/src/backend/window_runner.rs
index 81d20a9..0fcb0f0 100644
--- a/src/backend/window_runner.rs
+++ b/src/backend/window_runner.rs
@@ -4155,9 +4155,135 @@ impl<A: Application> wayland_client::Dispatch<ZwpPointerGesturePinchV1, ()> for
     }
 }
 
+/// Why a session's event loop stopped.
+enum SessionEnd {
+    /// The app asked to exit.
+    AppExit,
+    /// The compositor connection died. The `Application` is intact and can be
+    /// re-attached to a fresh connection.
+    ConnectionLost,
+}
+
+/// How many consecutive failed reconnects before giving up. Reset once a
+/// session has survived [`RECONNECT_RESET`], so a long-lived window that loses
+/// its connection twice in a day still gets a full budget the second time.
+const RECONNECT_ATTEMPTS: u32 = 8;
+const RECONNECT_RESET: std::time::Duration = std::time::Duration::from_secs(10);
+
+/// Run an [`Application`] to completion, surviving loss of the compositor
+/// connection.
+///
+/// A Wayland connection cannot be repaired once its transport state breaks — a
+/// single dropped file descriptor on a dmabuf-feedback event is enough, and
+/// libwayland then fails every dispatch with `EINVAL`. Exiting the process on
+/// that error (the old behavior) threw away everything the window held: a
+/// terminal's shell and scrollback, an editor's unsaved buffer.
+///
+/// So a connection is one *session*. Objects that belong to the connection —
+/// the Wayland globals, the surface, the swapchain, the renderer — are rebuilt
+/// per session. The things that carry user state outlive it: the `Application`
+/// itself, the calloop loop, and the message channel. Keeping the **same
+/// channel** matters as much as keeping the app: worker threads hold clones of
+/// its `Sender` (cce-terminal's pty reader is the canonical case), and a fresh
+/// channel would orphan them into a live-but-deaf process.
+///
+/// Caveat: GPU resources belong to the renderer, so a rebuild re-runs
+/// [`Application::renderer_init`]. Images uploaded outside it (e.g. in
+/// [`Application::new`]) are not replayed into the new renderer — upload from
+/// `renderer_init` if they must survive a reconnect.
 pub fn run<A: Application>() {
-    let conn = Connection::connect_to_env().unwrap();
-    let (globals, mut event_queue) = registry_queue_init(&conn).unwrap();
+    // Outlives every session: worker threads hold this Sender, and the app's
+    // own event sources are registered on this loop once.
+    let (sender, channel) = calloop::channel::channel::<A::Message>();
+    let mut event_loop = match EventLoop::try_new() {
+        Ok(l) => l,
+        Err(e) => {
+            log::error!("[window_runner] cannot create event loop: {e}");
+            return;
+        }
+    };
+    event_loop
+        .handle()
+        .insert_source(channel, |event, _metadata, app_state: &mut EngineState<A>| {
+            if let calloop::channel::Event::Msg(msg) = event {
+                let mut rebuild = false;
+                app_state.inner.as_mut().unwrap().update(msg, &mut rebuild, &mut app_state.exit);
+                if rebuild {
+                    app_state.redraw = true;
+                }
+            }
+        })
+        .unwrap();
+
+    let mut app: Option<A> = None;
+    let mut sources_registered = false;
+    let mut attempt: u32 = 0;
+
+    loop {
+        let started = std::time::Instant::now();
+        let (returned_app, end) =
+            run_session(&mut event_loop, sender.clone(), app.take(), !sources_registered);
+        app = returned_app;
+        sources_registered = true;
+
+        match end {
+            SessionEnd::AppExit => break,
+            SessionEnd::ConnectionLost => {
+                // Nothing to preserve if we never got as far as building the
+                // app — that is a failure to start, not a lost window.
+                if app.is_none() {
+                    log::error!("[window_runner] no compositor connection; giving up");
+                    break;
+                }
+                if started.elapsed() > RECONNECT_RESET {
+                    attempt = 0;
+                }
+                attempt += 1;
+                if attempt > RECONNECT_ATTEMPTS {
+                    log::error!(
+                        "[window_runner] connection lost; giving up after {} attempts",
+                        attempt - 1
+                    );
+                    break;
+                }
+                let backoff = std::time::Duration::from_millis(100 * (1 << attempt.min(6)));
+                log::warn!(
+                    "[window_runner] compositor connection lost; reconnecting in {backoff:?} (attempt {attempt})"
+                );
+                std::thread::sleep(backoff);
+            }
+        }
+    }
+
+    if let Some(mut app) = app {
+        app.on_exit();
+    }
+    crate::process::cleanup_spawned_processes();
+}
+
+/// One connection's lifetime: connect, build the surface and renderer, pump
+/// events until the app exits or the connection dies. Returns the
+/// `Application` so the caller can hand it to the next session.
+fn run_session<'l, A: Application>(
+    event_loop: &mut EventLoop<'l, EngineState<A>>,
+    sender: calloop::channel::Sender<A::Message>,
+    existing_app: Option<A>,
+    register_app_sources: bool,
+) -> (Option<A>, SessionEnd) {
+    let conn = match Connection::connect_to_env() {
+        Ok(c) => c,
+        Err(e) => {
+            log::error!("[window_runner] cannot connect to compositor: {e}");
+            return (existing_app, SessionEnd::ConnectionLost);
+        }
+    };
+    let (globals, mut event_queue) = match registry_queue_init(&conn) {
+        Ok(v) => v,
+        Err(e) => {
+            log::error!("[window_runner] registry init failed: {e}");
+            return (existing_app, SessionEnd::ConnectionLost);
+        }
+    };
     let qh = event_queue.handle();
 
     let compositor_state = CompositorState::bind(&globals, &qh).unwrap();
@@ -4167,7 +4293,6 @@ pub fn run<A: Application>() {
     let seat_state = SeatState::new(&globals, &qh);
     let output_state = OutputState::new(&globals, &qh);
 
-    let (sender, channel) = calloop::channel::channel::<A::Message>();
     let pointer_gestures: Option<ZwpPointerGesturesV1> = globals.bind(&qh, 1..=3, ()).ok();
 
     let mut engine_state = EngineState {
@@ -4221,13 +4346,22 @@ pub fn run<A: Application>() {
         dl_text_items: Vec::new(),
     };
 
-    event_queue.roundtrip(&mut engine_state).unwrap();
+    if let Err(e) = event_queue.roundtrip(&mut engine_state) {
+        log::error!("[window_runner] initial roundtrip failed: {e}");
+        return (existing_app, SessionEnd::ConnectionLost);
+    }
 
     let scale = detect_scale_factor(&engine_state.output_state);
     engine_state.scale_factor = scale;
     crate::scale::set_scale_factor(scale as f32);
 
-    let inner = A::new(&qh, engine_state.sender.clone());
+    // A reconnect re-attaches the SAME app: its state is the thing worth
+    // saving, and `A::new` would both discard it and hand a fresh Sender to
+    // worker threads that are still holding the original.
+    let inner = match existing_app {
+        Some(app) => app,
+        None => A::new(&qh, engine_state.sender.clone()),
+    };
     let settings = inner.settings();
     crate::scale::set_app_id(settings.app_id.clone());
     engine_state.logical_width = settings.width as f32;
@@ -4295,21 +4429,22 @@ pub fn run<A: Application>() {
         .unwrap()
         .renderer_init(engine_state.renderer.as_mut().unwrap());
 
-    let mut event_loop = EventLoop::try_new().unwrap();
     let loop_handle = event_loop.handle();
-    WaylandSource::new(conn.clone(), event_queue).insert(loop_handle.clone()).unwrap();
-
-    loop_handle.insert_source(channel, |event, _metadata, app_state: &mut EngineState<A>| {
-        if let calloop::channel::Event::Msg(msg) = event {
-            let mut rebuild = false;
-            app_state.inner.as_mut().unwrap().update(msg, &mut rebuild, &mut app_state.exit);
-            if rebuild {
-                app_state.redraw = true;
-            }
+    let wayland_token = match WaylandSource::new(conn.clone(), event_queue).insert(loop_handle.clone())
+    {
+        Ok(token) => token,
+        Err(e) => {
+            log::error!("[window_runner] cannot register the wayland source: {e}");
+            return (engine_state.inner.take(), SessionEnd::ConnectionLost);
         }
-    }).unwrap();
+    };
 
-    engine_state.inner.as_mut().unwrap().register_sources(&loop_handle);
+    // The app's own sources live on the persistent loop, so they are registered
+    // once for the process — re-registering per session would double-deliver
+    // every event on them.
+    if register_app_sources {
+        engine_state.inner.as_mut().unwrap().register_sources(&loop_handle);
+    }
 
     const KEY_REPEAT_DELAY: std::time::Duration = std::time::Duration::from_millis(500);
     const KEY_REPEAT_INTERVAL: std::time::Duration = std::time::Duration::from_millis(50);
@@ -4322,8 +4457,24 @@ pub fn run<A: Application>() {
         *FLAG.get_or_init(|| std::env::var_os("CCE_PRESENT_DEBUG").is_some())
     }
 
+    /// Seconds after session start at which to inject a simulated connection
+    /// loss, from `CCE_UI_FAULT_RECONNECT`. Resolved once: this is read from
+    /// the per-iteration path.
+    fn fault_reconnect_after() -> Option<std::time::Duration> {
+        static AFTER: std::sync::OnceLock<Option<std::time::Duration>> =
+            std::sync::OnceLock::new();
+        *AFTER.get_or_init(|| {
+            std::env::var("CCE_UI_FAULT_RECONNECT")
+                .ok()
+                .and_then(|v| v.parse::<f32>().ok())
+                .map(std::time::Duration::from_secs_f32)
+        })
+    }
+
     let mut last_title = settings.title.clone();
     let mut last_tick = std::time::Instant::now();
+    let mut end = SessionEnd::AppExit;
+    let session_start = std::time::Instant::now();
     loop {
         // Frame callbacks arrive with a p50 of 0ms but a ~0.5s tail, while the
         // compositor's own trace shows it firing them within one or two vsyncs
@@ -4336,7 +4487,8 @@ pub fn run<A: Application>() {
             None
         };
         if let Err(e) = event_loop.dispatch(std::time::Duration::from_millis(16), &mut engine_state) {
-            log::error!("[window_runner] Event loop error: {:?}", e);
+            log::error!("[window_runner] event loop error, ending session: {e:?}");
+            end = SessionEnd::ConnectionLost;
             break;
         }
         if let Some(start) = iter_start {
@@ -4358,9 +4510,24 @@ pub fn run<A: Application>() {
         // spins forever on a dead display while wayland-backend re-prints the
         // error on every flush attempt.
         if let Some(perr) = conn.protocol_error() {
-            log::error!("[window_runner] Fatal Wayland protocol error, exiting: {perr}");
+            log::error!("[window_runner] wayland protocol error, ending session: {perr}");
+            end = SessionEnd::ConnectionLost;
             break;
         }
+        // Fault injection for the reconnect path (`CCE_UI_FAULT_RECONNECT=<secs>`):
+        // real connection loss is a rare race that cannot be provoked on demand,
+        // so this drops the session exactly as a transport error would. One-shot
+        // per process, so the app reconnects and then stays up.
+        if let Some(after) = fault_reconnect_after() {
+            static FIRED: std::sync::atomic::AtomicBool = std::sync::atomic::AtomicBool::new(false);
+            if session_start.elapsed() >= after
+                && !FIRED.swap(true, std::sync::atomic::Ordering::Relaxed)
+            {
+                log::warn!("[window_runner] CCE_UI_FAULT_RECONNECT: dropping the session");
+                end = SessionEnd::ConnectionLost;
+                break;
+            }
+        }
         if engine_state.exit {
             break;
         }
@@ -4524,6 +4691,13 @@ pub fn run<A: Application>() {
             }
         }
     }
-    engine_state.inner.as_mut().unwrap().on_exit();
-    crate::process::cleanup_spawned_processes();
+
+    // Tear the session down: drop its Wayland source from the persistent loop
+    // (leaving it would leak a dead source per reconnect), then hand the app
+    // back before `engine_state` drops the renderer and the surface with it.
+    // `on_exit` and process cleanup belong to the app's real exit, in `run`.
+    loop_handle.remove(wayland_token);
+    let app = engine_state.inner.take();
+    drop(engine_state);
+    (app, end)
 }