muxplex/docs/plans/2026-04-08-federation-state-propagation-design.md

# Federation State Propagation Design

## Goal

Make federation a seamless experience where settings, hidden sessions, and activity state propagate across all connected muxplex servers. After this work, a user browsing through any server sees a consistent, unified experience -- settings follow you, hidden sessions disappear everywhere, and bell/activity indicators clear correctly regardless of which server owns the session.

## Background

Muxplex federation currently aggregates tmux sessions from multiple servers into a single dashboard, but state management remains per-server:

- **Settings are local.** Changing `fontSize` on Server A has no effect on Server B. A user who accesses both must configure each independently.
- **Hidden sessions only apply locally.** Hiding a session on Server A only hides it on Server A's dashboard. The same session remains visible if browsing Server B directly, or if the session lives on Server B and is viewed via federation from A.
- **Bell/activity clearing doesn't cross federation boundaries.** When viewing a remote session through federation, the remote server doesn't know anyone is watching. Its `should_clear_bell()` never fires, so activity indicators persist indefinitely. Browser-level indicators (tab title count, favicon badge) also don't filter hidden sessions from their counts.
- **A JS falsy-0 bug** in `getVisibleSessions()` causes sessions from the first remote instance (`remoteId: 0`) to be incorrectly subject to the hidden filter.

## Approach

Two-phase implementation. Phase 1 fixes four targeted bugs that cause the most visible UX pain without any new protocol. Phase 2 adds a minimal P2P settings sync protocol. The phases are independent -- Phase 1 can ship and provide immediate value, Phase 2 builds on top.

The key architectural insight (from COE review): muxplex is one person with multiple servers, not a distributed multi-user system. This means:

- **Settings sync** uses full-document push with a single timestamp per server, not per-key CRDTs. Simple and correct for the single-user case.
- **Bells are not a sync problem.** The bell data already flows through `/api/federation/sessions` on every 2-second poll. The fix is extending the heartbeat/presence system to work across federation boundaries.
- **Hidden sessions** are just another settings key that flows through the settings sync. No special treatment needed.

## Architecture

```
Phase 1: Fix existing federation gaps (no new protocol)

  Browser ──heartbeat──> Server A ──poll cycle──> Server B
                                   (bell/clear)
  
  getVisibleSessions() fixed: s.remoteId == null (not !s.remoteId)
  updatePageTitle() / updateFaviconBadge() filter through getVisibleSessions()

Phase 2: P2P settings sync (new protocol, piggybacks on poll cycle)

  Server A ──GET /api/settings/sync──> Server B
           <── {settings, settings_updated_at} ──
  
  If remote newer: adopt remote settings locally
  If local newer:  PUT /api/settings/sync to remote
  If equal:        no action
  
  Timestamps converge → no sync loops
  Offline servers catch up automatically on reconnect
```

## Components

### Phase 1: Federation Bug Fixes

#### Fix 1: `getVisibleSessions()` Falsy-0 Bug

**File:** `app.js` (~line 533)

Change `!s.remoteId` to `s.remoteId == null`. Currently `remoteId: 0` (first remote instance) is treated as falsy in JavaScript, causing sessions from the first remote to be incorrectly subject to the hidden filter when their name appears in `hidden_sessions`. One-line fix.

#### Fix 2: Heartbeat-Driven Bell Clearing Across Federation

**File:** `main.py` (poll cycle in `_run_poll_cycle`)

The core problem: the browser's heartbeat only goes to Server A (the one the browser is connected to). When viewing a remote session on Server B in fullscreen, Server B doesn't know anyone is watching. Its `should_clear_bell()` never fires for federation viewers.

The fix: in the poll loop on Server A, when the local heartbeat indicates a device is viewing a remote session in fullscreen with recent interaction, Server A fires `POST /api/federation/{remote_id}/sessions/{session}/bell/clear` to the remote. This piggybacks on the existing 2-second poll cycle and the existing federation bell/clear endpoint -- no new API surface needed. The remote server's bell state updates, and the next federation sessions poll reflects the cleared bell.

#### Fix 3: Browser Indicators Filter Hidden Sessions

**File:** `app.js` (`updatePageTitle()` and `updateFaviconBadge()`)

Both functions currently count bells from `_currentSessions` raw. Change both to use `getVisibleSessions(_currentSessions)` before counting, so hidden sessions don't contribute to the `(N)` tab title or amber favicon dot.

#### Fix 4: Sustained Bell Clearing While Viewing Remote Sessions

Currently `bell/clear` fires once on `openSession()`. If Server B fires a new bell while the user is actively watching via federation, it won't auto-clear. Fix 2 handles this -- the poll-cycle-driven bell/clear fires every 2 seconds as long as the user is viewing the remote session, not just on initial open.

### Phase 2: Federation Settings Sync Protocol

#### Syncable vs Local-Only Keys

Defined as an allowlist constant `SYNCABLE_KEYS` in `settings.py`.

| Syncs (user experience preferences) | Never syncs (per-machine identity/infra) |
|---|---|
| `fontSize` | `host` |
| `hoverPreviewDelay` | `port` |
| `gridColumns` | `auth` |
| `bellSound` | `session_ttl` |
| `viewMode` | `tls_cert` |
| `showDeviceBadges` | `tls_key` |
| `showHoverPreview` | `device_name` |
| `activityIndicator` | `federation_key` |
| `gridViewMode` | `remote_instances` |
| `sidebarOpen` | `multi_device_enabled` |
| `sort_order` | `new_session_template` |
| `hidden_sessions` | `delete_session_template` |
| `default_session` | |
| `window_size_largest` | |
| `auto_open_created` | |

Rule of thumb: if the setting is about "how I want my muxplex experience to look/behave" it syncs. If it's about "what this specific machine is and how it connects" it doesn't. Command templates don't sync because different machines might have different tmux setups.

#### `settings_updated_at` Timestamp

**File:** `settings.py`

Add `settings_updated_at` to `DEFAULT_SETTINGS` (default: `0.0`). Whenever `patch_settings()` is called for a syncable key, this bumps to `time.time()`. This is the single version clock for last-write-wins conflict resolution.

Critical for sync loop prevention: when applying synced settings from a remote, set `settings_updated_at` to the **incoming** timestamp, not `time.time()`. This ensures timestamps converge and the next poll sees them as equal.

#### New API Endpoints

**`GET /api/settings/sync`** -- Returns `{settings: {syncable keys only}, settings_updated_at: float}`. Authenticated via federation Bearer token (same auth as existing federation endpoints).

**`PUT /api/settings/sync`** -- Accepts `{settings: {syncable keys only}, settings_updated_at: float}`. If the incoming `settings_updated_at` is newer than the local one, applies the incoming syncable keys and adopts the incoming timestamp. If local is newer, responds with 409 and the local state so the caller knows it lost. Returns the final state either way. Authenticated via federation Bearer token.

#### Sync Logic in Poll Cycle

**File:** `main.py` (`_run_poll_cycle`)

Piggybacks on the existing 2-second poll cycle. After fetching federation sessions from each reachable remote:

1. `GET /api/settings/sync` from remote
2. Compare remote `settings_updated_at` vs local `settings_updated_at`
3. If remote is newer: adopt remote syncable keys locally, update local timestamp to remote's timestamp
4. If local is newer: `PUT /api/settings/sync` to remote with local syncable keys + timestamp
5. If equal: no action

#### Frontend Impact

No frontend changes needed for sync itself. The frontend already reads from `GET /api/settings` and the `_serverSettings` cache. When sync updates the server's `settings.json`, the next `loadServerSettings()` call (or the next time the user opens the settings dialog) picks up the synced values.

## Data Flow

### Phase 1: Bell Clearing Across Federation

```
1. User opens remote session (Server B) via Server A's UI
2. Server A fires one-time POST /api/federation/{B}/sessions/{name}/bell/clear
3. Every 2 seconds, Server A's poll cycle checks local heartbeat registry:
   - Device D is viewing session "build" on remoteId 1 in fullscreen
   - Last interaction < 60s ago
4. Server A fires POST /api/federation/1/sessions/build/bell/clear
5. Server B clears the bell in its state.json
6. Next poll: Server A fetches /api/federation/sessions → bell is cleared
7. Browser updates: title count decreases, favicon badge updates
```

### Phase 2: Settings Sync

```
1. User changes fontSize to 18 on Server A via the settings dialog
2. app.js calls PATCH /api/settings {fontSize: 18}
3. Server A's patch_settings() updates settings.json, bumps settings_updated_at to t=1712600000.0
4. Next poll cycle (within 2 seconds):
   a. Server A fetches GET /api/settings/sync from Server B
   b. Server B responds: {settings: {...}, settings_updated_at: 1712599990.0}
   c. Server A's timestamp (1712600000.0) > Server B's (1712599990.0)
   d. Server A sends PUT /api/settings/sync to Server B with its settings + timestamp
   e. Server B applies the syncable keys, sets its settings_updated_at to 1712600000.0
5. Timestamps now equal → no further sync until the next user edit
```

### Offline Catch-Up

```
1. Server B goes offline at t=100 (settings_updated_at: 100)
2. User changes settings on Server A at t=200 (settings_updated_at: 200)
3. Server B comes back online
4. Next poll cycle: Server A fetches GET /api/settings/sync from B
5. Server A (200) > Server B (100) → push to B
6. Server B adopts settings + timestamp 200
7. Fully caught up in one poll cycle
```

## Error Handling

### Phase 1

- **Bell clear when remote is unreachable:** Fire-and-forget, same as the existing `openSession()` bell/clear. If the remote is down, the bell stays uncleaned -- it clears on the next successful poll after the remote comes back.
- **Heartbeat-driven bell clear race:** The 2-second poll might fire a bell/clear just as the remote fires a new bell. The next poll cycle sees the new bell and clears it again. No atomicity concerns for a single-user system.
- **`remoteId: 0` fix backward compat:** Pure behavior correction. If someone had accidentally hidden a remote-0 session, it now correctly becomes visible.

### Phase 2

- **Sync loop prevention:** When applying synced settings, set `settings_updated_at` to the **incoming** timestamp, not `time.time()`. Timestamps converge and the next poll sees them as equal.
- **Partial reachability:** A can reach B but not C. B can reach C but not A. Settings flow A→B→C transitively through successive poll cycles. Eventually consistent -- the newest timestamp propagates across the mesh.
- **Concurrent edits:** User changes `fontSize` on A at t=100, then changes `sort_order` on B at t=101. B's document wins entirely -- A's `fontSize` change is lost. Acceptable for a single-user system where edits are infrequent. Full-document approach avoids per-key merge complexity.
- **`hidden_sessions` list merge:** Full-document last-write-wins applies to the entire list. If A hides "foo" at t=100 and B hides "bar" at t=101, B's list wins (only "bar" is hidden). Known limitation, acceptable for single-user.
- **Version mismatch:** If Server A is on a newer version with new syncable keys and Server B is older, the sync `PUT` sends keys B doesn't know about. B's `patch_settings()` already ignores unknown keys -- safe by default.
- **Rolling upgrades:** The `GET/PUT /api/settings/sync` endpoints will return 404/405 on older muxplex instances. The poll loop catches these errors and skips sync for that remote, enabling rolling upgrades without breakage.

## Testing Strategy

### Phase 1 Tests

- **`getVisibleSessions()` falsy-0 fix:** Test that a session with `remoteId: 0` is NOT hidden when its name doesn't appear in `hidden_sessions`, and IS visible even when a local session with the same name IS hidden.
- **Browser indicators filter hidden sessions:** Test that `updatePageTitle()` and `updateFaviconBadge()` count bells only from visible sessions.
- **Heartbeat-driven bell clear for remote sessions:** Python test verifying that the poll cycle calls the federation bell/clear endpoint when a device is viewing a remote session in fullscreen with recent interaction.

### Phase 2 Tests

- **`SYNCABLE_KEYS` allowlist:** Python test verifying the allowlist contains the expected keys and excludes local-only keys.
- **`settings_updated_at` bumps on patch:** Python test that `patch_settings()` updates `settings_updated_at` to a recent timestamp.
- **`GET /api/settings/sync`:** Returns only syncable keys + timestamp. Authenticated via federation key.
- **`PUT /api/settings/sync`:** Applies incoming settings when incoming timestamp is newer; rejects (409) when local is newer.
- **Sync loop prevention:** Test that applying synced settings uses the incoming timestamp (not `time.time()`), so two servers with the same timestamp don't ping-pong.
- **Unknown keys in sync payload ignored:** Test that a sync payload with keys not in `SYNCABLE_KEYS` are silently dropped.

### Manual Verification Checklist

1. Change `fontSize` on Server A → verify it appears on Server B within ~4 seconds
2. Hide a local session on Server A → verify it disappears on Server B's dashboard
3. Hide a Server B session from Server A's UI → verify it propagates to Server B and hides there too
4. View a remote session in fullscreen → verify bell clears on the remote within one poll cycle
5. Tab title shows correct `(N)` count excluding hidden sessions
6. Disconnect Server B → change settings on A → reconnect B → verify B catches up

## Compatibility

Builds on top of the settings consolidation work (v0.2.1). The new `GET/PUT /api/settings/sync` endpoints gracefully fail when connecting to older muxplex instances that don't have them -- the poll loop catches HTTP 404/405 and skips sync for that remote. This enables rolling upgrades.

## Open Questions

None -- all design decisions were validated during the brainstorm phase.