muxplex/docs/plans/2026-05-17-hidden-state-redesign-design.md

# Hidden State Redesign: Filter-at-Read, Single Visibility Helper

**Date:** 2026-05-17
**Status:** Design — ready for implementation
**Author:** bkrabach

## Background

The current implementation tangles two concerns: *placement* (which view a session
belongs to) and *presentation* (whether a session is hidden). The result is a
load-bearing invariant — "a session key cannot exist in both `hidden_sessions`
and any `view.sessions`" — enforced operationally on every add/remove/sync
(`views.py:enforce_mutual_exclusion`).

Because this invariant is enforced at write time but the count display reads
raw lengths at render time, the displayed counts can disagree with the rendered
grid. There are five count computation sites; only one of them goes through the
canonical filter (`getVisibleSessions`). The others read `v.sessions.length` or
`hidden_sessions.length` directly. See `app.js:894, 913, 1010, 1266, 2640`.

Concrete failure modes today:

- A dead tmux session leaves its key in `view.sessions` forever. The dropdown
  shows `Work (3)` but the grid renders 2 tiles.
- `hidden_sessions` accumulates stale keys across the lifetime of the install.
  The Hidden view label shows `(5)` but renders 0 tiles.
- `getVisibleSessions` does not cross-check `hidden_sessions` for user views. If
  the mutual-exclusion invariant is ever transiently violated (e.g., federation
  sync race), a hidden session **would** render in a user view.

## Principles

1. **Hidden is a property, not a placement.** A session can be in zero or more
   views and is independently marked hidden or not. View membership and hidden
   state are orthogonal.
2. **One filter function.** Every list-producing path and every count goes
   through a single canonical helper. No raw `.length` on stored arrays.
3. **Default exclude hidden, opt-in to include.** Display paths use the default.
   Management paths pass `include_hidden=True` explicitly.
4. **Compose, don't fuse.** Low-level data operations stay pure
   (`addMembership`, `hide`, `unhide`). User-intent operations compose them
   explicitly (`addSessionToView` = `unhide` + `addMembership`). The auto-unhide
   behavior remains, but it lives in a named operation a reviewer can see.
5. **Operational enforcement is replaced by read-time filtering.** The
   mutual-exclusion invariant is removed.

## Design

### Data model (no schema changes)

Storage is unchanged. Only the contract changes:

```json
{
  "views": [
    { "name": "Work", "sessions": ["dev1:proj", "dev1:build"] }
  ],
  "hidden_sessions": ["dev1:build", "dev1:old"]
}
```

The same key may now appear in both `view.sessions` and `hidden_sessions`. That
combination means: "this session is a member of Work, but is currently hidden."
It will not appear in the Work view by default; it will appear in the Hidden
view; it will appear *dimmed* in the Manage View panel for Work.

### The visibility helper (frontend)

Single function. Replaces all five count sites and `getVisibleSessions`.

```js
// app.js
function isHidden(key, hiddenSet) {
  return hiddenSet.has(key);
}

// Returns the canonical list of sessions for the requested view.
// view: "all" | "hidden" | <user view name>
// includeHidden: when true, do not filter hidden sessions out of user views
//                or "all". Ignored for "hidden" (always shows hidden only).
function filterVisible(sessions, settings, view, { includeHidden = false } = {}) {
  const hiddenSet = new Set(settings.hidden_sessions || []);
  const keyOf = s => s.sessionKey || s.name;

  // status-tile entries are never counted as sessions
  const live = (sessions || []).filter(s => !s.status);

  if (view === "hidden") {
    return live.filter(s => isHidden(keyOf(s), hiddenSet) || hiddenSet.has(s.name));
  }

  const visibilityPass = includeHidden
    ? () => true
    : s => !isHidden(keyOf(s), hiddenSet) && !hiddenSet.has(s.name);

  if (view === "all") {
    return live.filter(visibilityPass);
  }

  const userView = (settings.views || []).find(v => v.name === view);
  if (!userView) return [];
  const memberSet = new Set(userView.sessions || []);
  return live.filter(s =>
    (memberSet.has(keyOf(s)) || memberSet.has(s.name)) && visibilityPass(s)
  );
}

function visibleCount(...args) {
  return filterVisible(...args).length;
}
```

Counts that displayed `(N)` previously now call `visibleCount(_currentSessions,
settings, viewName)`. Manage View counts call
`visibleCount(_currentSessions, settings, viewName, { includeHidden: true })`
and additionally report the hidden-of-total breakdown (see UI section).

### Backend helper

If the server ever computes session lists (e.g., for sync or for a future API),
mirror the helper in Python:

```python
# views.py (replacing enforce_mutual_exclusion)

def is_hidden(key: str, settings: dict) -> bool:
    return key in (settings.get("hidden_sessions") or [])

def filter_visible(
    sessions: list[dict],
    settings: dict,
    view: str,
    *,
    include_hidden: bool = False,
) -> list[dict]:
    hidden = set(settings.get("hidden_sessions") or [])

    def key_of(s): return s.get("sessionKey") or s.get("name")
    def visibility_pass(s):
        if include_hidden:
            return True
        k = key_of(s)
        return k not in hidden and s.get("name") not in hidden

    live = [s for s in sessions if not s.get("status")]
    if view == "hidden":
        return [s for s in live if key_of(s) in hidden or s.get("name") in hidden]
    if view == "all":
        return [s for s in live if visibility_pass(s)]

    user_view = next((v for v in (settings.get("views") or []) if v["name"] == view), None)
    if not user_view:
        return []
    members = set(user_view.get("sessions") or [])
    return [
        s for s in live
        if (key_of(s) in members or s.get("name") in members) and visibility_pass(s)
    ]
```

### Operations: pure vs user-intent

Two layers. The UI calls user-intent operations. Tests verify both layers.

**Pure data operations** (low-level, no side effects beyond their name):

| Operation              | Effect                                                       |
|------------------------|--------------------------------------------------------------|
| `addMembership(k, v)`  | Append `k` to `settings.views[v].sessions` if absent         |
| `removeMembership(k, v)` | Remove `k` from `settings.views[v].sessions`               |
| `hide(k)`              | Append `k` to `settings.hidden_sessions` if absent           |
| `unhide(k)`            | Remove `k` from `settings.hidden_sessions`                   |

**User-intent operations** (high-level, used by UI):

| Operation                          | Composition                                                 |
|------------------------------------|-------------------------------------------------------------|
| `addSessionToView(k, v)`           | `unhide(k); addMembership(k, v)` — **auto-unhide here**     |
| `removeSessionFromView(k, v)`      | `removeMembership(k, v)` — does *not* hide                  |
| `hideSession(k)`                   | `hide(k)` — does *not* touch view membership                |
| `unhideSession(k)`                 | `unhide(k)` — does *not* touch view membership              |

The auto-unhide on "Add to View" is preserved as a deliberate user-intent
composition, **not** as an invariant enforced everywhere. Reviewers see
exactly one call to `unhide()` adjacent to `addMembership()` in
`addSessionToView`. Future operations that want to add membership without
unhiding (e.g., a bulk migration tool) can call `addMembership` directly and
the existence of that uncomposed path is obvious in code review.

### Federation sync

**Reality check first.** The current sync model (`settings.py:164-184`) is
**whole-list last-write-wins**: `apply_synced_settings()` replaces each
syncable key wholesale from the incoming payload, then runs
`enforce_mutual_exclusion` as a sanitization pass. Conflict resolution is via
a single `settings_updated_at` timestamp on the entire settings blob. There is
no per-key merge, no CRDT, no add-wins/remove-wins set semantics.

This refactor does **not** change that. We accept whole-list LWW for v1 and
document the consequences:

- If device A hides session X while device B adds X to a view, whichever
  device last wrote settings wins for *both* keys (`hidden_sessions` and
  `views`). The user may see a brief flicker between sync rounds. This is the
  same trade-off every other syncable setting already makes.
- Removing `enforce_mutual_exclusion` from `apply_synced_settings` is **not**
  safe in mixed-version federation (see Mixed-version safety below). It stays
  in place for v1 as a backstop. The read-time filter does not depend on the
  invariant for correctness; it just needs to handle overlap when overlap
  exists. Letting `enforce_mutual_exclusion` continue to strip overlap on
  incoming sync is fine — new code tolerates either presence or absence of
  overlap.

This means **Phase 3 (delete `enforce_mutual_exclusion`) is deferred** out of
the v1 plan. It becomes safe to remove only after all federated devices
publish a `_schema_version >= 2`. See Phase 3 in the implementation phases
section.

### Mixed-version safety (the blocking concern)

Without a schema version field, mixed-version federation will silently revert
user actions:

1. User on new device A hides session X (X is in view Work).
2. A's settings now have X in both `hidden_sessions` and `Work.sessions`.
3. Sync to old device B. B applies the new settings, then runs
   `enforce_mutual_exclusion` — silently strips X from `hidden_sessions`.
4. B saves with new `settings_updated_at`. Sync back to A. X is now unhidden.
5. User re-hides. Oscillation.

The mitigation: add a `_schema_version: int` field in **Phase 0** (a
prerequisite to all other phases). New clients write `_schema_version: 2`
into settings whenever they save. New clients detect old peers during
federation handshake (old peers send no `_schema_version` or `< 2`) and
behave conservatively:

- When sending to a legacy peer: do not produce settings where the same key
  appears in both `hidden_sessions` and any `view.sessions`. Pre-flatten by
  calling `enforce_mutual_exclusion` on the outgoing payload.
- When receiving from a legacy peer: accept the incoming state as-is (it
  already respects the old invariant).

This keeps mixed-version operation safe. Once all peers report
`_schema_version >= 2`, the pre-flatten step is unnecessary. Detection of
"all peers uniform" is a manual operator action for v1 (a log message
suffices); we do not need to automate it before deleting
`enforce_mutual_exclusion`.

### Stale key pruning (separate concern, local-only state)

Filtering at read time means stale keys no longer corrupt counts. But
`hidden_sessions` and `view.sessions` still accumulate dead keys forever and
sync them across the federation. Add a periodic pruning pass.

**Critical: pruning bookkeeping does NOT sync.** First-missed-at timestamps
are a per-device concern. Each device tracks which keys it has personally
failed to see, and prunes from its own settings when its grace period expires.
The actual prune (removing the key from `view.sessions` or `hidden_sessions`)
is a normal settings write that *does* sync via the existing LWW mechanism.
This avoids inventing new federation merge semantics.

Bookkeeping lives in a local sidecar file, not in `settings.json`:

```
~/.config/muxplex/pruning.json
{
  "first_missed_at": {
    "dev1:dead-session": 1747512345.0
  }
}
```

`pruning.json` is **not** in `SYNCABLE_KEYS` and is never sent to peers.

```python
def prune_stale_keys(settings: dict, live_keys: set[str], grace_seconds: float) -> dict:
    # 1. Update local first-missed-at bookkeeping based on live_keys.
    # 2. For each key in hidden_sessions or any view.sessions:
    #    - If key in live_keys: remove from first-missed-at (if present).
    #    - If key not in live_keys: ensure first-missed-at has this key.
    #      If first_missed_at[key] + grace_seconds < now: drop the key.
    # 3. Return modified settings (caller decides whether to save).
```

Runs on `_run_poll_cycle()` after the live session list is gathered
(`main.py:173`).

**Grace period rationale.** A tmux session disappears briefly during restart
cycles — minutes, occasionally hours. A user's intent to keep a session in a
view, however, can span days (a vacation, a long context-switch). The tension
is between cleaning up genuinely dead keys quickly and not surprising users.
Default: 24 hours, configurable via a new `stale_key_grace_hours` setting.
This is well above the worst-case restart duration and well below "I forgot
this view existed" timescales. Document the choice; revise based on use.

In a federated install, the grace period applies *per device*. A key not
seen on device A for 24h is dropped on A; B may still be tracking it.
B's eventual sync receives the deletion from A. This is the correct
behavior — A speaks for itself.

This phase is independent of the visibility refactor and can ship in its
own commit.

## UI changes

### Manage View panel

- Loads with `include_hidden=True`.
- Hidden sessions render dimmed (CSS class `session-hidden-dim`, ~50%
  opacity).
- Checkbox state for hidden sessions reflects membership (in view or not),
  independent of hidden state.
- Checking a hidden session that is not yet in the view calls
  `addSessionToView` — which unhides AND adds. The dim styling disappears.
- Unchecking a session calls `removeSessionFromView` (does not hide).
- A small "(hidden)" badge confirms hidden state for clarity.

### Settings panel view list

Show both numbers when relevant:

```
Work: 5 sessions (2 hidden)
Personal: 3 sessions
```

The bare "N sessions" form is used when no sessions in the view are hidden.
The "(M hidden)" suffix appears only when M > 0.

### Header & sidebar dropdowns

All counts go through `visibleCount(..., view, { includeHidden: false })`.
The bare numeric badge (e.g., `Work (3)`) reflects the count of sessions
actually rendered in the grid when that view is active. No exceptions.

### "Hidden" view label

The "Hidden" view count uses `visibleCount(..., "hidden")` — only counts
hidden sessions that are *live*. A stale entry in `hidden_sessions` for a
dead tmux session does not inflate this count. (The stale entry will be
removed by `prune_stale_keys` after the grace period.)

## Migration & backward compatibility

- **Settings files:** schema-compatible. Old files have no overlap between
  `hidden_sessions` and `view.sessions`; the new code handles that correctly.
  Until Phase 3 (deferred), new code never *writes* overlap either — it
  keeps `enforce_mutual_exclusion` as a backstop, so the on-disk schema does
  not change.
- **Schema version:** add `_schema_version: 2` to settings in Phase 0. Old
  clients ignore unknown keys (`settings.py:load_settings` only copies keys
  present in `DEFAULT_SETTINGS`), so this is safe to write. The version
  exists primarily as a marker for future Phase 3 — when all peers report
  `>= 2`, it is safe to delete `enforce_mutual_exclusion`.
- **Old clients reading new state:** identical to today (no overlap exists
  to drop, because we still call `enforce_mutual_exclusion` on save).
- **Key format normalization:** Phase 1 includes a one-time pass that walks
  `views[].sessions` and `hidden_sessions` on first load by the new code,
  upgrading bare-name entries to the canonical `device_id:name` form where a
  match exists in the current live session list. Entries that cannot be
  matched are preserved (they may match later). This pays the migration cost
  once and removes the need for dual-lookup at every read site.

## Implementation phases

Each phase is independently shippable. Order matters: Phase 0 is a
prerequisite to all others.

### Phase 0 — Schema version field

Tiny but load-bearing. Future-proofs the federation upgrade path.

- Add `_schema_version` to `DEFAULT_SETTINGS` (`settings.py:17-47`) with
  value `2`. Add to `SYNCABLE_KEYS` so peers see it.
- On every `save_settings`/`patch_settings` write, ensure
  `_schema_version` is set to `2` regardless of incoming patch (clients
  don't get to write older versions).
- Add a helper `peer_supports_v2(peer_settings)` that returns
  `peer_settings.get("_schema_version", 0) >= 2`.
- No behavioral change yet. This is just the marker.

### Phase 1 — Visibility helper + audit + key normalization

Single most important change. Source of truth for visibility lives here.

- Add `filterVisible()` and `visibleCount()` to `app.js`.
- Add `filter_visible()` to `views.py` (for backend symmetry, even if no
  current Python caller needs it).
- Replace every count site:
  - `app.js:894` (Hidden count) → `visibleCount(..., "hidden")`
  - `app.js:900-903` (All Sessions header) → `visibleCount(..., "all")`
  - `app.js:913` (User view header dropdown) → `visibleCount(..., v.name)`
  - `app.js:997-1000` (All Sessions sidebar) → same
  - `app.js:1010` (User view sidebar) → same
  - `app.js:1266` (Settings panel) → same, with `(M hidden)` suffix
  - `app.js:2640` (Manage View "in this view") →
    `visibleCount(..., view, { includeHidden: true })`
- Replace `getVisibleSessions()` body with a call to `filterVisible()`.
- Grep audit: search for `.sessions.length`, `hidden_sessions.length`,
  `hidden.length`. Each remaining occurrence must be justified in a comment
  or replaced.
- **One-time key normalization**: on first load by the new code (detected via
  absence of `_schema_version` in the on-disk settings, or via a new marker),
  walk `views[].sessions` and `hidden_sessions`. For each entry that is a
  bare `name`, check if any live session has a matching `name`; if so,
  upgrade the stored entry to `device_id:name` form. Entries that cannot be
  matched (no live session with that name) are left as-is. Run once, set
  `_schema_version: 2`, save. Subsequent loads skip the pass.
- After normalization lands, the `memberSet.has(keyOf(s)) || memberSet.has(s.name)`
  dual-lookup can be simplified to a single lookup. Keep the dual lookup
  initially — remove it in a follow-up commit once normalization has run on
  all production installs.

### Phase 2 — Operation layer

- Refactor the existing hide/unhide/add-to-view code paths into the four pure
  + four user-intent operations described above.
- Update existing call sites in `app.js` (`hideSession`, `unhideSession`,
  add-to-view flyout, Manage View checkbox handler) to call the user-intent
  operations.
- Document the layering in a header comment.

### Phase 3 — Remove `enforce_mutual_exclusion` *(deferred from v1)*

**Not part of v1.** Listed here for completeness; do not ship until all
federated peers report `_schema_version >= 2` and the operator confirms it
is safe to drop the legacy backstop.

When the time comes:
- Delete `views.py:enforce_mutual_exclusion` and its call site in
  `settings.py:apply_synced_settings()`.
- Delete the corresponding test cases in `tests/test_views.py`.
- Remove the outgoing-payload pre-flatten step added in Phase 0.
- Bump `_schema_version` to 3 if any further on-disk format change accompanies
  the removal.

Until this phase ships, the new code coexists with the old invariant. Counts
and rendering are correct; storage continues to enforce mutual exclusion. The
only user-visible cost is that the auto-unhide-on-add behavior cannot be
broken into "add without unhiding" without `enforce_mutual_exclusion`
reverting it on the next save. This is acceptable for v1 — that capability
is not exposed to users.

### Phase 4 — Stale key pruning (local-only bookkeeping)

- Add `prune_stale_keys()` in `views.py`. Reads & writes a local sidecar
  file `~/.config/muxplex/pruning.json` (see the design section above).
- The sidecar is **not** in `SYNCABLE_KEYS` and is never sent to peers.
- Wire into `_run_poll_cycle()` after the live session collection step
  (`main.py:173`).
- Add `stale_key_grace_hours` (default 24) to `DEFAULT_SETTINGS`. Syncable;
  affects every device that consumes it. Each device still tracks its own
  first-missed-at timestamps locally.

### Phase 5 — UI polish

- Manage View dim styling.
- Settings panel "(M hidden)" suffix.
- Any other discoverability improvements.

Phases 0–2 are the actual semantic change. Phase 4 is hygiene. Phase 5 is
presentation. Phase 3 is deferred. Don't combine them.

## Test plan

### New tests (frontend, `frontend/tests/test_app.mjs`)

- `filterVisible` matrix — every combination of:
  - view ∈ {"all", "hidden", `<user view>`, `<nonexistent view>`}
  - `includeHidden` ∈ {true, false}
  - session in / absent from `view.sessions`
  - key in / absent from `hidden_sessions`
  - key in both (the new permitted state; verify v1 storage never produces it,
    but the helper handles it correctly if it appears)
  - session-key format: `device_id:name`, bare `name`, both
- `visibleCount` matches `filterVisible().length` for the same inputs.
- Key normalization: bare-name entries upgrade to `device_id:name` when a
  match exists; remain bare when no match exists; idempotent across reloads.

### New tests (backend, `tests/test_views.py`, `tests/test_settings.py`)

- `filter_visible` mirroring the frontend matrix.
- `addSessionToView` unhides and adds membership.
- `addMembership` adds membership but does **not** unhide.
- `removeSessionFromView` removes membership but does **not** hide.
- `hide` and `unhide` do not touch view membership.
- Schema version: `save_settings` always writes `_schema_version: 2`.
- `peer_supports_v2`: returns true for `_schema_version >= 2`, false for
  missing or `< 2`.

### Federation tests (`tests/test_settings.py` or new `test_federation.py`)

- Round-trip: state with overlap, save, reload, assert content unchanged
  (with `enforce_mutual_exclusion` still active, overlap should be stripped
  on save — verify this is the v1 invariant).
- Mixed-version send: when serializing settings for a peer reporting
  `_schema_version < 2`, the outgoing payload has no overlap. Verify by
  constructing settings with overlap (force it past the backstop in a test
  helper) and asserting the outgoing form is flattened.
- Mixed-version receive: applying settings from a legacy peer is a no-op
  beyond the existing whole-list replacement. Verify `_schema_version` is
  not downgraded by an incoming payload that lacks it.

### Stale-key pruning tests (`tests/test_views.py`)

- Live key: pruning bookkeeping is cleared.
- Newly missing key: bookkeeping records `first_missed_at`, settings unchanged.
- Missing past grace: key removed from `view.sessions` and `hidden_sessions`,
  bookkeeping entry removed.
- Re-appeared after partial absence (less than grace): bookkeeping cleared,
  settings unchanged.
- Sidecar file: never written to `settings.json`, never sent over the wire.

### Updated tests

- Adapt existing view tests that asserted the old mutual-exclusion invariant.
  Several tests assume `add_to_view` removes from `hidden_sessions` as a side
  effect; rewrite them to call the user-intent operation explicitly and
  assert on both effects. (Keep the existing `enforce_mutual_exclusion`
  tests — that function stays in v1.)

### Removed tests

- None in v1. `enforce_mutual_exclusion` tests stay until Phase 3 ships.

## Out of scope

- API redesign for `/api/views` or `/api/sessions`. The wire format is
  unchanged.
- Conflict UI for federation merges. Last-write-wins on the hidden flag is
  acceptable for v1.
- Bulk operations (hide all in view, etc.). The current granularity is
  sufficient.
- Tooltip / hover state showing why a session is dimmed. The "(hidden)" badge
  is enough.

## Open questions

None that block implementation. Worth deciding before Phase 5:

- Should the Manage View panel default to `include_hidden=True`, or require
  an explicit "Show hidden" toggle? Recommendation: default to true. The
  whole point of the Manage View is management; hiding hidden items from a
  management UI defeats the purpose.

## References

- COE design review conversation, 2026-05-17 (workspace session
  e74700a2-5340-46be-8fae-df40a19204f3).
- Current implementation: `views.py:14-68`, `settings.py:24-65, 164-184`,
  `state.py:9`, `app.js:628-672, 894-1010, 1266, 2173-2236, 2640`.