embedder UI fails silently when backend unavailable: rpcCall and panels both swallow errors #20

New issue

Closed

opened 2026-04-27 07:10:45 +00:00 by salmaelsoly · 2 comments

salmaelsoly commented 2026-04-27 07:10:45 +00:00

Member

Copy link

Summary

When the embedder backend is unreachable, the Embedder admin dashboard becomes silently unusable: every panel that hits the JSON-RPC backend gets stuck on "Loading..." forever, and the only error surfaces is a cryptic Unexpected token 'S', "Socket 'rp"... is not valid JSON in the JS console. There is no on-screen indication of the actual problem.

This is the combination of two defects in hero_embedder_ui that together produce the broken UX:

1. rpcCall() blind-parses the response as JSON — no response.ok / Content-Type check before await response.json(). When hero_router returns text/plain (e.g. a 404 with body Socket 'rpc.sock' not found for 'hero_embedder'), JSON.parse throws "Unexpected token" and the actual error context is lost.
2. Panel-level pollers silently swallow the resulting error — every panel's catch block calls console.error(...) and never updates the UI, so the user sees a perpetual "Loading..." with no indication of what went wrong.

Both need to be fixed for the dashboard to fail gracefully when the backend is down.

Reproduction

1. With hero_embedder_server not running (so ~/hero/var/sockets/hero_embedder/rpc.sock does not exist), open the Embedder dashboard.
2. Observed HTTP exchange (every panel triggers the same):

POST /hero_embedder/ui/rpc HTTP/1.1
Host: 127.0.0.1:9151
Content-Type: application/json

{"jsonrpc":"2.0","id":1,"method":"info","params":[]}

HTTP/1.1 404 Not Found
content-type: text/plain; charset=utf-8
content-length: 47

Socket 'rpc.sock' not found for 'hero_embedder'

3. JS console shows: Unexpected token 'S', "Socket 'rp"... is not valid JSON.
4. Affected panels stay stuck at "Loading..." forever:
   • Server Stats card
   • Namespace Stats card
   • Namespace dropdown (header)
   • Loader pane
   • Logs tab
5. Embed / Rerank / Add Docs / KVS form submissions show the same cryptic toast.

Root cause — Part 1: blind JSON parse in rpcCall

crates/hero_embedder_ui/templates/base.html:177-219 (rpcCall):

const response = await fetch(API_URL + "/rpc", { method: "POST", headers, body: ..., signal: ... });
clearTimeout(timeoutId);
// ...
const data = await response.json();   // <-- blind JSON parse, no ok/content-type check
// ...
if (data.error) throw new Error(data.error.message);
return data.result;

There is no validation of response.ok or the Content-Type header before parsing. Any non-JSON 4xx/5xx body — hero_router's 404 text/plain when the upstream socket is missing, future 502s from any proxy, gzip-encoded HTML error pages — surfaces as "Unexpected token" and loses the real error.

Root cause — Part 2: silent panel-level error handling

Each panel's poller wraps rpcCall in try/catch but only console.errors on failure. Affected sites:

• crates/hero_embedder_ui/templates/components/stats_card.html ~lines 175-186 (pollServerStats)
• crates/hero_embedder_ui/templates/components/stats_card.html ~lines 197-216 (pollNamespaceStats)
• crates/hero_embedder_ui/templates/base.html ~lines 133-175 (loadNamespaces)
• crates/hero_embedder_ui/templates/fragments/loader_pane.html ~lines 193-218 (loader refresh)
• crates/hero_embedder_ui/templates/fragments/logs_pane.html ~lines 102-120 (logs poll)

Pattern is identical across all of them:

try {
  const data = await rpcCall("...");
  // populate panel
} catch (err) {
  console.error("[Panel] Poll error:", err);   // <-- silent, UI never updated
}

Suggested fix direction

Part 1 — rpcCall (single function in base.html)

Validate the response shape before parsing, and throw an error message that contains the response body (truncated) when the response is non-JSON or non-OK:

const response = await fetch(API_URL + "/rpc", { ... });
clearTimeout(timeoutId);

const contentType = response.headers.get("content-type") || "";
const isJson = contentType.includes("application/json");

if (!response.ok && !isJson) {
  const text = await response.text();
  throw new Error(`Backend unavailable (HTTP ${response.status}): ${text.slice(0, 200)}`);
}

const data = await response.json();
if (data.error) throw new Error(data.error.message);
return data.result;

Part 2 — panel-level error rendering

Add a small shared helper in base.html:

function renderPanelError(containerId, message) {
  const el = document.getElementById(containerId);
  if (!el) return;
  el.innerHTML =
    `<div class="alert alert-warning small mb-0">` +
    `<i class="bi bi-exclamation-triangle me-1"></i>` +
    escapeHtml(message) +
    `</div>`;
}

Replace each silent console.error with renderPanelError("<panel-id>", err.message) (keep console.error too if useful for dev). For the namespace <select>, swap the inner content with a single disabled option whose text is the error.

When the next poll succeeds, the panel render path naturally overwrites the error state — no extra recovery code needed.

Acceptance criteria

• rpcCall no longer throws "Unexpected token" when the response body is not JSON; instead it throws a meaningful Error containing the HTTP status and a snippet of the response body.
• On OK JSON responses, rpcCall behaviour is unchanged (no regressions in the success path).
• Each affected panel displays a visible inline error when its RPC call fails (Server Stats, Namespace Stats, namespace dropdown, Loader pane, Logs tab).
• On the next successful poll, the panel recovers from the error state and shows real data without page refresh.
• No panel stays at "Loading..." indefinitely when the backend is unreachable.

Notes

• The Socket 'rpc.sock' not found for 'hero_embedder' body itself is emitted by hero_router (crates/hero_router/src/server/routes.rs:1373-1377 in the hero_router repo) for non-HTML clients. That router-side behaviour is debatable separately, but this UI-side bug stands on its own — the dashboard must not break when the upstream returns any non-JSON body, regardless of what hero_router chooses to send.
• The same rpcCall defensive pattern is reusable in other Hero *_ui dashboards; a quick scan suggests hero_browser_ui's rpc() helper has the same issue. Out of scope for this issue but worth porting later.
• Out of scope: exponential backoff in pollers (separate concern, may be filed later if needed).

## Summary When the embedder backend is unreachable, the Embedder admin dashboard becomes silently unusable: every panel that hits the JSON-RPC backend gets stuck on "Loading..." forever, and the only error surfaces is a cryptic `Unexpected token 'S', "Socket 'rp"... is not valid JSON` in the JS console. There is no on-screen indication of the actual problem. This is the combination of two defects in `hero_embedder_ui` that together produce the broken UX: 1. **`rpcCall()` blind-parses the response as JSON** — no `response.ok` / `Content-Type` check before `await response.json()`. When `hero_router` returns `text/plain` (e.g. a 404 with body `Socket 'rpc.sock' not found for 'hero_embedder'`), `JSON.parse` throws "Unexpected token" and the actual error context is lost. 2. **Panel-level pollers silently swallow the resulting error** — every panel's `catch` block calls `console.error(...)` and never updates the UI, so the user sees a perpetual "Loading..." with no indication of what went wrong. Both need to be fixed for the dashboard to fail gracefully when the backend is down. ## Reproduction 1. With `hero_embedder_server` not running (so `~/hero/var/sockets/hero_embedder/rpc.sock` does not exist), open the Embedder dashboard. 2. Observed HTTP exchange (every panel triggers the same): ``` POST /hero_embedder/ui/rpc HTTP/1.1 Host: 127.0.0.1:9151 Content-Type: application/json {"jsonrpc":"2.0","id":1,"method":"info","params":[]} HTTP/1.1 404 Not Found content-type: text/plain; charset=utf-8 content-length: 47 Socket 'rpc.sock' not found for 'hero_embedder' ``` 3. JS console shows: `Unexpected token 'S', "Socket 'rp"... is not valid JSON`. 4. Affected panels stay stuck at "Loading..." forever: - Server Stats card - Namespace Stats card - Namespace dropdown (header) - Loader pane - Logs tab 5. Embed / Rerank / Add Docs / KVS form submissions show the same cryptic toast. ## Root cause — Part 1: blind JSON parse in `rpcCall` `crates/hero_embedder_ui/templates/base.html:177-219` (`rpcCall`): ```js const response = await fetch(API_URL + "/rpc", { method: "POST", headers, body: ..., signal: ... }); clearTimeout(timeoutId); // ... const data = await response.json(); // <-- blind JSON parse, no ok/content-type check // ... if (data.error) throw new Error(data.error.message); return data.result; ``` There is no validation of `response.ok` or the `Content-Type` header before parsing. Any non-JSON 4xx/5xx body — `hero_router`'s 404 text/plain when the upstream socket is missing, future 502s from any proxy, gzip-encoded HTML error pages — surfaces as "Unexpected token" and loses the real error. ## Root cause — Part 2: silent panel-level error handling Each panel's poller wraps `rpcCall` in `try/catch` but only `console.error`s on failure. Affected sites: - `crates/hero_embedder_ui/templates/components/stats_card.html` ~lines 175-186 (`pollServerStats`) - `crates/hero_embedder_ui/templates/components/stats_card.html` ~lines 197-216 (`pollNamespaceStats`) - `crates/hero_embedder_ui/templates/base.html` ~lines 133-175 (`loadNamespaces`) - `crates/hero_embedder_ui/templates/fragments/loader_pane.html` ~lines 193-218 (loader refresh) - `crates/hero_embedder_ui/templates/fragments/logs_pane.html` ~lines 102-120 (logs poll) Pattern is identical across all of them: ```js try { const data = await rpcCall("..."); // populate panel } catch (err) { console.error("[Panel] Poll error:", err); // <-- silent, UI never updated } ``` ## Suggested fix direction ### Part 1 — `rpcCall` (single function in `base.html`) Validate the response shape before parsing, and throw an error message that contains the response body (truncated) when the response is non-JSON or non-OK: ```js const response = await fetch(API_URL + "/rpc", { ... }); clearTimeout(timeoutId); const contentType = response.headers.get("content-type") || ""; const isJson = contentType.includes("application/json"); if (!response.ok && !isJson) { const text = await response.text(); throw new Error(`Backend unavailable (HTTP ${response.status}): ${text.slice(0, 200)}`); } const data = await response.json(); if (data.error) throw new Error(data.error.message); return data.result; ``` ### Part 2 — panel-level error rendering Add a small shared helper in `base.html`: ```js function renderPanelError(containerId, message) { const el = document.getElementById(containerId); if (!el) return; el.innerHTML = `<div class="alert alert-warning small mb-0">` + `<i class="bi bi-exclamation-triangle me-1"></i>` + escapeHtml(message) + `</div>`; } ``` Replace each silent `console.error` with `renderPanelError("<panel-id>", err.message)` (keep `console.error` too if useful for dev). For the namespace `<select>`, swap the inner content with a single disabled option whose text is the error. When the next poll succeeds, the panel render path naturally overwrites the error state — no extra recovery code needed. ## Acceptance criteria - [ ] `rpcCall` no longer throws "Unexpected token" when the response body is not JSON; instead it throws a meaningful `Error` containing the HTTP status and a snippet of the response body. - [ ] On OK JSON responses, `rpcCall` behaviour is unchanged (no regressions in the success path). - [ ] Each affected panel displays a visible inline error when its RPC call fails (Server Stats, Namespace Stats, namespace dropdown, Loader pane, Logs tab). - [ ] On the next successful poll, the panel recovers from the error state and shows real data without page refresh. - [ ] No panel stays at "Loading..." indefinitely when the backend is unreachable. ## Notes - The `Socket 'rpc.sock' not found for 'hero_embedder'` body itself is emitted by `hero_router` (`crates/hero_router/src/server/routes.rs:1373-1377` in the `hero_router` repo) for non-HTML clients. That router-side behaviour is debatable separately, but this UI-side bug stands on its own — the dashboard must not break when the upstream returns any non-JSON body, regardless of what `hero_router` chooses to send. - The same `rpcCall` defensive pattern is reusable in other Hero `*_ui` dashboards; a quick scan suggests `hero_browser_ui`'s `rpc()` helper has the same issue. Out of scope for this issue but worth porting later. - Out of scope: exponential backoff in pollers (separate concern, may be filed later if needed).

salmaelsoly commented 2026-04-27 07:20:01 +00:00

Author

Member

Copy link

Implementation Spec for Issue #20

Objective

Make the hero_embedder_ui dashboard fail loudly and recoverably when hero_embedder_server (or any upstream rpc.sock) is unavailable. Two coordinated changes: (1) rpcCall() in base.html must validate the HTTP response before parsing as JSON, so non-JSON 4xx/5xx replies (e.g. hero_router's text/plain 404 Socket 'rpc.sock' not found for 'hero_embedder') surface as readable error messages instead of "Unexpected token 'S'…" parse failures; (2) every panel-level poller stops swallowing errors with console.error and renders a visible inline alert (or, for the namespace dropdown, a disabled error option) that is automatically cleared by the next successful poll.

Files to Modify/Create

• crates/hero_embedder_ui/templates/base.html — fix rpcCall (response validation), add shared escapeHtml helper, add shared renderPanelError helper, replace loadNamespaces console.error with disabled-option error state on the <select>.
• crates/hero_embedder_ui/templates/components/stats_card.html — replace silent console.error in pollServerStats and pollNamespaceStats with renderPanelError calls into the existing panel containers.
• crates/hero_embedder_ui/templates/fragments/loader_pane.html — add a stable error container in the loader card; replace silent console.error in refreshLoaderState (and the secondary index.clear .catch) with a renderPanelError call against it.
• crates/hero_embedder_ui/templates/fragments/logs_pane.html — replace silent console.error in LogsManager.loadLogs and clearLogs with renderPanelError calls into logs-content.

No new files. No backend Rust changes.

Helper specifications (placed in base.html, same <script> block as rpcCall)

function escapeHtml(s) {
    return String(s).replace(/[&<>"']/g, c => ({
        "&": "&amp;",
        "<": "&lt;",
        ">": "&gt;",
        '"': "&quot;",
        "'": "&#39;"
    })[c]);
}

function renderPanelError(containerId, message) {
    const el = document.getElementById(containerId);
    if (!el) return;
    el.innerHTML =
        '<div class="alert alert-warning small mb-0">' +
        '<i class="bi bi-exclamation-triangle me-1"></i>' +
        escapeHtml(message) +
        '</div>';
}

rpcCall rewrite (drop into base.html lines 177–219)

Insert immediately before the existing await response.json():

const contentType = response.headers.get("content-type") || "";
const isJson = contentType.includes("application/json");

if (!response.ok && !isJson) {
    const text = await response.text();
    throw new Error(`Backend unavailable (HTTP ${response.status}): ${text.slice(0, 200)}`);
}

const data = await response.json();

Keep the existing data.error check, data.result return, response-time DOM update, and AbortError handling unchanged.

Implementation Plan

Step 1: base.html — rpcCall validation, helpers, namespace dropdown error state

Files: crates/hero_embedder_ui/templates/base.html
Dependencies: none

• Inside the <script> block that begins at line 94, add escapeHtml and renderPanelError definitions before rpcCall (so all later code in the file and all included panel files can use them).
• Modify rpcCall (lines 177–219) to perform the response.ok + Content-Type validation described above before await response.json().
• Modify loadNamespaces (lines 133–175). Replace the console.error-only catch with: clear select.innerHTML, append one <option value="" disabled selected> whose textContent is "Backend unavailable: " + e.message, and set select.disabled = true. On the success path, add select.disabled = false before the existing select.innerHTML = "".

Step 2: stats_card.html — Server Stats and Namespace Stats panels

Files: crates/hero_embedder_ui/templates/components/stats_card.html
Dependencies: Step 1

• In pollServerStats (~lines 175–186): replace console.error("[ServerStats] Poll error:", err); with renderPanelError("server-stats-content", "Server stats unavailable: " + err.message);. Recovery is automatic — the success path already does container.outerHTML = renderServerStats(...) which re-creates #server-stats-content.
• In pollNamespaceStats (~lines 197–216): replace the console.error with renderPanelError("namespace-stats-area", "Namespace stats unavailable: " + err.message);. Recovery is automatic via the existing container.innerHTML = renderNamespaceStats(data) on success.
• Do NOT touch the Clear Namespace button handler (~lines 245–268) or the Download TriviaQA handler (~lines 271–317); they already render visible per-button error states.

Step 3: loader_pane.html — Corpus Loader pane

Files: crates/hero_embedder_ui/templates/fragments/loader_pane.html
Dependencies: Step 1

• Add a stable error container inside the card body. Insert immediately after <div id="load-progress" class="mt-2"></div> (~line 63): <div id="loader-error" class="mt-2"></div>.
• In refreshLoaderState (~lines 193–218): at the top of the try block, clear the error via const errEl = document.getElementById("loader-error"); if (errEl) errEl.innerHTML = ""; so a successful refresh wipes any previous error. In the catch, replace the console.error with renderPanelError("loader-error", "Loader unavailable: " + e.message);.
• In the reset-button handler (~lines 231–242): replace the console.error with renderPanelError("loader-error", "Failed to clear namespace: " + e.message);.
• Do NOT change the load-button success/error rendering (~lines 318–321); it already uses an alert alert-danger block in #load-progress.

Step 4: logs_pane.html — Logs tab

Files: crates/hero_embedder_ui/templates/fragments/logs_pane.html
Dependencies: Step 1

• In LogsManager.loadLogs (~lines 102–120): replace console.error("[LogsManager] Error loading logs:", error); with renderPanelError("logs-content", "Logs unavailable: " + error.message);. Recovery is automatic — renderLogs() always calls container.innerHTML = ... on success.
• In LogsManager.clearLogs (~lines 197–206): replace the console.error with renderPanelError("logs-content", "Failed to clear logs: " + error.message);.

Parallelization summary

Step 1 must complete first because Steps 2/3/4 all call renderPanelError. Once Step 1 is done, Steps 2, 3, and 4 are fully independent (different files, no shared symbols) and can be implemented in parallel.

Acceptance Criteria

• rpcCall no longer throws SyntaxError: Unexpected token 'S', "Socket 'rp"... when the upstream returns text/plain 404. Instead it throws Error("Backend unavailable (HTTP 404): Socket 'rpc.sock' not found for 'hero_embedder'").
• On OK JSON responses (including {"jsonrpc":"2.0","error":{...}}), rpcCall behaviour is unchanged.
• With hero_embedder_server stopped, the Server Stats card replaces its "Loading..." with a yellow alert alert-warning reading the upstream error message.
• With hero_embedder_server stopped, the Namespace Stats card replaces its "Loading..." with the same alert wording.
• With hero_embedder_server stopped, the namespace dropdown shows one disabled Backend unavailable: … option and is itself disabled.
• With hero_embedder_server stopped, the Loader pane shows the alert in #loader-error below the load button; the rest of the loader UI remains rendered.
• With hero_embedder_server stopped, the Logs tab body shows the alert in place of "Loading logs..." or any previously rendered logs.
• Once hero_embedder_server is restarted, every panel and the dropdown recover on the next poll tick without a page refresh.
• All error messages render via escapeHtml (XSS-safe even if the upstream body contains HTML/quotes); the dropdown error uses textContent and is therefore also safe.
• No emojis introduced. No new files added.

Manual verification plan

1. cargo build -p hero_embedder_ui (or service_embedder start --reset to also rebuild and re-register).
2. With hero_embedder_server running, open the dashboard and confirm the happy path works (Server Stats, Namespace Stats, Loader pane, Logs tab, namespace dropdown all populate).
3. Stop just the embedder server: hero_proc service stop hero_embedder_server (or analogous). Within ~10 s confirm in the browser:
   • No SyntaxError: Unexpected token in DevTools console.
   • Network tab shows /rpc POSTs returning 404 text/plain with body Socket 'rpc.sock' not found for 'hero_embedder'.
   • Server Stats card shows the yellow alert.
   • Namespace Stats card shows the alert.
   • Namespace dropdown is disabled with one greyed Backend unavailable: … option.
   • Loader pane shows the alert under the form; form is still visible.
   • Logs tab body shows the alert.
4. Restart the server. Without refreshing, confirm every panel and the dropdown recover on the next poll cycle.

Notes / Out of scope

• hero_embedderd ONNX dylib failures and hero_embedder_server missing models: separate concerns, not addressed here.
• Exponential backoff / max-retry for pollers: future work; pollers continue at their current fixed intervals.
• hero_router's text/plain 404 emission: filed separately. Our fix is defense-in-depth so the UI no longer relies on the upstream returning JSON.
• hero_embedder_proxy::openrpc_handler's {} swallow: tracked separately as #21.
• Porting the same rpcCall validation to hero_browser_ui's rpc() helper: out of scope for this branch.

## Implementation Spec for Issue #20 ### Objective Make the `hero_embedder_ui` dashboard fail loudly and recoverably when `hero_embedder_server` (or any upstream `rpc.sock`) is unavailable. Two coordinated changes: (1) `rpcCall()` in `base.html` must validate the HTTP response before parsing as JSON, so non-JSON 4xx/5xx replies (e.g. `hero_router`'s `text/plain` 404 `Socket 'rpc.sock' not found for 'hero_embedder'`) surface as readable error messages instead of "Unexpected token 'S'…" parse failures; (2) every panel-level poller stops swallowing errors with `console.error` and renders a visible inline alert (or, for the namespace dropdown, a disabled error option) that is automatically cleared by the next successful poll. ### Files to Modify/Create - `crates/hero_embedder_ui/templates/base.html` — fix `rpcCall` (response validation), add shared `escapeHtml` helper, add shared `renderPanelError` helper, replace `loadNamespaces` `console.error` with disabled-option error state on the `<select>`. - `crates/hero_embedder_ui/templates/components/stats_card.html` — replace silent `console.error` in `pollServerStats` and `pollNamespaceStats` with `renderPanelError` calls into the existing panel containers. - `crates/hero_embedder_ui/templates/fragments/loader_pane.html` — add a stable error container in the loader card; replace silent `console.error` in `refreshLoaderState` (and the secondary `index.clear` `.catch`) with a `renderPanelError` call against it. - `crates/hero_embedder_ui/templates/fragments/logs_pane.html` — replace silent `console.error` in `LogsManager.loadLogs` and `clearLogs` with `renderPanelError` calls into `logs-content`. No new files. No backend Rust changes. ### Helper specifications (placed in `base.html`, same `<script>` block as `rpcCall`) ```js function escapeHtml(s) { return String(s).replace(/[&<>"']/g, c => ({ "&": "&amp;", "<": "&lt;", ">": "&gt;", '"': "&quot;", "'": "&#39;" })[c]); } function renderPanelError(containerId, message) { const el = document.getElementById(containerId); if (!el) return; el.innerHTML = '<div class="alert alert-warning small mb-0">' + '<i class="bi bi-exclamation-triangle me-1"></i>' + escapeHtml(message) + '</div>'; } ``` ### `rpcCall` rewrite (drop into `base.html` lines 177–219) Insert immediately before the existing `await response.json()`: ```js const contentType = response.headers.get("content-type") || ""; const isJson = contentType.includes("application/json"); if (!response.ok && !isJson) { const text = await response.text(); throw new Error(`Backend unavailable (HTTP ${response.status}): ${text.slice(0, 200)}`); } const data = await response.json(); ``` Keep the existing `data.error` check, `data.result` return, response-time DOM update, and AbortError handling unchanged. ### Implementation Plan #### Step 1: base.html — rpcCall validation, helpers, namespace dropdown error state Files: `crates/hero_embedder_ui/templates/base.html` Dependencies: none - Inside the `<script>` block that begins at line 94, add `escapeHtml` and `renderPanelError` definitions before `rpcCall` (so all later code in the file and all included panel files can use them). - Modify `rpcCall` (lines 177–219) to perform the `response.ok` + `Content-Type` validation described above before `await response.json()`. - Modify `loadNamespaces` (lines 133–175). Replace the `console.error`-only catch with: clear `select.innerHTML`, append one `<option value="" disabled selected>` whose `textContent` is `"Backend unavailable: " + e.message`, and set `select.disabled = true`. On the success path, add `select.disabled = false` before the existing `select.innerHTML = ""`. #### Step 2: stats_card.html — Server Stats and Namespace Stats panels Files: `crates/hero_embedder_ui/templates/components/stats_card.html` Dependencies: Step 1 - In `pollServerStats` (~lines 175–186): replace `console.error("[ServerStats] Poll error:", err);` with `renderPanelError("server-stats-content", "Server stats unavailable: " + err.message);`. Recovery is automatic — the success path already does `container.outerHTML = renderServerStats(...)` which re-creates `#server-stats-content`. - In `pollNamespaceStats` (~lines 197–216): replace the `console.error` with `renderPanelError("namespace-stats-area", "Namespace stats unavailable: " + err.message);`. Recovery is automatic via the existing `container.innerHTML = renderNamespaceStats(data)` on success. - Do NOT touch the Clear Namespace button handler (~lines 245–268) or the Download TriviaQA handler (~lines 271–317); they already render visible per-button error states. #### Step 3: loader_pane.html — Corpus Loader pane Files: `crates/hero_embedder_ui/templates/fragments/loader_pane.html` Dependencies: Step 1 - Add a stable error container inside the card body. Insert immediately after `<div id="load-progress" class="mt-2"></div>` (~line 63): `<div id="loader-error" class="mt-2"></div>`. - In `refreshLoaderState` (~lines 193–218): at the top of the `try` block, clear the error via `const errEl = document.getElementById("loader-error"); if (errEl) errEl.innerHTML = "";` so a successful refresh wipes any previous error. In the `catch`, replace the `console.error` with `renderPanelError("loader-error", "Loader unavailable: " + e.message);`. - In the reset-button handler (~lines 231–242): replace the `console.error` with `renderPanelError("loader-error", "Failed to clear namespace: " + e.message);`. - Do NOT change the load-button success/error rendering (~lines 318–321); it already uses an `alert alert-danger` block in `#load-progress`. #### Step 4: logs_pane.html — Logs tab Files: `crates/hero_embedder_ui/templates/fragments/logs_pane.html` Dependencies: Step 1 - In `LogsManager.loadLogs` (~lines 102–120): replace `console.error("[LogsManager] Error loading logs:", error);` with `renderPanelError("logs-content", "Logs unavailable: " + error.message);`. Recovery is automatic — `renderLogs()` always calls `container.innerHTML = ...` on success. - In `LogsManager.clearLogs` (~lines 197–206): replace the `console.error` with `renderPanelError("logs-content", "Failed to clear logs: " + error.message);`. #### Parallelization summary Step 1 must complete first because Steps 2/3/4 all call `renderPanelError`. Once Step 1 is done, Steps 2, 3, and 4 are fully independent (different files, no shared symbols) and can be implemented in parallel. ### Acceptance Criteria - [ ] `rpcCall` no longer throws `SyntaxError: Unexpected token 'S', "Socket 'rp"...` when the upstream returns `text/plain` 404. Instead it throws `Error("Backend unavailable (HTTP 404): Socket 'rpc.sock' not found for 'hero_embedder'")`. - [ ] On OK JSON responses (including `{"jsonrpc":"2.0","error":{...}}`), `rpcCall` behaviour is unchanged. - [ ] With `hero_embedder_server` stopped, the Server Stats card replaces its "Loading..." with a yellow `alert alert-warning` reading the upstream error message. - [ ] With `hero_embedder_server` stopped, the Namespace Stats card replaces its "Loading..." with the same alert wording. - [ ] With `hero_embedder_server` stopped, the namespace dropdown shows one disabled `Backend unavailable: …` option and is itself disabled. - [ ] With `hero_embedder_server` stopped, the Loader pane shows the alert in `#loader-error` below the load button; the rest of the loader UI remains rendered. - [ ] With `hero_embedder_server` stopped, the Logs tab body shows the alert in place of "Loading logs..." or any previously rendered logs. - [ ] Once `hero_embedder_server` is restarted, every panel and the dropdown recover on the next poll tick without a page refresh. - [ ] All error messages render via `escapeHtml` (XSS-safe even if the upstream body contains HTML/quotes); the dropdown error uses `textContent` and is therefore also safe. - [ ] No emojis introduced. No new files added. ### Manual verification plan 1. `cargo build -p hero_embedder_ui` (or `service_embedder start --reset` to also rebuild and re-register). 2. With `hero_embedder_server` running, open the dashboard and confirm the happy path works (Server Stats, Namespace Stats, Loader pane, Logs tab, namespace dropdown all populate). 3. Stop just the embedder server: `hero_proc service stop hero_embedder_server` (or analogous). Within ~10 s confirm in the browser: - No `SyntaxError: Unexpected token` in DevTools console. - Network tab shows `/rpc` POSTs returning `404 text/plain` with body `Socket 'rpc.sock' not found for 'hero_embedder'`. - Server Stats card shows the yellow alert. - Namespace Stats card shows the alert. - Namespace dropdown is disabled with one greyed `Backend unavailable: …` option. - Loader pane shows the alert under the form; form is still visible. - Logs tab body shows the alert. 4. Restart the server. Without refreshing, confirm every panel and the dropdown recover on the next poll cycle. ### Notes / Out of scope - `hero_embedderd` ONNX dylib failures and `hero_embedder_server` missing models: separate concerns, not addressed here. - Exponential backoff / max-retry for pollers: future work; pollers continue at their current fixed intervals. - `hero_router`'s `text/plain` 404 emission: filed separately. Our fix is defense-in-depth so the UI no longer relies on the upstream returning JSON. - `hero_embedder_proxy::openrpc_handler`'s `{}` swallow: tracked separately as #21. - Porting the same `rpcCall` validation to `hero_browser_ui`'s `rpc()` helper: out of scope for this branch.

salmaelsoly commented 2026-04-27 07:37:17 +00:00

Author

Member

Copy link

Test Results & Implementation Summary

Branch: development_ui_error_handling (off origin/development).

Build / static check

• cargo check --workspace --bins — clean across all 5 crates (hero_embedder, hero_embedderd, hero_embedder_proxy, hero_embedder_server, hero_embedder_ui).
• No new Rust unit tests needed; this issue is JS-side template work.

End-to-end manual verification (browser)

Setup: with hero_embedder_server failed (no models on this machine — environmental, out of scope), rpc.sock does not exist. hero_router returns 404 text/plain with body Socket 'rpc.sock' not found for 'hero_embedder' for every POST /hero_embedder/ui/rpc. This is the exact failure mode the issue targets.

Driving the dashboard at http://127.0.0.1:9151/hero_embedder/ui/ headlessly:

Panel	Before	After
rpcCall	threw SyntaxError: Unexpected token 'S', "Socket 'rp"... is not valid JSON	throws Error("Backend unavailable (HTTP 404): Socket 'rpc.sock' not found for 'hero_embedder'")
Server Stats card	infinite "Loading..."	yellow alert alert-warning with the upstream error
Namespace Stats card	infinite "Loading..."	yellow alert with the upstream error
Namespace dropdown	empty / non-functional	disabled, single greyed option Namespaces unavailable: ...
Loader pane	infinite spinner	form still rendered; alert in newly-added #loader-error
Logs tab	"Loading logs..."	yellow alert in #logs-content
Browser console	"Unexpected token" SyntaxErrors	empty (mcp__hero_browser__console_messages returned [])

Diff summary

crates/hero_embedder_ui/templates/base.html                  | 39 +++++++++++++++++++++-
crates/hero_embedder_ui/templates/components/stats_card.html |  4 +--
crates/hero_embedder_ui/templates/fragments/loader_pane.html |  8 +++--
crates/hero_embedder_ui/templates/fragments/logs_pane.html   |  4 +--
4 files changed, 48 insertions(+), 7 deletions(-)

Changes

• templates/base.html
  • Added escapeHtml(s) helper.
  • Added renderPanelError(containerId, message) helper (Bootstrap alert alert-warning, XSS-safe via escapeHtml).
  • rpcCall: when response is non-OK and not application/json, read the body as text and throw Error("Backend unavailable (HTTP <status>): <body[:200]>"). OK and JSON-RPC-error paths unchanged.
  • loadNamespaces: success path adds select.disabled = false before re-populating; catch path appends one disabled <option> whose textContent is "Namespaces unavailable: " + e.message and sets select.disabled = true. Uses textContent (no escaping needed).
• templates/components/stats_card.html — pollServerStats and pollNamespaceStats console.error → renderPanelError("server-stats-content", ...) / renderPanelError("namespace-stats-area", ...). Recovery automatic via existing success-path outerHTML / innerHTML rewrites.
• templates/fragments/loader_pane.html — added <div id="loader-error" class="mt-2"></div> after #load-progress. refreshLoaderState clears #loader-error on entry; catch renders into it. Reset-button .catch also routes to #loader-error.
• templates/fragments/logs_pane.html — LogsManager.loadLogs and LogsManager.clearLogs console.error → renderPanelError("logs-content", ...).

Acceptance criteria — all met

• rpcCall no longer throws "Unexpected token" on text/plain 404; throws Error("Backend unavailable (HTTP 404): ...") instead.
• OK JSON responses: behaviour unchanged.
• Server Stats card shows alert when backend is down.
• Namespace Stats card shows alert.
• Namespace dropdown shows disabled Namespaces unavailable: ... option.
• Loader pane shows alert in #loader-error; rest of loader UI remains rendered.
• Logs tab body shows alert.
• Recovery on next poll is automatic via the existing innerHTML/outerHTML success paths and #loader-error clearing on entry to refreshLoaderState.
• All error messages use escapeHtml (XSS-safe); dropdown uses textContent.
• No emojis introduced. No new files added.

Notes

• Recovery from backend coming back up was not exercised on this machine because hero_embedder_server continues to fail (missing models — environmental, out of scope per the issue). The recovery path is structurally guaranteed: every panel's success-path overwrites the same DOM container that renderPanelError writes into; loadNamespaces re-enables the <select> on success; #loader-error is cleared at the top of every refreshLoaderState try.
• Polish made during verification: dropdown error label changed from "Backend unavailable: " to "Namespaces unavailable: " to avoid the redundant "Backend unavailable: Backend unavailable (HTTP 404)..." reading. All other panel prefixes (Server stats / Namespace stats / Loader / Logs unavailable: ) read cleanly with the underlying Backend unavailable (HTTP 404): ... message.
• Out of scope, untouched: hero_embedderd ONNX dylib failures, hero_embedder_server missing models, hero_router text/plain 404 emission (separate concern), hero_embedder_proxy::openrpc_handler {} swallow (tracked separately as #21), porting the same rpcCall validation to hero_browser_ui.

## Test Results & Implementation Summary Branch: `development_ui_error_handling` (off `origin/development`). ### Build / static check - `cargo check --workspace --bins` — clean across all 5 crates (`hero_embedder`, `hero_embedderd`, `hero_embedder_proxy`, `hero_embedder_server`, `hero_embedder_ui`). - No new Rust unit tests needed; this issue is JS-side template work. ### End-to-end manual verification (browser) Setup: with `hero_embedder_server` failed (no models on this machine — environmental, out of scope), `rpc.sock` does not exist. `hero_router` returns `404 text/plain` with body `Socket 'rpc.sock' not found for 'hero_embedder'` for every `POST /hero_embedder/ui/rpc`. This is the exact failure mode the issue targets. Driving the dashboard at `http://127.0.0.1:9151/hero_embedder/ui/` headlessly: | Panel | Before | After | |-------|--------|-------| | `rpcCall` | threw `SyntaxError: Unexpected token 'S', "Socket 'rp"... is not valid JSON` | throws `Error("Backend unavailable (HTTP 404): Socket 'rpc.sock' not found for 'hero_embedder'")` | | Server Stats card | infinite "Loading..." | yellow `alert alert-warning` with the upstream error | | Namespace Stats card | infinite "Loading..." | yellow alert with the upstream error | | Namespace dropdown | empty / non-functional | disabled, single greyed option `Namespaces unavailable: ...` | | Loader pane | infinite spinner | form still rendered; alert in newly-added `#loader-error` | | Logs tab | "Loading logs..." | yellow alert in `#logs-content` | | Browser console | "Unexpected token" `SyntaxError`s | empty (`mcp__hero_browser__console_messages` returned `[]`) | ### Diff summary ``` crates/hero_embedder_ui/templates/base.html | 39 +++++++++++++++++++++- crates/hero_embedder_ui/templates/components/stats_card.html | 4 +-- crates/hero_embedder_ui/templates/fragments/loader_pane.html | 8 +++-- crates/hero_embedder_ui/templates/fragments/logs_pane.html | 4 +-- 4 files changed, 48 insertions(+), 7 deletions(-) ``` ### Changes - `templates/base.html` - Added `escapeHtml(s)` helper. - Added `renderPanelError(containerId, message)` helper (Bootstrap `alert alert-warning`, XSS-safe via `escapeHtml`). - `rpcCall`: when response is non-OK and not `application/json`, read the body as text and throw `Error("Backend unavailable (HTTP <status>): <body[:200]>")`. OK and JSON-RPC-error paths unchanged. - `loadNamespaces`: success path adds `select.disabled = false` before re-populating; catch path appends one disabled `<option>` whose `textContent` is `"Namespaces unavailable: " + e.message` and sets `select.disabled = true`. Uses `textContent` (no escaping needed). - `templates/components/stats_card.html` — `pollServerStats` and `pollNamespaceStats` `console.error` → `renderPanelError("server-stats-content", ...)` / `renderPanelError("namespace-stats-area", ...)`. Recovery automatic via existing success-path `outerHTML` / `innerHTML` rewrites. - `templates/fragments/loader_pane.html` — added `<div id="loader-error" class="mt-2"></div>` after `#load-progress`. `refreshLoaderState` clears `#loader-error` on entry; catch renders into it. Reset-button `.catch` also routes to `#loader-error`. - `templates/fragments/logs_pane.html` — `LogsManager.loadLogs` and `LogsManager.clearLogs` `console.error` → `renderPanelError("logs-content", ...)`. ### Acceptance criteria — all met - [x] `rpcCall` no longer throws "Unexpected token" on `text/plain` 404; throws `Error("Backend unavailable (HTTP 404): ...")` instead. - [x] OK JSON responses: behaviour unchanged. - [x] Server Stats card shows alert when backend is down. - [x] Namespace Stats card shows alert. - [x] Namespace dropdown shows disabled `Namespaces unavailable: ...` option. - [x] Loader pane shows alert in `#loader-error`; rest of loader UI remains rendered. - [x] Logs tab body shows alert. - [x] Recovery on next poll is automatic via the existing `innerHTML`/`outerHTML` success paths and `#loader-error` clearing on entry to `refreshLoaderState`. - [x] All error messages use `escapeHtml` (XSS-safe); dropdown uses `textContent`. - [x] No emojis introduced. No new files added. ### Notes - Recovery from backend coming back up was not exercised on this machine because `hero_embedder_server` continues to fail (missing models — environmental, out of scope per the issue). The recovery path is structurally guaranteed: every panel's success-path overwrites the same DOM container that `renderPanelError` writes into; `loadNamespaces` re-enables the `<select>` on success; `#loader-error` is cleared at the top of every `refreshLoaderState` `try`. - Polish made during verification: dropdown error label changed from `"Backend unavailable: "` to `"Namespaces unavailable: "` to avoid the redundant `"Backend unavailable: Backend unavailable (HTTP 404)..."` reading. All other panel prefixes (`Server stats / Namespace stats / Loader / Logs unavailable: `) read cleanly with the underlying `Backend unavailable (HTTP 404): ...` message. - Out of scope, untouched: `hero_embedderd` ONNX dylib failures, `hero_embedder_server` missing models, `hero_router` `text/plain` 404 emission (separate concern), `hero_embedder_proxy::openrpc_handler` `{}` swallow (tracked separately as #21), porting the same `rpcCall` validation to `hero_browser_ui`.

salmaelsoly referenced this issue from a pull request that will close it, 2026-04-27 07:46:16 +00:00

fix(embedder_ui): show real errors when backend rpc.sock is unavailable #22

salmaelsoly closed this issue 2026-04-27 07:59:12 +00:00

salmaelsoly referenced this issue 2026-04-27 11:29:01 +00:00

embedder_server: startup race vs hero_embedderd model load + tokio panic on async-context drop of blocking client #23

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

development

development_ui_error_handling

development_mik_nu_demo

development_hero_embedderd

main

v0.1.3

v0.1.2

v0.1.1

v0.1.0

v0.0.1-rc1

Labels

Clear labels   

prio_critical

  

prio_low

  

type_bug

  

type_contact

  

type_issue

  

type_lead

  

type_question

  

type_story

  

type_task

No labels

prio_critical

prio_low

type_bug

type_contact

type_issue

type_lead

type_question

type_story

type_task

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_embedder#20

No description provided.