lhumina_code/hero_db

Fork 0

hero_db_admin: HTTP 500 "Template 'index.html' not found" — templates loaded from env!("CARGO_MANIFEST_DIR") at runtime #33

New issue

Open

opened 2026-05-19 13:10:58 +00:00 by mahmoud · 1 comment

mahmoud commented

2026-05-19 13:10:58 +00:00

Owner

Symptom

A freshly built hero_db_admin from development HEAD (commit 1e00ce3 at filing time) returns HTTP 500 on every page route:

$ curl http://[…]::1:9988/hero_db/admin/
Template error: Template 'index.html' not found

Direct probe via UDS reproduces the same:

$ sudo -u common curl --unix-socket /home/common/hero/var/sockets/hero_db/admin.sock http://localhost/
Template error: Template 'index.html' not found

Observed on mahmoud-ashraf-devbox (65.21.46.99) after deploying a release binary I built on the same host. The Redis-RESP path and JSON-RPC paths still work (redis-cli … PING → PONG); only the web UI is broken.

Root cause

crates/hero_db_admin/src/web.rs:125-160 loads templates from disk at runtime, using env!("CARGO_MANIFEST_DIR") as the base:

pub const UI_CRATE_DIR: &str = env!("CARGO_MANIFEST_DIR");

// …inside create_web_router…
let template_dir = format!("{}/templates", UI_CRATE_DIR);
let mut tera = Tera::default();
let mut template_files: Vec<(String, Option<String>)> = Vec::new();

if let Ok(entries) = std::fs::read_dir(&template_dir) {  // <-- silently swallows EACCES
    …
}

env!("CARGO_MANIFEST_DIR") is evaluated at compile time, so the build host's path to the hero_db_admin source crate is baked into the binary. This makes the binary non-relocatable and non-portable across UIDs — even on the same host:

Move the binary to another machine → templates missing.
Run the binary as a different user that can't traverse the build dir → templates missing.
Delete the build tree after install → templates missing.

The if let Ok(...) further hides this: when read_dir returns Err, the template vec stays empty, Tera is initialized with zero templates, and every route that calls render(...) returns 500 with the cryptic "Template 'X' not found" error rather than logging the underlying I/O error.

Concrete repro on `mahmoud-ashraf-devbox`

Built on the box as mahmoud: CARGO_TARGET_DIR=/home/mahmoud/hero/build/target cargo build --release -p hero_db_admin. CARGO_MANIFEST_DIR resolves to /home/mahmoud/work/hotfix/hero_db/crates/hero_db_admin.
Installed binary to /home/common/hero/bin/hero_db_admin (chown common:common).
Restarted via hero_proc service.start hero_db_admin.
Process runs as common. /home/mahmoud has mode drwx------ (standard adduser default), so common cannot traverse it.
read_dir("/home/mahmoud/work/hotfix/hero_db/crates/hero_db_admin/templates") → Permission denied → swallowed → 500 on every page.

Proof of the permission issue:

$ sudo -u common ls /home/mahmoud/work/hotfix/hero_db/crates/hero_db_admin/templates
ls: cannot access '/home/mahmoud/work/hotfix/…/templates': Permission denied

$ strings /home/mahmoud/hero/build/target/release/hero_db_admin | grep templates
/templates

The path is baked into the binary (strings shows /templates suffix, prefixed by the manifest-dir constant at runtime).

Why the previously-deployed binary worked

The binary that was on the box before this swap (hero_db_admin.bak-…, 18.9 MB, built May 18) contains no index.html string and no /templates path string — it appears to predate the current Tera-from-disk codepath (likely from before the hero_db_ui → hero_db_admin rename, or built with a different asset-loading strategy). So the regression is latent: the moment anyone builds and deploys from current development, the UI breaks.

Suggested fix

Embed templates at compile time so the binary is self-contained, the same way static assets are now served through hero_admin_lib::assets::shared_static_handler. Options, in rough order of size of change:

rust-embed (smallest diff): derive RustEmbed on a Templates struct rooted at "templates/", iterate Templates::iter() in create_web_router, and call tera.add_raw_template(name, str_from_bytes). This is what hero_admin_lib already uses for shared static assets.
include_str! per template: simpler if the template set is small/stable. Lose hot-reload-from-disk, gain portability.
Runtime-configurable templates path with sensible default (e.g., HERO_DB_ADMIN_TEMPLATES env var, default <install_prefix>/share/hero_db_admin/templates), and have the release flow install the templates alongside the binary. Largest change but most flexible.

While fixing, please also surface the read_dir error explicitly — silently swallowing it via if let Ok(...) is what made this turn into a 500-with-cryptic-message rather than a startup failure that would have caught the bug immediately.

Workaround in place

Rolled hero_db_admin on mahmoud-ashraf-devbox back to the pre-existing binary so the UI keeps working. The new hero_db_server binary (this is the actual BrokenPipe log-spam fix in #32) is fully deployed and verified — that PR is unaffected by this issue.

Impact

Anyone rebuilding and deploying hero_db_admin from current development on a multi-user box, or shipping the binary from CI to a different host, gets a fully broken admin UI.
The Bug 2 fix in PR #32 (hyper connection-error filter in hero_db_admin) can be merged but cannot be deployed on the devbox until this lands.

## Symptom A freshly built `hero_db_admin` from `development` HEAD (commit `1e00ce3` at filing time) returns **HTTP 500** on every page route: ``` $ curl http://[…]::1:9988/hero_db/admin/ Template error: Template 'index.html' not found ``` Direct probe via UDS reproduces the same: ``` $ sudo -u common curl --unix-socket /home/common/hero/var/sockets/hero_db/admin.sock http://localhost/ Template error: Template 'index.html' not found ``` Observed on `mahmoud-ashraf-devbox` (65.21.46.99) after deploying a release binary I built on the same host. The Redis-RESP path and JSON-RPC paths still work (`redis-cli … PING` → `PONG`); only the web UI is broken. ## Root cause `crates/hero_db_admin/src/web.rs:125-160` loads templates from disk at runtime, using `env!("CARGO_MANIFEST_DIR")` as the base: ```rust pub const UI_CRATE_DIR: &str = env!("CARGO_MANIFEST_DIR"); // …inside create_web_router… let template_dir = format!("{}/templates", UI_CRATE_DIR); let mut tera = Tera::default(); let mut template_files: Vec<(String, Option<String>)> = Vec::new(); if let Ok(entries) = std::fs::read_dir(&template_dir) { // <-- silently swallows EACCES … } ``` `env!("CARGO_MANIFEST_DIR")` is evaluated at compile time, so the build host's path to the `hero_db_admin` source crate is baked into the binary. This makes the binary **non-relocatable** and **non-portable across UIDs** — even on the same host: - Move the binary to another machine → templates missing. - Run the binary as a different user that can't traverse the build dir → templates missing. - Delete the build tree after install → templates missing. The `if let Ok(...)` further hides this: when `read_dir` returns `Err`, the template vec stays empty, Tera is initialized with **zero** templates, and *every* route that calls `render(...)` returns 500 with the cryptic "Template 'X' not found" error rather than logging the underlying I/O error. ## Concrete repro on `mahmoud-ashraf-devbox` 1. Built on the box as `mahmoud`: `CARGO_TARGET_DIR=/home/mahmoud/hero/build/target cargo build --release -p hero_db_admin`. CARGO_MANIFEST_DIR resolves to `/home/mahmoud/work/hotfix/hero_db/crates/hero_db_admin`. 2. Installed binary to `/home/common/hero/bin/hero_db_admin` (chown common:common). 3. Restarted via hero_proc `service.start hero_db_admin`. 4. Process runs as `common`. `/home/mahmoud` has mode `drwx------` (standard adduser default), so `common` cannot traverse it. 5. `read_dir("/home/mahmoud/work/hotfix/hero_db/crates/hero_db_admin/templates")` → `Permission denied` → swallowed → 500 on every page. Proof of the permission issue: ``` $ sudo -u common ls /home/mahmoud/work/hotfix/hero_db/crates/hero_db_admin/templates ls: cannot access '/home/mahmoud/work/hotfix/…/templates': Permission denied $ strings /home/mahmoud/hero/build/target/release/hero_db_admin | grep templates /templates ``` The path is baked into the binary (`strings` shows `/templates` suffix, prefixed by the manifest-dir constant at runtime). ## Why the previously-deployed binary worked The binary that was on the box before this swap (`hero_db_admin.bak-…`, 18.9 MB, built May 18) contains **no `index.html` string** and **no `/templates` path string** — it appears to predate the current Tera-from-disk codepath (likely from before the `hero_db_ui` → `hero_db_admin` rename, or built with a different asset-loading strategy). So the regression is latent: the moment anyone builds and deploys from current `development`, the UI breaks. ## Suggested fix Embed templates at compile time so the binary is self-contained, the same way static assets are now served through `hero_admin_lib::assets::shared_static_handler`. Options, in rough order of size of change: 1. **`rust-embed`** (smallest diff): derive `RustEmbed` on a `Templates` struct rooted at `"templates/"`, iterate `Templates::iter()` in `create_web_router`, and call `tera.add_raw_template(name, str_from_bytes)`. This is what `hero_admin_lib` already uses for shared static assets. 2. **`include_str!` per template**: simpler if the template set is small/stable. Lose hot-reload-from-disk, gain portability. 3. **Runtime-configurable templates path with sensible default** (e.g., `HERO_DB_ADMIN_TEMPLATES` env var, default `<install_prefix>/share/hero_db_admin/templates`), and have the release flow install the templates alongside the binary. Largest change but most flexible. While fixing, please also surface the `read_dir` error explicitly — silently swallowing it via `if let Ok(...)` is what made this turn into a 500-with-cryptic-message rather than a startup failure that would have caught the bug immediately. ## Workaround in place Rolled `hero_db_admin` on `mahmoud-ashraf-devbox` back to the pre-existing binary so the UI keeps working. The new `hero_db_server` binary (this is the actual `BrokenPipe` log-spam fix in https://forge.ourworld.tf/lhumina_code/hero_db/pulls/32) is fully deployed and verified — that PR is unaffected by this issue. ## Impact - Anyone rebuilding and deploying `hero_db_admin` from current `development` on a multi-user box, or shipping the binary from CI to a different host, gets a fully broken admin UI. - The Bug 2 fix in PR #32 (hyper connection-error filter in `hero_db_admin`) can be merged but cannot be deployed on the devbox until this lands.

mahmoud referenced this issue

2026-05-19 13:11:39 +00:00

fix(hero_db): silence BrokenPipe ERROR spam on client disconnect #32

mahmoud commented

2026-05-19 13:35:08 +00:00

Author

Owner

Research outcomes (before coding)

Surveyed sibling admin crates across the org and the framework helpers to settle on a fix consistent with the rest of the ecosystem rather than a one-off for hero_db. Findings:

Canonical pattern in this codebase

All other admin crates compile templates into the binary at build time, so the binary is relocatable by construction:

Crate	Template engine	Asset embedding
`hero_proc_admin`	Askama (compile-time)	`rust-embed`
`hero_slides_admin`	Askama	`rust-embed`
`hero_books_admin`	Askama	—
`my_compute_zos_admin`	Askama	—
`my_compute_explorer_admin`	Askama	—
`hero_db_admin`	Tera + `env!("CARGO_MANIFEST_DIR")` ← this bug	—
`hero_aibroker_ui`	Tera + `env!("CARGO_MANIFEST_DIR")` ← same latent bug	—

Grep for env!("CARGO_MANIFEST_DIR") in admin/UI crates returned exactly two production hits: this crate and hero_aibroker_ui/src/main.rs:65-66 (Tera::new(format!("{}/templates/**/*.html", env!("CARGO_MANIFEST_DIR"))).expect(...)). hero_aibroker_ui actually .expect()s the result, so on a misaligned deploy it would panic at startup rather than fail with a runtime 500 — better diagnostic, same root cause.

Hidden gem: the framework already supports the Tera path

hero_website_lib/src/templates.rs:81-111 exposes:

pub fn load_embedded_templates<A: Embed>() -> Tera

Pass any #[derive(Embed)] #[folder = "templates/"] struct, get back a fully-loaded Tera. It even merges library-default templates on top. So a Tera-based service can become relocatable with ~4 lines of struct + 1 call.

Template-syntax audit

All 11 hero_db_admin templates (~5000 LoC) use only {% extends %}, {% block %}, {% include %}, {{ var }}. Zero Tera-specific filters (| default, | length, | safe, etc.). The templates would migrate to Askama or stay on Tera with no rewrites.

Documentation gap

skills/hero/ui/hero_ui_assets.md correctly documents rust-embed for static assets.
No skill currently says "templates must be embedded, never loaded from disk via CARGO_MANIFEST_DIR." That's how the bug slipped in.

Recommended fix

Two paths exist; recommend doing the small one first and treating the larger one as separate follow-up:

Path B (recommended for this issue) — Tera + load_embedded_templates. ~20 LoC, mostly deletions. Replace the UI_CRATE_DIR + read_dir(...) block in crates/hero_db_admin/src/web.rs:124-169 with:

use rust_embed::Embed;

#[derive(Embed)]
#[folder = "templates/"]
struct Templates;

// inside create_web_router:
let tera = hero_website_lib::templates::load_embedded_templates::<Templates>();

Add hero_website_lib = { workspace = true } (and confirm rust-embed = { workspace = true }) to crates/hero_db_admin/Cargo.toml. Delete the UI_CRATE_DIR const.

Why this and not Path A:

Same engine, same template syntax, same single render(...) call site → minimal regression risk.
Uses an existing framework helper rather than introducing a new pattern.
Errors surface at startup (the helper logs and returns an empty Tera that will fail on first render) — strictly better than today's silently-swallowed EACCES.

Path A (follow-up, optional) — migrate hero_db_admin to Askama. Justifiable on ecosystem-consistency grounds (5-of-6 admin crates use Askama), but a separate, judgment-call refactor with its own diff and review surface. Defer.

Additional work this issue should spawn

Apply the same Path B fix to hero_aibroker_ui — same bug, same blast radius. Track separately so it doesn't block the hero_db_admin fix.
Add a skill section at skills/hero/ui/ (extend hero_ui_assets.md or create hero_ui_templates.md) documenting: "Templates must be embedded into the binary. Use Askama for new services (preferred — typed, compile-time-checked) or hero_website_lib::templates::load_embedded_templates::<MyTemplates>() for existing Tera services. Never use env!("CARGO_MANIFEST_DIR") at runtime — it bakes the build host's path into the binary and breaks relocation, cross-UID deploys, and CI artifacts." Link from hero_ui_dashboard_admin.md.
Surface the silent-swallow bug — even with embedded templates, the if let Ok(read_dir(...)) pattern hid the underlying EACCES. Any future template-loading code should propagate errors so failures are loud.

Not taking action on this issue right now (returning to other work). Captured here so whoever picks it up has the full context and doesn't need to re-do the survey.

## Research outcomes (before coding) Surveyed sibling admin crates across the org and the framework helpers to settle on a fix consistent with the rest of the ecosystem rather than a one-off for hero_db. Findings: ### Canonical pattern in this codebase All other admin crates compile templates **into the binary** at build time, so the binary is relocatable by construction: | Crate | Template engine | Asset embedding | |---|---|---| | `hero_proc_admin` | Askama (compile-time) | `rust-embed` | | `hero_slides_admin` | Askama | `rust-embed` | | `hero_books_admin` | Askama | — | | `my_compute_zos_admin` | Askama | — | | `my_compute_explorer_admin` | Askama | — | | `hero_db_admin` | **Tera + `env!("CARGO_MANIFEST_DIR")`** ← this bug | — | | `hero_aibroker_ui` | **Tera + `env!("CARGO_MANIFEST_DIR")`** ← **same latent bug** | — | Grep for `env!("CARGO_MANIFEST_DIR")` in admin/UI crates returned exactly two production hits: this crate and `hero_aibroker_ui/src/main.rs:65-66` (`Tera::new(format!("{}/templates/**/*.html", env!("CARGO_MANIFEST_DIR"))).expect(...)`). `hero_aibroker_ui` actually `.expect()`s the result, so on a misaligned deploy it would *panic at startup* rather than fail with a runtime 500 — better diagnostic, same root cause. ### Hidden gem: the framework already supports the Tera path `hero_website_lib/src/templates.rs:81-111` exposes: ```rust pub fn load_embedded_templates<A: Embed>() -> Tera ``` Pass any `#[derive(Embed)] #[folder = "templates/"]` struct, get back a fully-loaded Tera. It even merges library-default templates on top. So a Tera-based service can become relocatable with ~4 lines of struct + 1 call. ### Template-syntax audit All 11 hero_db_admin templates (~5000 LoC) use only `{% extends %}`, `{% block %}`, `{% include %}`, `{{ var }}`. **Zero Tera-specific filters** (`| default`, `| length`, `| safe`, etc.). The templates would migrate to Askama or stay on Tera with no rewrites. ### Documentation gap - `skills/hero/ui/hero_ui_assets.md` correctly documents `rust-embed` for **static assets**. - No skill currently says "templates must be embedded, never loaded from disk via `CARGO_MANIFEST_DIR`." That's how the bug slipped in. ## Recommended fix Two paths exist; recommend doing the small one first and treating the larger one as separate follow-up: **Path B (recommended for this issue) — Tera + `load_embedded_templates`.** ~20 LoC, mostly deletions. Replace the `UI_CRATE_DIR` + `read_dir(...)` block in `crates/hero_db_admin/src/web.rs:124-169` with: ```rust use rust_embed::Embed; #[derive(Embed)] #[folder = "templates/"] struct Templates; // inside create_web_router: let tera = hero_website_lib::templates::load_embedded_templates::<Templates>(); ``` Add `hero_website_lib = { workspace = true }` (and confirm `rust-embed = { workspace = true }`) to `crates/hero_db_admin/Cargo.toml`. Delete the `UI_CRATE_DIR` const. Why this and not Path A: - Same engine, same template syntax, same single `render(...)` call site → minimal regression risk. - Uses an existing framework helper rather than introducing a new pattern. - Errors surface at startup (the helper logs and returns an empty Tera that will fail on first render) — strictly better than today's silently-swallowed `EACCES`. **Path A (follow-up, optional) — migrate `hero_db_admin` to Askama.** Justifiable on ecosystem-consistency grounds (5-of-6 admin crates use Askama), but a separate, judgment-call refactor with its own diff and review surface. Defer. ## Additional work this issue should spawn 1. **Apply the same Path B fix to `hero_aibroker_ui`** — same bug, same blast radius. Track separately so it doesn't block the hero_db_admin fix. 2. **Add a skill section** at `skills/hero/ui/` (extend `hero_ui_assets.md` or create `hero_ui_templates.md`) documenting: "Templates must be embedded into the binary. Use Askama for new services (preferred — typed, compile-time-checked) or `hero_website_lib::templates::load_embedded_templates::<MyTemplates>()` for existing Tera services. **Never** use `env!("CARGO_MANIFEST_DIR")` at runtime — it bakes the build host's path into the binary and breaks relocation, cross-UID deploys, and CI artifacts." Link from `hero_ui_dashboard_admin.md`. 3. **Surface the silent-swallow bug** — even with embedded templates, the `if let Ok(read_dir(...))` pattern hid the underlying EACCES. Any future template-loading code should propagate errors so failures are loud. Not taking action on this issue right now (returning to other work). Captured here so whoever picks it up has the full context and doesn't need to re-do the survey.