lhumina_code/home
23
0
Fork 0
Code Issues 6 Pull requests Projects Releases Packages Wiki Activity Actions

[nu-demo] service_embedder.nu missing ORT_LIB_LOCATION/LD_LIBRARY_PATH/EMBEDDER_MODELS env vars — embedderd panics with 'No embedder models found' after fresh install #166

New issue
Closed
opened 2026-04-24 14:48:15 +00:00 by mik-tf · 2 comments
mik-tf commented 2026-04-24 14:48:15 +00:00
Owner
Copy link

Symptom

On a fresh herodemo VM installed from hero_skills service_install_all, starting the hero_embedder service results in hero_embedderd panicking repeatedly:

Error: No embedder models found in /home/driver/hero/var/embedder/models
Mutex poisoned
thread '<unnamed>' panicked at ort-2.0.0-rc.11/src/util/mutex_std.rs:15:23:

The action registered by service_install_all has minimal env:

env  {"HERO_EMBEDDERD_PORT":"8092"}

Missing: ORT_LIB_LOCATION, LD_LIBRARY_PATH, EMBEDDER_MODELS.

Root cause

hero_skills/tools/modules/services/service_embedder.nu doesn't set any ONNX or models env vars in the action definition. The binary is shipped, but it can't find its shared library (ONNX Runtime) or models directory.

Additionally, ONNX Runtime itself is NOT installed by hero_skills install-all — it's a separate manual step. On heronu we did this by downloading the v1.23.2 release tarball into /usr/local/onnxruntime/.

Fix

Two changes to service_embedder.nu:

  1. Install ONNX Runtime as part of service_embedder install. Download the v1.23.2 tarball (or a pinned version) into /usr/local/onnxruntime/ or /data/opt/onnxruntime/ and verify the shared lib is loadable.

  2. Bake the env vars into the hero_embedderd + hero_embedder_server action specs:

let env = {
    HERO_EMBEDDERD_PORT: "8092"
    ORT_LIB_LOCATION: "/usr/local/onnxruntime/lib"
    LD_LIBRARY_PATH: "/usr/local/onnxruntime/lib"
    EMBEDDER_MODELS: $"($env.ROOTDIR)/var/embedder/models"
}

And for hero_embedder_server, additionally:

HERO_EMBEDDERD_URL: "http://127.0.0.1:8092"

Demo workaround (applied on herodemo 2026-04-24)

  1. Manually downloaded ONNX v1.23.2 tarball → /usr/local/onnxruntime/.
  2. Added the 3 env vars to both hero_embedderd and hero_embedder_server actions via a custom hero_proc action.set call (python script patch_embedderd_action.py).
  3. Restarted hero_embedder — daemon came up, embedder_server initially failed (race), retried manually → all good.

See sibling issue for the race condition: hero_embedder_server starts before hero_embedderd is ready — no dependency-wait.

Related

  • home#128 — hero_skills installers missing apt dev packages (similar category: missing install-time steps)
  • home#145 — hero_embedder block_in_place patch (different bug in same service)
  • home#160 — consolidated demo state

Signed-off-by: mik-tf

## Symptom On a fresh herodemo VM installed from `hero_skills service_install_all`, starting the `hero_embedder` service results in hero_embedderd panicking repeatedly: ``` Error: No embedder models found in /home/driver/hero/var/embedder/models Mutex poisoned thread '<unnamed>' panicked at ort-2.0.0-rc.11/src/util/mutex_std.rs:15:23: ``` The action registered by `service_install_all` has minimal env: ``` env {"HERO_EMBEDDERD_PORT":"8092"} ``` Missing: `ORT_LIB_LOCATION`, `LD_LIBRARY_PATH`, `EMBEDDER_MODELS`. ## Root cause `hero_skills/tools/modules/services/service_embedder.nu` doesn't set any ONNX or models env vars in the action definition. The binary is shipped, but it can't find its shared library (ONNX Runtime) or models directory. Additionally, ONNX Runtime itself is NOT installed by `hero_skills install-all` — it's a separate manual step. On heronu we did this by downloading the v1.23.2 release tarball into `/usr/local/onnxruntime/`. ## Fix Two changes to `service_embedder.nu`: 1. **Install ONNX Runtime as part of `service_embedder install`.** Download the v1.23.2 tarball (or a pinned version) into `/usr/local/onnxruntime/` or `/data/opt/onnxruntime/` and verify the shared lib is loadable. 2. **Bake the env vars into the hero_embedderd + hero_embedder_server action specs:** ```nu let env = { HERO_EMBEDDERD_PORT: "8092" ORT_LIB_LOCATION: "/usr/local/onnxruntime/lib" LD_LIBRARY_PATH: "/usr/local/onnxruntime/lib" EMBEDDER_MODELS: $"($env.ROOTDIR)/var/embedder/models" } ``` And for hero_embedder_server, additionally: ```nu HERO_EMBEDDERD_URL: "http://127.0.0.1:8092" ``` ## Demo workaround (applied on herodemo 2026-04-24) 1. Manually downloaded ONNX v1.23.2 tarball → `/usr/local/onnxruntime/`. 2. Added the 3 env vars to both `hero_embedderd` and `hero_embedder_server` actions via a custom `hero_proc action.set` call (python script `patch_embedderd_action.py`). 3. Restarted hero_embedder — daemon came up, embedder_server initially failed (race), retried manually → all good. See sibling issue for the race condition: `hero_embedder_server starts before hero_embedderd is ready — no dependency-wait`. ## Related - [home#128](https://forge.ourworld.tf/lhumina_code/home/issues/128) — hero_skills installers missing apt dev packages (similar category: missing install-time steps) - [home#145](https://forge.ourworld.tf/lhumina_code/home/issues/145) — hero_embedder block_in_place patch (different bug in same service) - [home#160](https://forge.ourworld.tf/lhumina_code/home/issues/160) — consolidated demo state Signed-off-by: mik-tf
mik-tf commented 2026-04-25 15:53:13 +00:00
Author
Owner
Copy link

Partially closed by hero_skills #125 (commit 1379cc0)

The apt-side gap is now codified in install_base: libssl-dev, pkg-config, wget, jq, sqlite3, libsqlite3-dev, python3, python3-pip, python3-venv, python3-openpyxl.

Note: this issue mentions multiple specific gaps. Some are now resolved upstream; others (e.g. uv install, ONNX runtime, Chrome) are tracked in the same Phase 2 effort and will land in a follow-up PR — see home#185 for the master tracker.

Signed-off-by: mik-tf

## Partially closed by hero_skills #125 (commit `1379cc0`) The apt-side gap is now codified in `install_base`: libssl-dev, pkg-config, wget, jq, sqlite3, libsqlite3-dev, python3, python3-pip, python3-venv, python3-openpyxl. Note: this issue mentions multiple specific gaps. Some are now resolved upstream; others (e.g. `uv` install, ONNX runtime, Chrome) are tracked in the same Phase 2 effort and will land in a follow-up PR — see [home#185](https://forge.ourworld.tf/lhumina_code/home/issues/185) for the master tracker. Signed-off-by: mik-tf
mik-tf commented 2026-04-25 16:22:01 +00:00
Author
Owner
Copy link

Resolved by lhumina_code/hero_skills@7c823d1 (PR lhumina_code/hero_skills#126).

Part of Phase 2 tracker #185.

Resolved by https://forge.ourworld.tf/lhumina_code/hero_skills/commit/7c823d1 (PR https://forge.ourworld.tf/lhumina_code/hero_skills/pulls/126). Part of Phase 2 tracker https://forge.ourworld.tf/lhumina_code/home/issues/185.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/home#166
No description provided.