lhumina_code/home
23
0
Fork 0
Code Issues 6 Pull requests Projects Releases Packages Wiki Activity Actions

[nu-demo] hero_agent triage classifier routes 'What is Hero OS?' to Knowledge path (no tools offered) — grounding never happens #152

New issue
Closed
opened 2026-04-24 01:50:01 +00:00 by mik-tf · 1 comment
mik-tf commented 2026-04-24 01:50:01 +00:00
Owner
Copy link

Symptom

Even with:

  • hero_agent/src/prompt.rs containing the MANDATORY directive (compiled into the binary, verified via strings hero_agent_server | grep -c MANDATORY = 1, same for Hero Vired, sovereign digital),
  • tool_router.rs listing search_hero_docs in always_include,
  • llm_client.rs emitting tool_choice: "required" whenever tools are offered (#150),
  • aibroker routing live (Claude Haiku 4.5 backing LLM),

…the plain question "What is Hero OS and what can it do?" still returns a generic OS description ("user-friendly and efficient, Lightweight performance, Runs smoothly on various devices") — no mention of "sovereign digital workspace," no doc citation.

Root cause (diagnosed on heronu 2026-04-24)

hero_agent has a triage classifier that routes every incoming chat message into one of (at least) two buckets:

  • triage=Knowledge → LLM called without the tools list. This is the "pure chat" path — the model gets only the system prompt + message, and composes an answer from training data. The MANDATORY directive is present in the system prompt, but with no tool offered, search_hero_docs can't be called; with no tool call, the model falls back to training priors; tool_choice: "required" is never emitted because tools is empty.
  • triage=Tools → LLM called with the full 58+ tool list (from tool_router.rs), tool_choice: "required" forces a tool call, and grounding works.

"What is Hero OS?" is being classified as Knowledge, so the hero-grounding chain is never invoked. The smoke test response proves it: no tool call, no doc citation, generic OS description.

When the user instead asks "Call search_hero_docs with query='Hero OS overview'", the triage classifier routes to Tools and grounding works perfectly — confirmed by repeated testing.

Why this is wrong for Hero OS

The triage split is useful in principle (skip tool overhead for pure chat), but the classifier treats Hero-specific questions as "general knowledge" when they are in fact the ONE class of question we always want grounded. A user asking about Hero OS in the built-in AI Assistant should get docs_hero content, not hallucinated marketing copy.

Proposed fixes

Three options, in order of pragmatism:

  1. Quick demo fix — always route to Tools when the message contains any hero_* keyword. Add a pre-triage override in hero_agent_server/src/routes.rs (or wherever triage lives): 
    let force_tools = msg.to_lowercase().contains("hero os")
    || HERO_SERVICE_NAMES.iter().any(|s| msg.contains(s));
let triage = if force_tools { Triage::Tools } else { classify(&msg) };
  2. Inject search on Knowledge path — when Knowledge is selected but the message mentions hero_*, proactively call search_hero_docs before asking the LLM and pass the snippets as context. No triage change; just augment the Knowledge handler.
  3. Proper fix — make the classifier itself LLM-driven and feed it the always_include tool names so it can route when any always-include tool is relevant. Removes the hard-coded keyword list.

For the demo, (1) is the smallest change and unambiguous.

Related

  • #148 — nu-demo architecture index
  • #149 — prompt.rs directive strengthening (prerequisite, already landed on development_mik_nu_demo)
  • #150 — tool_choice='required' support (prerequisite, already landed on development_mik_nu_demo)
  • Sibling: #152 — tools payload size + naming violations block Tools-path execution even after triage succeeds

Verification

After fix:

curl --unix-socket ~/hero/var/sockets/hero_agent/rpc.sock \
  -X POST http://localhost/rpc \
  -d '{"jsonrpc":"2.0","id":1,"method":"agent.chat","params":{"message":"What is Hero OS?","model":"claude-haiku-4.5"}}'

→ response must cite hero_os_guide or use phrases directly from docs_hero ("sovereign digital workspace," "browser-based," "local-first").

Signed-off-by: mik-tf

## Symptom Even with: - `hero_agent/src/prompt.rs` containing the MANDATORY directive (compiled into the binary, verified via `strings hero_agent_server | grep -c MANDATORY` = 1, same for `Hero Vired`, `sovereign digital`), - `tool_router.rs` listing `search_hero_docs` in `always_include`, - `llm_client.rs` emitting `tool_choice: "required"` whenever tools are offered (https://forge.ourworld.tf/lhumina_code/home/issues/150), - aibroker routing live (Claude Haiku 4.5 backing LLM), …the plain question `"What is Hero OS and what can it do?"` still returns a generic OS description ("user-friendly and efficient, Lightweight performance, Runs smoothly on various devices") — no mention of "sovereign digital workspace," no doc citation. ## Root cause (diagnosed on heronu 2026-04-24) `hero_agent` has a **triage classifier** that routes every incoming chat message into one of (at least) two buckets: - `triage=Knowledge` → LLM called **without** the tools list. This is the "pure chat" path — the model gets only the system prompt + message, and composes an answer from training data. The MANDATORY directive is present in the system prompt, but with no tool offered, `search_hero_docs` can't be called; with no tool call, the model falls back to training priors; `tool_choice: "required"` is never emitted because tools is empty. - `triage=Tools` → LLM called **with** the full 58+ tool list (from `tool_router.rs`), `tool_choice: "required"` forces a tool call, and grounding works. "What is Hero OS?" is being classified as `Knowledge`, so the hero-grounding chain is never invoked. The smoke test response proves it: no tool call, no doc citation, generic OS description. When the user instead asks `"Call search_hero_docs with query='Hero OS overview'"`, the triage classifier routes to `Tools` and grounding works perfectly — confirmed by repeated testing. ## Why this is wrong for Hero OS The triage split is useful in principle (skip tool overhead for pure chat), but the classifier treats Hero-specific questions as "general knowledge" when they are in fact the ONE class of question we always want grounded. A user asking about Hero OS in the built-in AI Assistant should get docs_hero content, not hallucinated marketing copy. ## Proposed fixes Three options, in order of pragmatism: 1. **Quick demo fix** — always route to `Tools` when the message contains any `hero_*` keyword. Add a pre-triage override in `hero_agent_server/src/routes.rs` (or wherever triage lives): ```rust let force_tools = msg.to_lowercase().contains("hero os") || HERO_SERVICE_NAMES.iter().any(|s| msg.contains(s)); let triage = if force_tools { Triage::Tools } else { classify(&msg) }; ``` 2. **Inject search on Knowledge path** — when Knowledge is selected but the message mentions `hero_*`, proactively call `search_hero_docs` before asking the LLM and pass the snippets as context. No triage change; just augment the Knowledge handler. 3. **Proper fix** — make the classifier itself LLM-driven and feed it the `always_include` tool names so it can route when any always-include tool is relevant. Removes the hard-coded keyword list. For the demo, (1) is the smallest change and unambiguous. ## Related - https://forge.ourworld.tf/lhumina_code/home/issues/148 — nu-demo architecture index - https://forge.ourworld.tf/lhumina_code/home/issues/149 — prompt.rs directive strengthening (prerequisite, already landed on `development_mik_nu_demo`) - https://forge.ourworld.tf/lhumina_code/home/issues/150 — tool_choice='required' support (prerequisite, already landed on `development_mik_nu_demo`) - Sibling: https://forge.ourworld.tf/lhumina_code/home/issues/152 — tools payload size + naming violations block Tools-path execution even after triage succeeds ## Verification After fix: ```bash curl --unix-socket ~/hero/var/sockets/hero_agent/rpc.sock \ -X POST http://localhost/rpc \ -d '{"jsonrpc":"2.0","id":1,"method":"agent.chat","params":{"message":"What is Hero OS?","model":"claude-haiku-4.5"}}' ``` → response must cite `hero_os_guide` or use phrases directly from docs_hero ("sovereign digital workspace," "browser-based," "local-first"). Signed-off-by: mik-tf
mik-tf commented 2026-04-27 04:42:47 +00:00
Author
Owner
Copy link

Fixed in hero_agent commit b650591 on development. Implements option 1 from the issue body — the smallest change with unambiguous behavior.

Root cause re-confirmed by reading the code: triage.rs::quick_triage runs through CHITCHAT_PATTERNS → TOOL_PATTERNS → short-message threshold → LLM fallback. None of those steps recognize "What is Hero OS?" as a tool-requiring query, so it ends up in the LLM-triage default → Knowledge → no tools offered → search_hero_docs never reaches the model → grounded answers are structurally impossible.

Fix at triage.rs::quick_triage — pre-empt every existing check with a hero_* pattern match at step 0:

// Closes home#152 — hero-stack questions always need the Tools path
// so search_hero_docs grounding fires.
if HERO_PATTERNS.is_match(trimmed) {
    return Some(TriageResult::Tools);
}

HERO_PATTERNS is one alternation regex covering every service prefix used in agent.rs::message_contains_hero_keyword (home#150) and prompt.rs's MANDATORY block (home#149):

(?i)\bhero[ _](os|books|agent|aibroker|router|osis|foundry|voice|proxy|browser|collab|livekit|proc|skills|archipelagos)\b

The inline doc on HERO_PATTERNS explicitly calls out that the three lists must stay in sync — adding a new hero_* service requires updating all three.

Tests added (3):

  • test_hero_questions_route_to_tools — sanity matrix of 6 different hero_* queries (What is Hero OS?, How does hero_books work?, Tell me about hero_aibroker., explain hero_router routing, Why is hero_voice so slow?, hero_archipelagos overview) — all must classify Tools.
  • test_hero_pattern_overrides_chitchat — "hi, what is hero_os?" must beat the chitchat-greeting regex (the new check sits at step 0, before chitchat patterns).
  • test_hero_pattern_does_not_match_unrelated_words — "Who is your hero?" and "Tell me about Hero Vired" stay ambiguous → defer to LLM triage. Disambiguates from generic "hero" without a service suffix.

Why this completes the grounding tripod:

Issue Layer Effect
home#149 prompt.rs system prompt Tells the model it MUST call search_hero_docs
home#150 llm_client.rs tool_choice Forces the model to call it (API hard constraint)
home#152 triage.rs quick_triage Routes the request to the path where #149 + #150 can run

Without all three, hero-stack questions fall back to training data. With all three: deterministic grounding via search_hero_docs, citations from the docs corpus, no hallucinated marketing copy.

Verification: cargo fmt, cargo check -p hero_agent clean, all 10 triage tests pass + 87 pre-existing tests still pass.

Meta-tracker: home#193.

Signed-off-by: mik-tf

Fixed in hero_agent commit `b650591` on `development`. Implements **option 1** from the issue body — the smallest change with unambiguous behavior. **Root cause re-confirmed by reading the code:** `triage.rs::quick_triage` runs through CHITCHAT_PATTERNS → TOOL_PATTERNS → short-message threshold → LLM fallback. None of those steps recognize "What is Hero OS?" as a tool-requiring query, so it ends up in the LLM-triage default → Knowledge → no tools offered → search_hero_docs never reaches the model → grounded answers are structurally impossible. **Fix** at `triage.rs::quick_triage` — pre-empt every existing check with a hero_* pattern match at step 0: ```rust // Closes home#152 — hero-stack questions always need the Tools path // so search_hero_docs grounding fires. if HERO_PATTERNS.is_match(trimmed) { return Some(TriageResult::Tools); } ``` `HERO_PATTERNS` is one alternation regex covering every service prefix used in `agent.rs::message_contains_hero_keyword` (home#150) and `prompt.rs`'s MANDATORY block (home#149): ``` (?i)\bhero[ _](os|books|agent|aibroker|router|osis|foundry|voice|proxy|browser|collab|livekit|proc|skills|archipelagos)\b ``` The inline doc on `HERO_PATTERNS` explicitly calls out that the three lists must stay in sync — adding a new `hero_*` service requires updating all three. **Tests added (3):** - `test_hero_questions_route_to_tools` — sanity matrix of 6 different `hero_*` queries (`What is Hero OS?`, `How does hero_books work?`, `Tell me about hero_aibroker.`, `explain hero_router routing`, `Why is hero_voice so slow?`, `hero_archipelagos overview`) — all must classify `Tools`. - `test_hero_pattern_overrides_chitchat` — `"hi, what is hero_os?"` must beat the chitchat-greeting regex (the new check sits at step 0, before chitchat patterns). - `test_hero_pattern_does_not_match_unrelated_words` — `"Who is your hero?"` and `"Tell me about Hero Vired"` stay ambiguous → defer to LLM triage. Disambiguates from generic "hero" without a service suffix. **Why this completes the grounding tripod:** | Issue | Layer | Effect | |---|---|---| | home#149 | `prompt.rs` system prompt | Tells the model it MUST call `search_hero_docs` | | home#150 | `llm_client.rs` `tool_choice` | Forces the model to call it (API hard constraint) | | **home#152** | `triage.rs` quick_triage | Routes the request to the path where #149 + #150 can run | Without all three, hero-stack questions fall back to training data. With all three: deterministic grounding via `search_hero_docs`, citations from the docs corpus, no hallucinated marketing copy. **Verification:** `cargo fmt`, `cargo check -p hero_agent` clean, all 10 triage tests pass + 87 pre-existing tests still pass. Meta-tracker: home#193. Signed-off-by: mik-tf
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/home#152
No description provided.