AI assistant making up CRM data — investigate hallucination source #39

New issue

Open

opened 2026-05-06 17:07:25 +00:00 by mik-tf · 1 comment

mik-tf commented

2026-05-06 17:07:25 +00:00

Owner

Overview

The CRM AI surface returns invented contacts / projects / tasks instead of grounding on real OSIS data. Casper hit this; Timur is investigating.

Why

Meeting 2026-05-06: "asked timur to view the issue / AI making up stuff / explained to Casper / Timur helping Casper on blockers".

This is related to but distinct from home#215 (assistant non-functional from missing Groq key). The hallucination here happens even when the assistant responds — the response is fluent but factually wrong.

Likely candidates for the root cause:

Per-context routing broken (hero_biz#37 + hero_rpc#42) — agent gets default-context data instead of the user's context, so its grounding is wrong.
Tool surface for CRM reads/writes missing or unreliable — agent falls back to generative answer.
System prompt doesn't enforce "if you don't have grounded data, say so".

Acceptance

Reproduce a clear hallucination case (one prompt, one wrong response)
Identify which of the candidate causes is the dominant one
Fix it (filed as separate issue if non-trivial) or escalate to hero_agent
Verify on demo VM with the same prompt

hero_biz#37 — context selection has no effect
hero_rpc#42 — UDS transport drops X-Hero-Context
home#215 — assistant non-functional
hero_agent#16 — Ambient AI parent

Owner: timur (investigating) + casper (reporting).

Source: meeting notes 2026-05-06.

## Overview The CRM AI surface returns invented contacts / projects / tasks instead of grounding on real OSIS data. Casper hit this; Timur is investigating. ## Why Meeting 2026-05-06: "asked timur to view the issue / AI making up stuff / explained to Casper / Timur helping Casper on blockers". This is *related to but distinct from* [home#215](https://forge.ourworld.tf/lhumina_code/home/issues/215) (assistant non-functional from missing Groq key). The hallucination here happens **even when the assistant responds** — the response is fluent but factually wrong. Likely candidates for the root cause: 1. Per-context routing broken ([hero_biz#37](https://forge.ourworld.tf/lhumina_code/hero_biz/issues/37) + [hero_rpc#42](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/42)) — agent gets default-context data instead of the user's context, so its grounding is wrong. 2. Tool surface for CRM reads/writes missing or unreliable — agent falls back to generative answer. 3. System prompt doesn't enforce "if you don't have grounded data, say so". ## Acceptance - [ ] Reproduce a clear hallucination case (one prompt, one wrong response) - [ ] Identify which of the candidate causes is the dominant one - [ ] Fix it (filed as separate issue if non-trivial) or escalate to hero_agent - [ ] Verify on demo VM with the same prompt ## Related - [hero_biz#37](https://forge.ourworld.tf/lhumina_code/hero_biz/issues/37) — context selection has no effect - [hero_rpc#42](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/42) — UDS transport drops X-Hero-Context - [home#215](https://forge.ourworld.tf/lhumina_code/home/issues/215) — assistant non-functional - [hero_agent#16](https://forge.ourworld.tf/lhumina_code/hero_agent/issues/16) — Ambient AI parent Owner: timur (investigating) + casper (reporting). Source: meeting notes 2026-05-06.

mik-tf added this to the ACTIVE project

2026-05-06 17:31:55 +00:00

mik-tf referenced this issue from lhumina_code/home

2026-05-06 17:45:03 +00:00

[ROADMAP] Phase ordering for ACTIVE board (project 13) #221

~~casper-stevens referenced this issue 2026-05-07 07:42:06 +00:00~~

fix(rpc): UDS transport silently drops X-Hero-Context headers — context isolation broken for direct socket calls #40

casper-stevens commented

2026-05-07 07:53:47 +00:00

Member

The dominant cause is likely candidate 2 — the AI tool surface may not exist yet. From reading the code, there appear to be no tool definitions, no function-calling schema, and no OSIS calls from the AI layer. build_entity_context() loads the currently-viewed entity and injects it as markdown into the prompt; that's likely the only OSIS data the LLM sees. Anything outside that one entity would then get fabricated.

Candidate 1 (context routing) is likely not the cause here. hero_biz → hero_osis traffic goes over HTTP through hero_router, and that path appears to already pass X-Hero-Context correctly. The UDS header bug in hero_rpc#42 is real but likely affects a different code path.

What would unblock this: wire CRM read tools into the assistant (search_persons, search_companies, list_tasks, get_deal) with a proper function-calling schema and dispatch in the assistant_chat() handler. Adding a grounding guardrail to the system prompt would help as a safety net regardless.

The dominant cause is **likely candidate 2** — the AI tool surface may not exist yet. From reading the code, there appear to be no tool definitions, no function-calling schema, and no OSIS calls from the AI layer. `build_entity_context()` loads the currently-viewed entity and injects it as markdown into the prompt; that's likely the only OSIS data the LLM sees. Anything outside that one entity would then get fabricated. **Candidate 1 (context routing)** is likely not the cause here. hero_biz → hero_osis traffic goes over HTTP through hero_router, and that path appears to already pass `X-Hero-Context` correctly. The UDS header bug in hero_rpc#42 is real but likely affects a different code path. **What would unblock this:** wire CRM read tools into the assistant (`search_persons`, `search_companies`, `list_tasks`, `get_deal`) with a proper function-calling schema and dispatch in the `assistant_chat()` handler. Adding a grounding guardrail to the system prompt would help as a safety net regardless.

casper-stevens referenced this issue from lhumina_code/hero_osis

2026-05-07 08:01:11 +00:00

fix: update Cargo.lock to pick up hero_rpc_client context fix (ed6e7eb3c0) #46