lhumina_code/home
21
0
Fork 0
Code Issues 10 Pull requests Projects Releases Packages Wiki Activity Actions

AI Agent knowledge pipeline — MCP integration + Hero OS context #75

New issue
Closed
opened 2026-03-22 16:44:52 +00:00 by mik-tf · 1 comment
mik-tf commented 2026-03-22 16:44:52 +00:00
Owner
Copy link

AI Agent Knowledge Pipeline — Complete Ecosystem Migration Spec

Context

hero_agent replaced hero_shrimp (#72) but the ecosystem wiring is incomplete. hero_agent works as a standalone AI agent (57 tools, SSE, voice) but doesn't integrate with the Hero OS service ecosystem the way Shrimp did.

This issue completes the migration to 100%.

Current State

Integration hero_shrimp had hero_agent has Status
LLM routing via AIBroker All calls through aibroker Config points to aibroker, OpenRouter fallback primary Partial
MCP tools via mcp_hero 5 tools (register, list, interface, generate, execute) mcp.json not wired Missing
hero_inspector discovery Appeared in service list No OpenRPC spec exposed Missing
hero_proc lifecycle Via socat bridge Direct socket binding (works) Functional
hero_books docs Could query via MCP No MCP wiring Missing
hero_proxy routing /hero_shrimp_ui/ /hero_agent/ Done
AI island frontend /hero_shrimp/api/chat /hero_agent/api/chat Done
System prompt context Generic Generic Both need improvement
Model naming AIBroker names (claude-sonnet) OpenRouter names (anthropic/claude-sonnet-4) Inconsistent
Smoke tests Basic Basic Needs expansion

Level 1: MCP Wiring (Quick — config only)

Goal: hero_agent discovers and uses all Hero service tools via mcp_hero.

  • Configure mcp.json in hero_agent data dir to connect to mcp_hero
  • mcp_hero runs as stdio subprocess of hero_aibroker (already built)
  • Verify hero_agent discovers all 5 mcp_hero tools at startup
  • Test: list_services → returns all Hero services
  • Test: register_service → discovers a service's OpenRPC spec
  • Test: generate_code + execute_code → generates and runs Python to interact with a service
  • Update hero_services entrypoint to ensure mcp.json is seeded with correct config

mcp.json format:

{
  "mcpServers": {
    "hero": {
      "command": "/root/hero/bin/mcp_hero",
      "transport": "stdio"
    }
  }
}

Level 2: System Prompt Enrichment (Quick — code change)

Goal: The LLM knows what Hero OS is and how to help users.

  • Update hero_agent/crates/hero_agent/src/prompt.rs to include:
    • What Hero OS is (unified platform, self-managing services, OpenRPC, MCP)
    • Available services and their purpose (from hero_inspector or hardcoded)
    • Available capabilities: "You can search docs via hero_books, manage processes via hero_proc, browse files via hero_foundry, etc."
    • Common user intents mapped to actions
    • "Use MCP tools to interact with Hero services"
  • Dynamic enrichment: query hero_inspector for live service list at startup, inject into prompt
  • Test: "What can I do with Hero?" → meaningful answer listing real capabilities

Level 3: LLM Routing Consistency (Medium)

Goal: All hero_agent LLM traffic goes through hero_aibroker.

  • hero_agent uses aibroker model names (not OpenRouter names)
    • claude-sonnet not anthropic/claude-sonnet-4
    • gpt-4o-mini not openai/gpt-4o-mini
    • llama-70b not meta-llama/llama-3.3-70b-instruct
  • When aibroker is available, ALL LLM calls go through it (model registry, key rotation, metrics)
  • OpenRouter direct is fallback only when aibroker is down
  • hero_agent.toml: comment out HERO_AGENT_OPENROUTER_MODELS so aibroker models are used by default
  • Verify aibroker model registry has the expected models configured

Level 4: Documentation Search Tool (Medium)

Goal: hero_agent can search and retrieve Hero OS documentation.

  • Add search_hero_docs built-in tool to hero_agent_tools/
  • Tool calls hero_books via Unix socket JSON-RPC:
    • search.query for semantic search (uses hero_embedder)
    • collections.getPage for direct page lookup
    • collections.list to show available documentation
  • Auto-triggered when user asks about features, architecture, configuration
  • Returns relevant doc snippets injected into conversation

Tool schema:

{
  "name": "search_hero_docs",
  "description": "Search Hero OS documentation (books, guides, API docs)",
  "parameters": {
    "query": {"type": "string", "description": "Search query"},
    "collection": {"type": "string", "description": "Optional: specific collection (hero-os-guide, developer-guide, etc.)"}
  }
}

Level 5: hero_proc SDK Lifecycle (Medium)

Goal: hero_agent properly integrates with hero_proc for managed lifecycle.

  • Add hero_proc_sdk dependency to hero_agent
  • Implement --start / --stop CLI flags
  • Register with hero_proc on startup (health check, log forwarding)
  • ProcLogger for dual logging (tracing + hero_proc logs tab)
  • Service shows in hero_proc dashboard with proper status

Level 6: OpenRPC Spec (Medium)

Goal: hero_agent is discoverable by hero_inspector and other services.

  • Define OpenRPC spec for hero_agent's API:
    • agent.chat — send message, get response
    • agent.conversations.list — list conversations
    • agent.conversations.get — get conversation history
    • agent.voice.transcribe — STT
    • agent.voice.tts — TTS
    • agent.config — get agent configuration
    • agent.stats — get statistics
  • Serve at /openrpc.json or via rpc.discover
  • hero_inspector auto-discovers hero_agent
  • MCP gateway exposes hero_agent tools to other agents

Level 7: Smoke Tests (Quick)

  • hero_agent health check in smoke.sh
  • hero_agent RPC connectivity test
  • hero_agent MCP tool discovery test (verify mcp_hero tools visible)
  • hero_agent voice transcription test (send test audio, verify transcript)
  • hero_agent connection status indicator test

Architecture: Complete Ecosystem Flow

User speaks/types in AI Assistant
  |
  +-- hero_agent receives message
  +-- System prompt includes:
  |     - Hero OS overview
  |     - Available services (from hero_inspector)
  |     - User memories
  |     - Current plan
  |
  +-- LLM call routes through hero_aibroker
  |     - Model registry (cheapest/best routing)
  |     - API key rotation
  |     - Usage metrics
  |
  +-- LLM decides to use tools:
  |     +-- MCP tools (via mcp_hero):
  |     |     - register_service → discover Hero service
  |     |     - generate_code → Python to interact with it
  |     |     - execute_code → run and return results
  |     +-- search_hero_docs → query hero_books
  |     +-- Built-in tools (file, git, search, shell, etc.)
  |
  +-- Response streams back via SSE
  +-- Optional: TTS playback (gpt-4o-mini-tts or browser Speech API)
  +-- Optional: auto-listen for next voice input (conversation mode)

References

  • Issue #18 — AI Broker OpenRPC design, mcp_hero implementation
  • Issue #72 — hero_agent replaced hero_shrimp
  • Issue #74 — Voice AI strategy
  • hero_aibroker/crates/mcp/mcp_hero/ — MCP tools source
  • hero_agent/crates/hero_agent/src/prompt.rs — System prompt
  • hero_agent/crates/hero_agent/src/mcp_client.rs — MCP client
  • hero_services/data/books/ — Seeded documentation

Signed-off-by: mik-tf

## AI Agent Knowledge Pipeline — Complete Ecosystem Migration Spec ### Context hero_agent replaced hero_shrimp (#72) but the ecosystem wiring is incomplete. hero_agent works as a standalone AI agent (57 tools, SSE, voice) but doesn't integrate with the Hero OS service ecosystem the way Shrimp did. This issue completes the migration to 100%. --- ### Current State | Integration | hero_shrimp had | hero_agent has | Status | |------------|----------------|----------------|--------| | LLM routing via AIBroker | All calls through aibroker | Config points to aibroker, OpenRouter fallback primary | Partial | | MCP tools via mcp_hero | 5 tools (register, list, interface, generate, execute) | mcp.json not wired | Missing | | hero_inspector discovery | Appeared in service list | No OpenRPC spec exposed | Missing | | hero_proc lifecycle | Via socat bridge | Direct socket binding (works) | Functional | | hero_books docs | Could query via MCP | No MCP wiring | Missing | | hero_proxy routing | /hero_shrimp_ui/ | /hero_agent/ | Done | | AI island frontend | /hero_shrimp/api/chat | /hero_agent/api/chat | Done | | System prompt context | Generic | Generic | Both need improvement | | Model naming | AIBroker names (claude-sonnet) | OpenRouter names (anthropic/claude-sonnet-4) | Inconsistent | | Smoke tests | Basic | Basic | Needs expansion | --- ### Level 1: MCP Wiring (Quick — config only) **Goal:** hero_agent discovers and uses all Hero service tools via mcp_hero. - [ ] Configure `mcp.json` in hero_agent data dir to connect to mcp_hero - [ ] mcp_hero runs as stdio subprocess of hero_aibroker (already built) - [ ] Verify hero_agent discovers all 5 mcp_hero tools at startup - [ ] Test: `list_services` → returns all Hero services - [ ] Test: `register_service` → discovers a service's OpenRPC spec - [ ] Test: `generate_code` + `execute_code` → generates and runs Python to interact with a service - [ ] Update hero_services entrypoint to ensure mcp.json is seeded with correct config **mcp.json format:** ```json { "mcpServers": { "hero": { "command": "/root/hero/bin/mcp_hero", "transport": "stdio" } } } ``` --- ### Level 2: System Prompt Enrichment (Quick — code change) **Goal:** The LLM knows what Hero OS is and how to help users. - [ ] Update `hero_agent/crates/hero_agent/src/prompt.rs` to include: - What Hero OS is (unified platform, self-managing services, OpenRPC, MCP) - Available services and their purpose (from hero_inspector or hardcoded) - Available capabilities: "You can search docs via hero_books, manage processes via hero_proc, browse files via hero_foundry, etc." - Common user intents mapped to actions - "Use MCP tools to interact with Hero services" - [ ] Dynamic enrichment: query hero_inspector for live service list at startup, inject into prompt - [ ] Test: "What can I do with Hero?" → meaningful answer listing real capabilities --- ### Level 3: LLM Routing Consistency (Medium) **Goal:** All hero_agent LLM traffic goes through hero_aibroker. - [ ] hero_agent uses aibroker model names (not OpenRouter names) - `claude-sonnet` not `anthropic/claude-sonnet-4` - `gpt-4o-mini` not `openai/gpt-4o-mini` - `llama-70b` not `meta-llama/llama-3.3-70b-instruct` - [ ] When aibroker is available, ALL LLM calls go through it (model registry, key rotation, metrics) - [ ] OpenRouter direct is fallback only when aibroker is down - [ ] hero_agent.toml: comment out `HERO_AGENT_OPENROUTER_MODELS` so aibroker models are used by default - [ ] Verify aibroker model registry has the expected models configured --- ### Level 4: Documentation Search Tool (Medium) **Goal:** hero_agent can search and retrieve Hero OS documentation. - [ ] Add `search_hero_docs` built-in tool to `hero_agent_tools/` - [ ] Tool calls hero_books via Unix socket JSON-RPC: - `search.query` for semantic search (uses hero_embedder) - `collections.getPage` for direct page lookup - `collections.list` to show available documentation - [ ] Auto-triggered when user asks about features, architecture, configuration - [ ] Returns relevant doc snippets injected into conversation **Tool schema:** ```json { "name": "search_hero_docs", "description": "Search Hero OS documentation (books, guides, API docs)", "parameters": { "query": {"type": "string", "description": "Search query"}, "collection": {"type": "string", "description": "Optional: specific collection (hero-os-guide, developer-guide, etc.)"} } } ``` --- ### Level 5: hero_proc SDK Lifecycle (Medium) **Goal:** hero_agent properly integrates with hero_proc for managed lifecycle. - [ ] Add `hero_proc_sdk` dependency to hero_agent - [ ] Implement `--start` / `--stop` CLI flags - [ ] Register with hero_proc on startup (health check, log forwarding) - [ ] ProcLogger for dual logging (tracing + hero_proc logs tab) - [ ] Service shows in hero_proc dashboard with proper status --- ### Level 6: OpenRPC Spec (Medium) **Goal:** hero_agent is discoverable by hero_inspector and other services. - [ ] Define OpenRPC spec for hero_agent's API: - `agent.chat` — send message, get response - `agent.conversations.list` — list conversations - `agent.conversations.get` — get conversation history - `agent.voice.transcribe` — STT - `agent.voice.tts` — TTS - `agent.config` — get agent configuration - `agent.stats` — get statistics - [ ] Serve at `/openrpc.json` or via `rpc.discover` - [ ] hero_inspector auto-discovers hero_agent - [ ] MCP gateway exposes hero_agent tools to other agents --- ### Level 7: Smoke Tests (Quick) - [ ] hero_agent health check in smoke.sh - [ ] hero_agent RPC connectivity test - [ ] hero_agent MCP tool discovery test (verify mcp_hero tools visible) - [ ] hero_agent voice transcription test (send test audio, verify transcript) - [ ] hero_agent connection status indicator test --- ### Architecture: Complete Ecosystem Flow ``` User speaks/types in AI Assistant | +-- hero_agent receives message +-- System prompt includes: | - Hero OS overview | - Available services (from hero_inspector) | - User memories | - Current plan | +-- LLM call routes through hero_aibroker | - Model registry (cheapest/best routing) | - API key rotation | - Usage metrics | +-- LLM decides to use tools: | +-- MCP tools (via mcp_hero): | | - register_service → discover Hero service | | - generate_code → Python to interact with it | | - execute_code → run and return results | +-- search_hero_docs → query hero_books | +-- Built-in tools (file, git, search, shell, etc.) | +-- Response streams back via SSE +-- Optional: TTS playback (gpt-4o-mini-tts or browser Speech API) +-- Optional: auto-listen for next voice input (conversation mode) ``` ### References - Issue #18 — AI Broker OpenRPC design, mcp_hero implementation - Issue #72 — hero_agent replaced hero_shrimp - Issue #74 — Voice AI strategy - hero_aibroker/crates/mcp/mcp_hero/ — MCP tools source - hero_agent/crates/hero_agent/src/prompt.rs — System prompt - hero_agent/crates/hero_agent/src/mcp_client.rs — MCP client - hero_services/data/books/ — Seeded documentation Signed-off-by: mik-tf
mik-tf commented 2026-03-22 17:50:46 +00:00
Author
Owner
Copy link

Completed — v0.6.0-dev

All 7 levels implemented and verified:

Level 1: MCP Wiring

  • Extended McpClient with stdio subprocess transport
  • mcp.json seeded in entrypoint with mcp_hero stdio config
  • hero_agent discovers mcp_hero tools at startup

Level 2: System Prompt Enrichment

  • prompt.rs enriched with Hero OS context, service descriptions, MCP instructions
  • Dynamic service list from MCP discovery injected into prompt

Level 3: LLM Routing

  • hero_agent.toml uses aibroker model names (claude-sonnet-4.5, claude-haiku-4.5, etc.)
  • OpenRouter commented out as fallback only
  • Default model prefers aibroker_models.first()

Level 4: search_hero_docs Tool

  • New built-in tool calling hero_books via Unix socket JSON-RPC
  • Supports query, book filter, and limit parameters

Level 5: hero_proc_sdk Lifecycle

  • New hero_agent_daemon crate with --start/--stop CLI
  • Registered in build-local.sh

Level 6: OpenRPC Spec

  • /openrpc.json endpoint + rpc.discover JSON-RPC method
  • Methods: agent.chat, agent.conversations., agent.config, agent.stats, agent.skills, agent.voice.

Level 7: Smoke Tests

  • 6 tests in smoke.sh: health, OpenRPC, config (7 models), skills (57 tools), voice
  • All passing

Repos touched

  • hero_agent (13 files)
  • hero_services (4 files)
  • hero_aibroker (1 file — daemon fix)

Release

## Completed — v0.6.0-dev All 7 levels implemented and verified: ### Level 1: MCP Wiring - Extended McpClient with stdio subprocess transport - mcp.json seeded in entrypoint with mcp_hero stdio config - hero_agent discovers mcp_hero tools at startup ### Level 2: System Prompt Enrichment - prompt.rs enriched with Hero OS context, service descriptions, MCP instructions - Dynamic service list from MCP discovery injected into prompt ### Level 3: LLM Routing - hero_agent.toml uses aibroker model names (claude-sonnet-4.5, claude-haiku-4.5, etc.) - OpenRouter commented out as fallback only - Default model prefers aibroker_models.first() ### Level 4: search_hero_docs Tool - New built-in tool calling hero_books via Unix socket JSON-RPC - Supports query, book filter, and limit parameters ### Level 5: hero_proc_sdk Lifecycle - New hero_agent_daemon crate with --start/--stop CLI - Registered in build-local.sh ### Level 6: OpenRPC Spec - /openrpc.json endpoint + rpc.discover JSON-RPC method - Methods: agent.chat, agent.conversations.*, agent.config, agent.stats, agent.skills, agent.voice.* ### Level 7: Smoke Tests - 6 tests in smoke.sh: health, OpenRPC, config (7 models), skills (57 tools), voice - All passing ### Repos touched - hero_agent (13 files) - hero_services (4 files) - hero_aibroker (1 file — daemon fix) ### Release - v0.6.0-dev: https://forge.ourworld.tf/lhumina_code/hero_services/releases/tag/v0.6.0-dev
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/home#75
No description provided.