lhumina_code/hero_demo

Fork 0

[infra][P1] post-deploy verification scripts — WASM, books refresh, embedder, AI grounding, demo seed, auth flow #55

New issue

Open

opened 2026-05-01 17:59:01 +00:00 by mik-tf · 0 comments

mik-tf commented

2026-05-01 17:59:01 +00:00

Owner

Summary

After a service_install_all deploy, "all services running" does not equal "demo flows actually work." Today's demo prep surfaced six classes of gap that aren't currently checked anywhere — they're caught (or not) by an operator manually clicking through and noticing something's off. That's not a runbook, it's a coping strategy.

This issue tracks adding a scripts/verify/ directory with six focused checks plus a master orchestrator. Each is small, idempotent, and exit-code-honest. Tonight (2026-05-01) we'll execute manual versions to ensure tomorrow's demo works; the captured commands become the seed for these scripts as PRs in the days after.

Drop these into hero_demo/scripts/verify/. Master runs all six in order; each is independently invokable.

Why this needs to exist

Six distinct silent-failure modes today. WASM bundle stale; hero_books didn't re-pull on restart; embedder reports "quality 1 not available"; AI Assistant grounds on stale docs; OnlyOffice has no test docs to open; auth flow has historically broken in subtle ways (project_auth_gating_status.md).
No single command answers "is the demo ready?" Operator has to manually click through 8 demo steps to find out. Each manual click takes minutes plus mental load.
Recovery is reactive, not proactive. When something breaks during a demo, we lose face. When something fails in a verify script we run BEFORE the demo, we just fix it.

Architecture

hero_demo/scripts/verify/:

verify/
├── 00_orchestrator.sh           # Master — runs all six in order, exits 0 on full pass
├── 10_wasm_currency.nu          # A
├── 20_books_refresh.sh          # B
├── 30_embedder_health.sh        # C
├── 40_ai_grounding_smoke.sh     # D
├── 50_demo_content_seed.sh      # E
├── 60_auth_flow.sh              # F
└── lib/                         # shared helpers
    ├── rpc_helpers.sh           # `rpc_call <sock> <method> <params>` etc.
    └── config.sh                # CONTEXTS, WEBDAV_BASE, SOCK_BASE etc.

Run via make verify from hero_demo/Makefile, or directly: bash scripts/verify/00_orchestrator.sh.

Each script:

Hits real services via Unix sockets (no mocks)
Idempotent — running twice is a no-op when nothing changed
Loud failure with clear diagnostic on stderr
Exits non-zero on fail; orchestrator stops on first fail unless --continue

A. WASM currency check (`10_wasm_currency.nu`)

Goal: Skip the 25-min WASM rebuild when the bundle is current. Force it when stale. Never silently use a stale bundle.

Mechanism: Stamp .build_sha into the WASM bundle dir at build time; compare to current hero_os HEAD on next deploy.

#!/usr/bin/env nu
# 10_wasm_currency.nu — exit 0 if WASM bundle matches current hero_os HEAD, 1 otherwise.
# Companion to service_os install: write .build_sha there after a successful WASM build.

const BUNDLE_DIR = "/home/driver/hero/share/hero_os/public"
const SHA_FILE   = "/home/driver/hero/share/hero_os/public/.build_sha"
const CODE_DIR   = "/data/home/driver/hero/code0/hero_os"

def main [] {
    if not ($BUNDLE_DIR | path exists) {
        print $"✗ WASM bundle dir missing: ($BUNDLE_DIR)"
        exit 1
    }
    if not ($SHA_FILE | path exists) {
        print $"✗ No .build_sha — bundle was never stamped or is from before this script existed. Rebuild required."
        exit 1
    }

    let bundled_sha = (open $SHA_FILE | str trim)
    let current_sha = (do { ^git -C $CODE_DIR rev-parse HEAD } | complete | get stdout | str trim)

    if $bundled_sha == $current_sha {
        print $"✓ WASM bundle current at ($current_sha | str substring 0..7)"
        exit 0
    } else {
        print $"✗ WASM bundle stale: bundled=($bundled_sha | str substring 0..7), current=($current_sha | str substring 0..7)"
        print "  Rebuild: cd ~/hero/code0/hero_os && make build && make install-assets-release"
        exit 1
    }
}

To make this useful, also add to service_os.nu: after a successful make install-assets-release, write git rev-parse HEAD > /home/driver/hero/share/hero_os/public/.build_sha. This is the missing primitive — without it, the comparison can't work.

Caveats:

Some hero_os development commits don't actually affect the WASM (e.g. CI tweaks, README). True dependency tracking would require a content-hash of the build inputs (Cargo.lock + src/), but commit SHA is a tighter lower-bound (rebuild more often than necessary, never less).
After the first time this runs and forces a rebuild, the hash is up-to-date and subsequent deploys correctly skip.

B. hero_books refresh (`20_books_refresh.sh`)

Goal: Force every registered library to git-pull + re-extract Q&A for changed pages + re-embed. Must run after every deploy because service_books restart does NOT reliably auto-pull libraries today (observed today: ~/hero/var/books/hero/ mtime stayed at Apr 23 across multiple restarts).

The .ai/<page>.toml hash gate already exists — running this script multiple times is bounded by actual content changes. Subsequent runs are fast.

#!/bin/bash
# 20_books_refresh.sh — force pull + reindex on every registered library.
# Bounded cost via existing .ai/<page>.toml hash check inside hero_books.
set -euo pipefail
SOCK=~/hero/var/sockets/hero_books/rpc.sock

rpc() {
    curl -s --unix-socket "$SOCK" -X POST -H "Content-Type: application/json" \
        -d "$1" http://localhost/rpc
}

# 1. Get registered libraries
LIBS_RESP=$(rpc '{"jsonrpc":"2.0","id":1,"method":"libraries.list","params":{}}')
echo "$LIBS_RESP" | jq -e '.result' >/dev/null || { echo "✗ libraries.list failed"; echo "$LIBS_RESP" | jq .; exit 1; }
LIBS=$(echo "$LIBS_RESP" | jq -r '.result.libraries[].namespace')

# 2. For each library: scan with --pull (git pull) + reindex (extract+embed changed)
for lib in $LIBS; do
    echo "→ $lib: pull + scan"
    rpc "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"books.scan\",\"params\":{\"library\":\"$lib\",\"pull\":true}}" \
        | jq -e '.result' >/dev/null || { echo "  ✗ scan failed"; exit 1; }

    echo "→ $lib: reindex"
    rpc "{\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"books.reindex\",\"params\":{\"library\":\"$lib\"}}" \
        | jq -e '.result' >/dev/null || { echo "  ✗ reindex failed"; exit 1; }

    # Sanity: confirm clone mtime got bumped (proves git pull actually happened)
    if [ -d "/home/driver/hero/var/books/$lib/.git" ]; then
        mtime=$(stat -c %Y "/home/driver/hero/var/books/$lib")
        age=$(( $(date +%s) - mtime ))
        if [ "$age" -gt 300 ]; then
            echo "  ⚠ $lib mtime is ${age}s old — pull may have silently failed"
        fi
    fi
done

echo "✓ all libraries refreshed"

Caveats:

The pull: true param on books.scan is what we BELIEVE the API to be — verify against the actual hero_books OpenRPC during manual run tonight. If not present, fall back to git -C ~/hero/var/books/<lib> pull directly + then call books.reindex.
The mtime sanity check catches the "pull silently no-oped" case we observed today.

C. embedder health (`30_embedder_health.sh`)

Goal: Confirm hero_embedderd is reachable, models are loaded, a real embed call returns a real vector. Specifically catch the "Embedder for quality 1 (Fast) not available" condition we hit today.

#!/bin/bash
# 30_embedder_health.sh — embedder health + a real embed test.
set -euo pipefail
SOCK=~/hero/var/sockets/hero_embedder/rpc.sock

rpc() {
    curl -s --unix-socket "$SOCK" -X POST -H "Content-Type: application/json" -d "$1" http://localhost/rpc
}

# 1. health endpoint
echo "→ embedder.health"
HEALTH=$(rpc '{"jsonrpc":"2.0","id":1,"method":"health","params":{}}')
echo "$HEALTH" | jq -e '.result.status == "ok"' >/dev/null || {
    echo "✗ embedder.health not OK"
    echo "$HEALTH" | jq .
    exit 1
}
echo "  ✓ healthy"

# 2. real embed call (catches "quality 1 not available" silently broken state)
echo "→ test embed"
EMBED=$(rpc '{"jsonrpc":"2.0","id":2,"method":"embed","params":{"text":"hello world","quality":1}}')
DIM=$(echo "$EMBED" | jq -r '.result.vector | length // 0')
if [ "$DIM" -lt 100 ]; then
    echo "✗ embed call returned no/short vector (dim=$DIM)"
    echo "$EMBED" | jq .
    exit 1
fi
echo "  ✓ embed call returned ${DIM}-dim vector"

# 3. namespace listing (confirms data dir is set up)
NS=$(rpc '{"jsonrpc":"2.0","id":3,"method":"namespace.list","params":{}}' | jq -r '.result.namespaces | length')
echo "  → ${NS} namespaces registered"

echo "✓ embedder fully healthy"

Caveats:

Method names assumed (health, embed, namespace.list) — verify against hero_embedder OpenRPC. The quality param matches the investigation notes from session 52.

D. AI grounding smoke (`40_ai_grounding_smoke.sh`)

Goal: Verify the AI Assistant has the LATEST docs_hero indexed by asking known questions and checking which page is cited. If we just merged session 52's Tier A pages, asking "what is agent_run?" must cite service_router.md — if it cites old content, hero_books didn't re-extract.

#!/bin/bash
# 40_ai_grounding_smoke.sh — battery of grounded questions, exit 0 if all hit expected pages.
set -uo pipefail
SOCK=~/hero/var/sockets/hero_books/rpc.sock

rpc_query() {
    local q="$1"
    curl -s --unix-socket "$SOCK" -X POST -H "Content-Type: application/json" \
        -d "$(jq -n --arg q "$q" '{jsonrpc:"2.0",id:1,method:"search.query",params:{query:$q,limit:3}}')" \
        http://localhost/rpc
}

# Expected: top result's `page` field after running search.query.
# Add new entries when new Tier A/B pages land.
declare -a TESTS=(
    "What is agent_run?|service_router"
    "How does the AI Assistant ground its answers?|service_agent"
    "How do I add a library to Ask the Librarian?|service_books"
    "What's the difference between hero_office and hero_onlyoffice?|service_office"
    "What voice does Hero use for TTS?|service_voice"
    "Where are my photos stored on disk?|service_photos"
    "How does hero_router expose every service as MCP?|service_router"
)

PASS=0; FAIL=0
for t in "${TESTS[@]}"; do
    IFS='|' read -r query expected <<< "$t"
    result=$(rpc_query "$query")
    top=$(echo "$result" | jq -r '.result.results[0].page // "none"')
    if [ "$top" = "$expected" ]; then
        echo "✓ '$query' → $top"
        PASS=$((PASS+1))
    else
        echo "✗ '$query' expected $expected, got $top"
        # Show top 3 for debugging
        echo "    top 3: $(echo "$result" | jq -r '.result.results[].page' | head -3 | tr '\n' ',')"
        FAIL=$((FAIL+1))
    fi
done

echo "─────"
echo "$PASS/${#TESTS[@]} passed"
exit $FAIL

Caveats:

Tests must be updated when Tier A/B pages are renamed or restructured. Worth adding a comment in the file pointing at the docs_hero collection so engineers update both.
Doesn't test the LLM round-trip — that's agent.chat not search.query. A future iteration can add an end-to-end agent.chat test that exercises the LLM provider too. For now, search.query is the first place stale content shows up.

E. demo content seed (`50_demo_content_seed.sh`)

Goal: Copy the demo PDFs into hero_foundry's webdav for every registered context, idempotent on sha256.

#!/bin/bash
# 50_demo_content_seed.sh — seed demo PDFs into per-context foundry webdav.
# Source: pass --src /path/to/docs_demo/
set -euo pipefail

SRC_DIR="${SRC_DIR:-/home/pctwo/Documents/temp/hero_work/docs_demo}"
WEBDAV_BASE=/home/driver/hero/var/hero_foundry/webdav
CONTEXTS="${HERO_CONTEXTS:-incubaid,geomind,threefold,ourworld}"

[ -d "$SRC_DIR" ] || { echo "✗ source dir not found: $SRC_DIR"; exit 1; }

IFS=',' read -ra CTX_ARRAY <<< "$CONTEXTS"
TOTAL_COPIED=0
TOTAL_SKIPPED=0
for ctx in "${CTX_ARRAY[@]}"; do
    dest_dir="$WEBDAV_BASE/$ctx/Office"
    mkdir -p "$dest_dir"
    chown driver:driver "$dest_dir"

    for f in "$SRC_DIR"/*.pdf "$SRC_DIR"/*.docx "$SRC_DIR"/*.xlsx; do
        [ -f "$f" ] || continue
        fn=$(basename "$f")
        src_sha=$(sha256sum "$f" | cut -d' ' -f1)
        if [ -f "$dest_dir/$fn" ]; then
            dst_sha=$(sha256sum "$dest_dir/$fn" | cut -d' ' -f1)
            if [ "$src_sha" = "$dst_sha" ]; then
                TOTAL_SKIPPED=$((TOTAL_SKIPPED+1))
                continue
            fi
        fi
        cp "$f" "$dest_dir/$fn"
        chown driver:driver "$dest_dir/$fn"
        echo "  → $ctx/Office/$fn"
        TOTAL_COPIED=$((TOTAL_COPIED+1))
    done
done

echo "✓ seeded ${TOTAL_COPIED} new files, skipped ${TOTAL_SKIPPED} unchanged across ${#CTX_ARRAY[@]} contexts"

Caveats:

Source directory currently lives only on the workstation. Either: (a) the script runs locally and SCP's, or (b) we vendor the PDFs into the hero_demo repo under seed/office/. Long-term (b) is right; short-term (a) is fine.
Permissions: hero_foundry runs as driver, so files must be owned by driver:driver to be readable.
May need parallel seeding for Photos/ and Videos/ later — same script shape, different source dir.

F. auth flow (`60_auth_flow.sh`)

Goal: Verify basic-auth gate, post-auth desktop loads, X-Hero-Context header propagates correctly to per-context endpoints.

#!/bin/bash
# 60_auth_flow.sh — auth gate + X-Hero-Context propagation smoke.
set -uo pipefail

BASE_URL="${HERODEMO_URL:-https://herodemo.gent01.grid.tf}"
ADMIN_USER="${HERODEMO_ADMIN_USER:-admin}"
ADMIN_PASS="${HERODEMO_ADMIN_PASS:-}"
[ -n "$ADMIN_PASS" ] || { echo "✗ HERODEMO_ADMIN_PASS not set"; exit 1; }

# 1. Unauthenticated: must be 401
echo "→ unauthenticated GET /"
code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE_URL/")
[ "$code" = "401" ] || { echo "  ✗ expected 401, got $code"; exit 1; }
echo "  ✓ 401"

# 2. Authenticated: must be 200 or 302
echo "→ authenticated GET /"
code=$(curl -s -o /dev/null -w "%{http_code}" -u "$ADMIN_USER:$ADMIN_PASS" "$BASE_URL/")
[[ "$code" =~ ^(200|302)$ ]] || { echo "  ✗ expected 200/302, got $code"; exit 1; }
echo "  ✓ $code"

# 3. WASM bundle reachable post-auth (catches blank-desktop-after-login)
echo "→ WASM asset reachable"
code=$(curl -s -o /dev/null -w "%{http_code}" -u "$ADMIN_USER:$ADMIN_PASS" "$BASE_URL/hero_os/")
[[ "$code" =~ ^(200|301|302)$ ]] || { echo "  ✗ hero_os UI not reachable: $code"; exit 1; }
echo "  ✓ hero_os UI reachable"

# 4. X-Hero-Context propagation: pick a per-context endpoint
echo "→ X-Hero-Context propagation"
for ctx in ${HERO_CONTEXTS:-incubaid geomind}; do
    resp=$(curl -s -u "$ADMIN_USER:$ADMIN_PASS" \
        -H "X-Hero-Context: $ctx" -H "Content-Type: application/json" \
        "$BASE_URL/hero_osis_business/rpc" \
        -X POST \
        -d '{"jsonrpc":"2.0","id":1,"method":"contact.list","params":{}}')
    count=$(echo "$resp" | jq -r '.result | length // 0' 2>/dev/null || echo "?")
    if [ "$count" = "?" ]; then
        echo "  ✗ $ctx: contact.list returned non-JSON or error"
        echo "    $resp" | head -c 200
        exit 1
    fi
    echo "  ✓ $ctx: $count contacts"
done

echo "✓ auth flow complete"

Caveats:

Assumes hero_proxy + hero_router URL paths. URL shape /hero_osis_business/rpc may need to be verified against current routing.
Doesn't currently distinguish "context X has 0 contacts" from "endpoint returned 0 due to context-fallback bug" — both look like count=0. A stricter test would seed a known contact in each context and verify it appears.

00. Master orchestrator (`00_orchestrator.sh`)

#!/bin/bash
# 00_orchestrator.sh — run all six verifies in order.
# Exits non-zero on first failure unless --continue passed.
set -uo pipefail

CONTINUE=false
[ "${1:-}" = "--continue" ] && CONTINUE=true

cd "$(dirname "$0")"

declare -a CHECKS=(
    "10_wasm_currency.nu|WASM bundle currency"
    "20_books_refresh.sh|hero_books library refresh"
    "30_embedder_health.sh|embedder health"
    "40_ai_grounding_smoke.sh|AI Assistant grounding"
    "50_demo_content_seed.sh|demo content seed"
    "60_auth_flow.sh|auth + X-Hero-Context"
)

FAILS=()
for entry in "${CHECKS[@]}"; do
    IFS='|' read -r script desc <<< "$entry"
    echo ""
    echo "═══ $desc ═══"
    if [ -x "$script" ] || [[ "$script" == *.nu ]]; then
        if [[ "$script" == *.nu ]]; then
            nu "$script" || { FAILS+=("$desc"); $CONTINUE || exit 1; }
        else
            bash "$script" || { FAILS+=("$desc"); $CONTINUE || exit 1; }
        fi
    else
        echo "✗ $script not executable"
        FAILS+=("$desc")
        $CONTINUE || exit 1
    fi
done

echo ""
echo "═══════════════════════════════════════"
if [ ${#FAILS[@]} -eq 0 ]; then
    echo "✓ all 6 checks passed — demo is verified"
    exit 0
else
    echo "✗ ${#FAILS[@]} failure(s):"
    for f in "${FAILS[@]}"; do echo "  - $f"; done
    exit 1
fi

Hookup into the runbook

In hero_demo/docs/ops/DEPLOYMENT.md, add a new section after §4 (Install hero services):

## §4.5 Verify (mandatory before declaring deploy complete)

After service_install_all completes, run:

  bash scripts/verify/00_orchestrator.sh

This runs six checks in order: WASM currency, books refresh, embedder
health, AI grounding, demo content seed, auth flow. All six must pass
before the demo is considered deploy-ready. If anything fails, the
script exits non-zero with a specific reason.

To override an individual failure (acknowledged risk):

  bash scripts/verify/00_orchestrator.sh --continue

Also adds to hero_demo/Makefile:

.PHONY: verify
verify:
	bash scripts/verify/00_orchestrator.sh

.PHONY: verify-continue
verify-continue:
	bash scripts/verify/00_orchestrator.sh --continue

Rollout

This issue is a META — each script lands as a separate small PR. Order:

50_demo_content_seed.sh — easy, no service-side changes, immediate value tonight (seeding PDFs)
30_embedder_health.sh — read-only, low risk
40_ai_grounding_smoke.sh — read-only, immediate signal on docs_hero currency
20_books_refresh.sh — needs verifying the hero_books RPC method names; may surface a small server-side gap (e.g. if books.scan doesn't take pull: true, file a separate hero_books issue)
60_auth_flow.sh — depends on getting clean URL paths via hero_router/hero_proxy
10_wasm_currency.nu — depends on amending service_os to write .build_sha. Slight ordering: PR the .build_sha write FIRST, then the verify script.

Once 1-6 land, master orchestrator (00) is one more PR.

Tonight's manual run as the seed

Tonight (2026-05-01) we'll execute the equivalent commands manually after install_all run 4 finishes, against herodemo, capturing every command + output. By tomorrow morning the team has:

A working demo for the agenda (Books, embedder, slides, office, biz, agent+CRM, whiteboard, collab)
Real raw data on what RPC method names actually exist (closing the "method names assumed" caveats above)
Validation that the script designs make sense

The post-demo PRs land each script with the manual-run commands as the seed.

Cross-refs

hero_demo#52 — vision — sovereignty pitch needs reproducible deploys
hero_demo#54 — CI artifacts — these scripts run on CI-built artifacts too once #54 lands
hero_proc#86 — reliability META (closed but informs the principles)
session 52's five deploy bugs — the kind of thing that would have been caught if #54 + this issue had existed before today.

## Summary After a `service_install_all` deploy, "all services running" does not equal "demo flows actually work." Today's demo prep surfaced six classes of gap that aren't currently checked anywhere — they're caught (or not) by an operator manually clicking through and noticing something's off. That's not a runbook, it's a coping strategy. This issue tracks adding a `scripts/verify/` directory with six focused checks plus a master orchestrator. Each is small, idempotent, and exit-code-honest. Tonight (2026-05-01) we'll execute manual versions to ensure tomorrow's demo works; the captured commands become the seed for these scripts as PRs in the days after. Drop these into `hero_demo/scripts/verify/`. Master runs all six in order; each is independently invokable. ## Why this needs to exist - **Six distinct silent-failure modes today.** WASM bundle stale; hero_books didn't re-pull on restart; embedder reports "quality 1 not available"; AI Assistant grounds on stale docs; OnlyOffice has no test docs to open; auth flow has historically broken in subtle ways (project_auth_gating_status.md). - **No single command answers "is the demo ready?"** Operator has to manually click through 8 demo steps to find out. Each manual click takes minutes plus mental load. - **Recovery is reactive, not proactive.** When something breaks during a demo, we lose face. When something fails in a verify script we run BEFORE the demo, we just fix it. ## Architecture `hero_demo/scripts/verify/`: ``` verify/ ├── 00_orchestrator.sh # Master — runs all six in order, exits 0 on full pass ├── 10_wasm_currency.nu # A ├── 20_books_refresh.sh # B ├── 30_embedder_health.sh # C ├── 40_ai_grounding_smoke.sh # D ├── 50_demo_content_seed.sh # E ├── 60_auth_flow.sh # F └── lib/ # shared helpers ├── rpc_helpers.sh # `rpc_call <sock> <method> <params>` etc. └── config.sh # CONTEXTS, WEBDAV_BASE, SOCK_BASE etc. ``` Run via `make verify` from `hero_demo/Makefile`, or directly: `bash scripts/verify/00_orchestrator.sh`. Each script: - Hits real services via Unix sockets (no mocks) - Idempotent — running twice is a no-op when nothing changed - Loud failure with clear diagnostic on stderr - Exits non-zero on fail; orchestrator stops on first fail unless `--continue` ## A. WASM currency check (`10_wasm_currency.nu`) **Goal:** Skip the 25-min WASM rebuild when the bundle is current. Force it when stale. Never silently use a stale bundle. **Mechanism:** Stamp `.build_sha` into the WASM bundle dir at build time; compare to current hero_os HEAD on next deploy. ```nu #!/usr/bin/env nu # 10_wasm_currency.nu — exit 0 if WASM bundle matches current hero_os HEAD, 1 otherwise. # Companion to service_os install: write .build_sha there after a successful WASM build. const BUNDLE_DIR = "/home/driver/hero/share/hero_os/public" const SHA_FILE = "/home/driver/hero/share/hero_os/public/.build_sha" const CODE_DIR = "/data/home/driver/hero/code0/hero_os" def main [] { if not ($BUNDLE_DIR | path exists) { print $"✗ WASM bundle dir missing: ($BUNDLE_DIR)" exit 1 } if not ($SHA_FILE | path exists) { print $"✗ No .build_sha — bundle was never stamped or is from before this script existed. Rebuild required." exit 1 } let bundled_sha = (open $SHA_FILE | str trim) let current_sha = (do { ^git -C $CODE_DIR rev-parse HEAD } | complete | get stdout | str trim) if $bundled_sha == $current_sha { print $"✓ WASM bundle current at ($current_sha | str substring 0..7)" exit 0 } else { print $"✗ WASM bundle stale: bundled=($bundled_sha | str substring 0..7), current=($current_sha | str substring 0..7)" print " Rebuild: cd ~/hero/code0/hero_os && make build && make install-assets-release" exit 1 } } ``` **To make this useful, also add to `service_os.nu`:** after a successful `make install-assets-release`, write `git rev-parse HEAD > /home/driver/hero/share/hero_os/public/.build_sha`. This is the missing primitive — without it, the comparison can't work. **Caveats:** - Some hero_os development commits don't actually affect the WASM (e.g. CI tweaks, README). True dependency tracking would require a content-hash of the build inputs (Cargo.lock + src/), but commit SHA is a tighter lower-bound (rebuild more often than necessary, never less). - After the first time this runs and forces a rebuild, the hash is up-to-date and subsequent deploys correctly skip. ## B. hero_books refresh (`20_books_refresh.sh`) **Goal:** Force every registered library to git-pull + re-extract Q&A for changed pages + re-embed. Must run after every deploy because `service_books restart` does NOT reliably auto-pull libraries today (observed today: `~/hero/var/books/hero/` mtime stayed at Apr 23 across multiple restarts). The `.ai/<page>.toml` hash gate already exists — running this script multiple times is bounded by actual content changes. Subsequent runs are fast. ```bash #!/bin/bash # 20_books_refresh.sh — force pull + reindex on every registered library. # Bounded cost via existing .ai/<page>.toml hash check inside hero_books. set -euo pipefail SOCK=~/hero/var/sockets/hero_books/rpc.sock rpc() { curl -s --unix-socket "$SOCK" -X POST -H "Content-Type: application/json" \ -d "$1" http://localhost/rpc } # 1. Get registered libraries LIBS_RESP=$(rpc '{"jsonrpc":"2.0","id":1,"method":"libraries.list","params":{}}') echo "$LIBS_RESP" | jq -e '.result' >/dev/null || { echo "✗ libraries.list failed"; echo "$LIBS_RESP" | jq .; exit 1; } LIBS=$(echo "$LIBS_RESP" | jq -r '.result.libraries[].namespace') # 2. For each library: scan with --pull (git pull) + reindex (extract+embed changed) for lib in $LIBS; do echo "→ $lib: pull + scan" rpc "{\"jsonrpc\":\"2.0\",\"id\":2,\"method\":\"books.scan\",\"params\":{\"library\":\"$lib\",\"pull\":true}}" \ | jq -e '.result' >/dev/null || { echo " ✗ scan failed"; exit 1; } echo "→ $lib: reindex" rpc "{\"jsonrpc\":\"2.0\",\"id\":3,\"method\":\"books.reindex\",\"params\":{\"library\":\"$lib\"}}" \ | jq -e '.result' >/dev/null || { echo " ✗ reindex failed"; exit 1; } # Sanity: confirm clone mtime got bumped (proves git pull actually happened) if [ -d "/home/driver/hero/var/books/$lib/.git" ]; then mtime=$(stat -c %Y "/home/driver/hero/var/books/$lib") age=$(( $(date +%s) - mtime )) if [ "$age" -gt 300 ]; then echo " ⚠ $lib mtime is ${age}s old — pull may have silently failed" fi fi done echo "✓ all libraries refreshed" ``` **Caveats:** - The `pull: true` param on `books.scan` is what we BELIEVE the API to be — verify against the actual hero_books OpenRPC during manual run tonight. If not present, fall back to `git -C ~/hero/var/books/<lib> pull` directly + then call `books.reindex`. - The mtime sanity check catches the "pull silently no-oped" case we observed today. ## C. embedder health (`30_embedder_health.sh`) **Goal:** Confirm hero_embedderd is reachable, models are loaded, a real embed call returns a real vector. Specifically catch the "Embedder for quality 1 (Fast) not available" condition we hit today. ```bash #!/bin/bash # 30_embedder_health.sh — embedder health + a real embed test. set -euo pipefail SOCK=~/hero/var/sockets/hero_embedder/rpc.sock rpc() { curl -s --unix-socket "$SOCK" -X POST -H "Content-Type: application/json" -d "$1" http://localhost/rpc } # 1. health endpoint echo "→ embedder.health" HEALTH=$(rpc '{"jsonrpc":"2.0","id":1,"method":"health","params":{}}') echo "$HEALTH" | jq -e '.result.status == "ok"' >/dev/null || { echo "✗ embedder.health not OK" echo "$HEALTH" | jq . exit 1 } echo " ✓ healthy" # 2. real embed call (catches "quality 1 not available" silently broken state) echo "→ test embed" EMBED=$(rpc '{"jsonrpc":"2.0","id":2,"method":"embed","params":{"text":"hello world","quality":1}}') DIM=$(echo "$EMBED" | jq -r '.result.vector | length // 0') if [ "$DIM" -lt 100 ]; then echo "✗ embed call returned no/short vector (dim=$DIM)" echo "$EMBED" | jq . exit 1 fi echo " ✓ embed call returned ${DIM}-dim vector" # 3. namespace listing (confirms data dir is set up) NS=$(rpc '{"jsonrpc":"2.0","id":3,"method":"namespace.list","params":{}}' | jq -r '.result.namespaces | length') echo " → ${NS} namespaces registered" echo "✓ embedder fully healthy" ``` **Caveats:** - Method names assumed (`health`, `embed`, `namespace.list`) — verify against hero_embedder OpenRPC. The `quality` param matches the investigation notes from session 52. ## D. AI grounding smoke (`40_ai_grounding_smoke.sh`) **Goal:** Verify the AI Assistant has the LATEST docs_hero indexed by asking known questions and checking which page is cited. If we just merged session 52's Tier A pages, asking "what is agent_run?" must cite `service_router.md` — if it cites old content, hero_books didn't re-extract. ```bash #!/bin/bash # 40_ai_grounding_smoke.sh — battery of grounded questions, exit 0 if all hit expected pages. set -uo pipefail SOCK=~/hero/var/sockets/hero_books/rpc.sock rpc_query() { local q="$1" curl -s --unix-socket "$SOCK" -X POST -H "Content-Type: application/json" \ -d "$(jq -n --arg q "$q" '{jsonrpc:"2.0",id:1,method:"search.query",params:{query:$q,limit:3}}')" \ http://localhost/rpc } # Expected: top result's `page` field after running search.query. # Add new entries when new Tier A/B pages land. declare -a TESTS=( "What is agent_run?|service_router" "How does the AI Assistant ground its answers?|service_agent" "How do I add a library to Ask the Librarian?|service_books" "What's the difference between hero_office and hero_onlyoffice?|service_office" "What voice does Hero use for TTS?|service_voice" "Where are my photos stored on disk?|service_photos" "How does hero_router expose every service as MCP?|service_router" ) PASS=0; FAIL=0 for t in "${TESTS[@]}"; do IFS='|' read -r query expected <<< "$t" result=$(rpc_query "$query") top=$(echo "$result" | jq -r '.result.results[0].page // "none"') if [ "$top" = "$expected" ]; then echo "✓ '$query' → $top" PASS=$((PASS+1)) else echo "✗ '$query' expected $expected, got $top" # Show top 3 for debugging echo " top 3: $(echo "$result" | jq -r '.result.results[].page' | head -3 | tr '\n' ',')" FAIL=$((FAIL+1)) fi done echo "─────" echo "$PASS/${#TESTS[@]} passed" exit $FAIL ``` **Caveats:** - Tests must be updated when Tier A/B pages are renamed or restructured. Worth adding a comment in the file pointing at the docs_hero collection so engineers update both. - Doesn't test the LLM round-trip — that's `agent.chat` not `search.query`. A future iteration can add an end-to-end `agent.chat` test that exercises the LLM provider too. For now, search.query is the first place stale content shows up. ## E. demo content seed (`50_demo_content_seed.sh`) **Goal:** Copy the demo PDFs into hero_foundry's webdav for every registered context, idempotent on sha256. ```bash #!/bin/bash # 50_demo_content_seed.sh — seed demo PDFs into per-context foundry webdav. # Source: pass --src /path/to/docs_demo/ set -euo pipefail SRC_DIR="${SRC_DIR:-/home/pctwo/Documents/temp/hero_work/docs_demo}" WEBDAV_BASE=/home/driver/hero/var/hero_foundry/webdav CONTEXTS="${HERO_CONTEXTS:-incubaid,geomind,threefold,ourworld}" [ -d "$SRC_DIR" ] || { echo "✗ source dir not found: $SRC_DIR"; exit 1; } IFS=',' read -ra CTX_ARRAY <<< "$CONTEXTS" TOTAL_COPIED=0 TOTAL_SKIPPED=0 for ctx in "${CTX_ARRAY[@]}"; do dest_dir="$WEBDAV_BASE/$ctx/Office" mkdir -p "$dest_dir" chown driver:driver "$dest_dir" for f in "$SRC_DIR"/*.pdf "$SRC_DIR"/*.docx "$SRC_DIR"/*.xlsx; do [ -f "$f" ] || continue fn=$(basename "$f") src_sha=$(sha256sum "$f" | cut -d' ' -f1) if [ -f "$dest_dir/$fn" ]; then dst_sha=$(sha256sum "$dest_dir/$fn" | cut -d' ' -f1) if [ "$src_sha" = "$dst_sha" ]; then TOTAL_SKIPPED=$((TOTAL_SKIPPED+1)) continue fi fi cp "$f" "$dest_dir/$fn" chown driver:driver "$dest_dir/$fn" echo " → $ctx/Office/$fn" TOTAL_COPIED=$((TOTAL_COPIED+1)) done done echo "✓ seeded ${TOTAL_COPIED} new files, skipped ${TOTAL_SKIPPED} unchanged across ${#CTX_ARRAY[@]} contexts" ``` **Caveats:** - Source directory currently lives only on the workstation. Either: (a) the script runs locally and SCP's, or (b) we vendor the PDFs into the hero_demo repo under `seed/office/`. Long-term (b) is right; short-term (a) is fine. - Permissions: hero_foundry runs as `driver`, so files must be owned by `driver:driver` to be readable. - May need parallel seeding for `Photos/` and `Videos/` later — same script shape, different source dir. ## F. auth flow (`60_auth_flow.sh`) **Goal:** Verify basic-auth gate, post-auth desktop loads, X-Hero-Context header propagates correctly to per-context endpoints. ```bash #!/bin/bash # 60_auth_flow.sh — auth gate + X-Hero-Context propagation smoke. set -uo pipefail BASE_URL="${HERODEMO_URL:-https://herodemo.gent01.grid.tf}" ADMIN_USER="${HERODEMO_ADMIN_USER:-admin}" ADMIN_PASS="${HERODEMO_ADMIN_PASS:-}" [ -n "$ADMIN_PASS" ] || { echo "✗ HERODEMO_ADMIN_PASS not set"; exit 1; } # 1. Unauthenticated: must be 401 echo "→ unauthenticated GET /" code=$(curl -s -o /dev/null -w "%{http_code}" "$BASE_URL/") [ "$code" = "401" ] || { echo " ✗ expected 401, got $code"; exit 1; } echo " ✓ 401" # 2. Authenticated: must be 200 or 302 echo "→ authenticated GET /" code=$(curl -s -o /dev/null -w "%{http_code}" -u "$ADMIN_USER:$ADMIN_PASS" "$BASE_URL/") [[ "$code" =~ ^(200|302)$ ]] || { echo " ✗ expected 200/302, got $code"; exit 1; } echo " ✓ $code" # 3. WASM bundle reachable post-auth (catches blank-desktop-after-login) echo "→ WASM asset reachable" code=$(curl -s -o /dev/null -w "%{http_code}" -u "$ADMIN_USER:$ADMIN_PASS" "$BASE_URL/hero_os/") [[ "$code" =~ ^(200|301|302)$ ]] || { echo " ✗ hero_os UI not reachable: $code"; exit 1; } echo " ✓ hero_os UI reachable" # 4. X-Hero-Context propagation: pick a per-context endpoint echo "→ X-Hero-Context propagation" for ctx in ${HERO_CONTEXTS:-incubaid geomind}; do resp=$(curl -s -u "$ADMIN_USER:$ADMIN_PASS" \ -H "X-Hero-Context: $ctx" -H "Content-Type: application/json" \ "$BASE_URL/hero_osis_business/rpc" \ -X POST \ -d '{"jsonrpc":"2.0","id":1,"method":"contact.list","params":{}}') count=$(echo "$resp" | jq -r '.result | length // 0' 2>/dev/null || echo "?") if [ "$count" = "?" ]; then echo " ✗ $ctx: contact.list returned non-JSON or error" echo " $resp" | head -c 200 exit 1 fi echo " ✓ $ctx: $count contacts" done echo "✓ auth flow complete" ``` **Caveats:** - Assumes hero_proxy + hero_router URL paths. URL shape `/hero_osis_business/rpc` may need to be verified against current routing. - Doesn't currently distinguish "context X has 0 contacts" from "endpoint returned 0 due to context-fallback bug" — both look like `count=0`. A stricter test would seed a known contact in each context and verify it appears. ## 00. Master orchestrator (`00_orchestrator.sh`) ```bash #!/bin/bash # 00_orchestrator.sh — run all six verifies in order. # Exits non-zero on first failure unless --continue passed. set -uo pipefail CONTINUE=false [ "${1:-}" = "--continue" ] && CONTINUE=true cd "$(dirname "$0")" declare -a CHECKS=( "10_wasm_currency.nu|WASM bundle currency" "20_books_refresh.sh|hero_books library refresh" "30_embedder_health.sh|embedder health" "40_ai_grounding_smoke.sh|AI Assistant grounding" "50_demo_content_seed.sh|demo content seed" "60_auth_flow.sh|auth + X-Hero-Context" ) FAILS=() for entry in "${CHECKS[@]}"; do IFS='|' read -r script desc <<< "$entry" echo "" echo "═══ $desc ═══" if [ -x "$script" ] || [[ "$script" == *.nu ]]; then if [[ "$script" == *.nu ]]; then nu "$script" || { FAILS+=("$desc"); $CONTINUE || exit 1; } else bash "$script" || { FAILS+=("$desc"); $CONTINUE || exit 1; } fi else echo "✗ $script not executable" FAILS+=("$desc") $CONTINUE || exit 1 fi done echo "" echo "═══════════════════════════════════════" if [ ${#FAILS[@]} -eq 0 ]; then echo "✓ all 6 checks passed — demo is verified" exit 0 else echo "✗ ${#FAILS[@]} failure(s):" for f in "${FAILS[@]}"; do echo " - $f"; done exit 1 fi ``` ## Hookup into the runbook In `hero_demo/docs/ops/DEPLOYMENT.md`, add a new section after §4 (Install hero services): ``` ## §4.5 Verify (mandatory before declaring deploy complete) After service_install_all completes, run: bash scripts/verify/00_orchestrator.sh This runs six checks in order: WASM currency, books refresh, embedder health, AI grounding, demo content seed, auth flow. All six must pass before the demo is considered deploy-ready. If anything fails, the script exits non-zero with a specific reason. To override an individual failure (acknowledged risk): bash scripts/verify/00_orchestrator.sh --continue ``` Also adds to `hero_demo/Makefile`: ```makefile .PHONY: verify verify: bash scripts/verify/00_orchestrator.sh .PHONY: verify-continue verify-continue: bash scripts/verify/00_orchestrator.sh --continue ``` ## Rollout This issue is a META — each script lands as a separate small PR. Order: 1. `50_demo_content_seed.sh` — easy, no service-side changes, immediate value tonight (seeding PDFs) 2. `30_embedder_health.sh` — read-only, low risk 3. `40_ai_grounding_smoke.sh` — read-only, immediate signal on docs_hero currency 4. `20_books_refresh.sh` — needs verifying the hero_books RPC method names; may surface a small server-side gap (e.g. if `books.scan` doesn't take `pull: true`, file a separate hero_books issue) 5. `60_auth_flow.sh` — depends on getting clean URL paths via hero_router/hero_proxy 6. `10_wasm_currency.nu` — depends on amending service_os to write `.build_sha`. Slight ordering: PR the .build_sha write FIRST, then the verify script. Once 1-6 land, master orchestrator (00) is one more PR. ## Tonight's manual run as the seed Tonight (2026-05-01) we'll execute the equivalent commands manually after install_all run 4 finishes, against herodemo, capturing every command + output. By tomorrow morning the team has: - A working demo for the agenda (Books, embedder, slides, office, biz, agent+CRM, whiteboard, collab) - Real raw data on what RPC method names actually exist (closing the "method names assumed" caveats above) - Validation that the script designs make sense The post-demo PRs land each script with the manual-run commands as the seed. ## Cross-refs - [hero_demo#52 — vision](https://forge.ourworld.tf/lhumina_code/hero_demo/issues/52) — sovereignty pitch needs reproducible deploys - [hero_demo#54 — CI artifacts](https://forge.ourworld.tf/lhumina_code/hero_demo/issues/54) — these scripts run on CI-built artifacts too once #54 lands - [hero_proc#86](https://forge.ourworld.tf/lhumina_code/hero_proc/issues/86) — reliability META (closed but informs the principles) - session 52's five deploy bugs — the kind of thing that would have been caught if #54 + this issue had existed before today.