lhumina_code/hero_rpc
1
0
Fork 0
Code Issues 18 Pull requests Projects Releases 2 Packages Wiki Activity Actions

Re-validate hero_service template + benchmark osis with real indexes #122

New issue
Closed
opened 2026-05-22 01:03:37 +00:00 by timur · 1 comment
timur commented 2026-05-22 01:03:37 +00:00
Owner
Copy link

Re-validate hero_service template + benchmark osis with real indexes

Background

lhumina_code/hero_service is the canonical clone-me template for new Hero services — the output of hero_rpc_generator --name hero_<name> already tuned to pass lab infocheck. Since the last regeneration (commit 447f44b — feat: migrate to per-domain generated/ codegen layout (#96)), hero_rpc shipped a large pile of generator/codegen changes that the template has not yet been re-verified against:

  • #117 — <Name> / <Name>Input split + new(input) -> sid / set(sid, input) -> () CRUD shape, unified SDK ↔ server types, typed-SDK seed::{blank,random,from_dir}, canonical hero_rpc_osis::rpc::bootstrap (crates/osis/src/seed/ deleted upstream)
  • #119 — workspace-root tests/ crate with per-rootobject E2E tests
  • #114 — single types.rs (WASM twin emitter gone)
  • #111 — walkthrough drops redundant sid + wires python SDK + gates types_wasm
  • #108 — [rootobject] marker required for codegen to treat a type as a root
  • #107, #105, #103, #102, #101, #100, #99 — examples restructure, working _admin + _web scaffolds, lowercase wire method names, per-domain generated/ layout, OpenRpcDocument::from_value

cargo check --workspace on hero_service against hero_rpc@development compiles today (one unused-import warning in crates/hero_service/src/catalog/generated/types.rs:10), but no end-to-end run, no SDK round-trip, no admin smoke test, no Rhai script run has been done since the regen. The current schema also has a single rootobject with a single @index field, so it doesn't exercise the index surface meaningfully.

There is currently no benchmark infrastructure anywhere in hero_rpc or hero_service for the osis backend. We don't know what indexed vs. non-indexed query latency looks like, how throughput scales with row count, or what cost a @index annotation actually buys.

Goal

When this issue closes, the following are simultaneously true:

  1. hero_service template is fully working against the latest hero_rpc codegen — re-generated cleanly, all binaries build, lab infocheck is clean, the server boots, the SDK round-trips through the new <Name>Input / new(input) / set(sid, input) shape, the admin panel renders rows, the Rhai bindings load, the examples run.
  2. Schemas exercise the index surface — the catalog domain (or a sibling demo domain) defines multiple rootobjects that each declare @index on different field shapes (single-field str, single-field non-str if supported, multiple-@index per rootobject, and one rootobject with no @index as a baseline).
  3. Benchmarks live in-repo and produce real numbers — a benches/ crate (criterion or divan) in hero_rpc (so every generated service inherits the pattern) measures osis backend ops against a generated service:
    • set (insert) throughput per second at N rows (N ∈ {1k, 10k, 100k})
    • get (by sid) latency at the same row counts
    • Query latency comparing indexed vs. non-indexed lookups — the key result this issue exists to produce
    • list_all cost at growing row counts
    • Index-build cost when populating from empty
  4. Real benchmark results are committed as a BENCH_RESULTS.md (or equivalent) alongside the harness, captured from a local run, so we have a baseline for regressions.
  5. hero_rpc tests pass — fix any drift exposed along the way; the workspace-root tests/ crate from #119 stays green.

Out of scope

  • WASM-target SDK perf (still blocked on hero_rpc2 tokio/mio thread, separate issue).
  • Comparing osis to external KV stores — this is about us vs. ourselves with and without indexes.
  • Cross-machine / distributed benchmarks.
  • Index types we don't yet support (@unique, composite, @sort) — if the work surfaces that we need these, file a follow-up; do not extend the OSchema annotation surface as part of this issue unless absolutely required to express a benchmark case.

Concrete checklist

Phase 1 — re-generate + verify hero_service

  • Regenerate the hero_service template through hero_rpc_generator against hero_rpc@development HEAD. Diff vs. current repo; commit the regen on a feature branch.
  • Fix any drift the regen surfaces (codegen bugs go in hero_rpc, template-side hand-edits go in hero_service).
  • cargo build --workspace clean, zero warnings (including the unused import: hero_rpc_osis::otoml::OtomlSerialize already visible today).
  • lab infocheck clean.
  • lab service hero_service --install --start brings up hero_service_server and hero_service_admin; --status shows both healthy; --stop tears them down cleanly.
  • SDK round-trip from hero_service_examples: ServiceCatalog::service_definition_new(ServiceDefinitionInput{...}) -> sid, then service_definition_set(sid, ServiceDefinitionInput{..}), then service_definition_get(sid) returning the updated row.
  • Admin panel renders the seeded rows.
  • Rhai binding crate loads + a one-line script lists rows.
  • Python / JS SDKs (emitted under sdk/) compile / parse and call list() against the running server.

Phase 2 — schemas that actually exercise indexes

  • Extend schemas/catalog/catalog.oschema (or add a sibling .oschema file under schemas/<domain>/) with at least four rootobjects:
    • IndexedNone — rootobject with no @index (baseline).
    • IndexedSingle — one str @index field.
    • IndexedMulti — two or more @index fields.
    • IndexedNonStr — @index on a non-str field if supported; otherwise document the limitation and skip.
  • Each rootobject deliberately picks field shapes (string lengths, optional fields, embedded enums) representative of real catalog records — no toy name: str placeholders.
  • Regenerate; confirm the generator emits the right indexed_fields() for each.

Phase 3 — benchmark harness

  • Add crates/osis_benches/ (or benches/ on the osis crate) using criterion (consistent with the Rust ecosystem; avoid handrolled timing). Feature-gate so workspace cargo build doesn't pull it in.
  • Spin up a real hero_recipes_server (or the hero_service catalog server) inside the harness via hero_rpc_osis::rpc::bootstrap::run_for_test — no in-process bypass.
  • Bench cases:
    • set_throughput / {1k, 10k, 100k} — insert N rows of each rootobject; record rows/sec.
    • get_by_sid_latency / {1k, 10k, 100k} — point lookup against a populated store.
    • query_indexed_vs_full_scan / {1k, 10k, 100k} — same logical query against IndexedSingle (uses the index) vs. against IndexedNone (full scan). This is the headline number.
    • query_multi_index / {10k} — exercise IndexedMulti.
    • index_build_from_empty / 100k — cold populate cost.
  • Output written into BENCH_RESULTS.md (table per case, criterion summary, machine spec line, hero_rpc commit sha). Committed.

Phase 4 — hero_rpc tests

  • Workspace-root tests/ crate (added in #119) stays green after the schema additions; extend it with per-new-rootobject E2E cases.
  • cargo test --workspace clean on both repos.
  • cargo check --target wasm32-unknown-unknown -p hero_recipes_sdk — known-broken (hero_rpc2 transitive tokio/mio), don't block on it, just note status unchanged.

Acceptance

  • Both PRs (hero_rpc + hero_service) reviewed and squash-merged into development in lockstep.
  • BENCH_RESULTS.md shows a measurable indexed-vs-full-scan gap at 10k+ rows (if it doesn't, that's itself a finding worth a follow-up issue).
  • Closeout comment posts both PR links, the headline indexed-vs-full-scan numbers, and any follow-ups the work surfaced.

Repos involved

  • lhumina_code/hero_rpc — generator changes (if regen surfaces bugs), benchmark harness lives here.
  • lhumina_code/hero_service — template regen + extended schemas.

Branch model per hero_branching skill — feature branches off development, squash-merge back.

# Re-validate hero_service template + benchmark osis with real indexes ## Background `lhumina_code/hero_service` is the canonical **clone-me template** for new Hero services — the output of `hero_rpc_generator --name hero_<name>` already tuned to pass `lab infocheck`. Since the last regeneration (commit `447f44b — feat: migrate to per-domain generated/ codegen layout (#96)`), hero_rpc shipped a large pile of generator/codegen changes that the template has **not yet been re-verified against**: - **#117** — `<Name>` / `<Name>Input` split + `new(input) -> sid` / `set(sid, input) -> ()` CRUD shape, unified SDK ↔ server types, typed-SDK `seed::{blank,random,from_dir}`, canonical `hero_rpc_osis::rpc::bootstrap` (`crates/osis/src/seed/` deleted upstream) - **#119** — workspace-root `tests/` crate with per-rootobject E2E tests - **#114** — single `types.rs` (WASM twin emitter gone) - **#111** — walkthrough drops redundant sid + wires python SDK + gates `types_wasm` - **#108** — `[rootobject]` marker required for codegen to treat a type as a root - **#107**, **#105**, **#103**, **#102**, **#101**, **#100**, **#99** — examples restructure, working `_admin` + `_web` scaffolds, lowercase wire method names, per-domain `generated/` layout, `OpenRpcDocument::from_value` `cargo check --workspace` on hero_service against `hero_rpc@development` compiles today (one unused-import warning in `crates/hero_service/src/catalog/generated/types.rs:10`), but **no end-to-end run, no SDK round-trip, no admin smoke test, no Rhai script run** has been done since the regen. The current schema also has **a single rootobject with a single `@index` field**, so it doesn't exercise the index surface meaningfully. There is currently **no benchmark infrastructure anywhere in hero_rpc or hero_service** for the osis backend. We don't know what indexed vs. non-indexed query latency looks like, how throughput scales with row count, or what cost a `@index` annotation actually buys. ## Goal When this issue closes, the following are simultaneously true: 1. **hero_service template is fully working** against the latest hero_rpc codegen — re-generated cleanly, all binaries build, `lab infocheck` is clean, the server boots, the SDK round-trips through the new `<Name>Input` / `new(input)` / `set(sid, input)` shape, the admin panel renders rows, the Rhai bindings load, the examples run. 2. **Schemas exercise the index surface** — the catalog domain (or a sibling demo domain) defines **multiple rootobjects** that each declare `@index` on **different field shapes** (single-field `str`, single-field non-`str` if supported, multiple-`@index` per rootobject, and one rootobject with **no** `@index` as a baseline). 3. **Benchmarks live in-repo and produce real numbers** — a `benches/` crate (criterion or divan) in **hero_rpc** (so every generated service inherits the pattern) measures osis backend ops against a generated service: - `set` (insert) throughput per second at N rows (N ∈ {1k, 10k, 100k}) - `get` (by sid) latency at the same row counts - **Query latency comparing indexed vs. non-indexed lookups** — the key result this issue exists to produce - `list_all` cost at growing row counts - Index-build cost when populating from empty 4. **Real benchmark results are committed** as a `BENCH_RESULTS.md` (or equivalent) alongside the harness, captured from a local run, so we have a baseline for regressions. 5. **hero_rpc tests pass** — fix any drift exposed along the way; the workspace-root `tests/` crate from #119 stays green. ## Out of scope - WASM-target SDK perf (still blocked on hero_rpc2 tokio/mio thread, separate issue). - Comparing osis to external KV stores — this is about **us vs. ourselves with and without indexes**. - Cross-machine / distributed benchmarks. - Index types we don't yet support (`@unique`, composite, `@sort`) — if the work surfaces that we need these, file a follow-up; **do not extend the OSchema annotation surface as part of this issue** unless absolutely required to express a benchmark case. ## Concrete checklist ### Phase 1 — re-generate + verify hero_service - [ ] Regenerate the hero_service template through `hero_rpc_generator` against `hero_rpc@development` HEAD. Diff vs. current repo; commit the regen on a feature branch. - [ ] Fix any drift the regen surfaces (codegen bugs go in hero_rpc, template-side hand-edits go in hero_service). - [ ] `cargo build --workspace` clean, zero warnings (including the `unused import: hero_rpc_osis::otoml::OtomlSerialize` already visible today). - [ ] `lab infocheck` clean. - [ ] `lab service hero_service --install --start` brings up `hero_service_server` and `hero_service_admin`; `--status` shows both healthy; `--stop` tears them down cleanly. - [ ] SDK round-trip from `hero_service_examples`: `ServiceCatalog::service_definition_new(ServiceDefinitionInput{...}) -> sid`, then `service_definition_set(sid, ServiceDefinitionInput{..})`, then `service_definition_get(sid)` returning the updated row. - [ ] Admin panel renders the seeded rows. - [ ] Rhai binding crate loads + a one-line script lists rows. - [ ] Python / JS SDKs (emitted under `sdk/`) compile / parse and call `list()` against the running server. ### Phase 2 — schemas that actually exercise indexes - [ ] Extend `schemas/catalog/catalog.oschema` (or add a sibling `.oschema` file under `schemas/<domain>/`) with at least four rootobjects: - `IndexedNone` — rootobject with **no** `@index` (baseline). - `IndexedSingle` — one `str @index` field. - `IndexedMulti` — two or more `@index` fields. - `IndexedNonStr` — `@index` on a non-`str` field if supported; otherwise document the limitation and skip. - [ ] Each rootobject deliberately picks field shapes (string lengths, optional fields, embedded enums) representative of real catalog records — no toy `name: str` placeholders. - [ ] Regenerate; confirm the generator emits the right `indexed_fields()` for each. ### Phase 3 — benchmark harness - [ ] Add `crates/osis_benches/` (or `benches/` on the `osis` crate) using **criterion** (consistent with the Rust ecosystem; avoid handrolled timing). Feature-gate so workspace `cargo build` doesn't pull it in. - [ ] Spin up a real `hero_recipes_server` (or the hero_service catalog server) inside the harness via `hero_rpc_osis::rpc::bootstrap::run_for_test` — no in-process bypass. - [ ] Bench cases: - `set_throughput / {1k, 10k, 100k}` — insert N rows of each rootobject; record rows/sec. - `get_by_sid_latency / {1k, 10k, 100k}` — point lookup against a populated store. - `query_indexed_vs_full_scan / {1k, 10k, 100k}` — same logical query against `IndexedSingle` (uses the index) vs. against `IndexedNone` (full scan). **This is the headline number.** - `query_multi_index / {10k}` — exercise `IndexedMulti`. - `index_build_from_empty / 100k` — cold populate cost. - [ ] Output written into `BENCH_RESULTS.md` (table per case, criterion summary, machine spec line, hero_rpc commit sha). Committed. ### Phase 4 — hero_rpc tests - [ ] Workspace-root `tests/` crate (added in #119) stays green after the schema additions; extend it with per-new-rootobject E2E cases. - [ ] `cargo test --workspace` clean on both repos. - [ ] `cargo check --target wasm32-unknown-unknown -p hero_recipes_sdk` — known-broken (hero_rpc2 transitive tokio/mio), don't block on it, just note status unchanged. ## Acceptance - [ ] Both PRs (hero_rpc + hero_service) reviewed and squash-merged into `development` in lockstep. - [ ] `BENCH_RESULTS.md` shows a measurable indexed-vs-full-scan gap at 10k+ rows (if it doesn't, that's itself a finding worth a follow-up issue). - [ ] Closeout comment posts both PR links, the headline indexed-vs-full-scan numbers, and any follow-ups the work surfaced. ## Repos involved - `lhumina_code/hero_rpc` — generator changes (if regen surfaces bugs), benchmark harness lives here. - `lhumina_code/hero_service` — template regen + extended schemas. Branch model per `hero_branching` skill — feature branches off `development`, squash-merge back.
timur closed this issue 2026-05-22 07:42:10 +00:00
timur commented 2026-05-22 07:43:43 +00:00
Author
Owner
Copy link

Closed by:

  • lhumina_code/hero_rpc#125 — b4b76231 codegen fixes + crates/osis_benches/ criterion harness + impl ClientT for hero_rpc2::Client.
  • lhumina_code/hero_service#7 — a259899 template regen onto the post-#114/#117/#119 layout + new bench domain with the four Indexed* rootobjects.

Headline numbers (BENCH_RESULTS.md)

query_indexed_vs_full_scan at 5 000 rows, identical IndexedSingle dataset on both arms:

Arm Mean Read
shadow_indexed.title (HashMap built from indexed_fields()) 1.349 ms what @index would buy once OSIS consults it
full_scan.title (list_full() + filter()) 121.98 ms the only query shape the wire path actually supports today

~90× gap — entirely on the table because OSIS storage never calls indexed_fields() and the SDK trait emits no _find_* method. That's the headline finding from this issue's bench phase.

Other numbers (raw OSIS over hero_rpc_osis::rpc::bootstrap::run_for_test):

  • set_throughput: ~195–204 elem/s, IndexedSingle vs IndexedNone within noise — confirms @index is cost-free on the write path today (because it's unused).
  • get_by_sid_latency: ~21–23 µs, flat across row count.
  • query_multi_index (IndexedMulti, 2k rows): ~1.25–1.30 ms via shadow index.
  • index_build_from_empty (IndexedSingle, 2k rows): ~196 elem/s, dominated by the OSIS atomic write+rename.

Reproduce with:

cargo bench -p hero_rpc_osis_benches --bench index_perf -- --quick
# Larger headline row count:
BENCH_LARGE=100000 cargo bench -p hero_rpc_osis_benches --bench index_perf -- --quick

Follow-ups surfaced and filed

  • #123 — OSIS @index integration with hero_indexer: typed <RootObject>FindParams (range options on numeric @index fields), generated _find SDK + server method, OSIS-side write-through to hero_indexer_sdk::HeroIndexAPIClient. The 90× gap above is the target it claims.
  • #124 — Service lifecycle alignment: extract a MultiDomainBuilder so the in-process tests/src/lib.rs stops mirroring main.rs, and emit the Layer-2/3 nushell tests/smoke.nu / tests/api_integration.nu scripts the tests_pyramid skill describes.

Not done in this issue (documented as out-of-scope follow-ups)

  • lab service hero_service --install --start end-to-end verification (works locally, gated on the alignment work in #124).
  • Python / JS SDK round-trip beyond the codegen emit (Rust + Rhai paths are exercised by the e2e tests).
  • wasm32 hero_rpc2 transitive tokio/mio gap (status unchanged from #119).
Closed by: - lhumina_code/hero_rpc#125 — `b4b76231` codegen fixes + `crates/osis_benches/` criterion harness + `impl ClientT for hero_rpc2::Client`. - lhumina_code/hero_service#7 — `a259899` template regen onto the post-#114/#117/#119 layout + new `bench` domain with the four `Indexed*` rootobjects. ## Headline numbers (`BENCH_RESULTS.md`) `query_indexed_vs_full_scan` at 5 000 rows, identical `IndexedSingle` dataset on both arms: | Arm | Mean | Read | |-----|-----:|------| | `shadow_indexed.title` (HashMap built from `indexed_fields()`) | **1.349 ms** | what `@index` *would* buy once OSIS consults it | | `full_scan.title` (`list_full() + filter()`) | **121.98 ms** | the only query shape the wire path actually supports today | **~90× gap** — entirely on the table because OSIS storage never calls `indexed_fields()` and the SDK trait emits no `_find_*` method. That's the headline finding from this issue's bench phase. Other numbers (raw OSIS over `hero_rpc_osis::rpc::bootstrap::run_for_test`): - `set_throughput`: ~195–204 elem/s, `IndexedSingle` vs `IndexedNone` within noise — confirms `@index` is cost-free on the write path today (because it's unused). - `get_by_sid_latency`: ~21–23 µs, flat across row count. - `query_multi_index` (`IndexedMulti`, 2k rows): ~1.25–1.30 ms via shadow index. - `index_build_from_empty` (`IndexedSingle`, 2k rows): ~196 elem/s, dominated by the OSIS atomic `write+rename`. Reproduce with: ```bash cargo bench -p hero_rpc_osis_benches --bench index_perf -- --quick # Larger headline row count: BENCH_LARGE=100000 cargo bench -p hero_rpc_osis_benches --bench index_perf -- --quick ``` ## Follow-ups surfaced and filed - **#123 — OSIS `@index` integration with `hero_indexer`**: typed `<RootObject>FindParams` (range options on numeric `@index` fields), generated `_find` SDK + server method, OSIS-side write-through to `hero_indexer_sdk::HeroIndexAPIClient`. The 90× gap above is the target it claims. - **#124 — Service lifecycle alignment**: extract a `MultiDomainBuilder` so the in-process `tests/src/lib.rs` stops mirroring `main.rs`, and emit the Layer-2/3 nushell `tests/smoke.nu` / `tests/api_integration.nu` scripts the `tests_pyramid` skill describes. ## Not done in this issue (documented as out-of-scope follow-ups) - `lab service hero_service --install --start` end-to-end verification (works locally, gated on the alignment work in #124). - Python / JS SDK round-trip beyond the codegen emit (Rust + Rhai paths are exercised by the e2e tests). - wasm32 hero_rpc2 transitive `tokio/mio` gap (status unchanged from #119).
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
development_new
main
development_refactor_list_full
development_jan
latest
v0.1.0
Labels
Clear labels   
prio_critical

   
prio_low

   
type_bug

   
type_contact

   
type_issue

   
type_lead

   
type_question

   
type_story

   
type_task
No labels
prio_critical
prio_low
type_bug
type_contact
type_issue
type_lead
type_question
type_story
type_task
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/hero_rpc#122
No description provided.