Generator + scaffolder: emit per-root-object Criterion benchmarks from OSchema #113

New issue

Closed

opened 2026-05-21 12:16:43 +00:00 by timur · 2 comments

timur commented 2026-05-21 12:16:43 +00:00

Owner

Copy link

Context

The scaffolder already emits a complete shape from an OSchema: typed SDK (#60), CRUD round-trip tests (tests_emit.rs), runnable examples (examples.rs), per-root-object _admin + _web UI (#98). The missing peer is benchmarks — there's no canonical way today to track per-operation latency for a Hero service, nor a regression signal when codegen / dispatch changes (rpc2_adapter, hero_rpc2).

This issue adds per-root-object Criterion benchmarks, emitted from the schema the same way the existing test/example/UI scaffolds are, with output that lives in a workspace-root benches/ directory.

What this does

1. New emitter crates/generator/src/generate/benches_emit.rs (mirrors the existing tests_emit.rs / examples.rs pattern).
2. Scaffolder wiring writes the parent benches/ directory at workspace root, scaffolded once + generated/ subfolder ignored per #96.
3. Per-root-object benches measure the typed SDK CRUD methods landed in #98 — new, get, set, list, list_full, delete, exists.
4. Optional: service-method benches for any non-CRUD methods declared in the .oschema.

Concrete deliverables

Workspace-root layout

<workspace>/
├── benches/                              # scaffolded once at workspace root
│   ├── Cargo.toml                        # bench crate, depends on sdk + criterion + tokio + hero_rpc2
│   ├── .gitignore                        # `generated/`
│   ├── runner.rs                         # scaffolded-once handwritten entry — spins up the service
│   │                                     #   in-process via hero_rpc2::ServerBuilder, then dispatches
│   │                                     #   to the generated bench groups
│   └── generated/                        # gitignored
│       ├── mod.rs                        # `pub mod <entity>_crud; ...` per root object
│       ├── <entity>_crud.rs              # criterion group: bench_<entity>_new, _get, _set,
│       │                                 #   _list, _list_full, _delete, _exists
│       └── <domain>_methods.rs           # optional — one bench per non-CRUD service method

The Cargo.toml declares each generated file as a [[bench]] entry with harness = false so Criterion drives the loop. Standard cargo bench runs everything; cargo bench -- <entity> filters per object.

Bench shape (per root object)

// benches/generated/recipe_crud.rs — generated
use criterion::{criterion_group, Criterion};
use hero_recipes_sdk::Client;

pub fn bench_recipe(c: &mut Criterion, client: &Client) {
    let mut g = c.benchmark_group("recipe");
    g.bench_function("new", |b| b.iter(|| client.recipe_new(/* fixture */)));
    g.bench_function("get", |b| b.iter(|| client.recipe_get(sid)));
    g.bench_function("set", |b| b.iter(|| client.recipe_set(/* fixture */)));
    g.bench_function("list", |b| b.iter(|| client.recipe_list()));
    g.bench_function("list_full", |b| b.iter(|| client.recipe_list_full()));
    g.bench_function("delete", |b| b.iter(|| client.recipe_delete(sid)));
    g.bench_function("exists", |b| b.iter(|| client.recipe_exists(sid)));
    g.finish();
}

runner.rs (scaffolded once) wires up the in-process service + tokio runtime and calls each generated bench_<entity> group. Contributor edits runner.rs to customize fixtures or add custom groups; regenerates generated/ freely.

Fixture generation

The emitter introspects each root object's fields and produces a default fixture (zero/empty values per type) so the benches compile out of the box. The contributor refines fixtures in runner.rs for realistic payloads.

What to do

1. Design first — post a comment with the proposed emitter layout, the per-bench-group shape, and how runner.rs spins up the service in-process via hero_rpc2::ServerBuilder. Wait for sign-off.
2. Implement crates/generator/src/generate/benches_emit.rs mirroring tests_emit.rs / examples.rs.
3. Wire into crates/generator/src/build/scaffold.rs and crates/generator/src/bin/scaffold.rs (default-on; --no-benches to skip).
4. Apply to example/recipe_server/ — cargo bench runs clean against the in-process service.
5. Update hero_service_scaffold.md skill in hero_skills to document the benches/ folder + --no-benches flag.

Acceptance

• recipe_server/benches/ exists, gets regenerated on cargo build, and cargo bench runs all CRUD benches against an in-process service with no manual setup.
• git status clean after cargo build (benches/generated/ ignored).
• --no-benches skips the entire benches/ scaffold.
• Skill doc reflects the new surface.
• Fresh lab service new <name> produces a complete bench scaffold alongside the existing test/example/UI scaffolds.

Out of scope

• Storing bench results / regression dashboards (criterion --baseline is enough out of the box).
• Custom Criterion configuration per bench (warmup time, sample size) — contributor edits runner.rs.
• Benchmarks for non-OSIS service methods that take complex hand-built fixtures — generate the wiring, contributor fills the fixture.
• Concurrent-load / stress benches — separate issue if needed.

Related

• Parent META: hero_skills#262
• Generated-vs-handwritten layout: #96
• UI scaffolds (same scaffolder hook + per-entity introspection pattern): #98
• CRUD methods on the typed SDK that these benchmark: #60 + #98 sub-deliverable (b)
• Existing emitter siblings to mirror: crates/generator/src/generate/tests_emit.rs, examples.rs

## Context The scaffolder already emits a complete shape from an OSchema: typed SDK ([#60](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/60)), CRUD round-trip tests (`tests_emit.rs`), runnable examples (`examples.rs`), per-root-object `_admin` + `_web` UI ([#98](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/98)). The missing peer is **benchmarks** — there's no canonical way today to track per-operation latency for a Hero service, nor a regression signal when codegen / dispatch changes (`rpc2_adapter`, `hero_rpc2`). This issue adds **per-root-object Criterion benchmarks**, emitted from the schema the same way the existing test/example/UI scaffolds are, with output that lives in a workspace-root `benches/` directory. ## What this does 1. **New emitter** `crates/generator/src/generate/benches_emit.rs` (mirrors the existing `tests_emit.rs` / `examples.rs` pattern). 2. **Scaffolder wiring** writes the parent `benches/` directory at workspace root, scaffolded once + `generated/` subfolder ignored per [#96](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/96). 3. **Per-root-object benches** measure the typed SDK CRUD methods landed in [#98](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/98) — `new`, `get`, `set`, `list`, `list_full`, `delete`, `exists`. 4. Optional: **service-method benches** for any non-CRUD methods declared in the `.oschema`. ## Concrete deliverables ### Workspace-root layout ``` <workspace>/ ├── benches/ # scaffolded once at workspace root │ ├── Cargo.toml # bench crate, depends on sdk + criterion + tokio + hero_rpc2 │ ├── .gitignore # `generated/` │ ├── runner.rs # scaffolded-once handwritten entry — spins up the service │ │ # in-process via hero_rpc2::ServerBuilder, then dispatches │ │ # to the generated bench groups │ └── generated/ # gitignored │ ├── mod.rs # `pub mod <entity>_crud; ...` per root object │ ├── <entity>_crud.rs # criterion group: bench_<entity>_new, _get, _set, │ │ # _list, _list_full, _delete, _exists │ └── <domain>_methods.rs # optional — one bench per non-CRUD service method ``` The `Cargo.toml` declares each generated file as a `[[bench]]` entry with `harness = false` so Criterion drives the loop. Standard `cargo bench` runs everything; `cargo bench -- <entity>` filters per object. ### Bench shape (per root object) ```rust // benches/generated/recipe_crud.rs — generated use criterion::{criterion_group, Criterion}; use hero_recipes_sdk::Client; pub fn bench_recipe(c: &mut Criterion, client: &Client) { let mut g = c.benchmark_group("recipe"); g.bench_function("new", |b| b.iter(|| client.recipe_new(/* fixture */))); g.bench_function("get", |b| b.iter(|| client.recipe_get(sid))); g.bench_function("set", |b| b.iter(|| client.recipe_set(/* fixture */))); g.bench_function("list", |b| b.iter(|| client.recipe_list())); g.bench_function("list_full", |b| b.iter(|| client.recipe_list_full())); g.bench_function("delete", |b| b.iter(|| client.recipe_delete(sid))); g.bench_function("exists", |b| b.iter(|| client.recipe_exists(sid))); g.finish(); } ``` `runner.rs` (scaffolded once) wires up the in-process service + tokio runtime and calls each generated `bench_<entity>` group. Contributor edits `runner.rs` to customize fixtures or add custom groups; regenerates `generated/` freely. ### Fixture generation The emitter introspects each root object's fields and produces a default fixture (zero/empty values per type) so the benches compile out of the box. The contributor refines fixtures in `runner.rs` for realistic payloads. ## What to do 1. **Design first** — post a comment with the proposed emitter layout, the per-bench-group shape, and how `runner.rs` spins up the service in-process via `hero_rpc2::ServerBuilder`. Wait for sign-off. 2. Implement `crates/generator/src/generate/benches_emit.rs` mirroring `tests_emit.rs` / `examples.rs`. 3. Wire into `crates/generator/src/build/scaffold.rs` and `crates/generator/src/bin/scaffold.rs` (default-on; `--no-benches` to skip). 4. Apply to `example/recipe_server/` — `cargo bench` runs clean against the in-process service. 5. Update `hero_service_scaffold.md` skill in `hero_skills` to document the `benches/` folder + `--no-benches` flag. ## Acceptance - `recipe_server/benches/` exists, gets regenerated on `cargo build`, and `cargo bench` runs all CRUD benches against an in-process service with no manual setup. - `git status` clean after `cargo build` (`benches/generated/` ignored). - `--no-benches` skips the entire `benches/` scaffold. - Skill doc reflects the new surface. - Fresh `lab service new <name>` produces a complete bench scaffold alongside the existing test/example/UI scaffolds. ## Out of scope - Storing bench results / regression dashboards (`criterion --baseline` is enough out of the box). - Custom Criterion configuration per bench (warmup time, sample size) — contributor edits `runner.rs`. - Benchmarks for non-OSIS service methods that take complex hand-built fixtures — generate the wiring, contributor fills the fixture. - Concurrent-load / stress benches — separate issue if needed. ## Related - Parent META: [hero_skills#262](https://forge.ourworld.tf/lhumina_code/hero_skills/issues/262) - Generated-vs-handwritten layout: [#96](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/96) - UI scaffolds (same scaffolder hook + per-entity introspection pattern): [#98](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/98) - CRUD methods on the typed SDK that these benchmark: [#60](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/60) + [#98](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/98) sub-deliverable (b) - Existing emitter siblings to mirror: `crates/generator/src/generate/tests_emit.rs`, `examples.rs`

timur referenced this issue 2026-05-21 12:56:44 +00:00

Generator + scaffolder: emit end-to-end tests crate from OSchema (workspace-root tests/) #115

timur commented 2026-05-21 12:57:03 +00:00

Author

Owner

Copy link

Design — per-root-object Criterion benches scaffold

Posting before coding per the workflow ask. Below is the proposed shape; flagging two judgement calls inline that need a thumbs-up before I implement.

1. Emitter layout

New emitter at crates/generator/src/generate/benches_emit.rs, sibling to tests_emit.rs and examples.rs. Same module structure (impl Generator { pub(in crate::generate) fn generate_benches_file(&self, result: &mut GenerationResult) -> Result<()> }), same call site from build/scaffold.rs. Like tests_emit.rs it walks Schema::definitions, filters for obj.is_root_object || has_sid, and emits one bench_<entity>(c: &mut Criterion, client: &Arc<Client>, rt: &tokio::runtime::Handle) function per root object.

The benches_emit.rs codegen is invoked from the codegen path (OschemaBuilder), not from the one-shot scaffolder — same as tests_emit.rs. It outputs to <workspace>/benches/generated/<entity>_crud.rs, with a <workspace>/benches/generated/mod.rs that re-exports each module so runner.rs can use generated::*;.

Workspace-level benches/ (Cargo.toml + .gitignore + runner.rs) is scaffolded-once from build/scaffold.rs::generate_benches_dir, parallel to generate_examples_crate. Codegen never overwrites these three.

2. Per-bench-group shape

Each generated bench_<entity> function takes the shared client + tokio Handle, so the runner only spins up the in-process server once. Criterion's block_on pattern for async via rt.block_on(...):

// benches/generated/recipe_crud.rs — generated
use std::sync::Arc;
use criterion::Criterion;
use hero_recipes_sdk::recipes::{Recipe, RecipesClient};

pub fn bench_recipe(c: &mut Criterion, client: &Arc<dyn jsonrpsee::core::client::ClientT>, rt: &tokio::runtime::Handle) {
    let mut g = c.benchmark_group("recipe");

    // Seed a sid for read-path benches. Done once per group via a tokio block_on.
    let seed_sid: String = rt.block_on(async {
        RecipesClient::recipe_set(&**client, None, Recipe::default()).await.expect("seed set")
    });

    g.bench_function("new", |b| b.iter(|| rt.block_on(RecipesClient::recipe_new(&**client, None))));
    g.bench_function("get", |b| b.iter(|| rt.block_on(RecipesClient::recipe_get(&**client, None, seed_sid.clone()))));
    g.bench_function("set", |b| b.iter(|| rt.block_on(RecipesClient::recipe_set(&**client, None, Recipe::default()))));
    g.bench_function("list", |b| b.iter(|| rt.block_on(RecipesClient::recipe_list(&**client, None))));
    g.bench_function("list_full", |b| b.iter(|| rt.block_on(RecipesClient::recipe_list_full(&**client, None))));
    g.bench_function("exists", |b| b.iter(|| rt.block_on(RecipesClient::recipe_exists(&**client, None, seed_sid.clone()))));

    // delete last so the seed sid stays valid for the read benches above.
    // We re-create + delete on each iter to keep the bench self-contained.
    g.bench_function("delete", |b| b.iter_batched(
        || rt.block_on(RecipesClient::recipe_set(&**client, None, Recipe::default())).unwrap(),
        |sid| rt.block_on(RecipesClient::recipe_delete(&**client, None, sid)),
        criterion::BatchSize::SmallInput,
    ));

    g.finish();
}

Notes:

• .bench_function everywhere except delete which uses .bench_with_input (via iter_batched) to set up a fresh sid per iteration without measuring the setup. Justified because delete mutates global state; the other six are idempotent against the seeded sid.
• All async via rt.block_on(...) — Criterion does not natively support async; this is the standard criterion::async_executor::TokioExecutor alternative (which requires criterion = { features = ["async_tokio"] } — happy to switch to that if preferred, slightly less verbose).
• Service methods (non-CRUD entries in OSchema) emit to <domain>_methods.rs as a sibling — same shape, one g.bench_function(method_name, ...) per declared method. Fixture defaults per param type (see §5).

3. runner.rs — in-process spin-up

Mirrors examples/recipe_server/crates/hero_recipes_examples/examples/01_walkthrough.rs step 6 verbatim. Scaffolded-once so contributors can customize fixtures / add custom groups:

//! Bench runner — scaffolded once. Re-running the generator never touches
//! this file. Edit freely.

use std::sync::Arc;
use criterion::{criterion_main, Criterion};
use hero_recipes_server::recipes::OsisRecipes;
use hero_rpc2::prelude::*;
use hero_rpc_osis::rpc::{OsisDomainInit, rpc2_adapter};

mod generated;

fn main() {
    let rt = tokio::runtime::Runtime::new().expect("runtime");
    let handle = rt.handle().clone();

    // 1. spin up the recipes domain in-process
    let data_dir = std::env::temp_dir().join(format!("hero-recipes-bench-{}", std::process::id()));
    let _ = std::fs::remove_dir_all(&data_dir);
    std::fs::create_dir_all(&data_dir).unwrap();
    let app: Arc<OsisRecipes> = <OsisRecipes as OsisDomainInit>::create(data_dir.to_str().unwrap(), 0).unwrap();
    let module = rpc2_adapter::module_for(app).unwrap();

    let sock = std::env::temp_dir().join(format!("hero-recipes-bench-{}.sock", std::process::id()));
    let _ = std::fs::remove_file(&sock);

    let (server, client) = rt.block_on(async {
        let server = ServerBuilder::new(module).serve_http(&sock).await.unwrap();
        let client = ClientBuilder::new().connect_http(&sock).await.unwrap();
        (server, Arc::new(client))
    });

    // 2. run all generated bench groups against the shared client
    let mut c = Criterion::default().configure_from_args();
    generated::recipe_crud::bench_recipe(&mut c, &client, &handle);
    generated::collection_crud::bench_collection(&mut c, &client, &handle);
    c.final_summary();

    // 3. tear down
    rt.block_on(async { server.shutdown().await });
    let _ = std::fs::remove_file(&sock);
    let _ = std::fs::remove_dir_all(&data_dir);
}

Per-entity bench_<entity> calls inside main are themselves codegen output (in generated/mod.rs as a pub fn run_all(c: &mut Criterion, client: &Arc<…>, rt: &Handle) { bench_recipe(…); bench_collection(…); }) so runner.rs stays one-line per regeneration:

    generated::run_all(&mut c, &client, &handle);

That way runner.rs remains truly handwritten — adding/removing root objects in the schema doesn't require editing it.

4. Cargo.toml + [[bench]] entries

The issue body says "declares each generated file as a [[bench]] entry". Reading that literally, each <entity>_crud.rs would be its own Cargo [[bench]] target — but each generated file is just a pub fn bench_<entity>(…) library module, not a runnable harness with criterion_main!(). To make them standalone runnable bins they'd each need their own server spin-up, which would slow cargo bench to a crawl and complicate fixture sharing.

Proposed: one [[bench]] entry pointing at runner.rs, which is the single criterion driver and composes all the generated groups. Filtering per entity stays available via cargo bench -- recipe (Criterion's regex filter on group names). This matches how tests.rs is structured (one file, many #[test] fns).

# benches/Cargo.toml — scaffolded once
[package]
name = "hero_recipes_benches"
version.workspace = true
edition.workspace = true
publish = false

[dependencies]
hero_recipes_sdk    = { path = "../sdk/rust" }
hero_recipes_server = { path = "../crates/hero_recipes_server" }
hero_rpc2           = { path = "../../../crates/hero_rpc2", features = ["client", "uds-http"] }  # scaffolded as git-dep in user repos
hero_rpc_osis       = { path = "../../../crates/osis" }
tokio               = { workspace = true }
jsonrpsee           = { version = "0.26", features = ["client-core", "async-client"] }

[dev-dependencies]
criterion = { version = "0.5", features = [] }

[[bench]]
name = "runner"
path = "runner.rs"
harness = false

Open question: OK with the single-entry shape? If you'd rather have one [[bench]] per entity (separate processes per entity bench run), I'll switch — it's a one-line emitter change but doubles the cold-start cost.

The benches crate becomes a workspace member (added to root Cargo.toml's [workspace] members). Gating: skipped entirely when --no-benches is passed (mirroring --no-web).

5. Fixture defaults per OSchema field type

Same table as the _admin/_web form-input defaults (hero_rpc#98, see ui_emit::FieldKind::input_html), specialized for Rust-side defaults rather than HTML inputs:

FieldKind	Rust fixture
Str	String::new()
SignedInt	0
UnsignedInt	0
Float	0.0
Bool	false
OTime	OTime::default()
Enum(_)	<Enum>::default() (codegen already emits #[derive(Default)] + #[default] on the first variant)
PrimitiveList	Vec::new()
Json	Default::default() (struct types all derive Default)

Since every generated [rootobject] already has #[derive(Default)] (verified on Recipe / Collection in the recipe_server), the emitter can collapse the per-field fixture to a single <Type>::default() — no need to enumerate fields. For service-method benches, the emitter inspects each param type and emits the per-field defaults inline.

6. Scaffolder wiring

Mirroring with_web / without_web / --no-web exactly (PR #103 / ac70c7e):

• WorkspaceScaffolder::generate_benches: bool field, default true
• with_benches() / without_benches() builders
• bin/scaffold.rs: --no-benches flag → scaffolder.without_benches()
• scaffold() calls generate_benches_dir(&workspace_dir, &mut result)? when the flag is on, after generate_examples_crate
• create_workspace_dirs adds the benches/ directory creation
• generate_workspace_cargo_toml adds "benches" to [workspace] members when the flag is on
• generate_benches_dir writes the three preserved files: Cargo.toml, .gitignore (generated/), runner.rs

Tests added to crates/generator/src/build/scaffold.rs::tests:

• test_scaffolder_benches_default_on_with_without_toggle (mirrors test_scaffolder_web_default_on_with_without_toggle)
• test_scaffold_emits_benches_crate (mirrors test_scaffold_emits_examples_crate)
• test_scaffold_no_benches_skips_dir

7. Known risk — OTime serde mismatch

01_walkthrough.rs calls out a pre-existing serde shape mismatch on OTime between SDK and OSIS server — recipe.set + recipe.get with a real Recipe round-trip cleanly when the schema doesn't use OTime (the recipe schema doesn't), but a future schema that does will fail the set / get bench paths. If this bites during recipe_server validation I'll either (a) skip set/get/delete/exists benches for objects whose schema contains OTime fields with a // FIXME: see hero_rpc#… comment, or (b) seed via the OSIS domain handler directly inside runner.rs (bypassing wire ser/deser for setup only — the bench still measures the wire path).

I'll start with (b) since it's strictly more robust and keeps the bench surface uniform. Acceptable?

Acceptance plan

Recipe_server validation steps:

1. cd examples/recipe_server && cargo bench — all 14 benches (7 ops × 2 root objects) run green.
2. cargo bench -- recipe/list — filter works.
3. git status clean (benches/generated/ ignored).
4. cargo run -p hero_rpc_generator -- --name foo --workspace-dir /tmp/foo --no-benches — no benches/ dir.
5. cargo run -p hero_rpc_generator -- --name foo --workspace-dir /tmp/foo — benches/ dir present, cargo build clean.

---

Sign-off needed on:

1. Single [[bench]] for runner.rs vs. one per entity (§4) — proposing single.
2. Seed-via-OSIS fallback for OTime-sensitive schemas (§7) — proposing enable from the start.
3. criterion = { features = ["async_tokio"] } vs. manual rt.block_on (§2) — proposing manual block_on (one fewer feature flag in a crate that already has plenty).

Will hold on coding until the thumbs-up. Worktree at /tmp/hero_rpc_113, branch issue-113-benches already created off origin/development (HEAD at 2aeee22).

## Design — per-root-object Criterion benches scaffold Posting before coding per the workflow ask. Below is the proposed shape; flagging two judgement calls inline that need a thumbs-up before I implement. ### 1. Emitter layout New emitter at `crates/generator/src/generate/benches_emit.rs`, sibling to `tests_emit.rs` and `examples.rs`. Same module structure (`impl Generator { pub(in crate::generate) fn generate_benches_file(&self, result: &mut GenerationResult) -> Result<()> }`), same call site from `build/scaffold.rs`. Like `tests_emit.rs` it walks `Schema::definitions`, filters for `obj.is_root_object || has_sid`, and emits one `bench_<entity>(c: &mut Criterion, client: &Arc<Client>, rt: &tokio::runtime::Handle)` function per root object. The `benches_emit.rs` codegen is invoked from the codegen path (`OschemaBuilder`), not from the one-shot scaffolder — same as `tests_emit.rs`. It outputs to `<workspace>/benches/generated/<entity>_crud.rs`, with a `<workspace>/benches/generated/mod.rs` that re-exports each module so `runner.rs` can `use generated::*;`. Workspace-level `benches/` (Cargo.toml + `.gitignore` + `runner.rs`) is scaffolded-once from `build/scaffold.rs::generate_benches_dir`, parallel to `generate_examples_crate`. Codegen never overwrites these three. ### 2. Per-bench-group shape Each generated `bench_<entity>` function takes the shared `client` + tokio `Handle`, so the runner only spins up the in-process server once. Criterion's `block_on` pattern for async via `rt.block_on(...)`: ```rust // benches/generated/recipe_crud.rs — generated use std::sync::Arc; use criterion::Criterion; use hero_recipes_sdk::recipes::{Recipe, RecipesClient}; pub fn bench_recipe(c: &mut Criterion, client: &Arc<dyn jsonrpsee::core::client::ClientT>, rt: &tokio::runtime::Handle) { let mut g = c.benchmark_group("recipe"); // Seed a sid for read-path benches. Done once per group via a tokio block_on. let seed_sid: String = rt.block_on(async { RecipesClient::recipe_set(&**client, None, Recipe::default()).await.expect("seed set") }); g.bench_function("new", |b| b.iter(|| rt.block_on(RecipesClient::recipe_new(&**client, None)))); g.bench_function("get", |b| b.iter(|| rt.block_on(RecipesClient::recipe_get(&**client, None, seed_sid.clone())))); g.bench_function("set", |b| b.iter(|| rt.block_on(RecipesClient::recipe_set(&**client, None, Recipe::default())))); g.bench_function("list", |b| b.iter(|| rt.block_on(RecipesClient::recipe_list(&**client, None)))); g.bench_function("list_full", |b| b.iter(|| rt.block_on(RecipesClient::recipe_list_full(&**client, None)))); g.bench_function("exists", |b| b.iter(|| rt.block_on(RecipesClient::recipe_exists(&**client, None, seed_sid.clone())))); // delete last so the seed sid stays valid for the read benches above. // We re-create + delete on each iter to keep the bench self-contained. g.bench_function("delete", |b| b.iter_batched( || rt.block_on(RecipesClient::recipe_set(&**client, None, Recipe::default())).unwrap(), |sid| rt.block_on(RecipesClient::recipe_delete(&**client, None, sid)), criterion::BatchSize::SmallInput, )); g.finish(); } ``` Notes: - `.bench_function` everywhere except `delete` which uses `.bench_with_input` (via `iter_batched`) to set up a fresh sid per iteration without measuring the setup. Justified because `delete` mutates global state; the other six are idempotent against the seeded sid. - All async via `rt.block_on(...)` — Criterion does not natively support async; this is the standard `criterion::async_executor::TokioExecutor` alternative (which requires `criterion = { features = ["async_tokio"] }` — happy to switch to that if preferred, slightly less verbose). - Service methods (non-CRUD entries in OSchema) emit to `<domain>_methods.rs` as a sibling — same shape, one `g.bench_function(method_name, ...)` per declared method. Fixture defaults per param type (see §5). ### 3. `runner.rs` — in-process spin-up Mirrors `examples/recipe_server/crates/hero_recipes_examples/examples/01_walkthrough.rs` step 6 verbatim. Scaffolded-once so contributors can customize fixtures / add custom groups: ```rust //! Bench runner — scaffolded once. Re-running the generator never touches //! this file. Edit freely. use std::sync::Arc; use criterion::{criterion_main, Criterion}; use hero_recipes_server::recipes::OsisRecipes; use hero_rpc2::prelude::*; use hero_rpc_osis::rpc::{OsisDomainInit, rpc2_adapter}; mod generated; fn main() { let rt = tokio::runtime::Runtime::new().expect("runtime"); let handle = rt.handle().clone(); // 1. spin up the recipes domain in-process let data_dir = std::env::temp_dir().join(format!("hero-recipes-bench-{}", std::process::id())); let _ = std::fs::remove_dir_all(&data_dir); std::fs::create_dir_all(&data_dir).unwrap(); let app: Arc<OsisRecipes> = <OsisRecipes as OsisDomainInit>::create(data_dir.to_str().unwrap(), 0).unwrap(); let module = rpc2_adapter::module_for(app).unwrap(); let sock = std::env::temp_dir().join(format!("hero-recipes-bench-{}.sock", std::process::id())); let _ = std::fs::remove_file(&sock); let (server, client) = rt.block_on(async { let server = ServerBuilder::new(module).serve_http(&sock).await.unwrap(); let client = ClientBuilder::new().connect_http(&sock).await.unwrap(); (server, Arc::new(client)) }); // 2. run all generated bench groups against the shared client let mut c = Criterion::default().configure_from_args(); generated::recipe_crud::bench_recipe(&mut c, &client, &handle); generated::collection_crud::bench_collection(&mut c, &client, &handle); c.final_summary(); // 3. tear down rt.block_on(async { server.shutdown().await }); let _ = std::fs::remove_file(&sock); let _ = std::fs::remove_dir_all(&data_dir); } ``` Per-entity `bench_<entity>` calls inside `main` are themselves codegen output (in `generated/mod.rs` as a `pub fn run_all(c: &mut Criterion, client: &Arc<…>, rt: &Handle) { bench_recipe(…); bench_collection(…); }`) so `runner.rs` stays one-line per regeneration: ```rust generated::run_all(&mut c, &client, &handle); ``` That way `runner.rs` remains truly handwritten — adding/removing root objects in the schema doesn't require editing it. ### 4. Cargo.toml + `[[bench]]` entries The issue body says "declares each generated file as a `[[bench]]` entry". Reading that literally, each `<entity>_crud.rs` would be its own Cargo `[[bench]]` target — but each generated file is just a `pub fn bench_<entity>(…)` library module, not a runnable harness with `criterion_main!()`. To make them standalone runnable bins they'd each need their own server spin-up, which would slow `cargo bench` to a crawl and complicate fixture sharing. **Proposed:** one `[[bench]]` entry pointing at `runner.rs`, which is the single criterion driver and composes all the generated groups. Filtering per entity stays available via `cargo bench -- recipe` (Criterion's regex filter on group names). This matches how `tests.rs` is structured (one file, many `#[test]` fns). ```toml # benches/Cargo.toml — scaffolded once [package] name = "hero_recipes_benches" version.workspace = true edition.workspace = true publish = false [dependencies] hero_recipes_sdk = { path = "../sdk/rust" } hero_recipes_server = { path = "../crates/hero_recipes_server" } hero_rpc2 = { path = "../../../crates/hero_rpc2", features = ["client", "uds-http"] } # scaffolded as git-dep in user repos hero_rpc_osis = { path = "../../../crates/osis" } tokio = { workspace = true } jsonrpsee = { version = "0.26", features = ["client-core", "async-client"] } [dev-dependencies] criterion = { version = "0.5", features = [] } [[bench]] name = "runner" path = "runner.rs" harness = false ``` **Open question:** OK with the single-entry shape? If you'd rather have one `[[bench]]` per entity (separate processes per entity bench run), I'll switch — it's a one-line emitter change but doubles the cold-start cost. The benches crate becomes a workspace member (added to root `Cargo.toml`'s `[workspace] members`). Gating: skipped entirely when `--no-benches` is passed (mirroring `--no-web`). ### 5. Fixture defaults per OSchema field type Same table as the `_admin`/`_web` form-input defaults (hero_rpc#98, see `ui_emit::FieldKind::input_html`), specialized for Rust-side defaults rather than HTML inputs: | `FieldKind` | Rust fixture | |-----------------|-------------------------| | `Str` | `String::new()` | | `SignedInt` | `0` | | `UnsignedInt` | `0` | | `Float` | `0.0` | | `Bool` | `false` | | `OTime` | `OTime::default()` | | `Enum(_)` | `<Enum>::default()` (codegen already emits `#[derive(Default)]` + `#[default]` on the first variant) | | `PrimitiveList` | `Vec::new()` | | `Json` | `Default::default()` (struct types all derive `Default`) | Since every generated `[rootobject]` already has `#[derive(Default)]` (verified on `Recipe` / `Collection` in the recipe_server), the emitter can collapse the per-field fixture to a single `<Type>::default()` — no need to enumerate fields. For service-method benches, the emitter inspects each param type and emits the per-field defaults inline. ### 6. Scaffolder wiring Mirroring `with_web` / `without_web` / `--no-web` exactly (PR #103 / ac70c7e): - `WorkspaceScaffolder::generate_benches: bool` field, default `true` - `with_benches()` / `without_benches()` builders - `bin/scaffold.rs`: `--no-benches` flag → `scaffolder.without_benches()` - `scaffold()` calls `generate_benches_dir(&workspace_dir, &mut result)?` when the flag is on, after `generate_examples_crate` - `create_workspace_dirs` adds the `benches/` directory creation - `generate_workspace_cargo_toml` adds `"benches"` to `[workspace] members` when the flag is on - `generate_benches_dir` writes the three preserved files: `Cargo.toml`, `.gitignore` (`generated/`), `runner.rs` Tests added to `crates/generator/src/build/scaffold.rs::tests`: - `test_scaffolder_benches_default_on_with_without_toggle` (mirrors `test_scaffolder_web_default_on_with_without_toggle`) - `test_scaffold_emits_benches_crate` (mirrors `test_scaffold_emits_examples_crate`) - `test_scaffold_no_benches_skips_dir` ### 7. Known risk — OTime serde mismatch `01_walkthrough.rs` calls out a pre-existing serde shape mismatch on `OTime` between SDK and OSIS server — `recipe.set` + `recipe.get` with a real `Recipe` round-trip cleanly when the schema doesn't use `OTime` (the recipe schema doesn't), but a future schema that does will fail the `set` / `get` bench paths. If this bites during recipe_server validation I'll either (a) skip `set`/`get`/`delete`/`exists` benches for objects whose schema contains `OTime` fields with a `// FIXME: see hero_rpc#…` comment, or (b) seed via the OSIS domain handler directly inside `runner.rs` (bypassing wire ser/deser for setup only — the bench still measures the wire path). I'll start with (b) since it's strictly more robust and keeps the bench surface uniform. Acceptable? ### Acceptance plan Recipe_server validation steps: 1. `cd examples/recipe_server && cargo bench` — all 14 benches (7 ops × 2 root objects) run green. 2. `cargo bench -- recipe/list` — filter works. 3. `git status` clean (`benches/generated/` ignored). 4. `cargo run -p hero_rpc_generator -- --name foo --workspace-dir /tmp/foo --no-benches` — no `benches/` dir. 5. `cargo run -p hero_rpc_generator -- --name foo --workspace-dir /tmp/foo` — `benches/` dir present, `cargo build` clean. --- **Sign-off needed on:** 1. Single `[[bench]]` for runner.rs vs. one per entity (§4) — proposing **single**. 2. Seed-via-OSIS fallback for OTime-sensitive schemas (§7) — proposing **enable from the start**. 3. `criterion = { features = ["async_tokio"] }` vs. manual `rt.block_on` (§2) — proposing **manual block_on** (one fewer feature flag in a crate that already has plenty). Will hold on coding until the thumbs-up. Worktree at `/tmp/hero_rpc_113`, branch `issue-113-benches` already created off `origin/development` (HEAD at 2aeee22).

timur referenced this issue 2026-05-21 14:02:00 +00:00

Generator: unify SDK ↔ server types and seeding (single source of truth) #117

timur referenced this issue 2026-05-21 14:04:36 +00:00

Generator + scaffolder: emit end-to-end tests crate from OSchema (workspace-root tests/) #115

timur referenced this issue 2026-05-21 15:11:51 +00:00

Generator: unify SDK ↔ server types and seeding (single source of truth) #117

timur referenced this issue 2026-05-21 15:12:30 +00:00

Generator + scaffolder: emit end-to-end tests crate from OSchema (workspace-root tests/) #115

timur referenced this issue 2026-05-21 15:24:52 +00:00

Generator: unify SDK ↔ server types and seeding (single source of truth) #117

timur referenced this issue from a commit 2026-05-21 17:19:54 +00:00

feat(scaffold): per-root-object workspace-root E2E tests crate (closes #115)

timur referenced this issue 2026-05-21 17:20:24 +00:00

feat(scaffold): per-root-object workspace-root E2E tests crate (closes #115) #119

timur commented 2026-05-22 10:09:14 +00:00

Author

Owner

Copy link

Closing as superseded.

Two issues have absorbed the original scope:

• #125 landed crates/osis_benches/ — the Criterion harness for OSIS storage / index / wire-path benches against the template's new bench domain. That's the perf-regression surface this was meant to provide.
• #124 establishes that benches run via the same lab service --start --ephemeral subprocess as cargo tests / nu scripts. The per-service per-rootobject SDK-trait micro-bench scaffold the original design described would diverge from that single-bootstrap principle.

If we later want per-rootobject SDK-trait CRUD micro-benches on top of the osis_benches harness, file as a new issue against the post-#124 shape. The original design comment is preserved here for context.

Closing as superseded. Two issues have absorbed the original scope: - **[#125](https://forge.ourworld.tf/lhumina_code/hero_rpc/pulls/125)** landed `crates/osis_benches/` — the Criterion harness for OSIS storage / index / wire-path benches against the template's new `bench` domain. That's the perf-regression surface this was meant to provide. - **[#124](https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/124)** establishes that benches run via the same `lab service --start --ephemeral` subprocess as cargo tests / nu scripts. The per-service per-rootobject SDK-trait micro-bench scaffold the original design described would diverge from that single-bootstrap principle. If we later want per-rootobject SDK-trait CRUD micro-benches *on top of* the `osis_benches` harness, file as a new issue against the post-#124 shape. The original design comment is preserved here for context.

timur closed this issue 2026-05-22 10:09:14 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

development

development_new

main

development_refactor_list_full

development_jan

latest

v0.1.0

Labels

Clear labels   

prio_critical

  

prio_low

  

type_bug

  

type_contact

  

type_issue

  

type_lead

  

type_question

  

type_story

  

type_task

No labels

prio_critical

prio_low

type_bug

type_contact

type_issue

type_lead

type_question

type_story

type_task

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_blueprint#113

No description provided.