[WIP] Phase A+B: oschema-driven SDK regeneration (DRAFT — gated on characterization tests) #30
No reviewers
Labels
No labels
prio_critical
prio_low
type_bug
type_contact
type_issue
type_lead
type_question
type_story
type_task
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
lhumina_code/hero_indexer!30
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "oschema-migration"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Phase A + B of hero_indexer#28 (sub-issue of hero_rpc#132).
Status: DRAFT — do not merge. Blocked on characterization tests landing green against this PR's head.
What this PR does
Migrates
hero_indexer_sdkoff the legacyhero_rpc_derive::openrpc_client!proc-macro pipeline onto oschema-driven codegen — the canonical path per hero_skills#262 closeout.Phase A (commit
4f6450e) — six methods-only oschemas declaring the existing 19-method API surface across six namespaces (server,db,schema,doc,index,search). Zero rootobjects.Phase B (commit
2179a51) — drop theopenrpc_client!macro;hero_indexer_sdk/build.rsnow runs the generator against the Phase A schemas. Per-domain hero_rpc2#[rpc(server, client)]trait files emit intosrc/generated/; OpenRPC specs (one per domain + aggregate) emit intodocs/.Generated SDK surface
538 LoC total, zero hand-written. Six per-domain openrpc.json files plus an aggregate
docs/openrpc.json.Hiccups for Phase C reviewers
hero_rpc_osisnow optionally depends onhero_indexer_sdkfrom git (via itsrpcfeature). The build-dep chainhero_indexer_sdk → hero_rpc_osis → hero_indexer_sdkcycles. Phase B works around this by depending directly onhero_rpc_generatorinstead ofhero_rpc_osisin the SDK's build script. Phase C's server adoption will face the same constraint..generate_models()is the right target for the SDK crate;.generate_rpc()and.generate_server()emit server handler scaffolds into the current crate. The generator also emits vestigial<domain>/mod.rsbarrels and duplicates the SDK output at./sdk/rust/(becausesdk_dirdefaults to./sdk). Both gitignored here; worth a follow-up to tighten generator defaults for SDK-only consumers.Validation
cargo build -p hero_indexer_sdkclean.docs/<domain>/openrpc.json+ aggregate atdocs/openrpc.json).BLOCKED
Do not merge until the characterization-tests issue (sibling to #28, filed separately) lands green against this PR's head (
2179a51). Those tests capture wire-path goldens for all 17 methods + persistence state againstorigin/development, then re-run against2179a51. Both must produce identical goldens to prove "before/after equivalence". Phase C (server adoption) and Phase D (cross-repo hero_rpc revalidation) follow in separate PRs once we have that proof.Links
openrpc_client!→ oschema migration)Step 1 of the migration to oschema-driven codegen (hero_rpc#132 sub-issue). Each domain corresponds to one of the existing `<namespace>.*` namespaces from `crates/hero_indexer_server/openrpc.json`: - server.* → schemas/server/server.oschema (ping, stats, exit) - db.* → schemas/db/db.oschema (list, create, delete, close, select, info) - schema.* → schemas/schema/schema.oschema (get) - doc.* → schemas/doc/doc.oschema (add, add_batch, delete) - index.* → schemas/index/index.oschema (commit, reload) - search.* → schemas/search/search.oschema (query, count) Each schema is methods-only — no rootobjects — exactly matching the existing wire surface. Service-block names (Server/Db/Schema/Doc/ Index/Search) lower-case to the JSON-RPC namespace prefix so existing clients see no method-name change. Design decisions: - `any` for db.create(schema), doc.add(document) and search.query(query): Tantivy's index schema, the document shape, and the in-house QueryDefinition are all too recursive to model in oschema today. Kept as `serde_json::Value` (= `any` in oschema) — same wire shape, same semantics. Documented as a known constraint; a follow-up generator extension would let us tighten these to typed shapes. - Typed result structs (PingResult, StatsResult, …) instead of opaque `any` returns: the legacy `openrpc.json` left every result as `{type: object}`, so consumers reached for `serde_json::Value::get()`. Typed shapes mean OSIS-side consumers (notably hero_rpc's OsisIndexer) can pattern-match instead. - `Doc.delete` returns `DocDeleteResult` (not `DeleteResult`) so there's no name collision with `Db.delete`'s `DeleteResult` when the SDK re-exports both through a single namespace. The matching server adoption, SDK regen, hero_indexer_admin update, and the deletion of `crates/hero_indexer_server/openrpc.json` land in subsequent commits.Phase A+B: oschema-driven SDK regeneration (DRAFT — gated on characterization tests)to [WIP] Phase A+B: oschema-driven SDK regeneration (DRAFT — gated on characterization tests)BLOCKER — Phase B does not compile on
2179a51I'm the characterization-tests agent (hero_indexer#29). Before writing a single test file I ran the standard "does Phase B even build cleanly?" gate and it failed. Phase C cannot start until this is fixed.
TL;DR
cargo build --workspaceonorigin/development→ clean.cargo build --workspaceon2179a51(PR #30's tip) → 13 compile errors inhero_indexer(CLI bin) andhero_indexer_examples(integration tests).hero_indexer_sdk) fromopenrpc_client!proc-macro to oschema-driven#[rpc(server, client)]traits, but did not migrate the in-repo SDK call-sites that depend on the old client surface (HeroIndexAPIClient,*Inputstructs). Phase C is scoped to the server crate; it will not touch these call-sites either.What's broken on
2179a51The pre-migration SDK exported a flat
HeroIndexAPIClientwith*Input/*Outputtypes andclient.db_list(DbListInput {})-style methods. The post-migration SDK on2179a51exports per-domain trait modules —hero_indexer_sdk::db::DbClient,Dbtrait, typedDatabaseList/CreateResult/InfoResultstructs, methods with(ctx, name, ...)argument shape. The two surfaces are not source-compatible.Two crates in the workspace still reference the old shape:
crates/hero_indexer/src/main.rs— the CLI binary. UsesHeroIndexAPIClient::connect_socket(...)and 9 distinct*Inputtypes.crates/hero_indexer_examples/tests/integration.rs— the existing#[ignore]'d integration tests. Same client + 5 distinct*Inputtypes.Exact build failure on
2179a51Same pattern (12 add'l errors) on
cargo build --workspace --testsfromhero_indexer_examples/tests/integration.rs.Representative call-site (
crates/hero_indexer/src/main.rs:307-363):Each of these needs to be rewritten against the new per-domain trait surface (
use hero_indexer_sdk::db::DbClient;+client.db_create(None /* ctx */, name, schema).await?) — but that's migration agent work, not characterization-test work.Why this blocks #29
The acceptance criteria in #29 require running
lab service hero_indexer --testgreen on bothorigin/developmentAND2179a51.lab --testinvokescargo test --workspaceas Layer 1. The workspace doesn't compile on2179a51, so no test — characterization or otherwise — can run there. The byte-for-byte golden equivalence gate cannot even be attempted until Phase B is finished.The #29 prompt also assumes the existing
hero_indexer_sdktrait surface (HeroIndexAPIClient) is callable as-is on both branches. That's not true: on2179a51,HeroIndexAPIClientdoesn't exist in any form. Any characterization test file written against dev's SDK will compile-fail on2179a51exactly the same way the CLI binary does today. So I can't write a singletests/tree that lands green on both branches.What Phase B needs to do before I can resume
Either:
HeroIndexAPIClientexported (e.g. as a thin compat wrapper over the new per-domain clients) so the CLI binary + examples + characterization tests can use one shape against both branches. Single-pipeline-friendly; preserves the locked decision from hero_skills#262.2179a51itself — portcrates/hero_indexer/src/main.rsandcrates/hero_indexer_examples/tests/integration.rsonto the new per-domain*Clienttrait surface so the workspace compiles. Then I can write characterization tests against the new shape and forward-port them to dev (writing a compat layer on dev that maps to the new shape) — but this defeats the "same test file on both branches" gate, so option 1 is cleaner.I'm standing down on #29. The characterization-tests agent does not fix Phase B. Reactivate the migration agent — once
cargo build --workspaceis green on the Phase B branch tip, I can proceed.Environmental notes (not blockers, just FYI)
labCLI (/Users/timurgordon/.local/bin/lab) exposes thelab service hero_indexer --testverb when invoked from inside the repo (the verb is not surfaced inlab service hero_indexer --help, which only lists--start|--stop|--status|--install|--info, but it works in practice).lab pathrequiresPATH_ROOTinhero_cfg.toml, which is not present on this machine. Not relevant to this blocker but worth flagging if other agents needlab pathmachinery.crates/hero_indexer_server/openrpc.json) is byte-identical to dev (sha b743f83…). So wire-level behavior is unchanged on2179a51— the regression is purely at the Rust-API/source-compat level. Whenever Phase B's call-sites get fixed, golden equivalence should hold.Files touched on this branch
None. I have not committed anything. The
/private/tmp/hero_indexer_29_baselineworktree was never created — there was no point scaffolding tests against a non-compiling Phase B.Standing down. Re-ping me on #29 when the workspace compiles on the PR's head.
View command line instructions
Checkout
From your project repository, check out a new branch and test the changes.Merge
Merge the changes and update on Forgejo.Warning: The "Autodetect manual merge" setting is not enabled for this repository, you will have to mark this pull request as manually merged afterwards.