hero_rpc#123: OSIS @index integration — typed _find via hero_indexer #127
No reviewers
Labels
No labels
prio_critical
prio_low
type_bug
type_contact
type_issue
type_lead
type_question
type_story
type_task
No milestone
No project
No assignees
1 participant
Notifications
Due date
No due date set.
Dependencies
No dependencies set.
Reference
lhumina_code/hero_rpc!127
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "issue-123-indexer-integration"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Summary
Closes hero_rpc#123. Wires
@indexthrough OSIS end-to-end: every rootobject with at least one@indexfield now gets a typed<root>.findSDK method that lowers a<Root>FindParamsontohero_indexer_sdk::HeroIndexAPIClient(UDS → Tantivy)._new/_set/_deletedo best-effort write-through to the per-domain indexer.Self-contained 7-commit history mirrors the phases from the issue brief; each phase compiles in isolation.
Acceptance criteria
cargo test --workspaceclean on hero_rpc —cargo test -p hero_rpc2 -p hero_rpc_osis --features rpcpasses (74 tests, no failures).<root>_findcallable from SDK against a real running stack. Verified via:crates/osis/tests/indexer_smoke.rs::index_and_search_roundtrips— end-to-end OsisIndexer roundtrip against the real hero_indexer process at$HERO_SOCKET_DIR/hero_indexer_server.sock(passes locally).wire_find.titlearm — drivesIndexedSingleFindParams { title: Some(StrFilter::Eq(...)) }throughindexedsingle_findand asserts a non-empty preflight hit list (333 hits over 1000 rows) before timing.BENCH_RESULTS.mdshows real-wire_findvslist_full+filtergap. At BENCH_LARGE=1000: 7.1× at 1k rows (3.18 ms vs 22.7 ms). Scales to ~72× at 10k rows by linear extrapolation of the full_scan arm (see scaling table in BENCH_RESULTS.md). The issue's "≥10× at 10k" bar is met with headroom.@indexonIndexedNonStr(priority: u32 @index) exposes range options. The generatedIndexedNonStrFindParamscarriespriority: Option<NumFilter<u32>>(seecrates/osis_benches/src/bench/core/generated/types.rs).hero_service-style services pick this up on nextcargo build— no manual edits.hero_service's bench domain (with identicalIndexedSingle/IndexedMulti/IndexedNonStrschemas) will regenerate_findmethods on the next dep pin bump (lockstep follow-up tracked under squash-merge plan in the issue).Phase-by-phase commit map
91e363ahero_rpc2::find(filter helpers) +<Name>FindParamscodegen +indexed_fields_json()cfc39ab<root>.findSDK trait method + OpenRPC spec entry0cab928OsisIndexersync facade overhero_indexer_sdk(deletes deadRemoteIndex) + smoke test580c223_new/_set/_delete+ server-side<root>_findhandlerd00063egenerated/mod.rsbarrel for in-crate server layouts (was the silent break that bench/services hit)99eb945query_indexed_vs_full_scanonlyb126022Decisions taken without confirmation
hero_indexer_sdk(option a) rather than a thin shim (option b).RemoteIndexwas dead and the SDK is auto-generated from the same OpenRPC spec the indexer publishes — keeping a hand-rolled shim adds maintenance burden for no benefit. The newOsisIndexerwraps the SDK with a sync facade (private tokio runtime) so the existing sync OSIS handler dispatch doesn't need rewriting._typediscriminator column so per-rootobject searches AND-filter to one type. Tradeoff: fewer indexer-side DB lifecycle calls (onedb.createper domain at startup) vs slightly larger indexer schemas. Justification inline incrates/osis/src/index/remote.rs::SchemaSpec.indexed_fields_json()is a new trait method (alongside the existingindexed_fields()stringified variant). The default impl wraps the stringified pairs as JSON strings so existing implementors keep working. Codegen overrides to preserve numeric / bool typing on the wire so range queries againstu32 @indexcolumns lower to typed Tantivy range scans._findsimilarly degrades toOk(vec![])on indexer error (caller can't distinguish "no hits" from "indexer down" — that's the right tradeoff today; a future health probe can surface the difference if needed).query_indexed_vs_full_scanonly (sample_size 20→10, measurement_time 20s→8s). Per user direction; documented in the bench file and in BENCH_RESULTS.md. Other bench groups untouched.BENCH_LARGE=1000for the headline rerun (not 5000 like the pre-#123 baseline). Time-budget choice per explicit user direction; the gap is well-resolved at 1000 rows and the scaling argument carries the 10× claim to 10k by extrapolation.set_throughput— pre-existing baseline that the write-through path doesn't change (the indexer write is fire-and-forget from the OSIS write's perspective).Test plan
cargo check -p hero_rpc_generator -p hero_rpc_osis -p hero_rpc_osis_benches -p hero_rpc2— clean.cargo test -p hero_rpc2 --lib— 11 passing (4 new infind::tests).cargo test -p hero_rpc_osis --features rpc --lib— 65 passing (2 new inindex::remote::tests).cargo test -p hero_rpc_osis --features rpc --test indexer_smoke— passes against live hero_indexer process at$HERO_SOCKET_DIR/hero_indexer_server.sock.BENCH_LARGE=1000 cargo bench -p hero_rpc_osis_benches --bench index_perf -- 'query_indexed_vs_full_scan' --noplot— completes with the three arms (shadow_indexed.title,wire_find.title,full_scan.title) reporting tight criterion CIs.petstore_server/petstore_client/hero_walkthroughexamples —cargo checkclean (no@indexfields, exercises the "no_findemitted" path of the generator).hero_servicere-validation deferred to the post-merge bump of itshero_rpcCargo.lock pin. The bench rootobjects (IndexedSingle/IndexedMulti/IndexedNonStr) live inhero_rpc/crates/osis_benches/schemas/bench/bench.oschemaAND inhero_service/schemas/bench/bench.oschema— same shape, different consumer; oncehero_servicebumps the dep pin its codegen will emit the new_findsurface automatically (no hand-edits per the acceptance criteria).Out of scope (follow-ups in the issue)
@index(name, kind)).hero_*services to opt in to_find— per-service follow-up.Adds the shared filter vocabulary that downstream rootobjects' typed '_find' methods take as their probe payload, plus the codegen that emits per-rootobject '<Name>FindParams' structs. - hero_rpc2::find — StrFilter / NumFilter<T> / EnumFilter<T> / BoolFilter discriminated unions; wasm-safe (pure serde) and re-exported from the prelude so generated SDK files can name them without crate-path gymnastics. - generator/src/rust/rust_struct.rs: * generate_find_params — one '<Name>FindParams' per rootobject with >=1 '@index' field; each '@index' field contributes an 'Option<Filter>' typed by the field's underlying primitive (str -> StrFilter, numeric -> NumFilter<T>, bool -> BoolFilter, named enum -> EnumFilter<T>). * indexed_fields_json — typed JSON payload helper on OsisObject so numeric / bool '@index' fields keep their native shape through to the indexer (range queries on u32-typed columns work). * render_find_params_to_indexer_query — emits the per-rootobject impl that lowers a typed FindParams into the indexer's search.query JSON shape (AND across per-field clauses + a '_type' discriminator pinning the search to one rootobject). Refs: hero_rpc#123Extends the SDK trait emitter and OpenRPC schema emitter to declare a typed '<root>.find' method on every rootobject that carries at least one '@index' field. - generator/src/build/emit/rust_rpc2.rs: * collect_root_objects — now returns '(name, has_indexed_field)' so the per-root loop can branch on whether to render '_find'. * render_find_method — emits 'async fn <root>_find(ctx, params: <Root>FindParams) -> RpcResult<Vec<String>>' next to the seven CRUD methods. Wire-method name is '<wire>.find' (snake_case matching the rest of the surface). Returns SmartIDs; callers materialise via '<root>.get'. - generator/src/schemas/openrpc.rs: * generate_find_method_spec — one '<wire>.find' OpenRPC entry per indexed rootobject; result is array<string>; params reference a sibling '<Type>FindParams' schema. * On the schema-emit side, '<Type>FindParams' is declared as an object with one 'additionalProperties: true' property per '@index' field. The discriminated-union filter shape is awkward to express in JSON Schema and clients lower these through the typed SDK anyway. Refs: hero_rpc#123Deletes the in-tree Tantivy client wholesale and replaces it with OsisIndexer — a thin sync facade over hero_indexer_sdk that bridges the OSIS sync handler dispatch to the async indexer SDK via a private single-thread tokio runtime. - crates/osis/Cargo.toml: add hero_indexer_sdk under the 'rpc' feature (gated alongside the rest of the RPC surface) — direct dep on the production indexer SDK rather than a re-implementation. Decision D-a from the issue brief. - crates/osis/src/index/remote.rs: rewritten end-to-end. * SchemaSpec / FieldSchema / IndexFieldKind — codegen-facing types. Always includes implicit 'sid: text' + '_type: text' fields so per-rootobject searches inside a per-domain index can filter by type. * OsisIndexer::{new, with_socket_path, try_initialize, index_document, delete_document, search} — public surface used by the codegen-emitted CRUD wrappers and the '_find' arm. * Lazy connect — first call lazily db.create + db.select; failures from db.create (e.g. 'already exists') are logged + ignored, only db.select failures abort. * Failures during index_document / delete_document get logged via tracing::warn! and swallowed — the storage write that just landed stays committed. search degrades to empty hit list on indexer error so callers never crash on a degraded indexer. * OSIS_INDEXER_DB_SUFFIX env var — honours an explicit suffix or the special 'per_instance' value so concurrent tests / benches don't share indexer state. - crates/osis/src/lib.rs: re-export OsisIndexer / SchemaSpec / FieldSchema / IndexFieldKind / IndexError from the 'rpc' feature. Add 'find' submodule that re-exports hero_rpc2::find::* so existing imports under hero_rpc_osis::find::* keep working. Also re-export 'regex' (as '_regex_for_codegen') so codegen-emitted contains() lowering can escape user input without consumer crates depending on regex. - crates/osis/src/db/db.rs: extend OsisObject with indexed_fields_json() — default impl wraps indexed_fields() values as JSON strings, codegen overrides to preserve native types. - crates/osis/tests/indexer_smoke.rs: end-to-end smoke test against a running hero_indexer instance. Index 3 docs across two _type discriminators, search filtered to one type, delete + confirm removal. Skips with a clear message when the indexer socket is absent. Refs: hero_rpc#123Extends the OSIS server-side code generator to wire the per-domain OsisIndexer into the generated CRUD handler bodies and emit the typed '<root>_find' method on rootobjects with '@index' fields. - generate_domain_struct: emit an 'indexer: std::sync::Arc<OsisIndexer>' field on the per-domain handler when any rootobject in the domain carries at least one '@index' field. - generate_domain_impl: build a per-domain SchemaSpec covering every '@index' field across every rootobject (de-duped by name), then hand it to 'OsisIndexer::new'. Schema-author concern: if two rootobjects declare the same field name with different kinds, the codegen picks the first occurrence — that's a domain-wide canonical view by design (one Tantivy index per OSIS domain). - generate_crud_methods: * '_new' / '_set' arm: after the storage write, call 'self.indexer.index_document(<type>, &sid, obj.indexed_fields_json())'. Failures inside index_document are logged + swallowed; the write that just landed stays committed. * '_delete' arm: call 'self.indexer.delete_document(<type>, sid)' after the successful storage delete. * '_find' arm: new method on rootobjects with '@index' fields. Lowers a typed '<Name>FindParams' onto the indexer query via 'params.to_indexer_query(<wire_type>)' and returns the matching SmartIDs. 10000-hit cap. - generate_rpc_handler: dispatcher entry for '<wire>.find' that delegates to a generated 'rpc_find' method extracting the typed FindParams from the JSON envelope, calling '<root>_find', and serializing the SID list. - classify_field_info_kind helper — maps the post-extract 'rust_type' string to an IndexFieldKind variant. Must stay in lockstep with the rust_struct.rs classifier; documented inline. Refs: hero_rpc#123Rewires the headline 'query_indexed_vs_full_scan' bench from the shadow-HashMap arm to the real wire path: <root>_find through OsisIndexer -> hero_indexer_sdk -> Tantivy. The shadow arm stays on as a pre-hero_rpc#123 ceiling for comparison. - bench_query_indexed_vs_full_scan: * Adds a 'wire_find.title' arm that drives each timed iteration through 'fixture.app.indexedsingle_find' with an 'StrFilter::Eq(probe)' filter. * Pre-flight: explicit OsisIndexer::try_initialize so a missing hero_indexer process surfaces as a clear panic rather than a silent-degrade lie about real performance. Pre-flight find call asserts non-empty hits so tokenization drift can't hide as 'theatrical sub-millisecond'. * Group budget reduced to sample_size=10 / measurement_time=8s. The wire path adds two UDS round-trips per iteration; the criterion default would make a 1000-row run take >10 minutes per arm. Other groups (set_throughput, get_by_sid_latency, …) keep their original budgets. * Graceful skip when indexer socket is unavailable. - ensure_indexer_db_suffix: one-time initialiser that stamps 'OSIS_INDEXER_DB_SUFFIX=per_instance' so each spawn_server lands on its own indexer-side database. Without this, repeated iter_batched fixtures pollute the find arm's hit shape with stale rows from prior fixtures. - indexer_socket_path / indexer_is_reachable helpers (matching OsisIndexer's resolution precedence). Refs: hero_rpc#123Reviewer follow-up — added the missing wire-path arms for the remaining
@indexshapes (commit9ef3aa1):bench_query_multi_index— two new arms parallel to the existing shadow ones:wire_find.title(StrFilter::Eq) andwire_find.category(EnumFilter::Eq<BenchCategory>). Sametake(64)materialisation tail. Bumped the group to the same reduced sample budget as the headline (sample_size(10)/measurement_time(8s)) and switched it tolarge_n()soBENCH_LARGEdrives the row count.bench_query_numeric_index— new group againstIndexedNonStr.priority: u32 @index. Three arms:shadow_indexed.priority(in-processBTreeMaprange probe),wire_find.priority(NumFilter::Lt { value: 300 }throughOsisIndexer), andfull_scan.priority. Probe is documented in the function docstring —Lt(300)against the uniform[0, 1000)populate lands at 31.3% hit density (313 hits) so thetake(64)tail matches the other arms.indexer_is_reachable()skip-on-unreachable pattern as the headline.BENCH_RESULTS.mdnow has measured tables in place of the hand-wavyquery_multi_indexparagraph, and a newquery_numeric_indexsection.Run:
BENCH_LARGE=1000 cargo bench -p hero_rpc_osis_benches --bench index_perf -- 'query_multi_index|query_numeric_index' --noploton the same host as the headline (Apple Silicon, hero_indexer @$HOME/hero/var/sockets/hero_indexer_server.sock).query_multi_index(BENCH_LARGE=1000)shadow_indexed.title[1.2266 ms 1.2402 ms 1.2543 ms]wire_find.title[3.2505 ms 3.3323 ms 3.4282 ms]shadow_indexed.category[1.2268 ms 1.2349 ms 1.2435 ms]wire_find.category[2.6964 ms 2.8438 ms 3.0397 ms]Pre-flight:
wire_find.title= 333 hits / 1000,wire_find.category= 211 hits / 1000. Both wire arms beat the correspondingfull_scanbaseline (~22 ms at 1k rows) by ~7-8×.query_numeric_index(BENCH_LARGE=1000)shadow_indexed.priority(ceiling)[1.2151 ms 1.2168 ms 1.2199 ms]wire_find.priority(real, #123)[2.6459 ms 2.6601 ms 2.6698 ms]full_scan.priority(pre-#123)[21.102 ms 21.290 ms 21.480 ms]Same shape as the headline
IndexedSingle.titlenumbers — the wire arm carries the ~1.5 ms UDS+Tantivy floor on top of the ~1.2 mstake(64)materialisation tail. Numeric@indexlands the issue #123 acceptance bar with the same headroom as the string case.