Implement hero_memory #1
Loading…
Add table
Add a link
Reference in a new issue
No description provided.
Delete branch "%!s()"
Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?
Master tracking issue for hero_memory — the memory infrastructure for the Hero stack: a single service that ingests documents and code, organises them into collections, extracts Q&A pairs and ontological structure, generates embeddings, and serves retrieval to agents.
Spec:
docs/prd/anddocs/adr/ondevelopment.Strategy
The PRD describes the product. This issue tracks the build path. Each phase below produces a working subsystem with tests; phases land via commits referenced from this issue.
Phase 1 — Dimensions, provenance, extractor registry
Ground the schemas before anything depends on them.
hero_memory_lib::dimensions— canonical Rust enum per ADR-0004 with description, scope-guard, and target-count for each entryhero_memory_lib::provenance—_src_*key constants and helpers per ADR-0003hero_memory_lib::extractors— versioned extractor registry per ADR-0008Phase 2 — Collection registry
Build the org unit and change-detection plumbing.
hero_memory_lib::collections— registry, file→hash map, scan, change detection, purge-by-source helperscollections.redb, per-collectionfiles.redbcollections.create | list | get | scan | process | delete_src_*keys into the existingindex.addpath so every embedding doc carries provenancePhase 3 — Document conversion
hero_memory_lib::convert— PDF, PPTX, DOCX, HTML, code, plain (per PRD §04)content_hashconvert@v1extractor registeredPhase 4 — Q&A extraction
hero_memory_lib::ai— hero_aibroker client wrapper per ADR-0006hero_memory_lib::qa— per-dimension Q&A extraction with JSON-schema-constrained outputQ: ... A: ...per ADR-0007; doc id{collection}::{path}::{anchor}::{dimension}::{idx}qa.extract | list | search | purgePhase 5 — Ontology extraction
hero_memory_lib::ontology— load ontology via hero_db SDK, build prompt, validate response, write nodes/edges with_src_*provenanceontology@v1extractor registeredontology.attach | extract | stats | purgePhase 6 — Memory search
memory.searchRPC — high-level Q&A retrieval with provenance per PRD §07Phase 7 — Service script + deployment
service_memory— three modes (all-in-one,--inference --root,--userspace)hero_procregistration per thehero_proc_service_selfstartpatternPhase 8 — UI
hero_memory_uiupdated to show collections, dimensions, extractor stats, and source-purge toolsCross-repo dependencies
These will become separate issues in their respective repos when the dependent phase is unblocked:
GRAPH.PURGE_BY_SOURCEhelper; document the_src_*reserved-key convention; report row counts per_src_extractorfromGRAPH.STATSchat.complete(messages, model_hint?, json_schema?, max_tokens?, timeout?)with provider-agnostic JSON-schema-constrained outputAcceptance criteria (from PRD §01)
memory.search(query, dimension?)and get answers with full provenancehero_dbreports row counts per_src_extractorand supports purging by sourcehero_procin all three deployment modesUpdates
This issue is the strategy log. Each commit / PR that lands a piece of the work above gets a comment summarising what changed and which acceptance bullets it advances.
Phase 1 landed — dimensions, provenance, extractor registry
Commit:
dbefcb9(and lockfile sync532ee95)crates/hero_memory_lib/src/:dimensions.rs— canonical 17-variantDimensionenum per ADR-0004: 8 document, 3 code, 6 agent_memory. Each variant exposesid(),description(),scope_guard(),target_count(),kind().Dimension::for_kind(kind)filters the catalogue. Snake-case ids round-trip through serde andFromStr.provenance.rs—Provenancestruct +_src_*key constants per ADR-0003. Builder API (new+with_chunk/with_topic/with_model).to_property_pairs()emits ordered (key, value) pairs ready for graph node/edge inserts;from_properties()reads them back, validating required keys.extractors.rs—ExtractorIdenum (Qa | Ontology | Embed | Convert, version per variant) per ADR-0008.Displayrenders asname@vN;FromStrparses; serde uses the string form. ConstantsQA_V1/ONTOLOGY_V1/EMBED_V1/CONVERT_V1andALL_V1slice.pub used fromhero_memory_lib::{Dimension, DimensionKind, ExtractorId, Provenance}.Tests: 17 unit tests, all passing — id roundtrips, kind partition (8+3+6=17), serde format, provenance property-pair roundtrip, missing/invalid key handling, prefix detection.
Build:
cargo check -p hero_memory_lib --testsclean.What this unblocks
Every later phase builds on these three modules. Specifically:
Provenanceintoindex.addso embedding docs already carry source metadata.convert@v1against the file registry.Dimension::for_kindto iterate per collection-kind and writes provenance withtopicset.GRAPH.NODE.ADD/GRAPH.EDGE.ADD.Next: Phase 2 — Collection registry
The
hero_memory_lib::collectionsmodule: redb-backed registry, file→hash map, scan, change detection,_src_*-aware purge helpers. Will surface ascollections.*RPC methods onhero_memory_server.Phase 2 landed — collection registry, scan, provenance threading
Commit:
db053c0Collections subsystem (
hero_memory_lib::collections)types.rs—Collection,CollectionKind(Document/Code/AgentLog),CollectionState(Ready/Scanning/Processing),FileRecord,FileKind,ScanReport,NewCollection,CollectionsError,validate_name.CollectionKind::dimension_kind()andCollection::applicable_dimensions()connect a collection to the dimension catalogue from Phase 1.store.rs—CollectionsStore(registry redb at<data>/collections.redb) andFilesStore(per-collection redb at<data>/collections/<name>/files.redb). Public ops:create | get | list | delete, state transitions (update_state,mark_scanned,mark_processed),list_pending_for(extractor)for "what files still need this extractor."scan.rs—scan(root, files)walks the filesystem, sha256-hashes every file, and reconciles theFilesStore: added / changed / unchanged / removed. A hash change clearsextractor_runsandconverted_hashso downstream rows are known stale and Phase 3+ extractors will re-run. Hidden files and standard build dirs (target,node_modules,.git,__pycache__, …) are skipped.Server wiring
AppStategainsArc<CollectionsStore>.run_serverinitialises it fromHERO_MEMORY_DATA(renamed fromEMBEDDER_DATA), default~/hero/var/memory/data.hero_memory_server:collections.create(name, root, kind, dimensions?, namespace?)→Collectioncollections.list()→{collections: [Collection]}collections.get(name)→{collection: Collection?}collections.scan(name)→{name, report: ScanReport}collections.delete(name)→{name, deleted: true}Provenance threading (per ADR-0003)
Docgains optionalprovenance: Provenance.index.addmerges_src_*keys intometadataviaProvenance::merge_into_metadatabefore storing — every embedding row now carries its origin.Provenance::to_json_object()returns a typed JSON map (preservesi64for_src_extracted_at) for embedding metadata, alongside the string-onlyto_property_pairs()used for graph node/edge inserts._src_*keys win on merge collision (per the ADR).Verification
cargo checkclean.What this unblocks
FileRecord.kindto pick a converter, write the cached markdown to<data>/collections/<name>/cache/<rel>.md, and stampextractor_runs["convert"] = 1+converted_hash.Collection::applicable_dimensions()per file, callaibroker.chat.complete, and embed each pair viaindex.addwithprovenanceset — the_src_*keys are already wired through.Provenance::to_property_pairs()directly when callingGRAPH.NODE.ADD/GRAPH.EDGE.ADD.Next: Phase 3 — Document conversion
hero_memory_lib::convert— PDF / PPTX / DOCX / HTML / code / text → cached normalised markdown, keyed bycontent_hash. Registers asconvert@v1in the extractor map. New RPC:collections.processwill start to mean something (initially: run conversion over dirty files).Phase 3 landed — document conversion (
convert@v1)Commit:
aa5643aConversion subsystem (
hero_memory_lib::convert)cache.rs—ConvertCacherooted at<collection_dir>/cache.markdown_path("a/b/c.md")→<cache>/a/b/c.md.md;images_dir("a/b/c.md")→<cache>/a/b/c.md.images. Suffixing rather than replacing the source extension keepsfoo.mdandfoo.rsdistinct in the cache. Parent-traversal in source paths is dropped defensively.converter.rs—convert_file(input, kind, md_path, images_dir)dispatches onFileKind:\r\n/\rnormalisation and a trailing newline.unpdf/pptx-to-md/anytomdwith image extraction to the per-sourceimages_dir.html-to-markdown-rs.ConvertError::UnsupportedFormat.converted_hashrecorded on theFileRecord), optional images dir.RPC handler
convert.run(collection, path?, force?)— ifpathis set, runs that one file; otherwise iterates the registry. Skips records that already haveconvert@v1stamped at the currentcontent_hashunlessforceis true. Per-file outcome (converted|skipped|failed) returned indetailswith reason /converted_hash/markdown_path. On success,FileRecord.converted_hashandextractor_runs["convert"] = 1are persisted.Workspace deps added
unpdf 0.2,pptx-to-md 0.4,anytomd 1.2,html-to-markdown-rs 2.Verification
cargo checkclean.What this unblocks
ConvertCache::read_markdown(rel)for each file, then callsaibroker.chat.completeper applicable dimension. Re-runningqa@v1on an unchanged file is free because conversion is already done and the markdown text is content-addressed.hero_db.Next: Phase 4 — Q&A extraction
hero_memory_lib::ai(hero_aibroker client wrapper) +hero_memory_lib::qa(per-dimension Q&A pair generation with JSON-schema-constrained output). New RPCs:qa.extract | list | search | purge. Pairs embed viaindex.addwithprovenanceset, so the_src_*keys flow through automatically.Phase 7 landed — three deploy modes via CLI flags
Commit:
1b1d0a8The
hero_memoryCLI is now the single entry point for all three deploy modes (per PRD §08). No Nu script — the binary itself owns service registration viahero_proc_sdk(the hero_proc_service_selfstart pattern).Surface
Implementation
DeployMode { AllInOne, InferenceOnly, Userspace }selected by--inference/--userspace(mutually exclusive); default isAllInOne.build_service_definition(mode, inference_url_override)composes thehero_procservice definition with only the actions that mode needs.--inference-urlorHERO_MEMORY_INFERENCE_URLfails fast (exit code 2).HERO_MEMORY_INFERENCE_URLenv is set to the override URL; for AllInOne it points at the locally registered daemon.Cleanup of stale references
Cargo.toml: dropped a bogus "Linux: service_memory install_ort --root" line (no such Nu script in this repo) → replaced with the actual download-and-extract instruction.Makefilecheck-depsmatches.crates/hero_memory_inference/src/main.rs: doc + bind-list comment no longer referenceservice_memory.nu.docs/prd/08-deployment-and-ops.md: the deployment section now describes the actualhero_memory --startCLI rather than a Nu script that does not exist.Verification
Full workspace
cargo checkclean. No remainingservice_memory/service_embeddertext in the repo.Cross-repo:
hero_aibrokerissue filedPhases 4 and 5 need JSON-schema-constrained chat completion in
hero_aibroker, which is missing today. Filed as hero_aibroker#59 — small change per the audit (addjson_schema: Option<Value>toChatRequest, pass through to OpenAI-compatible providers).Next
Phase 4 (Q&A) and Phase 5 (ontology) are blocked on aibroker#59. While that lands, Phase 8 (UI updates for collections + convert) can proceed.