Migrate hero_proc storage from SQLite to OSIS #28

New issue

Closed

opened 2026-03-25 13:53:32 +00:00 by timur · 2 comments

timur commented

2026-03-25 13:53:32 +00:00

Owner

Context

Per discussion in hero_archipelagos#40, hero_proc currently uses SQLite for all persistence (jobs, runs, actions, secrets, logs). The goal is to explore migrating to OSIS as the data layer, which would:

Unify storage across the hero ecosystem
Enable auto-generated Rhai engine modules per root object type
Allow direct save/fetch of objects from OSIS in Rhai scripts
Share workflow state between hero_proc and hero_osis

Scope

Evaluate which hero_proc data models map to OSIS root objects
Design OSchema definitions for Job, Run, Action, etc.
Implement OSIS adapter alongside existing SQLite (feature flag)
Migrate existing data paths incrementally

References

hero_archipelagos#40: Intelligent automation app
hero_osis: https://forge.ourworld.tf/lhumina_code/hero_osis

This is a backlog item — not blocking the initial workflow UI implementation.

## Context Per discussion in hero_archipelagos#40, hero_proc currently uses SQLite for all persistence (jobs, runs, actions, secrets, logs). The goal is to explore migrating to OSIS as the data layer, which would: - Unify storage across the hero ecosystem - Enable auto-generated Rhai engine modules per root object type - Allow direct save/fetch of objects from OSIS in Rhai scripts - Share workflow state between hero_proc and hero_osis ## Scope - Evaluate which hero_proc data models map to OSIS root objects - Design OSchema definitions for Job, Run, Action, etc. - Implement OSIS adapter alongside existing SQLite (feature flag) - Migrate existing data paths incrementally ## References - hero_archipelagos#40: Intelligent automation app - hero_osis: https://forge.ourworld.tf/lhumina_code/hero_osis This is a **backlog** item — not blocking the initial workflow UI implementation.

timur referenced this issue from lhumina_code/hero_archipelagos

2026-03-25 14:15:12 +00:00

Intelligent automation app #40

timur commented

2026-03-26 09:49:30 +00:00

Author

Owner

Initial Analysis: SQLite → OSIS Migration Path

Current SQLite Data Models

hero_proc stores these entity types in SQLite:

Jobs — execution records with phase lifecycle (pending→running→succeeded/failed)
Runs — orchestration units grouping multiple jobs with inter-run dependencies
Actions — action specifications with scripts, interpreters, dependencies, health checks
Secrets — context-scoped encrypted configuration values
Logs — structured log entries with dotted-notation sources

Mapping to OSIS Root Objects

SQLite Entity	OSIS Root Object	Notes
`ActionSpec`	`Action` [rootobject]	Natural fit — has name, script, dependencies
`Job`	`Job` [rootobject]	Execution record with phase tracking
`Run`	`Run` [rootobject]	Groups jobs, tracks orchestration status
`Secret`	`Secret` [rootobject]	Context-scoped key-value pairs
`LogEntry`	Possibly stay in SQLite	High-volume, time-series data — OSIS may not be ideal for append-heavy logs

Benefits of OSIS

Auto-generated Rhai engine modules per root object type
Shared storage namespace across hero services in the same context
Type-safe serialization (OTOML format)
Full-text search indexing via Tantivy
Filesystem-based storage (easy backup/inspection)

Proposed Approach

Define OSchema files for Action, Job, Run, Secret
Implement OSIS adapter behind a feature flag (storage-osis)
Keep SQLite as default for now (storage-sqlite, default)
Migrate one entity at a time, starting with Actions (simplest lifecycle)
Logs can remain in SQLite (high volume, append-only pattern)

Open Questions

Should OSIS storage use the same context namespace as hero_osis_server?
How to handle the high-frequency job status updates (OSIS is filesystem-based)?
Migration tool for existing SQLite data?

## Initial Analysis: SQLite → OSIS Migration Path ### Current SQLite Data Models hero_proc stores these entity types in SQLite: 1. **Jobs** — execution records with phase lifecycle (pending→running→succeeded/failed) 2. **Runs** — orchestration units grouping multiple jobs with inter-run dependencies 3. **Actions** — action specifications with scripts, interpreters, dependencies, health checks 4. **Secrets** — context-scoped encrypted configuration values 5. **Logs** — structured log entries with dotted-notation sources ### Mapping to OSIS Root Objects | SQLite Entity | OSIS Root Object | Notes | |--------------|-----------------|-------| | `ActionSpec` | `Action` [rootobject] | Natural fit — has name, script, dependencies | | `Job` | `Job` [rootobject] | Execution record with phase tracking | | `Run` | `Run` [rootobject] | Groups jobs, tracks orchestration status | | `Secret` | `Secret` [rootobject] | Context-scoped key-value pairs | | `LogEntry` | Possibly stay in SQLite | High-volume, time-series data — OSIS may not be ideal for append-heavy logs | ### Benefits of OSIS - Auto-generated Rhai engine modules per root object type - Shared storage namespace across hero services in the same context - Type-safe serialization (OTOML format) - Full-text search indexing via Tantivy - Filesystem-based storage (easy backup/inspection) ### Proposed Approach 1. Define OSchema files for Action, Job, Run, Secret 2. Implement OSIS adapter behind a feature flag (`storage-osis`) 3. Keep SQLite as default for now (`storage-sqlite`, default) 4. Migrate one entity at a time, starting with Actions (simplest lifecycle) 5. Logs can remain in SQLite (high volume, append-only pattern) ### Open Questions - Should OSIS storage use the same context namespace as hero_osis_server? - How to handle the high-frequency job status updates (OSIS is filesystem-based)? - Migration tool for existing SQLite data?