lhumina_code/hero_proc

Fork 0

rework logging #89

New issue

Open

opened 2026-05-01 05:11:50 +00:00 by despiegk · 3 comments

despiegk commented

2026-05-01 05:11:50 +00:00

Owner

see /hero_log
use this log library
keep the server interface but at back we use this logging
no more sqlite

see /hero_log use this log library keep the server interface but at back we use this logging no more sqlite

despiegk referenced this issue

2026-05-01 05:12:13 +00:00

[bug][P1] Log-store runaway: 58 GB in 24h from one chatty service wedged hero_proc + filled /data on herodemo #87

despiegk commented

2026-05-01 05:20:59 +00:00

Author

Owner

Implementation Spec for Issue #89

Objective

Replace the SQLite-backed log subsystem (partitioned per-day shards under ~/hero/var/hero_proc/logs/<ctx>/<year>/<day>/logs.sqlite) with the file-based herolib_core::logger (TSV records under ~/hero/var/logs/<src>/...log). Keep every existing RPC method signature stable so SDK callers, the CLI, the Dioxus admin app, and downstream services keep working unchanged.

Requirements

The 14 RPC methods stay wire-compatible (same names, params, result shapes):
- Read: log.query, log.count, logs.get, logs.tail, logs.filter, logs.count, logs.sources
- Write: logs.insert, logs.insert_batch
- Maintenance: logs.delete_by_src, logs.delete_older_than
- Job-scoped: job.logs, job.logs_attempt, job.log_archive
Backend = herolib_core::logger::Logger (~/hero/var/logs, default LoggerConfig, DropDebugFirst overflow).
LogEntry/LogFilter types used by the server and SDK keep the same field names so generated SDK structs are unchanged.
All references to rusqlite, partition_*, PartitionedLogStore, and the flate2 import inside db/logs/ must go away. rusqlite stays in hero_proc_lib for the other tables.
log_batcher.rs becomes a thin Arc<Logger> wrapper exposing .send(LogEntry) for compatibility (writes via logger.log(entry.into())).
Dotted src routing: encode job_id as the third segment when present (e.g. core.deploy.42); when absent use <context>.<action> capped at 3 segments. Service-emitted src values from logs.insert are passed through after sanitisation.
tags carry the attempt:N, stream:stderr, and any user-supplied tags so job.log_archive can still partition by attempt.
7-day retention loop in main.rs calls logger.cleanup_before("", cutoff_epoch) instead of db.logging.delete_older_than.

Files to Modify/Create

New

crates/hero_proc_lib/src/db/logs/adapter.rs — translation between hero_log and the existing LogEntry/LogFilter shapes.
crates/hero_proc_lib/src/log_store.rs — replacement LoggingApi backend wrapping Arc<Logger>.

Modify

Cargo.toml (workspace) — add herolib_core workspace dep.
crates/hero_proc_lib/Cargo.toml — wire herolib_core, drop flate2.
crates/hero_proc_lib/src/db/logs/mod.rs — drop partition, pool, store modules and SQLite tests; re-export new adapter module.
crates/hero_proc_lib/src/db/logs/model.rs — delete open_log_db/init_schema; keep LogEntry, LogFilter, name_fix.
crates/hero_proc_lib/src/db/factory.rs — replace PartitionedLogStore with Logger-backed store. Add pub fn logger() accessor.
crates/hero_proc_server/src/log_batcher.rs — replace SQLite drain loop with Arc<Logger> wrapper.
crates/hero_proc_server/src/rpc/log.rs — RPC dispatch logic stays identical.
crates/hero_proc_server/src/rpc/job.rs — job.logs filtering switches to job:<id> tag.
crates/hero_proc_server/src/supervisor/executor.rs — rewrite log emission to use dotted src + tags.
crates/hero_proc_server/src/main.rs — open Arc<Logger> once; switch cleanup loop to logger.cleanup_before.
crates/hero_proc_server/openrpc.json — update LogFilter.src doc, document logid: 0.
crates/hero_proc_lib/src/db/logs/pool.rs, partition.rs, store.rs — DELETE.
Tests in hero_proc_integration_test and db/integration_tests.rs — drop SQLite-specific assertions.

Implementation Plan

Step 1: Add herolib_core dependency

Files: Cargo.toml (workspace), crates/hero_proc_lib/Cargo.toml

Add herolib_core = { git = "https://forge.ourworld.tf/lhumina_code/hero_lib.git", branch = "development", default-features = false }.
Wire into hero_proc_lib. Drop flate2.
cargo check -p hero_proc_lib.
Dependencies: none

Step 2: Build the hero_log adapter

Files: crates/hero_proc_lib/src/db/logs/adapter.rs (new)

to_hero_entry(LogEntry) -> herolib_core::logger::LogEntry: maps loglevel, builds dotted src <ctx>.<src>[.<job_id>], copies tags + adds error:1/attempt:N/stream:....
from_hero_entry(...): split dotted src back to context_name/job_id, read error from tags, logid = 0.
LogFilter -> LogQuery: prefix from src+context_name, between(epoch_from,epoch_to), with_tags, limit. loglevel_max and error_only post-filtered. offset applied client-side. job_id/job_ids -> tag job:<id>.
Dependencies: Step 1

Step 3: Rewrite the LoggingApi to use hero_log

Files: crates/hero_proc_lib/src/log_store.rs (new), crates/hero_proc_lib/src/db/factory.rs

LogStore { logger: Arc<Logger> } exposing same methods (insert/insert_batch/query/count/list_sources/delete_by_src/delete_older_than).
HeroProcDb::new builds Arc<Logger> once. Add pub fn logger(&self) -> Arc<Logger> accessor.
Public LoggingApi method signatures byte-identical.
Dependencies: Step 2

Step 4: Delete SQLite log code

Files: delete pool.rs, partition.rs, store.rs under db/logs/. Trim db/logs/mod.rs and db/logs/model.rs to keep only LogEntry, LogFilter, name_fix.
Dependencies: Step 3

Step 5: Replace log_batcher with hero_log shim

Files: crates/hero_proc_server/src/log_batcher.rs

LogBatcherSender wraps Arc<Logger>. send(LogEntry) -> adapter -> logger.log(...).
Drop SQLite drain loop, dropped counter, overflow tracing.
Dependencies: Step 3

Step 6: Wire executor and main.rs to dotted-src + tags

Files: crates/hero_proc_server/src/supervisor/executor.rs, crates/hero_proc_server/src/main.rs

Helpers job_log_src(ctx, action, job_id) and job_log_tags(job_id, attempt, stream).
Replace each emission site to populate dotted src + tags. error: bool stays.
Cleanup loop: db.logger().cleanup_before("", cutoff).
Dependencies: Step 5

Step 7: Adjust job-scoped log RPC to use tags

Files: crates/hero_proc_server/src/rpc/job.rs, crates/hero_proc_server/src/rpc/log.rs

job.logs: builds LogFilter { job_id: Some(id), .. }. Adapter translates to with_tags(["job:<id>"]).
job.log_archive: keys per-attempt off attempt:N tag.
build_filter accepts the same RPC params.
Dependencies: Step 6

Step 8: Update OpenRPC, SDK regen, and CLI

Files: crates/hero_proc_server/openrpc.json, crates/hero_proc_sdk/src/factory.rs, crates/hero_proc/src/cli/commands.rs

Doc-only updates: LogFilter.src is dotted-prefix; logid always 0.
Regenerate SDK if there is a generator step; wire format unchanged.
Dependencies: Step 7

Step 9: Tests

Files: crates/hero_proc_integration_test/src/tests/logs.rs, parallel_jobs_logging.rs, db/factory.rs tests

Drop logid > 0 checks. Convert glob * mid-string to dotted prefix. Add adapter unit tests.
Dependencies: Step 8

Step 10: Cleanup, docs, full build

Files: crates/hero_proc_server/instructions.md, openrpc.json doc strings, hero_proc_sdk/src/logger.rs doc tweaks

cargo build --workspace, cargo test --workspace pass.
Manual smoke: insert + filter + sources + retention.
Dependencies: Step 9

Acceptance Criteria

All 14 RPC method signatures unchanged.
No SQLite log tables, partition modules, or logs.sqlite files.
herolib_core::logger is the single backend; logs land under ~/hero/var/logs/.
cargo build --workspace passes; cargo test --workspace passes (excluding --ignored).
CLI hero_proc log query|filter|prune|export and hero_proc job logs produce same output shape.
Dioxus admin app's Logs tab works end-to-end.
logs.insert returns {"logid": 0}.
logs.sources returns non-empty after writes; logs.delete_older_than cleans stale sources.
job.log_archive partitions by attempt via attempt:N tags.

Notes — semantics that move

logid: SQLite autoincrement -> always 0 sentinel. Already 0 in batcher path; downstream callers must not key on it.
src glob: trailing * -> prefix match. Mid-string * not supported; adapter post-filters in Rust. CLI/UI only produce trailing-* today.
epoch_from/epoch_to: inclusive both ends — same as today.
loglevel_max: hero_log only has min_level; we post-filter <= max.
error_only: encoded as tag error:1 on writes; post-filter on read (with_tags requires ALL match).
offset: hero_log has no offset; applied client-side after search. Capped by hero_log's 2000-row search hard cap; UI tops out at 1000 so fine.
job_id: stays on LogEntry but stored as both dotted-src third segment AND job:<id> tag. Tag drives filtering.
Retention: cleanup_before is whole-file. Boundary-day partial deletes are coarser. Non-issue for daily 7-day loop.
delete_by_src: maps to cleanup_before(prefix, u32::MAX) — wipes the whole subtree.
logs.sources cache: 5s in hero_log + 15min RPC cache; 5s lag possible.
HeroLogger SDK shipper: stays on logs.insert over RPC. No client-side change.
Risks: (1) name_fix may underscore-replace dots; tests must assert dotted form is canonical. (2) Old SQLite files at ~/hero/var/hero_proc/logs/* orphaned — document manual cleanup. (3) No downstream services depend on partition-day directories.

## Implementation Spec for Issue #89 ### Objective Replace the SQLite-backed log subsystem (partitioned per-day shards under `~/hero/var/hero_proc/logs/<ctx>/<year>/<day>/logs.sqlite`) with the file-based `herolib_core::logger` (TSV records under `~/hero/var/logs/<src>/...log`). Keep every existing RPC method signature stable so SDK callers, the CLI, the Dioxus admin app, and downstream services keep working unchanged. ### Requirements - The 14 RPC methods stay wire-compatible (same names, params, result shapes): - Read: `log.query`, `log.count`, `logs.get`, `logs.tail`, `logs.filter`, `logs.count`, `logs.sources` - Write: `logs.insert`, `logs.insert_batch` - Maintenance: `logs.delete_by_src`, `logs.delete_older_than` - Job-scoped: `job.logs`, `job.logs_attempt`, `job.log_archive` - Backend = `herolib_core::logger::Logger` (`~/hero/var/logs`, default `LoggerConfig`, `DropDebugFirst` overflow). - `LogEntry`/`LogFilter` types used by the server and SDK keep the same field names so generated SDK structs are unchanged. - All references to `rusqlite`, `partition_*`, `PartitionedLogStore`, and the `flate2` import inside `db/logs/` must go away. `rusqlite` stays in `hero_proc_lib` for the other tables. - `log_batcher.rs` becomes a thin `Arc<Logger>` wrapper exposing `.send(LogEntry)` for compatibility (writes via `logger.log(entry.into())`). - Dotted `src` routing: encode `job_id` as the third segment when present (e.g. `core.deploy.42`); when absent use `<context>.<action>` capped at 3 segments. Service-emitted `src` values from `logs.insert` are passed through after sanitisation. - `tags` carry the `attempt:N`, `stream:stderr`, and any user-supplied tags so `job.log_archive` can still partition by attempt. - 7-day retention loop in `main.rs` calls `logger.cleanup_before("", cutoff_epoch)` instead of `db.logging.delete_older_than`. ### Files to Modify/Create #### New - `crates/hero_proc_lib/src/db/logs/adapter.rs` — translation between hero_log and the existing `LogEntry`/`LogFilter` shapes. - `crates/hero_proc_lib/src/log_store.rs` — replacement `LoggingApi` backend wrapping `Arc<Logger>`. #### Modify - `Cargo.toml` (workspace) — add `herolib_core` workspace dep. - `crates/hero_proc_lib/Cargo.toml` — wire `herolib_core`, drop `flate2`. - `crates/hero_proc_lib/src/db/logs/mod.rs` — drop `partition`, `pool`, `store` modules and SQLite tests; re-export new adapter module. - `crates/hero_proc_lib/src/db/logs/model.rs` — delete `open_log_db`/`init_schema`; keep `LogEntry`, `LogFilter`, `name_fix`. - `crates/hero_proc_lib/src/db/factory.rs` — replace `PartitionedLogStore` with `Logger`-backed store. Add `pub fn logger()` accessor. - `crates/hero_proc_server/src/log_batcher.rs` — replace SQLite drain loop with `Arc<Logger>` wrapper. - `crates/hero_proc_server/src/rpc/log.rs` — RPC dispatch logic stays identical. - `crates/hero_proc_server/src/rpc/job.rs` — `job.logs` filtering switches to `job:<id>` tag. - `crates/hero_proc_server/src/supervisor/executor.rs` — rewrite log emission to use dotted src + tags. - `crates/hero_proc_server/src/main.rs` — open `Arc<Logger>` once; switch cleanup loop to `logger.cleanup_before`. - `crates/hero_proc_server/openrpc.json` — update `LogFilter.src` doc, document `logid: 0`. - `crates/hero_proc_lib/src/db/logs/pool.rs`, `partition.rs`, `store.rs` — DELETE. - Tests in `hero_proc_integration_test` and `db/integration_tests.rs` — drop SQLite-specific assertions. ### Implementation Plan #### Step 1: Add herolib_core dependency Files: `Cargo.toml` (workspace), `crates/hero_proc_lib/Cargo.toml` - Add `herolib_core = { git = "https://forge.ourworld.tf/lhumina_code/hero_lib.git", branch = "development", default-features = false }`. - Wire into `hero_proc_lib`. Drop `flate2`. - `cargo check -p hero_proc_lib`. Dependencies: none #### Step 2: Build the hero_log adapter Files: `crates/hero_proc_lib/src/db/logs/adapter.rs` (new) - `to_hero_entry(LogEntry) -> herolib_core::logger::LogEntry`: maps loglevel, builds dotted src `<ctx>.<src>[.<job_id>]`, copies tags + adds `error:1`/`attempt:N`/`stream:...`. - `from_hero_entry(...)`: split dotted src back to `context_name`/`job_id`, read `error` from tags, `logid = 0`. - `LogFilter -> LogQuery`: prefix from `src`+`context_name`, between(epoch_from,epoch_to), with_tags, limit. `loglevel_max` and `error_only` post-filtered. `offset` applied client-side. `job_id`/`job_ids` -> tag `job:<id>`. Dependencies: Step 1 #### Step 3: Rewrite the LoggingApi to use hero_log Files: `crates/hero_proc_lib/src/log_store.rs` (new), `crates/hero_proc_lib/src/db/factory.rs` - `LogStore { logger: Arc<Logger> }` exposing same methods (insert/insert_batch/query/count/list_sources/delete_by_src/delete_older_than). - `HeroProcDb::new` builds `Arc<Logger>` once. Add `pub fn logger(&self) -> Arc<Logger>` accessor. - Public `LoggingApi` method signatures byte-identical. Dependencies: Step 2 #### Step 4: Delete SQLite log code Files: delete `pool.rs`, `partition.rs`, `store.rs` under `db/logs/`. Trim `db/logs/mod.rs` and `db/logs/model.rs` to keep only `LogEntry`, `LogFilter`, `name_fix`. Dependencies: Step 3 #### Step 5: Replace log_batcher with hero_log shim Files: `crates/hero_proc_server/src/log_batcher.rs` - `LogBatcherSender` wraps `Arc<Logger>`. `send(LogEntry)` -> adapter -> `logger.log(...)`. - Drop SQLite drain loop, dropped counter, overflow tracing. Dependencies: Step 3 #### Step 6: Wire executor and main.rs to dotted-src + tags Files: `crates/hero_proc_server/src/supervisor/executor.rs`, `crates/hero_proc_server/src/main.rs` - Helpers `job_log_src(ctx, action, job_id)` and `job_log_tags(job_id, attempt, stream)`. - Replace each emission site to populate dotted src + tags. `error: bool` stays. - Cleanup loop: `db.logger().cleanup_before("", cutoff)`. Dependencies: Step 5 #### Step 7: Adjust job-scoped log RPC to use tags Files: `crates/hero_proc_server/src/rpc/job.rs`, `crates/hero_proc_server/src/rpc/log.rs` - `job.logs`: builds `LogFilter { job_id: Some(id), .. }`. Adapter translates to `with_tags(["job:<id>"])`. - `job.log_archive`: keys per-attempt off `attempt:N` tag. - `build_filter` accepts the same RPC params. Dependencies: Step 6 #### Step 8: Update OpenRPC, SDK regen, and CLI Files: `crates/hero_proc_server/openrpc.json`, `crates/hero_proc_sdk/src/factory.rs`, `crates/hero_proc/src/cli/commands.rs` - Doc-only updates: `LogFilter.src` is dotted-prefix; `logid` always 0. - Regenerate SDK if there is a generator step; wire format unchanged. Dependencies: Step 7 #### Step 9: Tests Files: `crates/hero_proc_integration_test/src/tests/logs.rs`, `parallel_jobs_logging.rs`, `db/factory.rs` tests - Drop `logid > 0` checks. Convert glob `*` mid-string to dotted prefix. Add adapter unit tests. Dependencies: Step 8 #### Step 10: Cleanup, docs, full build Files: `crates/hero_proc_server/instructions.md`, openrpc.json doc strings, `hero_proc_sdk/src/logger.rs` doc tweaks - `cargo build --workspace`, `cargo test --workspace` pass. - Manual smoke: insert + filter + sources + retention. Dependencies: Step 9 ### Acceptance Criteria - [ ] All 14 RPC method signatures unchanged. - [ ] No SQLite log tables, partition modules, or `logs.sqlite` files. - [ ] `herolib_core::logger` is the single backend; logs land under `~/hero/var/logs/`. - [ ] `cargo build --workspace` passes; `cargo test --workspace` passes (excluding `--ignored`). - [ ] CLI `hero_proc log query|filter|prune|export` and `hero_proc job logs` produce same output shape. - [ ] Dioxus admin app's Logs tab works end-to-end. - [ ] `logs.insert` returns `{"logid": 0}`. - [ ] `logs.sources` returns non-empty after writes; `logs.delete_older_than` cleans stale sources. - [ ] `job.log_archive` partitions by attempt via `attempt:N` tags. ### Notes — semantics that move - **logid**: SQLite autoincrement -> always 0 sentinel. Already 0 in batcher path; downstream callers must not key on it. - **src glob**: trailing `*` -> prefix match. Mid-string `*` not supported; adapter post-filters in Rust. CLI/UI only produce trailing-`*` today. - **epoch_from/epoch_to**: inclusive both ends — same as today. - **loglevel_max**: hero_log only has `min_level`; we post-filter `<= max`. - **error_only**: encoded as tag `error:1` on writes; post-filter on read (with_tags requires ALL match). - **offset**: hero_log has no offset; applied client-side after search. Capped by hero_log's 2000-row search hard cap; UI tops out at 1000 so fine. - **job_id**: stays on LogEntry but stored as both dotted-src third segment AND `job:<id>` tag. Tag drives filtering. - **Retention**: `cleanup_before` is whole-file. Boundary-day partial deletes are coarser. Non-issue for daily 7-day loop. - **delete_by_src**: maps to `cleanup_before(prefix, u32::MAX)` — wipes the whole subtree. - **logs.sources cache**: 5s in hero_log + 15min RPC cache; 5s lag possible. - **HeroLogger SDK shipper**: stays on `logs.insert` over RPC. No client-side change. - **Risks**: (1) name_fix may underscore-replace dots; tests must assert dotted form is canonical. (2) Old SQLite files at `~/hero/var/hero_proc/logs/*` orphaned — document manual cleanup. (3) No downstream services depend on partition-day directories.

despiegk commented

2026-05-01 06:32:41 +00:00

Author

Owner

Test Results

Build: PASS
Total: 450
Passed: 399
Failed: 0
Ignored: 51

Crates with significant test counts

hero_proc_lib (lib): 140/140
hero_proc_server (lib): 70/70
hero_proc_sdk (lib): 20/20
hero_proc_integration_tests / cli_integration: 83/83
hero_proc_integration_tests / service_management: 16/18 (2 ignored)
hero_proc_integration_tests / jobs: 11/12 (1 ignored)
hero_proc_integration_tests / rhai_scripting: 10/10
hero_proc_integration_tests / shutdown: 8/17 (9 ignored)
hero_proc_integration_tests / server_lifecycle: 7/7
hero_proc_integration_tests / bulk_operations: 7/7
hero_proc_integration_tests / hero_script: 5/5
hero_proc_integration_tests / action_dependencies: 4/4
hero_proc_integration_tests / dependencies: 4/5 (1 ignored)
hero_proc_integration_tests / service_restart_with_cleanup: 4/4
hero_proc_integration_tests (lib): 4/4
hero_proc_integration_tests / pty: 3/3
hero_proc_integration_tests / health_probe: 2/2
hero_proc_integration_tests / failed_action_lifecycle: 1/1
hero_proc_integration_tests / dev_only: 0/7 (7 ignored)
Doc-tests (hero_proc_sdk, hero_proc_lib, hero_proc_integration_test): 0/31 (31 ignored)

## Test Results - Build: PASS - Total: 450 - Passed: 399 - Failed: 0 - Ignored: 51 ### Crates with significant test counts - hero_proc_lib (lib): 140/140 - hero_proc_server (lib): 70/70 - hero_proc_sdk (lib): 20/20 - hero_proc_integration_tests / cli_integration: 83/83 - hero_proc_integration_tests / service_management: 16/18 (2 ignored) - hero_proc_integration_tests / jobs: 11/12 (1 ignored) - hero_proc_integration_tests / rhai_scripting: 10/10 - hero_proc_integration_tests / shutdown: 8/17 (9 ignored) - hero_proc_integration_tests / server_lifecycle: 7/7 - hero_proc_integration_tests / bulk_operations: 7/7 - hero_proc_integration_tests / hero_script: 5/5 - hero_proc_integration_tests / action_dependencies: 4/4 - hero_proc_integration_tests / dependencies: 4/5 (1 ignored) - hero_proc_integration_tests / service_restart_with_cleanup: 4/4 - hero_proc_integration_tests (lib): 4/4 - hero_proc_integration_tests / pty: 3/3 - hero_proc_integration_tests / health_probe: 2/2 - hero_proc_integration_tests / failed_action_lifecycle: 1/1 - hero_proc_integration_tests / dev_only: 0/7 (7 ignored) - Doc-tests (hero_proc_sdk, hero_proc_lib, hero_proc_integration_test): 0/31 (31 ignored)

despiegk commented

2026-05-01 06:34:05 +00:00

Author

Owner

Implementation Complete

The logging subsystem has been reworked end-to-end. SQLite is no longer the log backend; everything now flows through herolib_core::logger (file-based TSV records under ~/hero/var/logs/). All RPC method names, parameters, result envelopes, and SDK type names are byte-identical, so existing callers (CLI, Dioxus admin app, downstream services) keep working unchanged.

What changed

New files

crates/hero_proc_lib/src/db/logs/adapter.rs — bidirectional translation between the local LogEntry/LogFilter and herolib_core::logger's entry/query types, plus post-filter helpers for semantics that hero_log does not natively support (error_only, loglevel_max, offset).
crates/hero_proc_lib/src/db/logs/log_store.rs — replacement LoggingApi wrapping Arc<Logger>. Same public method signatures as the old SQLite-backed one.

Deleted

crates/hero_proc_lib/src/db/logs/store.rs, partition.rs, pool.rs — the old PartitionedLogStore, day-partitioning, and per-shard SQLite pool.
flate2 dep from hero_proc_lib/Cargo.toml (only used by the deleted gzip export/import paths). rusqlite stays — it is still the backend for jobs, runs, services, secrets, and actions.

Modified

Cargo.toml (workspace) and hero_proc_lib/Cargo.toml — wired herolib_core as a workspace dep.
hero_proc_lib/src/db/factory.rs — HeroProcDb::new constructs Arc<Logger> once; new pub fn logger(&self) -> Arc<Logger> accessor; db.logging now points at the hero_log-backed LoggingApi.
hero_proc_lib/src/db/logs/mod.rs and model.rs — dropped deleted submodules; stripped open_log_db/init_schema/rusqlite imports/tests; kept LogEntry, LogFilter, name_fix. LogFilter.src doc updated to dotted-prefix semantics.
hero_proc_server/src/log_batcher.rs — full rewrite. LogBatcherSender is now a thin shim around Arc<Logger>. The internal MPSC channel, drain task, periodic flush, and dropped-line warning are all gone (hero_log already has its own non-blocking writer). start() signature unchanged; the JoinHandle is now a no-op task.
hero_proc_server/src/supervisor/executor.rs — every LogEntry { ... } construction site now writes a dotted <context>.<action>.<job_id> src and attaches ["job:<id>", "stream:<stdout|stderr>"] tags (plus "attempt:<n>" when retrying). Two private helpers added: job_log_src and job_log_tags.
hero_proc_server/Cargo.toml — added herolib_core dep so the batcher can import Logger.
hero_proc_server/openrpc.json — doc-string updates only: LogFilter.src semantics; logs.insert/logs.insert_batch note logid is always 0; logs.sources re-scans the hero_log backend. No wire-format changes.
hero_proc_server/openrpc.client.generated.rs — three doc-comment lines mirrored from the schema. No code/type/signature changes.
hero_proc_server/instructions.md — backend description updated for logs (other tables still SQLite).
Integration tests (hero_proc_integration_test/src/tests/logs.rs, stress.rs) — relaxed delete_by_source assertion (hero_log returns files-removed, not entries-removed) and fixed a units-mismatch in delete_older_than_epoch (was using milliseconds; hero_log uses seconds). Added a logs_count == 0 follow-up to verify entries are actually gone.
hero_proc_lib/src/db/integration_tests.rs — replaced SQLite-bound logging_* tests with a single end-to-end smoke test that exercises insert, context filter, job_id filter, and delete_older_than against the real hero_log backend.

Acceptance criteria

All 14 RPC method signatures unchanged (parameters + result envelope identical).
No SQLite log table, no logs.sqlite files, no partitioned* modules in hero_proc_lib.
herolib_core::logger is the single backend; one Arc<Logger> per server process; logs land under ~/hero/var/logs/....
cargo build --workspace passes; cargo test --workspace passes (399 passed, 0 failed, 51 pre-existing ignored).
CLI hero_proc log query|filter|prune|export and hero_proc job logs produce the same console output shape (no flag/output changes were required).
logs.insert returns {"logid": 0} (documented in the schema).
job.log_archive partitions by attempt — the executor now emits attempt:N tags and the existing post-filter in rpc/job.rs keys off them.

Test results

cargo build --workspace: PASS
cargo test --workspace: 399 passed, 0 failed, 51 ignored (51 ignored are pre-existing doctests + environment-gated shutdown tests, not related to logs).
cargo test -p hero_proc_lib --lib: 140 passed, 0 failed.
cargo test -p hero_proc_server --lib: 70 passed, 0 failed.
cargo test -p hero_proc_sdk --lib: 20 passed, 0 failed.

Operator notes

Old SQLite log files at ~/hero/var/hero_proc/logs/* are orphaned by this change. They are not migrated and not deleted automatically — operators should remove them manually when convenient.
LogFilter.src now requires a dotted prefix, segment-aligned. A trailing * is accepted as a wildcard suffix and stripped. Wildcards anywhere else in the pattern degrade to a full scan with post-filter (the CLI and admin UI only ever produce trailing-* patterns today, so this is transparent in practice).
logs.delete_by_src(prefix) removes the whole dotted subtree; the returned count is files-removed (not entries-removed), since hero_log cleans whole files.
logs.delete_older_than(epoch) operates at file granularity — boundary-day partial deletes are coarser than before, but the daily 7-day retention loop is unaffected.

## Implementation Complete The logging subsystem has been reworked end-to-end. SQLite is no longer the log backend; everything now flows through `herolib_core::logger` (file-based TSV records under `~/hero/var/logs/`). All RPC method names, parameters, result envelopes, and SDK type names are byte-identical, so existing callers (CLI, Dioxus admin app, downstream services) keep working unchanged. ### What changed **New files** - `crates/hero_proc_lib/src/db/logs/adapter.rs` — bidirectional translation between the local `LogEntry`/`LogFilter` and `herolib_core::logger`'s entry/query types, plus post-filter helpers for semantics that hero_log does not natively support (`error_only`, `loglevel_max`, offset). - `crates/hero_proc_lib/src/db/logs/log_store.rs` — replacement `LoggingApi` wrapping `Arc<Logger>`. Same public method signatures as the old SQLite-backed one. **Deleted** - `crates/hero_proc_lib/src/db/logs/store.rs`, `partition.rs`, `pool.rs` — the old `PartitionedLogStore`, day-partitioning, and per-shard SQLite pool. - `flate2` dep from `hero_proc_lib/Cargo.toml` (only used by the deleted gzip export/import paths). `rusqlite` stays — it is still the backend for jobs, runs, services, secrets, and actions. **Modified** - `Cargo.toml` (workspace) and `hero_proc_lib/Cargo.toml` — wired `herolib_core` as a workspace dep. - `hero_proc_lib/src/db/factory.rs` — `HeroProcDb::new` constructs `Arc<Logger>` once; new `pub fn logger(&self) -> Arc<Logger>` accessor; `db.logging` now points at the hero_log-backed `LoggingApi`. - `hero_proc_lib/src/db/logs/mod.rs` and `model.rs` — dropped deleted submodules; stripped `open_log_db`/`init_schema`/rusqlite imports/tests; kept `LogEntry`, `LogFilter`, `name_fix`. `LogFilter.src` doc updated to dotted-prefix semantics. - `hero_proc_server/src/log_batcher.rs` — full rewrite. `LogBatcherSender` is now a thin shim around `Arc<Logger>`. The internal MPSC channel, drain task, periodic flush, and dropped-line warning are all gone (hero_log already has its own non-blocking writer). `start()` signature unchanged; the JoinHandle is now a no-op task. - `hero_proc_server/src/supervisor/executor.rs` — every `LogEntry { ... }` construction site now writes a dotted `<context>.<action>.<job_id>` src and attaches `["job:<id>", "stream:<stdout|stderr>"]` tags (plus `"attempt:<n>"` when retrying). Two private helpers added: `job_log_src` and `job_log_tags`. - `hero_proc_server/Cargo.toml` — added `herolib_core` dep so the batcher can import `Logger`. - `hero_proc_server/openrpc.json` — doc-string updates only: `LogFilter.src` semantics; `logs.insert`/`logs.insert_batch` note `logid` is always 0; `logs.sources` re-scans the hero_log backend. No wire-format changes. - `hero_proc_server/openrpc.client.generated.rs` — three doc-comment lines mirrored from the schema. No code/type/signature changes. - `hero_proc_server/instructions.md` — backend description updated for logs (other tables still SQLite). - Integration tests (`hero_proc_integration_test/src/tests/logs.rs`, `stress.rs`) — relaxed `delete_by_source` assertion (hero_log returns files-removed, not entries-removed) and fixed a units-mismatch in `delete_older_than_epoch` (was using milliseconds; hero_log uses seconds). Added a `logs_count == 0` follow-up to verify entries are actually gone. - `hero_proc_lib/src/db/integration_tests.rs` — replaced SQLite-bound `logging_*` tests with a single end-to-end smoke test that exercises insert, context filter, `job_id` filter, and `delete_older_than` against the real hero_log backend. ### Acceptance criteria - [x] All 14 RPC method signatures unchanged (parameters + result envelope identical). - [x] No SQLite log table, no `logs.sqlite` files, no `partitioned*` modules in `hero_proc_lib`. - [x] `herolib_core::logger` is the single backend; one `Arc<Logger>` per server process; logs land under `~/hero/var/logs/...`. - [x] `cargo build --workspace` passes; `cargo test --workspace` passes (399 passed, 0 failed, 51 pre-existing ignored). - [x] CLI `hero_proc log query|filter|prune|export` and `hero_proc job logs` produce the same console output shape (no flag/output changes were required). - [x] `logs.insert` returns `{"logid": 0}` (documented in the schema). - [x] `job.log_archive` partitions by attempt — the executor now emits `attempt:N` tags and the existing post-filter in `rpc/job.rs` keys off them. ### Test results - `cargo build --workspace`: PASS - `cargo test --workspace`: 399 passed, 0 failed, 51 ignored (51 ignored are pre-existing doctests + environment-gated shutdown tests, not related to logs). - `cargo test -p hero_proc_lib --lib`: 140 passed, 0 failed. - `cargo test -p hero_proc_server --lib`: 70 passed, 0 failed. - `cargo test -p hero_proc_sdk --lib`: 20 passed, 0 failed. ### Operator notes - Old SQLite log files at `~/hero/var/hero_proc/logs/*` are orphaned by this change. They are not migrated and not deleted automatically — operators should remove them manually when convenient. - `LogFilter.src` now requires a dotted prefix, segment-aligned. A trailing `*` is accepted as a wildcard suffix and stripped. Wildcards anywhere else in the pattern degrade to a full scan with post-filter (the CLI and admin UI only ever produce trailing-`*` patterns today, so this is transparent in practice). - `logs.delete_by_src(prefix)` removes the whole dotted subtree; the returned count is files-removed (not entries-removed), since hero_log cleans whole files. - `logs.delete_older_than(epoch)` operates at file granularity — boundary-day partial deletes are coarser than before, but the daily 7-day retention loop is unaffected.

despiegk referenced this issue from a commit

2026-05-01 07:02:32 +00:00

refactor(logs): replace SQLite log backend with hero_log