use zinit jobs & logs for all long running jobs #24

New issue

Open

opened 2026-03-10 04:40:14 +00:00 by despiegk · 8 comments

despiegk commented 2026-03-10 04:40:14 +00:00

Owner

Copy link

e.g. importing books

make sure it happens in async mode on server and all logs sent to zinit on right source
then show what that source is e.g. servicename.import.something

now go over all long running jobs

• some of them can be done as job using a command line
• some are in server e.g. book import async but they need still to do log

make sure nowhere we do logging in the component itself

each service needs to integrate with zinit sdk for logging, process management, job mgmt, ...

e.g. importing books  make sure it happens in async mode on server and all logs sent to zinit on right source then show what that source is e.g. servicename.import.something now go over all long running jobs - some of them can be done as job using a command line - some are in server e.g. book import async but they need still to do log make sure nowhere we do logging in the component itself each service needs to integrate with zinit sdk for logging, process management, job mgmt, ...

image.png

179 KiB

timur was assigned by despiegk 2026-03-10 04:40:14 +00:00

despiegk added this to the ACTIVE project 2026-03-10 04:40:14 +00:00

timur referenced this issue from lhumina_code/hero_books 2026-03-10 08:59:32 +00:00

Integrate zinit SDK: ZinitLifecycle for all binaries, logging via zinit, health checks #77

timur commented 2026-03-10 08:59:41 +00:00

Owner

Copy link

hero_books zinit integration issue

Created detailed tracking issue for hero_books: lhumina_code/hero_books#77

Covers 12 areas of improvement:

1. Background initialization pipeline → zinit jobs (10-step startup sequence)
2. Library rescan → zinit job (currently std::thread::spawn)
3. Library re-embed → zinit job
4. Replace custom OperationLogger with zinit logs API
5. Collection scanning (git ops) → zinit job
6. AI Q&A extraction → zinit job
7. PDF generation → zinit job
8. Search indexing → zinit job
9. Enhanced service health checks
10. Move shell script orchestration into Rust via zinit_sdk
11. Structured log source naming convention (hero_books.{op}.{target})
12. Update zinit_sdk dependency to development_kristof branch

## hero_books zinit integration issue Created detailed tracking issue for hero_books: https://forge.ourworld.tf/lhumina_code/hero_books/issues/77 Covers 12 areas of improvement: 1. Background initialization pipeline → zinit jobs (10-step startup sequence) 2. Library rescan → zinit job (currently `std::thread::spawn`) 3. Library re-embed → zinit job 4. Replace custom `OperationLogger` with zinit logs API 5. Collection scanning (git ops) → zinit job 6. AI Q&A extraction → zinit job 7. PDF generation → zinit job 8. Search indexing → zinit job 9. Enhanced service health checks 10. Move shell script orchestration into Rust via zinit_sdk 11. Structured log source naming convention (`hero_books.{op}.{target}`) 12. Update zinit_sdk dependency to `development_kristof` branch

timur commented 2026-03-10 09:17:38 +00:00

Owner

Copy link

Related issues for the overall zinit lifecycle effort

• lhumina_code/hero_rpc#7 — Hero RPC servers should use zinit for lifecycle (ZinitLifecycle pattern)
• lhumina_code/hero_skills#51 — Skill documenting OServer::run_cli() / ZinitLifecycle standard
• lhumina_code/home#6 — Core services (AI Broker, Embedder, Indexer, Redis) must all start via zinit

### Related issues for the overall zinit lifecycle effort - https://forge.ourworld.tf/lhumina_code/hero_rpc/issues/7 — Hero RPC servers should use zinit for lifecycle (`ZinitLifecycle` pattern) - https://forge.ourworld.tf/lhumina_code/hero_skills/issues/51 — Skill documenting `OServer::run_cli()` / `ZinitLifecycle` standard - https://forge.ourworld.tf/lhumina_code/home/issues/6 — Core services (AI Broker, Embedder, Indexer, Redis) must all start via zinit

timur referenced this issue from lhumina_code/hero_embedder 2026-03-10 09:28:27 +00:00

Integrate zinit SDK: ZinitLifecycle for UI, logging via zinit, remove legacy scripts #9

timur commented 2026-03-10 09:28:38 +00:00

Owner

Copy link

hero_embedder zinit integration issue

Created tracking issue: lhumina_code/hero_embedder#9

Already done: Server has full ZinitLifecycle (Pattern B) with run/start/stop/status/logs subcommands.

Still needed (7 items):

1. Corpus download → zinit job (3.5MB gzip → 92k Q&A pairs, currently blocks RPC)
2. Corpus load/batch embedding → zinit job (heaviest op, minutes for 1000+ docs)
3. Model loading at startup → zinit job tracking for visibility
4. Replace custom OperationLogger (500-entry buffer) with zinit logs.insert()
5. Add ZinitLifecycle to hero_embedder_ui (currently standalone, no subcommands)
6. Namespace create/delete → zinit job for large namespaces
7. Remove legacy run.sh

## hero_embedder zinit integration issue Created tracking issue: https://forge.ourworld.tf/lhumina_code/hero_embedder/issues/9 **Already done:** Server has full `ZinitLifecycle` (Pattern B) with `run`/`start`/`stop`/`status`/`logs` subcommands. **Still needed (7 items):** 1. Corpus download → zinit job (3.5MB gzip → 92k Q&A pairs, currently blocks RPC) 2. Corpus load/batch embedding → zinit job (heaviest op, minutes for 1000+ docs) 3. Model loading at startup → zinit job tracking for visibility 4. Replace custom `OperationLogger` (500-entry buffer) with zinit `logs.insert()` 5. Add `ZinitLifecycle` to hero_embedder_ui (currently standalone, no subcommands) 6. Namespace create/delete → zinit job for large namespaces 7. Remove legacy `run.sh`

timur referenced this issue from lhumina_code/hero_voice 2026-03-10 09:38:06 +00:00

Integrate zinit SDK: ZinitLifecycle for both binaries, logging via zinit, Makefile update #6

timur commented 2026-03-10 09:38:16 +00:00

Owner

Copy link

hero_voice zinit integration issue

Created tracking issue: lhumina_code/hero_voice#6

Current state: Zero zinit SDK integration — uses raw zinit CLI calls from Makefile.

10 items needed:

1. Add ZinitLifecycle to hero_voice_server (no subcommands today)
2. Add ZinitLifecycle to hero_voice_ui (no subcommands today)
3. Transcription (Groq Whisper API) → zinit job with timeout + retry
4. Text transformation (DeepSeek API, 14 styles) → zinit job
5. Audio conversion (WAV→OGG, currently silent fire-and-forget) → zinit job
6. Logging → zinit logs.insert() with structured source names
7. Health check integration with zinit service registration
8. Makefile → binary subcommands (no more direct zinit CLI)
9. Add zinit_sdk + hero_rpc_server dependencies
10. Graceful shutdown coordination (currently process::exit(0))

## hero_voice zinit integration issue Created tracking issue: https://forge.ourworld.tf/lhumina_code/hero_voice/issues/6 **Current state:** Zero zinit SDK integration — uses raw `zinit` CLI calls from Makefile. **10 items needed:** 1. Add `ZinitLifecycle` to hero_voice_server (no subcommands today) 2. Add `ZinitLifecycle` to hero_voice_ui (no subcommands today) 3. Transcription (Groq Whisper API) → zinit job with timeout + retry 4. Text transformation (DeepSeek API, 14 styles) → zinit job 5. Audio conversion (WAV→OGG, currently silent fire-and-forget) → zinit job 6. Logging → zinit `logs.insert()` with structured source names 7. Health check integration with zinit service registration 8. Makefile → binary subcommands (no more direct `zinit` CLI) 9. Add `zinit_sdk` + `hero_rpc_server` dependencies 10. Graceful shutdown coordination (currently `process::exit(0)`)

timur referenced this issue from lhumina_code/hero_aibroker 2026-03-10 10:03:20 +00:00

Integrate zinit SDK: ZinitLifecycle for all binaries, logging via zinit, health checks #24

timur commented 2026-03-10 10:03:31 +00:00

Owner

Copy link

hero_aibroker zinit integration issue

Created tracking issue: lhumina_code/hero_aibroker#24

Already done: OpenRPC server has full ZinitLifecycle (Pattern B) with all subcommands.

Still needed (7 items):

1. Add ZinitLifecycle to hero_aibroker_http (currently no lifecycle, direct TCP/socket bind)
2. Streaming chat completions → zinit job tracking (fire-and-forget today)
3. Replace custom OperationLogger (500-entry buffer) with zinit logs.insert()
4. Clean up 40+ println!/eprintln! → tracing::
5. MCP server lifecycle → consider zinit process management or health monitoring
6. Health check improvements (currently plain text "OK")
7. Graceful shutdown for HTTP service (no drain/cleanup today)

## hero_aibroker zinit integration issue Created tracking issue: https://forge.ourworld.tf/lhumina_code/hero_aibroker/issues/24 **Already done:** OpenRPC server has full `ZinitLifecycle` (Pattern B) with all subcommands. **Still needed (7 items):** 1. Add `ZinitLifecycle` to hero_aibroker_http (currently no lifecycle, direct TCP/socket bind) 2. Streaming chat completions → zinit job tracking (fire-and-forget today) 3. Replace custom `OperationLogger` (500-entry buffer) with zinit `logs.insert()` 4. Clean up 40+ `println!`/`eprintln!` → `tracing::` 5. MCP server lifecycle → consider zinit process management or health monitoring 6. Health check improvements (currently plain text `"OK"`) 7. Graceful shutdown for HTTP service (no drain/cleanup today)

timur commented 2026-03-10 11:28:05 +00:00

Owner

Copy link

Correction applied to all child issues

All 4 repo issues have been updated to correct the scope of zinit integration. In-process long-running operations stay in-process — they work with in-memory state (loaded models, Chrome sessions, vector indexes, streaming connections) and cannot be externalized to zinit subprocess jobs.

What zinit IS used for (across all repos)

1. ZinitLifecycle pattern — standard run/start/stop/status/logs/serve subcommands for every binary
2. Logging via logs.insert() — structured source names like hero_books.pdf.mybook, hero_embedder.embed.docs
3. Health checks — configured in zinit service registration
4. Replace raw zinit CLI calls in Makefiles/scripts with binary subcommands

What zinit is NOT used for

In-process operations that depend on runtime memory state:

• PDF generation (Chrome CDP session + base64-inlined HTML)
• AI Q&A extraction (LLM API calls on loaded collections)
• Embedding generation (cached ONNX models + HNSW indexes)
• Streaming chat completions (live SSE sessions)
• Audio transcription/transformation (WebSocket + VAD state)
• Book export, library rescan, corpus load, etc.

These stay as tokio::spawn / std::thread::spawn / spawn_blocking but log through zinit for centralized visibility.

Updated child issues

Repo	Issue	Status
hero_books	lhumina_code/hero_books#77	Updated — lifecycle for 3 binaries + logging + health
hero_embedder	lhumina_code/hero_embedder#9	Updated — UI lifecycle + logging
hero_voice	lhumina_code/hero_voice#6	Updated — lifecycle for 2 binaries + logging + Makefile
hero_aibroker	lhumina_code/hero_aibroker#24	Updated — HTTP lifecycle + logging + println cleanup

## Correction applied to all child issues All 4 repo issues have been updated to correct the scope of zinit integration. **In-process long-running operations stay in-process** — they work with in-memory state (loaded models, Chrome sessions, vector indexes, streaming connections) and cannot be externalized to zinit subprocess jobs. ### What zinit IS used for (across all repos) 1. **`ZinitLifecycle` pattern** — standard `run`/`start`/`stop`/`status`/`logs`/`serve` subcommands for every binary 2. **Logging via `logs.insert()`** — structured source names like `hero_books.pdf.mybook`, `hero_embedder.embed.docs` 3. **Health checks** — configured in zinit service registration 4. **Replace raw `zinit` CLI calls** in Makefiles/scripts with binary subcommands ### What zinit is NOT used for In-process operations that depend on runtime memory state: - PDF generation (Chrome CDP session + base64-inlined HTML) - AI Q&A extraction (LLM API calls on loaded collections) - Embedding generation (cached ONNX models + HNSW indexes) - Streaming chat completions (live SSE sessions) - Audio transcription/transformation (WebSocket + VAD state) - Book export, library rescan, corpus load, etc. These stay as `tokio::spawn` / `std::thread::spawn` / `spawn_blocking` but **log through zinit** for centralized visibility. ### Updated child issues | Repo | Issue | Status | |------|-------|--------| | hero_books | https://forge.ourworld.tf/lhumina_code/hero_books/issues/77 | Updated — lifecycle for 3 binaries + logging + health | | hero_embedder | https://forge.ourworld.tf/lhumina_code/hero_embedder/issues/9 | Updated — UI lifecycle + logging | | hero_voice | https://forge.ourworld.tf/lhumina_code/hero_voice/issues/6 | Updated — lifecycle for 2 binaries + logging + Makefile | | hero_aibroker | https://forge.ourworld.tf/lhumina_code/hero_aibroker/issues/24 | Updated — HTTP lifecycle + logging + println cleanup |

timur commented 2026-03-10 11:44:50 +00:00

Owner

Copy link

Status update — 3 of 4 repos complete

All code verified compiling, committed, and pushed.

Repo	Issue	Status	Commit
hero_books	#77	✅ Closed	963f8ca — ZinitLifecycle for 3 binaries, removed custom logging
hero_embedder	#9	✅ Closed	642f88a — UI lifecycle, zinit logging, in-process job tracking, SDK fix
hero_voice	#6	✅ Closed	972a2e7 — lifecycle for 2 binaries, Makefile updated
hero_aibroker	#24	⏳ Open	OpenRPC done, HTTP binary lifecycle still needed

Key architecture decision: In-process long-running operations (PDF generation, embedding, transcription, streaming chat, etc.) stay as in-process async tasks — NOT zinit subprocess jobs. They work with in-memory state that can't cross process boundaries. Zinit is used for service lifecycle (ZinitLifecycle pattern), logging (logs.insert()), and health checks.

## Status update — 3 of 4 repos complete All code verified compiling, committed, and pushed. | Repo | Issue | Status | Commit | |------|-------|--------|--------| | hero_books | #77 | ✅ **Closed** | `963f8ca` — ZinitLifecycle for 3 binaries, removed custom logging | | hero_embedder | #9 | ✅ **Closed** | `642f88a` — UI lifecycle, zinit logging, in-process job tracking, SDK fix | | hero_voice | #6 | ✅ **Closed** | `972a2e7` — lifecycle for 2 binaries, Makefile updated | | hero_aibroker | #24 | ⏳ **Open** | OpenRPC done, HTTP binary lifecycle still needed | **Key architecture decision:** In-process long-running operations (PDF generation, embedding, transcription, streaming chat, etc.) stay as in-process async tasks — NOT zinit subprocess jobs. They work with in-memory state that can't cross process boundaries. Zinit is used for **service lifecycle** (`ZinitLifecycle` pattern), **logging** (`logs.insert()`), and **health checks**.

timur commented 2026-03-10 11:48:10 +00:00

Owner

Copy link

hero_aibroker #24 now also closed

Crates were renamed (_openrpc → _server, _http → _ui, _client → _sdk) and ZinitLifecycle was already added to the UI binary (commit aaebac1). Both binaries now have full lifecycle.

All 4 repos complete:

Repo	Issue	Status
hero_books	#77	✅ Closed
hero_embedder	#9	✅ Closed
hero_voice	#6	✅ Closed
hero_aibroker	#24	✅ Closed

## hero_aibroker #24 now also closed Crates were renamed (`_openrpc` → `_server`, `_http` → `_ui`, `_client` → `_sdk`) and ZinitLifecycle was already added to the UI binary (commit aaebac1). Both binaries now have full lifecycle. **All 4 repos complete:** | Repo | Issue | Status | |------|-------|--------| | hero_books | #77 | ✅ Closed | | hero_embedder | #9 | ✅ Closed | | hero_voice | #6 | ✅ Closed | | hero_aibroker | #24 | ✅ Closed |

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

development

development_timur

development_projects_island

development_ci_fix

v0.1.3

v0.1.2

v0.1.1

v0.1.0

v0.0.1-rc1

Labels

Clear labels

No items

No labels

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

ACTIVE

Assignees

Clear assignees

No assignees

timur

2 participants

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_os#24

No description provided.