single binary release | hero_compute runs all services (server, explorer, ui) #45

Closed
opened 2026-03-25 10:50:56 +00:00 by mahmoud · 3 comments
Owner

Problem

hero_compute ships currently as three separate binaries:

  • hero_compute_server
  • hero_compute_explorer
  • hero_compute_ui

This creates operational overhead:

  • Three separate hero_proc service registrations
  • Three separate make start/stop calls
  • Three separate build artifacts to distribute
  • Users must understand the role of each binary
  • Deployment on a new node requires configuring all three separately

Solution: Unified CLI Orchestrator

A single hero_compute CLI binary that orchestrates all three services via hero_proc, following the hero_proc_service_selfstart pattern. The three service binaries remain separate crates — the CLI registers and manages them as actions under one hero_compute service.

# Lifecycle (orchestrates all 3 via hero_proc)
hero_compute --start                                    # Local mode (default)
hero_compute --start --mode master                      # Explorer hub
hero_compute --start --mode worker --master-ip X.X.X    # Worker node
hero_compute --stop                                     # Stop all
hero_compute --status                                   # Show service states

Configurable ports:

hero_compute --start --port 80 --explorer-port 9002 --rpc-port 9003

Architecture

Multi-binary orchestrator — NOT a single embedded binary. The CLI manages lifecycle; each service runs as its own process.

crates/
  hero_compute/               # CLI orchestrator (hero_proc_service_selfstart pattern)
    src/main.rs               # --start/--stop/--status, env generation, service registration
  hero_compute_server/        # JSON-RPC daemon (unchanged, own binary)
  hero_compute_explorer/      # Multi-node gateway (unchanged, own binary)
  hero_compute_ui/            # Web dashboard (unchanged, own binary)
  hero_compute_sdk/           # OpenRPC client lib (unchanged)

HeroProc Integration

The CLI registers one hero_compute service with 3 actions:

Service: hero_compute
  Action: hero_compute_server   → exec: hero_compute_server [--tcp-port 9003]
  Action: hero_compute_ui       → exec: hero_compute_ui --port 80
  Action: hero_compute_explorer → exec: hero_compute_explorer [--tcp-port 9002]

Each action has:

  • Health checks (OpenRPC socket or TCP port)
  • Kill-other cleanup (socket + port)
  • Retry policies with backoff
  • Graceful stop via SIGTERM

hero_compute --start registers and starts all three actions.
hero_compute --stop stops all three.
hero_compute --status queries hero_proc for service state and displays it.

Deployment Modes

Mode Server UI Explorer Notes
local (default) Single node, no networking
master + heartbeat + TCP 9002 Explorer hub, workers connect here
worker + TCP 9003 Sends heartbeats to master

Environment Management

The CLI auto-generates .env with mode-specific variables:

  • HERO_PROC_SOCKET — auto-detected
  • EXPLORER_ADDRESSES — set per mode
  • MASTER_IP — worker mode only
  • HERO_COMPUTE_ADVERTISE_ADDRESS — worker auto-detects outbound IP
  • CHVM_MYCELIUM_PEERS — 10 default public peers

User-set variables are preserved across regeneration.

Current State (development_cli branch)

Already implemented:

  • hero_compute CLI crate with --start and --stop
  • Service registration with hero_proc (3 actions, health checks, retry, kill-other)
  • --mode local|master|worker with --master-ip
  • Configurable ports (--port, --explorer-port, --rpc-port)
  • Auto .env generation with mode-specific vars
  • Makefile updated (make start/stop/status)
  • scripts/configure.sh overhauled
  • docs/setup.md updated
  • Old shell scripts removed (start.sh, stop.sh, status.sh, dev-*.sh)

Remaining Work

  1. Add --status flag — query hero_proc SDK for hero_compute service state, display formatted output (actions, PIDs, health, uptime)
  2. CI release update — uncomment publish step in build-linux.yaml, upload all 4 binaries per architecture
  3. Docs polish — ensure docs/setup.md and README reflect final CLI interface

Definition of Done

  • hero_compute CLI crate built and working
  • hero_compute --start registers and starts all 3 services via hero_proc
  • hero_compute --stop stops all services
  • hero_compute --status displays service state from hero_proc
  • --mode local|master|worker with --master-ip support
  • Makefile updated — make start calls hero_compute --start
  • docs/setup.md updated with CLI usage
  • CI releases all binaries as artifacts
  • Old shell scripts removed
### Problem hero_compute ships currently as three separate binaries: - `hero_compute_server` - `hero_compute_explorer` - `hero_compute_ui` This creates operational overhead: - Three separate hero_proc service registrations - Three separate make start/stop calls - Three separate build artifacts to distribute - Users must understand the role of each binary - Deployment on a new node requires configuring all three separately ### Solution: Unified CLI Orchestrator A single `hero_compute` CLI binary that **orchestrates** all three services via hero_proc, following the `hero_proc_service_selfstart` pattern. The three service binaries remain separate crates — the CLI registers and manages them as actions under one `hero_compute` service. ```bash # Lifecycle (orchestrates all 3 via hero_proc) hero_compute --start # Local mode (default) hero_compute --start --mode master # Explorer hub hero_compute --start --mode worker --master-ip X.X.X # Worker node hero_compute --stop # Stop all hero_compute --status # Show service states ``` Configurable ports: ```bash hero_compute --start --port 80 --explorer-port 9002 --rpc-port 9003 ``` ### Architecture **Multi-binary orchestrator** — NOT a single embedded binary. The CLI manages lifecycle; each service runs as its own process. ``` crates/ hero_compute/ # CLI orchestrator (hero_proc_service_selfstart pattern) src/main.rs # --start/--stop/--status, env generation, service registration hero_compute_server/ # JSON-RPC daemon (unchanged, own binary) hero_compute_explorer/ # Multi-node gateway (unchanged, own binary) hero_compute_ui/ # Web dashboard (unchanged, own binary) hero_compute_sdk/ # OpenRPC client lib (unchanged) ``` ### HeroProc Integration The CLI registers one `hero_compute` service with 3 actions: ``` Service: hero_compute Action: hero_compute_server → exec: hero_compute_server [--tcp-port 9003] Action: hero_compute_ui → exec: hero_compute_ui --port 80 Action: hero_compute_explorer → exec: hero_compute_explorer [--tcp-port 9002] ``` Each action has: - Health checks (OpenRPC socket or TCP port) - Kill-other cleanup (socket + port) - Retry policies with backoff - Graceful stop via SIGTERM `hero_compute --start` registers and starts all three actions. `hero_compute --stop` stops all three. `hero_compute --status` queries hero_proc for service state and displays it. ### Deployment Modes | Mode | Server | UI | Explorer | Notes | |------|--------|----|----------|-------| | `local` (default) | ✅ | ✅ | ✅ | Single node, no networking | | `master` | ✅ + heartbeat | ✅ | ✅ + TCP 9002 | Explorer hub, workers connect here | | `worker` | ✅ + TCP 9003 | ✅ | ❌ | Sends heartbeats to master | ### Environment Management The CLI auto-generates `.env` with mode-specific variables: - `HERO_PROC_SOCKET` — auto-detected - `EXPLORER_ADDRESSES` — set per mode - `MASTER_IP` — worker mode only - `HERO_COMPUTE_ADVERTISE_ADDRESS` — worker auto-detects outbound IP - `CHVM_MYCELIUM_PEERS` — 10 default public peers User-set variables are preserved across regeneration. ### Current State (`development_cli` branch) Already implemented: - [x] `hero_compute` CLI crate with `--start` and `--stop` - [x] Service registration with hero_proc (3 actions, health checks, retry, kill-other) - [x] `--mode local|master|worker` with `--master-ip` - [x] Configurable ports (`--port`, `--explorer-port`, `--rpc-port`) - [x] Auto `.env` generation with mode-specific vars - [x] Makefile updated (`make start/stop/status`) - [x] `scripts/configure.sh` overhauled - [x] `docs/setup.md` updated - [x] Old shell scripts removed (start.sh, stop.sh, status.sh, dev-*.sh) ### Remaining Work 1. **Add `--status` flag** — query hero_proc SDK for `hero_compute` service state, display formatted output (actions, PIDs, health, uptime) 2. **CI release update** — uncomment publish step in `build-linux.yaml`, upload all 4 binaries per architecture 3. **Docs polish** — ensure `docs/setup.md` and README reflect final CLI interface ### Definition of Done - [x] `hero_compute` CLI crate built and working - [x] `hero_compute --start` registers and starts all 3 services via hero_proc - [x] `hero_compute --stop` stops all services - [ ] `hero_compute --status` displays service state from hero_proc - [x] `--mode local|master|worker` with `--master-ip` support - [x] Makefile updated — `make start` calls `hero_compute --start` - [x] `docs/setup.md` updated with CLI usage - [ ] CI releases all binaries as artifacts - [x] Old shell scripts removed
mahmoud self-assigned this 2026-03-25 11:03:39 +00:00
mahmoud added this to the ACTIVE project 2026-03-25 11:03:43 +00:00
mahmoud added this to the now milestone 2026-03-25 11:03:45 +00:00
Author
Owner

Gap Analysis: development_cli branch vs Issue Requirements

What's Already Done (5 commits on development_cli)

The development_cli branch has a working hero_compute CLI crate (crates/hero_compute/) that:

  • Registers all 3 services (server, UI, explorer) as hero_proc actions under one hero_compute service
  • Supports --start and --stop lifecycle flags
  • Supports --mode local|master|worker with --master-ip for worker mode
  • Configurable ports: --port (UI), --explorer-port, --rpc-port
  • Auto-generates .env with mode-specific variables + symlinks to $HOME/.env
  • Health checks per action (OpenRPC socket for server/explorer, TCP port for UI)
  • Kill-other cleanup (socket + port) before starting
  • Retry policies per action with backoff
  • Auto-detects worker outbound IP
  • Makefile updated: make start [MODE=...] [MASTER_IP=...], make stop, make status
  • scripts/configure.sh overhauled (412 lines — downloads cloud-hypervisor, mycelium, hero_proc, my_hypervisor)
  • docs/setup.md updated with new CLI usage

Gaps: What's Missing or Diverges from the Issue

1. CLI Interface: Flags vs Subcommands

Issue proposes:

hero_compute start
hero_compute stop
hero_compute status
hero_compute server serve
hero_compute server logs

Current implementation:

hero_compute --start
hero_compute --stop
# No status, no per-service subcommands

Decision needed: The flag-based approach (--start/--stop) works and follows the hero_proc_service_selfstart pattern used by other Hero services. The subcommand approach from the issue would be more user-friendly but diverges from the ecosystem pattern. Recommend: keep flags — consistent with hero_proc conventions. Add --status flag.

2. Still 4 Binaries, Not 1

The issue says "single binary release" but the current implementation is an orchestrator binary that launches 3 separate binaries (hero_compute_server, hero_compute_explorer, hero_compute_ui) via hero_proc.

Why this matters:

  • Release still ships 4 binaries (build target copies all 4 to ~/hero/bin/)
  • resolve_bin() in the CLI looks for sibling binaries on disk
  • If any binary is missing, startup fails

True single-binary approach would require:

  • Each service crate exposes a pub async fn run(args) in lib.rs
  • The unified binary imports and calls these directly
  • hero_proc actions point to hero_compute server serve (same binary, different subcommand)

Assessment: This is the biggest gap. The current approach is a multi-binary orchestrator, not a true single binary. Making it truly single-binary requires:

  1. Adding subcommands: hero_compute server serve, hero_compute ui serve, hero_compute explorer serve
  2. Each service crate must have a lib.rs with a pub async fn run() entry point (server and explorer already have lib.rs stubs)
  3. The unified crate depends on all 3 service crates as library dependencies
  4. hero_proc actions call hero_compute server serve instead of hero_compute_server

3. No hero_compute status Command

The issue lists hero_compute status but it's not implemented. The Makefile has a status target that shells out to hero_proc status hero_compute, which works but isn't part of the binary itself.

Recommendation: Add --status flag that calls hero_proc_sdk to query and display service state.

4. No Per-Service logs Subcommand

The issue proposes hero_compute server logs etc. Currently no log viewing at all from the CLI.

Recommendation: If going subcommand route for single-binary, add logs subcommand per service using hero_proc_sdk log query API.

5. CI Release Not Updated

The build-linux.yaml workflow:

  • Still uses build_binaries from build_lib.sh (builds all workspace binaries)
  • Publish step is commented out entirely
  • No artifact upload step
  • Doesn't produce a single downloadable binary

Needed: Uncomment and update publish step to upload the unified binary (or all 4 if staying multi-binary).

6. No Deprecation Notices

The issue says "Old three-binary setup deprecated but not removed until stable." The old shell scripts (scripts/start.sh, scripts/stop.sh, etc.) were removed entirely rather than deprecated.

Assessment: This is fine — the old scripts were internal. The 3 individual binaries still build and have their own main.rs, so they can still be run standalone. No action needed.


Phase 1: True Single Binary (High Priority)

  1. Add clap subcommands to hero_compute CLI:

    hero_compute --start [--mode ...]
    hero_compute --stop
    hero_compute --status              # NEW
    hero_compute server serve           # NEW - runs server foreground
    hero_compute explorer serve         # NEW - runs explorer foreground
    hero_compute ui serve [--port N]    # NEW - runs UI foreground
    hero_compute server logs            # NEW - view server logs
    hero_compute explorer logs          # NEW - view explorer logs  
    hero_compute ui logs                # NEW - view UI logs
    
  2. Wire up service entry points:

    • hero_compute_server/src/lib.rs → expose pub async fn serve(tcp_port: Option<u16>)
    • hero_compute_explorer/src/lib.rs → expose pub async fn serve(tcp_port: Option<u16>)
    • hero_compute_ui/src/lib.rs → expose pub async fn serve(port: u16, server_socket: Option<String>, explorer_socket: Option<String>)
    • Extract main.rs logic into these serve() functions
  3. Add library dependencies to crates/hero_compute/Cargo.toml:

    hero_compute_server = { path = "../hero_compute_server" }
    hero_compute_explorer = { path = "../hero_compute_explorer" }
    hero_compute_ui = { path = "../hero_compute_ui" }
    
  4. Update hero_proc action commands to use self-dispatch:

    // Instead of:
    let server_bin = resolve_bin(&bin_dir, "hero_compute_server");
    // Use:
    let self_bin = std::env::current_exe()?;
    let server_cmd = format!("{} server serve", self_bin.display());
    

Phase 2: Status & Logs

  1. Implement --status — query hero_proc for hero_compute service state, display formatted table
  2. Implement {service} logs — query hero_proc log API for each action's logs

Phase 3: CI & Release

  1. Update build-linux.yaml:
    • Build only the hero_compute binary (not all 4)
    • Uncomment and fix publish step
    • Upload single artifact per architecture
  2. Update Makefile build target to only copy hero_compute binary

Phase 4: Cleanup

  1. Remove individual binary [[bin]] sections from service Cargo.tomls (keep lib only)
  2. Update docs/setup.md to reflect single binary
  3. Update scripts/configure.sh if needed

Updated Definition of Done

  • hero_compute binary is the only binary needed (embeds server/explorer/ui)
  • hero_compute --start/--stop/--status manages all services
  • hero_compute {service} serve runs individual service foreground (for hero_proc)
  • hero_compute {service} logs shows per-service logs
  • hero_proc actions point to hero_compute {service} serve (self-dispatch)
  • Makefile build target produces one binary
  • make start calls hero_compute --start
  • CI releases hero_compute as single artifact (x86_64 + aarch64)
  • docs/setup.md documents single-binary install
  • Old individual binaries can optionally still build for development
## Gap Analysis: `development_cli` branch vs Issue Requirements ### What's Already Done (5 commits on `development_cli`) The `development_cli` branch has a working `hero_compute` CLI crate (`crates/hero_compute/`) that: - [x] Registers all 3 services (server, UI, explorer) as hero_proc actions under one `hero_compute` service - [x] Supports `--start` and `--stop` lifecycle flags - [x] Supports `--mode local|master|worker` with `--master-ip` for worker mode - [x] Configurable ports: `--port` (UI), `--explorer-port`, `--rpc-port` - [x] Auto-generates `.env` with mode-specific variables + symlinks to `$HOME/.env` - [x] Health checks per action (OpenRPC socket for server/explorer, TCP port for UI) - [x] Kill-other cleanup (socket + port) before starting - [x] Retry policies per action with backoff - [x] Auto-detects worker outbound IP - [x] Makefile updated: `make start [MODE=...] [MASTER_IP=...]`, `make stop`, `make status` - [x] `scripts/configure.sh` overhauled (412 lines — downloads cloud-hypervisor, mycelium, hero_proc, my_hypervisor) - [x] `docs/setup.md` updated with new CLI usage --- ### Gaps: What's Missing or Diverges from the Issue #### 1. CLI Interface: Flags vs Subcommands **Issue proposes:** ``` hero_compute start hero_compute stop hero_compute status hero_compute server serve hero_compute server logs ``` **Current implementation:** ``` hero_compute --start hero_compute --stop # No status, no per-service subcommands ``` **Decision needed:** The flag-based approach (`--start`/`--stop`) works and follows the `hero_proc_service_selfstart` pattern used by other Hero services. The subcommand approach from the issue would be more user-friendly but diverges from the ecosystem pattern. **Recommend: keep flags** — consistent with hero_proc conventions. Add `--status` flag. #### 2. Still 4 Binaries, Not 1 The issue says "single binary release" but the current implementation is an **orchestrator binary** that launches 3 separate binaries (`hero_compute_server`, `hero_compute_explorer`, `hero_compute_ui`) via hero_proc. **Why this matters:** - Release still ships 4 binaries (build target copies all 4 to `~/hero/bin/`) - `resolve_bin()` in the CLI looks for sibling binaries on disk - If any binary is missing, startup fails **True single-binary approach would require:** - Each service crate exposes a `pub async fn run(args)` in `lib.rs` - The unified binary imports and calls these directly - hero_proc actions point to `hero_compute server serve` (same binary, different subcommand) **Assessment:** This is the biggest gap. The current approach is a **multi-binary orchestrator**, not a true single binary. Making it truly single-binary requires: 1. Adding subcommands: `hero_compute server serve`, `hero_compute ui serve`, `hero_compute explorer serve` 2. Each service crate must have a `lib.rs` with a `pub async fn run()` entry point (server and explorer already have `lib.rs` stubs) 3. The unified crate depends on all 3 service crates as library dependencies 4. hero_proc actions call `hero_compute server serve` instead of `hero_compute_server` #### 3. No `hero_compute status` Command The issue lists `hero_compute status` but it's not implemented. The Makefile has a `status` target that shells out to `hero_proc status hero_compute`, which works but isn't part of the binary itself. **Recommendation:** Add `--status` flag that calls `hero_proc_sdk` to query and display service state. #### 4. No Per-Service `logs` Subcommand The issue proposes `hero_compute server logs` etc. Currently no log viewing at all from the CLI. **Recommendation:** If going subcommand route for single-binary, add `logs` subcommand per service using `hero_proc_sdk` log query API. #### 5. CI Release Not Updated The `build-linux.yaml` workflow: - Still uses `build_binaries` from `build_lib.sh` (builds all workspace binaries) - Publish step is **commented out** entirely - No artifact upload step - Doesn't produce a single downloadable binary **Needed:** Uncomment and update publish step to upload the unified binary (or all 4 if staying multi-binary). #### 6. No Deprecation Notices The issue says "Old three-binary setup deprecated but not removed until stable." The old shell scripts (`scripts/start.sh`, `scripts/stop.sh`, etc.) were **removed entirely** rather than deprecated. **Assessment:** This is fine — the old scripts were internal. The 3 individual binaries still build and have their own `main.rs`, so they can still be run standalone. No action needed. --- ### Recommended Implementation Plan #### Phase 1: True Single Binary (High Priority) 1. **Add clap subcommands** to `hero_compute` CLI: ``` hero_compute --start [--mode ...] hero_compute --stop hero_compute --status # NEW hero_compute server serve # NEW - runs server foreground hero_compute explorer serve # NEW - runs explorer foreground hero_compute ui serve [--port N] # NEW - runs UI foreground hero_compute server logs # NEW - view server logs hero_compute explorer logs # NEW - view explorer logs hero_compute ui logs # NEW - view UI logs ``` 2. **Wire up service entry points:** - `hero_compute_server/src/lib.rs` → expose `pub async fn serve(tcp_port: Option<u16>)` - `hero_compute_explorer/src/lib.rs` → expose `pub async fn serve(tcp_port: Option<u16>)` - `hero_compute_ui/src/lib.rs` → expose `pub async fn serve(port: u16, server_socket: Option<String>, explorer_socket: Option<String>)` - Extract `main.rs` logic into these `serve()` functions 3. **Add library dependencies** to `crates/hero_compute/Cargo.toml`: ```toml hero_compute_server = { path = "../hero_compute_server" } hero_compute_explorer = { path = "../hero_compute_explorer" } hero_compute_ui = { path = "../hero_compute_ui" } ``` 4. **Update hero_proc action commands** to use self-dispatch: ```rust // Instead of: let server_bin = resolve_bin(&bin_dir, "hero_compute_server"); // Use: let self_bin = std::env::current_exe()?; let server_cmd = format!("{} server serve", self_bin.display()); ``` #### Phase 2: Status & Logs 5. **Implement `--status`** — query hero_proc for `hero_compute` service state, display formatted table 6. **Implement `{service} logs`** — query hero_proc log API for each action's logs #### Phase 3: CI & Release 7. **Update `build-linux.yaml`:** - Build only the `hero_compute` binary (not all 4) - Uncomment and fix publish step - Upload single artifact per architecture 8. **Update Makefile `build` target** to only copy `hero_compute` binary #### Phase 4: Cleanup 9. Remove individual binary `[[bin]]` sections from service Cargo.tomls (keep lib only) 10. Update `docs/setup.md` to reflect single binary 11. Update `scripts/configure.sh` if needed --- ### Updated Definition of Done - [ ] `hero_compute` binary is the **only** binary needed (embeds server/explorer/ui) - [ ] `hero_compute --start/--stop/--status` manages all services - [ ] `hero_compute {service} serve` runs individual service foreground (for hero_proc) - [ ] `hero_compute {service} logs` shows per-service logs - [ ] hero_proc actions point to `hero_compute {service} serve` (self-dispatch) - [ ] Makefile `build` target produces one binary - [ ] `make start` calls `hero_compute --start` - [ ] CI releases `hero_compute` as single artifact (x86_64 + aarch64) - [ ] `docs/setup.md` documents single-binary install - [ ] Old individual binaries can optionally still build for development
Author
Owner

Decision: Keep Multi-Binary Orchestrator

After review, the approach is:

  1. Keep multi-binary orchestrator — the hero_compute CLI manages 3 separate service binaries via hero_proc. No need to embed all services into a single binary.
  2. Keep --start/--stop flags — follows hero_proc_service_selfstart pattern, consistent with the Hero ecosystem.
  3. Add --status flag — the only remaining feature gap.

The development_cli branch already has most of the work done. Remaining:

  • Add --status flag (query hero_proc SDK, display service state)
  • Update CI release workflow
  • Final docs polish
## Decision: Keep Multi-Binary Orchestrator After review, the approach is: 1. **Keep multi-binary orchestrator** — the `hero_compute` CLI manages 3 separate service binaries via hero_proc. No need to embed all services into a single binary. 2. **Keep `--start`/`--stop` flags** — follows `hero_proc_service_selfstart` pattern, consistent with the Hero ecosystem. 3. **Add `--status` flag** — the only remaining feature gap. The `development_cli` branch already has most of the work done. Remaining: - [ ] Add `--status` flag (query hero_proc SDK, display service state) - [ ] Update CI release workflow - [ ] Final docs polish
Author
Owner

Implementation committed: d797d18

Browse: d797d18
Branch: development_cli

Changes:

  • crates/hero_compute/src/main.rs — added --status flag with self_status() (queries hero_proc for service state + per-action details)
  • buildenv.sh — added hero_compute to BINARIES list
  • .forgejo/workflows/build-linux.yaml — uncommented publish step, gated on tag push
  • Makefilemake status now uses hero_compute --status
Implementation committed: `d797d18` Browse: https://forge.ourworld.tf/lhumina_code/hero_compute/commit/d797d18 Branch: `development_cli` Changes: - `crates/hero_compute/src/main.rs` — added `--status` flag with `self_status()` (queries hero_proc for service state + per-action details) - `buildenv.sh` — added `hero_compute` to BINARIES list - `.forgejo/workflows/build-linux.yaml` — uncommented publish step, gated on tag push - `Makefile` — `make status` now uses `hero_compute --status`
Sign in to join this conversation.
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/hero_compute#45
No description provided.