[Linux]: Deploy VM fails with 'Permission denied (os error 13)' — my_hypervisor VM state dir owned by root #98

New issue

Closed

opened 2026-04-21 12:21:23 +00:00 by rawan · 5 comments

rawan commented 2026-04-21 12:21:23 +00:00

Member

Copy link

Summary

Deploying a VM through the hero_compute UI (or directly via ComputeService.deploy_vm RPC) fails instantly. Every deploy_vm job ends in phase=failed, exit_code=1 and the VM state ends up as error.

Reproduction

1. Start hero_proc, hero_router, hero_compute_server, hero_compute_ui (they are all running in my environment).
2. Call the RPC directly:

   curl -s -X POST http://127.0.0.1:9001/rpc \
     -H 'Content-Type: application/json' \
     -d '{"jsonrpc":"2.0","id":1,"method":"ComputeService.deploy_vm","params":{"name":"test-vm-deploy","slice_count":1,"secret":"test","image":"ubuntu-24.04","ssh_keys":[]}}'
   

3. Initial response looks fine (state provisioning, job id assigned).
4. A few seconds later, list_vms shows:

   state: "error"
   log_status: "failed"
   logs: [
     ...,
     "Provisioning via hero_proc job 100",
     "ERROR: exited with code 1",
     "  > Permission denied (os error 13)",
     "  > Caused by:",
     "  > Error: IO error: Permission denied (os error 13)"
   ]
   

Out of all past deploy_vm jobs in my environment (ids 72, 73, 82, 83, 96–100), every single one has phase: failed, exit_code: 1 — the feature has been broken end-to-end.

Root cause

The job built by crates/hero_compute_server/src/cloud/rpc.rs:1403-1407 is:

my_hypervisor create --name <vm_sid> --cpus N --memory M --storage-quota XG --disk-size XG <image> && \
my_hypervisor start <vm_sid>

hero_proc runs this as the rawan user, but my_hypervisor immediately tries to take a file lock on every existing VM state directory under ~/.my_hypervisor/vms/<id>/state.lock, and those directories are owned by root:root:

$ ls -la ~/.my_hypervisor/vms/ | head -5
drwxr-xr-x 30 root root 4096 Apr 21 12:22 .
drwxr-xr-x  8 root root 4096 Mar 24 12:17 ..
drwxr-xr-x  2 root root 4096 Mar 24 12:18 096bc9b35870
...

$ ls -la ~/.my_hypervisor/vms/096bc9b35870/
-rw-r--r--  1 root root    0 Mar 24 12:18 state.lock

strace confirms the call that fails:

open("/home/rawan/.my_hypervisor/vms/096bc9b35870/state.lock",
     O_RDWR|O_CREAT|O_LARGEFILE|O_CLOEXEC, 0666) = -1 EACCES (Permission denied)

Running the same my_hypervisor create … or even my_hypervisor list from the rawan shell reproduces the same error. my_hypervisor doctor still reports “All checks passed”, which masks the problem.

Impact

• Deploy VM is completely non-functional on this node — the UI’s “Deploy VM” button silently produces a VM stuck in error with an opaque message.
• The error surfaces in the UI as Permission denied (os error 13) with no indication that the cause is a filesystem ownership mismatch in ~/.my_hypervisor/vms/.
• This is easy to get into: any earlier my_hypervisor invocation via sudo/doas leaves root-owned state dirs behind, and subsequent user-mode runs are broken forever.

Suggested fix (for discussion)

Several options, not mutually exclusive:

1. hero_compute side — fail fast with a useful error. Before building the deploy job, probe my_hypervisor list (or stat ~/.my_hypervisor/vms/*) and surface a specific error like my_hypervisor VM state dir is not writable by the current user — check ownership of ~/.my_hypervisor/vms/ instead of relaying the raw os error 13.
2. Deploy path — run my_hypervisor under the same user consistently. Document in the hero_compute install/setup scripts that my_hypervisor must always be invoked as the hero_compute service user, and extend scripts/configure.sh / setup to chown -R the ~/.my_hypervisor/ tree to that user.
3. my_hypervisor side (separate repo, but worth linking): doctor should also check that ~/.my_hypervisor/vms/**/state.lock are writable by the current user, and list should skip (not hard-fail on) directories it can’t lock.

Environment

• hero_compute: branch development, commit 13cac45
• hero_compute_server binary: target/x86_64-unknown-linux-musl/release/hero_compute_server
• hero_compute_ui port: 9001
• hero_proc socket: /home/rawan/hero/var/sockets/hero_proc/rpc.sock
• my_hypervisor: 0.1.5 (6a74899-dirty), binary at /home/rawan/.cargo/bin/my_hypervisor
• OS: Linux 6.17.0-14-generic

## Summary Deploying a VM through the hero_compute UI (or directly via `ComputeService.deploy_vm` RPC) fails instantly. Every `deploy_vm` job ends in `phase=failed, exit_code=1` and the VM state ends up as `error`. ## Reproduction 1. Start hero_proc, hero_router, hero_compute_server, hero_compute_ui (they are all running in my environment). 2. Call the RPC directly: ```bash curl -s -X POST http://127.0.0.1:9001/rpc \ -H 'Content-Type: application/json' \ -d '{"jsonrpc":"2.0","id":1,"method":"ComputeService.deploy_vm","params":{"name":"test-vm-deploy","slice_count":1,"secret":"test","image":"ubuntu-24.04","ssh_keys":[]}}' ``` 3. Initial response looks fine (state `provisioning`, job id assigned). 4. A few seconds later, `list_vms` shows: ``` state: "error" log_status: "failed" logs: [ ..., "Provisioning via hero_proc job 100", "ERROR: exited with code 1", " > Permission denied (os error 13)", " > Caused by:", " > Error: IO error: Permission denied (os error 13)" ] ``` Out of all past `deploy_vm` jobs in my environment (ids 72, 73, 82, 83, 96–100), every single one has `phase: failed, exit_code: 1` — the feature has been broken end-to-end. ## Root cause The job built by [crates/hero_compute_server/src/cloud/rpc.rs:1403-1407](crates/hero_compute_server/src/cloud/rpc.rs#L1403-L1407) is: ``` my_hypervisor create --name <vm_sid> --cpus N --memory M --storage-quota XG --disk-size XG <image> && \ my_hypervisor start <vm_sid> ``` hero_proc runs this as the `rawan` user, but `my_hypervisor` immediately tries to take a file lock on **every existing VM state directory** under `~/.my_hypervisor/vms/<id>/state.lock`, and those directories are owned by `root:root`: ``` $ ls -la ~/.my_hypervisor/vms/ | head -5 drwxr-xr-x 30 root root 4096 Apr 21 12:22 . drwxr-xr-x 8 root root 4096 Mar 24 12:17 .. drwxr-xr-x 2 root root 4096 Mar 24 12:18 096bc9b35870 ... $ ls -la ~/.my_hypervisor/vms/096bc9b35870/ -rw-r--r-- 1 root root 0 Mar 24 12:18 state.lock ``` strace confirms the call that fails: ``` open("/home/rawan/.my_hypervisor/vms/096bc9b35870/state.lock", O_RDWR|O_CREAT|O_LARGEFILE|O_CLOEXEC, 0666) = -1 EACCES (Permission denied) ``` Running the same `my_hypervisor create …` or even `my_hypervisor list` from the `rawan` shell reproduces the same error. `my_hypervisor doctor` still reports “All checks passed”, which masks the problem. ## Impact - Deploy VM is completely non-functional on this node — the UI’s “Deploy VM” button silently produces a VM stuck in `error` with an opaque message. - The error surfaces in the UI as `Permission denied (os error 13)` with no indication that the cause is a filesystem ownership mismatch in `~/.my_hypervisor/vms/`. - This is easy to get into: any earlier `my_hypervisor` invocation via `sudo`/`doas` leaves root-owned state dirs behind, and subsequent user-mode runs are broken forever. ## Suggested fix (for discussion) Several options, not mutually exclusive: 1. **hero_compute side — fail fast with a useful error.** Before building the deploy job, probe `my_hypervisor list` (or `stat ~/.my_hypervisor/vms/*`) and surface a specific error like `my_hypervisor VM state dir is not writable by the current user — check ownership of ~/.my_hypervisor/vms/` instead of relaying the raw `os error 13`. 2. **Deploy path — run `my_hypervisor` under the same user consistently.** Document in the hero_compute install/setup scripts that `my_hypervisor` must always be invoked as the hero_compute service user, and extend `scripts/configure.sh` / `setup` to `chown -R` the `~/.my_hypervisor/` tree to that user. 3. **my_hypervisor side** (separate repo, but worth linking): `doctor` should also check that `~/.my_hypervisor/vms/**/state.lock` are writable by the current user, and `list` should skip (not hard-fail on) directories it can’t lock. ## Environment - hero_compute: branch `development`, commit `13cac45` - hero_compute_server binary: `target/x86_64-unknown-linux-musl/release/hero_compute_server` - hero_compute_ui port: 9001 - hero_proc socket: `/home/rawan/hero/var/sockets/hero_proc/rpc.sock` - my_hypervisor: 0.1.5 (6a74899-dirty), binary at `/home/rawan/.cargo/bin/my_hypervisor` - OS: Linux 6.17.0-14-generic

rawan changed title from Deploy VM fails with 'Permission denied (os error 13)' — my_hypervisor VM state dir owned by root to [Linux]: Deploy VM fails with 'Permission denied (os error 13)' — my_hypervisor VM state dir owned by root 2026-04-21 12:22:17 +00:00

rawan self-assigned this 2026-04-21 12:22:25 +00:00

rawan commented 2026-04-21 14:01:02 +00:00

Author

Member

Copy link

Implementation Specification: Pre-flight my_hypervisor Probe for deploy_vm

Issue: #98 — [Linux]: Deploy VM fails with 'Permission denied (os error 13)' — my_hypervisor VM state dir owned by root
Repository: hero_compute
Scope: hero_compute-side fix only (suggestion #1 from the issue, plus trivial setup doc/script touch-ups from #2). Suggestion #3 (my_hypervisor doctor / list robustness) lives in a different repo and is out of scope.

---

Objective

When ComputeService.deploy_vm is called, run a short, bounded pre-flight probe of my_hypervisor list before enqueueing the hero_proc deploy job. If the probe fails with a permission-related error (typical symptom: root-owned state dirs under ~/.my_hypervisor/vms/ that the service user cannot lock), return a specific, user-friendly error to the RPC caller instead of letting the deploy job fail later with an opaque Permission denied (os error 13). For healthy systems the probe must be cheap enough not to noticeably slow deployments.

---

Requirements

• Add a pre-flight probe that invokes my_hypervisor list (or an equivalent quick-exit subcommand) with a bounded timeout (default 3 seconds).
• Run the probe before any slice allocation / VM record creation in deploy_vm, so a failed probe leaves no stray VM(error) records or InUse slices behind.
• Detect permission-class failures: exit code non-zero AND (stderr OR stdout) contains any of Permission denied, os error 13, or EACCES.
• On permission failure, return ComputeServiceError::Internal(message) where message is a specific, actionable string that includes the ~/.my_hypervisor/vms/ path (resolved from $HOME at runtime when possible). Example message:
  my_hypervisor VM state dir is not writable by the current user — check ownership of /home/<user>/.my_hypervisor/vms/ (often caused by a prior sudo/doas invocation). Run: sudo chown -R <user>:<user> /home/<user>/.my_hypervisor/
• On non-permission probe failures (missing binary, generic error, timeout), emit a warning in tracing and allow the deploy to proceed. The original failure surface (hero_proc job stderr → vm_log_fail) remains the source of truth for those cases. This keeps the probe strictly additive for diagnosability.
• Fast-path when probe succeeds: zero additional RPC or filesystem operations beyond the single subprocess invocation — no caching required, but the probe must not block other deploys (the deploy_lock mutex is acquired after the probe).
• Skip the probe entirely when running under StubDriver (non-Linux / driver init failed). In that environment my_hypervisor is not on PATH and the existing stub error path is preserved.
• Gate the probe behind an opt-out env var HERO_COMPUTE_SKIP_HYPERVISOR_PROBE (set to 1 to skip). Useful for CI and for users whose my_hypervisor binary lives outside $PATH.
• Post-enqueue vm_log_fail messaging for jobs that fail after enqueue remains unchanged (so running VMs' error log format is stable).
• Add setup hygiene: scripts/configure.sh (Step 6, install_my_hypervisor) chown -R "$SUDO_USER:$SUDO_USER" "$HOME/.my_hypervisor" when running under sudo, to prevent root-ownership drift across runs.
• Add a unit test that exercises the permission-detection helper with synthetic probe output and asserts the friendly error text.

---

Files to Modify / Create

Modify

• crates/hero_compute_server/src/cloud/rpc.rs

  • Add a small hypervisor_probe submodule (inline or #[path]-referenced) exposing:
    • fn probe_my_hypervisor(timeout: Duration) -> ProbeResult
    • enum ProbeResult { Ok, PermissionDenied { path: String, raw: String }, OtherFailure(String), BinaryMissing, Skipped, Timeout }
    • fn classify_probe_output(exit_code: i32, stdout: &str, stderr: &str) -> ProbeResult — pure, testable.
    • fn hypervisor_state_dir() -> String — returns $HOME/.my_hypervisor/vms/ with fallback.
  • In deploy_vm (around line 1253, before let _lock = self.deploy_lock.lock()...), insert:
    1. Early-return Ok if self.hypervisor is StubDriver (use a downcast_ref or add a marker method is_stub() on the HypervisorDriver trait).
    2. Early-return Ok if HERO_COMPUTE_SKIP_HYPERVISOR_PROBE=1.
    3. Call probe_my_hypervisor(Duration::from_secs(3)).
    4. On ProbeResult::PermissionDenied { path, .. }: tracing::error! + return ComputeServiceError::Internal(format!("my_hypervisor VM state dir is not writable by the current user — check ownership of {} (often caused by a prior sudo/doas invocation). Run: sudo chown -R $USER:$USER {}", path, parent_of(path))).
    5. On any other non-Ok variant: tracing::warn! and continue (deploy proceeds as today).

• crates/hero_compute_server/src/cloud/constants.rs

  • Add pub const HYPERVISOR_PROBE_TIMEOUT_SECS: u64 = 3;
  • Add pub const HYPERVISOR_PROBE_SKIP_ENV: &str = "HERO_COMPUTE_SKIP_HYPERVISOR_PROBE";

• crates/hero_compute_server/src/cloud/hypervisor.rs

  • Extend HypervisorDriver trait with a default fn is_stub(&self) -> bool { false }; override StubDriver::is_stub to return true. This lets deploy_vm skip the probe cleanly without adding conditional compilation.

• crates/hero_compute_server/src/cloud/tests.rs

  • Append a #[test] fn test_classify_probe_output_permission_denied() that feeds synthetic (exit_code=1, stderr="Error: IO error: Permission denied (os error 13)") into classify_probe_output and asserts it returns ProbeResult::PermissionDenied { .. }. Add a second case for Ok (exit 0) and a third for OtherFailure (exit 1 with an unrelated stderr). Tests must go through the super::server::* re-export path used by the existing tests in that file.

• scripts/configure.sh

  • In install_my_hypervisor (around line 312), after mkdir -p "$HOME/.cargo/bin" "$HOME/.my_hypervisor/bin" and again at the end of check_hypervisor_deps, add a chown_hypervisor_dirs helper that, when SUDO_USER is set and non-empty, runs chown -R "$SUDO_USER:$SUDO_USER" "/home/$SUDO_USER/.my_hypervisor" 2>/dev/null || true. Emits ok "Corrected ownership of ~/.my_hypervisor for $SUDO_USER".

Create

No new files. The probe module is kept inline inside rpc.rs (pattern matches the existing hypervisor_probe peers there: hardware.rs, hero_proc_jobs.rs, stats.rs are separate files, but the probe is small enough — ~80 lines — that inline keeps the diff tight and avoids a new #[path] include).

Optional (only if the reviewer prefers a dedicated file): crates/hero_compute_server/src/cloud/hypervisor_probe.rs wired in via #[path = "hypervisor_probe.rs"] mod hypervisor_probe; at the top of rpc.rs next to the other mod declarations (line 14–25). In that case, tests.rs imports via use super::server::rpc::hypervisor_probe::*;.

---

Step-by-Step Implementation Plan

Step 1 — Extend the HypervisorDriver trait with is_stub()

Files: crates/hero_compute_server/src/cloud/hypervisor.rs
Depends on: —

• Add fn is_stub(&self) -> bool { false } to the HypervisorDriver trait with a default impl.
• Override in impl HypervisorDriver for StubDriver to return true.
• No changes needed to MyHypervisorDriver (the default is correct).
• Runs standalone: cargo check -p hero_compute_server must pass.

Step 2 — Add probe constants

Files: crates/hero_compute_server/src/cloud/constants.rs
Depends on: —

• Add HYPERVISOR_PROBE_TIMEOUT_SECS and HYPERVISOR_PROBE_SKIP_ENV constants documented with the same comment style as existing entries.
• Runs in parallel with Step 1.

Step 3 — Add the probe module (classification + subprocess)

Files: crates/hero_compute_server/src/cloud/rpc.rs
Depends on: Step 2

• Add a mod hypervisor_probe { ... } block near the top of rpc.rs (after the existing thread_local! / helper definitions, before ComputeServiceHandler). It exposes:
  • pub enum ProbeResult { Ok, PermissionDenied { path: String, raw: String }, OtherFailure(String), BinaryMissing, Timeout, Skipped }
  • pub fn hypervisor_state_dir() -> String — reads $HOME, returns format!("{}/.my_hypervisor/vms/", home); falls back to "~/.my_hypervisor/vms/" if $HOME is unset.
  • pub fn classify_probe_output(exit_code: i32, stdout: &str, stderr: &str) -> ProbeResult — pure function. Returns Ok on exit_code 0; returns PermissionDenied when exit_code != 0 AND the concatenated stdout + stderr (case-insensitive) contains any of "permission denied", "os error 13", "eacces". Otherwise OtherFailure(stderr.trim().to_string()).
  • pub fn probe_my_hypervisor(timeout: Duration) -> ProbeResult — uses std::process::Command::new("my_hypervisor").arg("list"), matching the pattern at rpc.rs:947–972. Implements timeout via Command::spawn() + a watchdog thread that calls child.kill() if elapsed > timeout (simple, no extra deps — mirrors how bash -c 'timeout 5 …' is done in the health check). On ErrorKind::NotFound, return BinaryMissing. On timeout, return Timeout.
• Keep the whole module <120 lines and self-contained.

Step 4 — Wire the probe into deploy_vm

Files: crates/hero_compute_server/src/cloud/rpc.rs
Depends on: Step 1, Step 3

• In fn deploy_vm (line 1197), immediately after the input-validation block that ends at line 1252 and before let _lock = self.deploy_lock.lock()... at line 1256, insert the probe:

  if !self.hypervisor.is_stub() && std::env::var(constants::HYPERVISOR_PROBE_SKIP_ENV).ok().as_deref() != Some("1") {
      match hypervisor_probe::probe_my_hypervisor(
          Duration::from_secs(constants::HYPERVISOR_PROBE_TIMEOUT_SECS)
      ) {
          hypervisor_probe::ProbeResult::PermissionDenied { path, raw } => {
              tracing::error!("deploy_vm: my_hypervisor probe failed (permission): {}", raw);
              return Err(ComputeServiceError::Internal(format!(
                  "my_hypervisor VM state dir is not writable by the current user — \
                   check ownership of {path} (often caused by a prior sudo/doas \
                   invocation). Run: sudo chown -R \"$USER:$USER\" {}",
                  std::path::Path::new(&path).parent()
                      .map(|p| p.display().to_string())
                      .unwrap_or_else(|| "~/.my_hypervisor".to_string())
              )));
          }
          hypervisor_probe::ProbeResult::Ok => {}
          other => {
              tracing::warn!("deploy_vm: my_hypervisor probe non-fatal result: {:?}", other);
          }
      }
  }
  

• Because this runs before the lock + slice allocation, a failed probe returns instantly without any side-effects on OSIS state — exactly the property the issue requires.

Step 5 — Unit test for the classifier

Files: crates/hero_compute_server/src/cloud/tests.rs
Depends on: Step 3

• Add tests that call classify_probe_output directly (pure function, no I/O). Cover:
  • exit_code=0, "", "" → Ok
  • exit_code=1, "", "Error: IO error: Permission denied (os error 13)" → PermissionDenied
  • exit_code=1, "", "EACCES: ~/.my_hypervisor/vms/abc/state.lock" → PermissionDenied
  • exit_code=1, "", "No kernels found" → OtherFailure
• Also add one test for hypervisor_state_dir() that sets HOME via std::env::set_var (within a #[serial]-style single-thread block, or just assert the returned string ends with /.my_hypervisor/vms/).
• Tests import via the existing use super::server::*; pattern. The probe module must be pub(crate) visible through that path.

Step 6 — Setup script hygiene (chown)

Files: scripts/configure.sh
Depends on: —

• In install_my_hypervisor (line 312), add a trailing call to a new chown_hypervisor_dirs helper placed just below check_hypervisor_deps:

  chown_hypervisor_dirs() {
    if [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != "root" ]; then
      local target_home
      target_home=$(getent passwd "$SUDO_USER" | cut -d: -f6)
      if [ -d "$target_home/.my_hypervisor" ]; then
        chown -R "$SUDO_USER:$SUDO_USER" "$target_home/.my_hypervisor" 2>/dev/null || true
        ok "Corrected ownership of $target_home/.my_hypervisor for $SUDO_USER"
      fi
    fi
  }
  

• Call it at the end of install_my_hypervisor, after check_hypervisor_deps.
• Runs in parallel with all code steps.

Step 7 — Manual verification (scripted smoke)

Depends on: Steps 1-5

• Build the workspace: cargo build --workspace from the repo root.
• Reproduce the bug: sudo my_hypervisor list once to create a root-owned state dir, then rm -rf ~/.my_hypervisor/vms/* && sudo mkdir -p ~/.my_hypervisor/vms/junk && sudo chown root:root ~/.my_hypervisor/vms/junk && sudo touch ~/.my_hypervisor/vms/junk/state.lock && sudo chown root:root ~/.my_hypervisor/vms/junk/state.lock.
• Call ComputeService.deploy_vm via the same curl from the issue. Expect the JSON-RPC response to contain the new friendly error message and no new Vm record in error state.
• Fix ownership (sudo chown -R $USER:$USER ~/.my_hypervisor) and rerun; expect the deploy to start normally.

---

Acceptance Criteria

• HypervisorDriver::is_stub() exists and returns true for StubDriver, false for MyHypervisorDriver.
• hypervisor_probe::classify_probe_output is a pure function that maps (exit_code, stdout, stderr) to a ProbeResult.
• classify_probe_output returns PermissionDenied for all of: "Permission denied (os error 13)", "Error: IO error: Permission denied (os error 13)", "EACCES"; returns Ok on exit_code 0; returns OtherFailure for other non-zero exits.
• probe_my_hypervisor respects HYPERVISOR_PROBE_TIMEOUT_SECS and kills the child on timeout.
• deploy_vm calls the probe before acquiring deploy_lock, allocating slices, or creating a Vm record.
• A permission-class probe failure returns ComputeServiceError::Internal whose message contains the literal substring my_hypervisor VM state dir is not writable and the resolved ~/.my_hypervisor/vms/ path.
• A permission-class probe failure produces no new Vm rows, no InUse slices, no hero_proc jobs.
• HERO_COMPUTE_SKIP_HYPERVISOR_PROBE=1 short-circuits the probe and restores the pre-change deploy_vm behaviour exactly.
• When the hypervisor driver is StubDriver (e.g. on macOS dev machines), the probe is skipped.
• Non-permission probe failures (binary missing, timeout, generic error) do not block the deploy — they only emit a tracing::warn!.
• cargo test -p hero_compute_server --features cloud passes, including the three new classify_probe_output cases.
• Post-enqueue failure path (hero_proc job returns non-zero) still uses vm_log_fail with identical wording to today (no behavioural change for deploys that succeed the probe but fail later).
• scripts/configure.sh runs chown -R $SUDO_USER:$SUDO_USER ~/.my_hypervisor when executed under sudo by a non-root user.
• The healthy-system fast path adds at most one my_hypervisor list subprocess call (~tens of milliseconds) per deploy_vm invocation.

---

Notes

• Why Internal and not a new variant? ComputeServiceError in crates/hero_compute_server/src/cloud/rpc_generated.rs is generated from the cloud oschema; adding a new variant there would require regenerating the schema and all clients. Internal(String) is already used for analogous "server-side precondition failed" conditions (e.g. "No node registered. Call node_register first." at rpc.rs:1303) and the UI surfaces its payload verbatim. If a future iteration wants a typed PreflightFailed variant, that is a separate schema-regen change outside this fix.
• Why probe my_hypervisor list rather than stat the directory? list exercises the exact same state.lock acquisition path that create/start use, so it catches precisely the failure mode from the issue. A stat on the directory would miss permission problems on per-VM state.lock files.
• Timeout implementation. Keep it dependency-free: Command::spawn() + a background thread::spawn(move || { sleep(timeout); let _ = child.kill(); }) pattern. No tokio::process needed because the trait method is synchronous. This matches the style already in use in rpc.rs:947–972.
• Out of scope (explicit):
  • my_hypervisor-side fixes (doctor check, list skipping unreadable dirs) — tracked in the my_hypervisor repo per issue suggestion #3.
  • Adding a new typed error variant to ComputeServiceError or the generated cloud schema.
  • Changes to start_vm / stop_vm / restart_vm / delete_vm — they share the same underlying hazard, but the issue specifically scopes the fix to deploy_vm. A follow-up issue should consider hoisting the probe into a shared preflight_hypervisor() helper reused across all four entry points.
  • Caching probe results across requests. Probe cost is low enough (~30–80 ms on a warm system) that caching is premature; revisit only if benchmarks show otherwise.
  • UI-side changes in hero_compute_ui — the new error message will surface automatically because the UI already renders error.message on RPC failure.
• Regression risk. The probe runs only on deploy_vm. It adds one subprocess exec per call on the healthy path and zero persistent state. The only observable behavioural change for a healthy system is +30-80ms latency on deploy_vm. All other RPCs are untouched.

# Implementation Specification: Pre-flight `my_hypervisor` Probe for `deploy_vm` **Issue:** #98 — [Linux]: Deploy VM fails with 'Permission denied (os error 13)' — my_hypervisor VM state dir owned by root **Repository:** `hero_compute` **Scope:** hero_compute-side fix only (suggestion #1 from the issue, plus trivial setup doc/script touch-ups from #2). Suggestion #3 (my_hypervisor `doctor` / `list` robustness) lives in a different repo and is **out of scope**. --- ## Objective When `ComputeService.deploy_vm` is called, run a short, bounded pre-flight probe of `my_hypervisor list` **before** enqueueing the hero_proc deploy job. If the probe fails with a permission-related error (typical symptom: root-owned state dirs under `~/.my_hypervisor/vms/` that the service user cannot lock), return a specific, user-friendly error to the RPC caller instead of letting the deploy job fail later with an opaque `Permission denied (os error 13)`. For healthy systems the probe must be cheap enough not to noticeably slow deployments. --- ## Requirements - Add a pre-flight probe that invokes `my_hypervisor list` (or an equivalent quick-exit subcommand) with a bounded timeout (default 3 seconds). - Run the probe **before** any slice allocation / VM record creation in `deploy_vm`, so a failed probe leaves no stray `VM(error)` records or `InUse` slices behind. - Detect permission-class failures: exit code non-zero AND (stderr OR stdout) contains any of `Permission denied`, `os error 13`, or `EACCES`. - On permission failure, return `ComputeServiceError::Internal(message)` where `message` is a specific, actionable string that includes the `~/.my_hypervisor/vms/` path (resolved from `$HOME` at runtime when possible). Example message: `my_hypervisor VM state dir is not writable by the current user — check ownership of /home/<user>/.my_hypervisor/vms/ (often caused by a prior sudo/doas invocation). Run: sudo chown -R <user>:<user> /home/<user>/.my_hypervisor/` - On non-permission probe failures (missing binary, generic error, timeout), emit a warning in tracing and **allow** the deploy to proceed. The original failure surface (hero_proc job stderr → `vm_log_fail`) remains the source of truth for those cases. This keeps the probe strictly additive for diagnosability. - Fast-path when probe succeeds: zero additional RPC or filesystem operations beyond the single subprocess invocation — no caching required, but the probe must not block other deploys (the `deploy_lock` mutex is acquired *after* the probe). - Skip the probe entirely when running under `StubDriver` (non-Linux / driver init failed). In that environment `my_hypervisor` is not on PATH and the existing stub error path is preserved. - Gate the probe behind an opt-out env var `HERO_COMPUTE_SKIP_HYPERVISOR_PROBE` (set to `1` to skip). Useful for CI and for users whose `my_hypervisor` binary lives outside `$PATH`. - Post-enqueue `vm_log_fail` messaging for jobs that fail *after* enqueue remains unchanged (so running VMs' error log format is stable). - Add setup hygiene: `scripts/configure.sh` (Step 6, `install_my_hypervisor`) `chown -R "$SUDO_USER:$SUDO_USER" "$HOME/.my_hypervisor"` when running under sudo, to prevent root-ownership drift across runs. - Add a unit test that exercises the permission-detection helper with synthetic probe output and asserts the friendly error text. --- ## Files to Modify / Create ### Modify - `crates/hero_compute_server/src/cloud/rpc.rs` - Add a small `hypervisor_probe` submodule (inline or `#[path]`-referenced) exposing: - `fn probe_my_hypervisor(timeout: Duration) -> ProbeResult` - `enum ProbeResult { Ok, PermissionDenied { path: String, raw: String }, OtherFailure(String), BinaryMissing, Skipped, Timeout }` - `fn classify_probe_output(exit_code: i32, stdout: &str, stderr: &str) -> ProbeResult` — pure, testable. - `fn hypervisor_state_dir() -> String` — returns `$HOME/.my_hypervisor/vms/` with fallback. - In `deploy_vm` (around line 1253, before `let _lock = self.deploy_lock.lock()...`), insert: 1. Early-return `Ok` if `self.hypervisor` is `StubDriver` (use a `downcast_ref` or add a marker method `is_stub()` on the `HypervisorDriver` trait). 2. Early-return `Ok` if `HERO_COMPUTE_SKIP_HYPERVISOR_PROBE=1`. 3. Call `probe_my_hypervisor(Duration::from_secs(3))`. 4. On `ProbeResult::PermissionDenied { path, .. }`: `tracing::error!` + return `ComputeServiceError::Internal(format!("my_hypervisor VM state dir is not writable by the current user — check ownership of {} (often caused by a prior sudo/doas invocation). Run: sudo chown -R $USER:$USER {}", path, parent_of(path)))`. 5. On any other non-Ok variant: `tracing::warn!` and continue (deploy proceeds as today). - `crates/hero_compute_server/src/cloud/constants.rs` - Add `pub const HYPERVISOR_PROBE_TIMEOUT_SECS: u64 = 3;` - Add `pub const HYPERVISOR_PROBE_SKIP_ENV: &str = "HERO_COMPUTE_SKIP_HYPERVISOR_PROBE";` - `crates/hero_compute_server/src/cloud/hypervisor.rs` - Extend `HypervisorDriver` trait with a default `fn is_stub(&self) -> bool { false }`; override `StubDriver::is_stub` to return `true`. This lets `deploy_vm` skip the probe cleanly without adding conditional compilation. - `crates/hero_compute_server/src/cloud/tests.rs` - Append a `#[test] fn test_classify_probe_output_permission_denied()` that feeds synthetic `(exit_code=1, stderr="Error: IO error: Permission denied (os error 13)")` into `classify_probe_output` and asserts it returns `ProbeResult::PermissionDenied { .. }`. Add a second case for `Ok` (exit 0) and a third for `OtherFailure` (exit 1 with an unrelated stderr). Tests must go through the `super::server::*` re-export path used by the existing tests in that file. - `scripts/configure.sh` - In `install_my_hypervisor` (around line 312), after `mkdir -p "$HOME/.cargo/bin" "$HOME/.my_hypervisor/bin"` and again at the end of `check_hypervisor_deps`, add a `chown_hypervisor_dirs` helper that, when `SUDO_USER` is set and non-empty, runs `chown -R "$SUDO_USER:$SUDO_USER" "/home/$SUDO_USER/.my_hypervisor" 2>/dev/null || true`. Emits `ok "Corrected ownership of ~/.my_hypervisor for $SUDO_USER"`. ### Create No new files. The probe module is kept inline inside `rpc.rs` (pattern matches the existing `hypervisor_probe` peers there: `hardware.rs`, `hero_proc_jobs.rs`, `stats.rs` are separate files, but the probe is small enough — ~80 lines — that inline keeps the diff tight and avoids a new `#[path]` include). *Optional (only if the reviewer prefers a dedicated file):* `crates/hero_compute_server/src/cloud/hypervisor_probe.rs` wired in via `#[path = "hypervisor_probe.rs"] mod hypervisor_probe;` at the top of `rpc.rs` next to the other mod declarations (line 14–25). In that case, `tests.rs` imports via `use super::server::rpc::hypervisor_probe::*;`. --- ## Step-by-Step Implementation Plan ### Step 1 — Extend the `HypervisorDriver` trait with `is_stub()` **Files:** `crates/hero_compute_server/src/cloud/hypervisor.rs` **Depends on:** — - Add `fn is_stub(&self) -> bool { false }` to the `HypervisorDriver` trait with a default impl. - Override in `impl HypervisorDriver for StubDriver` to return `true`. - No changes needed to `MyHypervisorDriver` (the default is correct). - Runs standalone: `cargo check -p hero_compute_server` must pass. ### Step 2 — Add probe constants **Files:** `crates/hero_compute_server/src/cloud/constants.rs` **Depends on:** — - Add `HYPERVISOR_PROBE_TIMEOUT_SECS` and `HYPERVISOR_PROBE_SKIP_ENV` constants documented with the same comment style as existing entries. - Runs in parallel with Step 1. ### Step 3 — Add the probe module (classification + subprocess) **Files:** `crates/hero_compute_server/src/cloud/rpc.rs` **Depends on:** Step 2 - Add a `mod hypervisor_probe { ... }` block near the top of `rpc.rs` (after the existing `thread_local!` / helper definitions, before `ComputeServiceHandler`). It exposes: - `pub enum ProbeResult { Ok, PermissionDenied { path: String, raw: String }, OtherFailure(String), BinaryMissing, Timeout, Skipped }` - `pub fn hypervisor_state_dir() -> String` — reads `$HOME`, returns `format!("{}/.my_hypervisor/vms/", home)`; falls back to `"~/.my_hypervisor/vms/"` if `$HOME` is unset. - `pub fn classify_probe_output(exit_code: i32, stdout: &str, stderr: &str) -> ProbeResult` — pure function. Returns `Ok` on exit_code 0; returns `PermissionDenied` when exit_code != 0 AND the concatenated `stdout + stderr` (case-insensitive) contains any of `"permission denied"`, `"os error 13"`, `"eacces"`. Otherwise `OtherFailure(stderr.trim().to_string())`. - `pub fn probe_my_hypervisor(timeout: Duration) -> ProbeResult` — uses `std::process::Command::new("my_hypervisor").arg("list")`, matching the pattern at `rpc.rs:947–972`. Implements timeout via `Command::spawn()` + a watchdog thread that calls `child.kill()` if elapsed > timeout (simple, no extra deps — mirrors how `bash -c 'timeout 5 …'` is done in the health check). On `ErrorKind::NotFound`, return `BinaryMissing`. On timeout, return `Timeout`. - Keep the whole module <120 lines and self-contained. ### Step 4 — Wire the probe into `deploy_vm` **Files:** `crates/hero_compute_server/src/cloud/rpc.rs` **Depends on:** Step 1, Step 3 - In `fn deploy_vm` (line 1197), immediately after the input-validation block that ends at line 1252 and **before** `let _lock = self.deploy_lock.lock()...` at line 1256, insert the probe: ``` if !self.hypervisor.is_stub() && std::env::var(constants::HYPERVISOR_PROBE_SKIP_ENV).ok().as_deref() != Some("1") { match hypervisor_probe::probe_my_hypervisor( Duration::from_secs(constants::HYPERVISOR_PROBE_TIMEOUT_SECS) ) { hypervisor_probe::ProbeResult::PermissionDenied { path, raw } => { tracing::error!("deploy_vm: my_hypervisor probe failed (permission): {}", raw); return Err(ComputeServiceError::Internal(format!( "my_hypervisor VM state dir is not writable by the current user — \ check ownership of {path} (often caused by a prior sudo/doas \ invocation). Run: sudo chown -R \"$USER:$USER\" {}", std::path::Path::new(&path).parent() .map(|p| p.display().to_string()) .unwrap_or_else(|| "~/.my_hypervisor".to_string()) ))); } hypervisor_probe::ProbeResult::Ok => {} other => { tracing::warn!("deploy_vm: my_hypervisor probe non-fatal result: {:?}", other); } } } ``` - Because this runs before the lock + slice allocation, a failed probe returns instantly without any side-effects on OSIS state — exactly the property the issue requires. ### Step 5 — Unit test for the classifier **Files:** `crates/hero_compute_server/src/cloud/tests.rs` **Depends on:** Step 3 - Add tests that call `classify_probe_output` directly (pure function, no I/O). Cover: - `exit_code=0, "", ""` → `Ok` - `exit_code=1, "", "Error: IO error: Permission denied (os error 13)"` → `PermissionDenied` - `exit_code=1, "", "EACCES: ~/.my_hypervisor/vms/abc/state.lock"` → `PermissionDenied` - `exit_code=1, "", "No kernels found"` → `OtherFailure` - Also add one test for `hypervisor_state_dir()` that sets `HOME` via `std::env::set_var` (within a `#[serial]`-style single-thread block, or just assert the returned string ends with `/.my_hypervisor/vms/`). - Tests import via the existing `use super::server::*;` pattern. The probe module must be `pub(crate)` visible through that path. ### Step 6 — Setup script hygiene (chown) **Files:** `scripts/configure.sh` **Depends on:** — - In `install_my_hypervisor` (line 312), add a trailing call to a new `chown_hypervisor_dirs` helper placed just below `check_hypervisor_deps`: ``` chown_hypervisor_dirs() { if [ -n "${SUDO_USER:-}" ] && [ "$SUDO_USER" != "root" ]; then local target_home target_home=$(getent passwd "$SUDO_USER" | cut -d: -f6) if [ -d "$target_home/.my_hypervisor" ]; then chown -R "$SUDO_USER:$SUDO_USER" "$target_home/.my_hypervisor" 2>/dev/null || true ok "Corrected ownership of $target_home/.my_hypervisor for $SUDO_USER" fi fi } ``` - Call it at the end of `install_my_hypervisor`, after `check_hypervisor_deps`. - Runs in parallel with all code steps. ### Step 7 — Manual verification (scripted smoke) **Depends on:** Steps 1-5 - Build the workspace: `cargo build --workspace` from the repo root. - Reproduce the bug: `sudo my_hypervisor list` once to create a root-owned state dir, then `rm -rf ~/.my_hypervisor/vms/* && sudo mkdir -p ~/.my_hypervisor/vms/junk && sudo chown root:root ~/.my_hypervisor/vms/junk && sudo touch ~/.my_hypervisor/vms/junk/state.lock && sudo chown root:root ~/.my_hypervisor/vms/junk/state.lock`. - Call `ComputeService.deploy_vm` via the same curl from the issue. Expect the JSON-RPC response to contain the new friendly error message and **no** new `Vm` record in `error` state. - Fix ownership (`sudo chown -R $USER:$USER ~/.my_hypervisor`) and rerun; expect the deploy to start normally. --- ## Acceptance Criteria - [ ] `HypervisorDriver::is_stub()` exists and returns `true` for `StubDriver`, `false` for `MyHypervisorDriver`. - [ ] `hypervisor_probe::classify_probe_output` is a pure function that maps `(exit_code, stdout, stderr)` to a `ProbeResult`. - [ ] `classify_probe_output` returns `PermissionDenied` for all of: `"Permission denied (os error 13)"`, `"Error: IO error: Permission denied (os error 13)"`, `"EACCES"`; returns `Ok` on exit_code 0; returns `OtherFailure` for other non-zero exits. - [ ] `probe_my_hypervisor` respects `HYPERVISOR_PROBE_TIMEOUT_SECS` and kills the child on timeout. - [ ] `deploy_vm` calls the probe **before** acquiring `deploy_lock`, allocating slices, or creating a `Vm` record. - [ ] A permission-class probe failure returns `ComputeServiceError::Internal` whose message contains the literal substring `my_hypervisor VM state dir is not writable` and the resolved `~/.my_hypervisor/vms/` path. - [ ] A permission-class probe failure produces no new `Vm` rows, no `InUse` slices, no hero_proc jobs. - [ ] `HERO_COMPUTE_SKIP_HYPERVISOR_PROBE=1` short-circuits the probe and restores the pre-change deploy_vm behaviour exactly. - [ ] When the hypervisor driver is `StubDriver` (e.g. on macOS dev machines), the probe is skipped. - [ ] Non-permission probe failures (binary missing, timeout, generic error) do **not** block the deploy — they only emit a `tracing::warn!`. - [ ] `cargo test -p hero_compute_server --features cloud` passes, including the three new `classify_probe_output` cases. - [ ] Post-enqueue failure path (hero_proc job returns non-zero) still uses `vm_log_fail` with identical wording to today (no behavioural change for deploys that succeed the probe but fail later). - [ ] `scripts/configure.sh` runs `chown -R $SUDO_USER:$SUDO_USER ~/.my_hypervisor` when executed under sudo by a non-root user. - [ ] The healthy-system fast path adds at most one `my_hypervisor list` subprocess call (~tens of milliseconds) per `deploy_vm` invocation. --- ## Notes - **Why `Internal` and not a new variant?** `ComputeServiceError` in `crates/hero_compute_server/src/cloud/rpc_generated.rs` is generated from the cloud oschema; adding a new variant there would require regenerating the schema and all clients. `Internal(String)` is already used for analogous "server-side precondition failed" conditions (e.g. `"No node registered. Call node_register first."` at rpc.rs:1303) and the UI surfaces its payload verbatim. If a future iteration wants a typed `PreflightFailed` variant, that is a separate schema-regen change outside this fix. - **Why probe `my_hypervisor list` rather than stat the directory?** `list` exercises the exact same `state.lock` acquisition path that `create`/`start` use, so it catches precisely the failure mode from the issue. A `stat` on the directory would miss permission problems on per-VM `state.lock` files. - **Timeout implementation.** Keep it dependency-free: `Command::spawn()` + a background `thread::spawn(move || { sleep(timeout); let _ = child.kill(); })` pattern. No `tokio::process` needed because the trait method is synchronous. This matches the style already in use in `rpc.rs:947–972`. - **Out of scope (explicit):** - my_hypervisor-side fixes (`doctor` check, `list` skipping unreadable dirs) — tracked in the my_hypervisor repo per issue suggestion #3. - Adding a new typed error variant to `ComputeServiceError` or the generated cloud schema. - Changes to `start_vm` / `stop_vm` / `restart_vm` / `delete_vm` — they share the same underlying hazard, but the issue specifically scopes the fix to `deploy_vm`. A follow-up issue should consider hoisting the probe into a shared `preflight_hypervisor()` helper reused across all four entry points. - Caching probe results across requests. Probe cost is low enough (~30–80 ms on a warm system) that caching is premature; revisit only if benchmarks show otherwise. - UI-side changes in `hero_compute_ui` — the new error message will surface automatically because the UI already renders `error.message` on RPC failure. - **Regression risk.** The probe runs only on `deploy_vm`. It adds one subprocess exec per call on the healthy path and zero persistent state. The only observable behavioural change for a healthy system is `+30-80ms` latency on `deploy_vm`. All other RPCs are untouched.

rawan commented 2026-04-21 14:36:27 +00:00

Author

Member

Copy link

Test Results

• Total tests: 32
• Passed: 30
• Failed: 0
• Ignored: 2

New tests for pre-flight probe (all passing)

• test_classify_probe_output_ok
• test_classify_probe_output_permission_denied_os_error_13
• test_classify_probe_output_permission_denied_eacces
• test_classify_probe_output_other_failure
• test_hypervisor_state_dir_ends_with_expected_suffix

Build

cargo build --workspace — pass (no errors)

Summary

All existing tests pass, 5 new unit tests cover the probe classifier and state-dir helper.

## Test Results - Total tests: 32 - Passed: 30 - Failed: 0 - Ignored: 2 ### New tests for pre-flight probe (all passing) - `test_classify_probe_output_ok` - `test_classify_probe_output_permission_denied_os_error_13` - `test_classify_probe_output_permission_denied_eacces` - `test_classify_probe_output_other_failure` - `test_hypervisor_state_dir_ends_with_expected_suffix` ### Build `cargo build --workspace` — pass (no errors) ### Summary All existing tests pass, 5 new unit tests cover the probe classifier and state-dir helper.

rawan commented 2026-04-21 14:37:48 +00:00

Author

Member

Copy link

Implementation summary

Added a pre-flight my_hypervisor probe to ComputeService.deploy_vm so the
"Permission denied (os error 13)" failure reported in this issue now surfaces
as a specific, actionable error message instead of an opaque failure after the
hero_proc job has already been enqueued.

Changes

crates/hero_compute_server/src/cloud/hypervisor.rs

• Added fn is_stub(&self) -> bool { false } to the HypervisorDriver trait
  with a default implementation.
• Overrode it in impl HypervisorDriver for StubDriver to return true so
  the new probe is skipped when the stub driver is in use (non-Linux / driver
  init failed).

crates/hero_compute_server/src/cloud/constants.rs

• HYPERVISOR_PROBE_TIMEOUT_SECS: u64 = 3
• HYPERVISOR_PROBE_SKIP_ENV: &str = "HERO_COMPUTE_SKIP_HYPERVISOR_PROBE"

crates/hero_compute_server/src/cloud/rpc.rs

• New pub(crate) mod hypervisor_probe with:
  • ProbeResult { Ok, PermissionDenied { path, raw }, OtherFailure, BinaryMissing, Timeout, Skipped }
  • classify_probe_output(exit_code, stdout, stderr) — pure, unit-tested.
    Returns PermissionDenied when the exit code is non-zero and the
    concatenated output contains any of "permission denied", "os error 13",
    or "eacces" (case-insensitive).
  • hypervisor_state_dir() — resolves $HOME/.my_hypervisor/vms/.
  • probe_my_hypervisor(timeout) — spawns my_hypervisor list with a
    bounded wall-clock timeout and a watchdog thread that kills the child on
    timeout. Dependency-free.
• Wired the probe into ComputeServiceHandler::deploy_vm before
  deploy_lock is acquired, slices are allocated, or a VM record is created.
  On PermissionDenied the caller gets a ComputeServiceError::Internal
  whose message names the offending path and shows the exact chown command to
  fix it. Non-permission failures (binary missing, timeout, other) only emit
  a tracing::warn! and let the deploy proceed — the existing post-enqueue
  failure path is untouched.
• Respects HERO_COMPUTE_SKIP_HYPERVISOR_PROBE=1 for CI / opt-out.

crates/hero_compute_server/src/cloud/server/mod.rs

• Test-only re-export #[cfg(test)] pub(crate) use rpc::hypervisor_probe; so
  unit tests can reach the probe module through the existing
  super::server::* path.

crates/hero_compute_server/src/cloud/tests.rs

• Five new unit tests covering classify_probe_output (Ok / PermissionDenied
  via os error 13 / PermissionDenied via EACCES / OtherFailure) and
  hypervisor_state_dir.

scripts/configure.sh

• New chown_hypervisor_dirs helper. When the script is invoked under sudo
  by a non-root user, it chown -R $SUDO_USER:$SUDO_USER $HOME/.my_hypervisor
  to prevent the root-ownership drift that triggers this bug in the first
  place. Called from both exit paths of install_my_hypervisor.

Test results

• cargo test -p hero_compute_server: 30 passed, 0 failed, 2 ignored (doctests).
• cargo build --workspace: pass, no warnings.
• All five new hypervisor_probe tests pass:
  • test_classify_probe_output_ok
  • test_classify_probe_output_permission_denied_os_error_13
  • test_classify_probe_output_permission_denied_eacces
  • test_classify_probe_output_other_failure
  • test_hypervisor_state_dir_ends_with_expected_suffix

Notes / caveats

• The probe adds one my_hypervisor list subprocess call per deploy_vm
  invocation on the healthy path (~tens of ms). All other RPCs are untouched.
• Scope is intentionally limited to deploy_vm. start_vm / stop_vm /
  restart_vm / delete_vm share the same underlying hazard; hoisting the
  probe into a shared preflight_hypervisor() helper is left as a follow-up.
• my_hypervisor-side fixes (suggestion #3 from the issue — doctor
  state.lock check and list skipping unreadable dirs) live in the
  my_hypervisor repo and are out of scope here.
• The existing post-enqueue failure path (hero_proc job returns non-zero →
  vm_log_fail) is unchanged, so running VMs' error log format is stable.

## Implementation summary Added a pre-flight `my_hypervisor` probe to `ComputeService.deploy_vm` so the "Permission denied (os error 13)" failure reported in this issue now surfaces as a specific, actionable error message instead of an opaque failure after the hero_proc job has already been enqueued. ### Changes **`crates/hero_compute_server/src/cloud/hypervisor.rs`** - Added `fn is_stub(&self) -> bool { false }` to the `HypervisorDriver` trait with a default implementation. - Overrode it in `impl HypervisorDriver for StubDriver` to return `true` so the new probe is skipped when the stub driver is in use (non-Linux / driver init failed). **`crates/hero_compute_server/src/cloud/constants.rs`** - `HYPERVISOR_PROBE_TIMEOUT_SECS: u64 = 3` - `HYPERVISOR_PROBE_SKIP_ENV: &str = "HERO_COMPUTE_SKIP_HYPERVISOR_PROBE"` **`crates/hero_compute_server/src/cloud/rpc.rs`** - New `pub(crate) mod hypervisor_probe` with: - `ProbeResult { Ok, PermissionDenied { path, raw }, OtherFailure, BinaryMissing, Timeout, Skipped }` - `classify_probe_output(exit_code, stdout, stderr)` — pure, unit-tested. Returns `PermissionDenied` when the exit code is non-zero and the concatenated output contains any of `"permission denied"`, `"os error 13"`, or `"eacces"` (case-insensitive). - `hypervisor_state_dir()` — resolves `$HOME/.my_hypervisor/vms/`. - `probe_my_hypervisor(timeout)` — spawns `my_hypervisor list` with a bounded wall-clock timeout and a watchdog thread that kills the child on timeout. Dependency-free. - Wired the probe into `ComputeServiceHandler::deploy_vm` before `deploy_lock` is acquired, slices are allocated, or a VM record is created. On `PermissionDenied` the caller gets a `ComputeServiceError::Internal` whose message names the offending path and shows the exact chown command to fix it. Non-permission failures (binary missing, timeout, other) only emit a `tracing::warn!` and let the deploy proceed — the existing post-enqueue failure path is untouched. - Respects `HERO_COMPUTE_SKIP_HYPERVISOR_PROBE=1` for CI / opt-out. **`crates/hero_compute_server/src/cloud/server/mod.rs`** - Test-only re-export `#[cfg(test)] pub(crate) use rpc::hypervisor_probe;` so unit tests can reach the probe module through the existing `super::server::*` path. **`crates/hero_compute_server/src/cloud/tests.rs`** - Five new unit tests covering `classify_probe_output` (Ok / PermissionDenied via `os error 13` / PermissionDenied via `EACCES` / OtherFailure) and `hypervisor_state_dir`. **`scripts/configure.sh`** - New `chown_hypervisor_dirs` helper. When the script is invoked under sudo by a non-root user, it `chown -R $SUDO_USER:$SUDO_USER $HOME/.my_hypervisor` to prevent the root-ownership drift that triggers this bug in the first place. Called from both exit paths of `install_my_hypervisor`. ### Test results - `cargo test -p hero_compute_server`: 30 passed, 0 failed, 2 ignored (doctests). - `cargo build --workspace`: pass, no warnings. - All five new `hypervisor_probe` tests pass: - `test_classify_probe_output_ok` - `test_classify_probe_output_permission_denied_os_error_13` - `test_classify_probe_output_permission_denied_eacces` - `test_classify_probe_output_other_failure` - `test_hypervisor_state_dir_ends_with_expected_suffix` ### Notes / caveats - The probe adds one `my_hypervisor list` subprocess call per `deploy_vm` invocation on the healthy path (~tens of ms). All other RPCs are untouched. - Scope is intentionally limited to `deploy_vm`. `start_vm` / `stop_vm` / `restart_vm` / `delete_vm` share the same underlying hazard; hoisting the probe into a shared `preflight_hypervisor()` helper is left as a follow-up. - my_hypervisor-side fixes (suggestion #3 from the issue — `doctor` state.lock check and `list` skipping unreadable dirs) live in the my_hypervisor repo and are out of scope here. - The existing post-enqueue failure path (hero_proc job returns non-zero → `vm_log_fail`) is unchanged, so running VMs' error log format is stable.

rawan commented 2026-04-21 21:29:16 +00:00

Author

Member

Copy link

Follow-up: end-to-end deploy fix

After running the chown from this fix, the original Permission denied (os error 13) is gone, but deploys then hit a second failure that blocks the same UX:

Error: Storage error: mount failed (try running with sudo/doas):
  mount: /home/<user>/.my_hypervisor/vms/<id>/rootfs.mnt:
  failed to setup loop device for /home/<user>/.my_hypervisor/vms/<id>/rootfs.img.

my_hypervisor create/start/stop need root to set up loop devices and mount rootfs images. Setting file capabilities (cap_sys_admin,cap_net_admin+ep) on the binary did not unblock the loop-device setup on a stock Ubuntu host, so the PR wraps those invocations with doas -n instead.

Additional changes in this PR

crates/hero_compute_server/src/cloud/constants.rs

• HYPERVISOR_CMD_PREFIX_ENV = "HERO_COMPUTE_HYPERVISOR_CMD_PREFIX"
• HYPERVISOR_CMD_PREFIX_DEFAULT = "doas -n "

crates/hero_compute_server/src/cloud/rpc.rs

• New helper hypervisor_cmd_prefix() reads the env var and returns either
  the override or the default. The literal strings "none" or "" disable
  the wrapper entirely (direct invocation).
• All four my_hypervisor invocations in the cloud service (deploy_vm,
  start_vm, stop_vm, restart_vm) now prepend the prefix. The pre-flight
  probe still runs my_hypervisor list directly since list does not need
  elevated privileges.

scripts/configure.sh

• New install_hypervisor_doas_rule helper. When doas is installed and the
  script can write /etc/doas.conf, it appends a minimal rule:
  permit nopass <service_user> cmd <path/to/my_hypervisor>. Idempotent — it
  skips if the exact rule is already present. When doas is missing or the
  script is not running with write access, it prints guidance instead.
• Called from both exit paths of install_my_hypervisor, alongside
  chown_hypervisor_dirs.

End-to-end verification

On the affected machine, after rebuilding, restarting the hero_compute
service, and issuing the same curl from the original report:

state: running
log_status: completed
logs: [
  ...,
  "Provisioning via hero_proc job 109",
  "Running health check...",
  "deploy_vm completed successfully"
]

Deploy completes. VM is reachable (Cloud Hypervisor started, TAP networking
configured).

Opt-out

On hosts that grant my_hypervisor elevated privileges another way (e.g.
systemd AmbientCapabilities, or a dedicated service user that is
effectively root), set HERO_COMPUTE_HYPERVISOR_CMD_PREFIX=none (or the
empty string) in hero_compute's environment to invoke my_hypervisor
directly without a wrapper.

Test results

• cargo test -p hero_compute_server: 30 passed, 0 failed, 2 ignored (doctests).
• cargo build --workspace: pass.

## Follow-up: end-to-end deploy fix After running the chown from this fix, the original `Permission denied (os error 13)` is gone, but deploys then hit a second failure that blocks the same UX: ``` Error: Storage error: mount failed (try running with sudo/doas): mount: /home/<user>/.my_hypervisor/vms/<id>/rootfs.mnt: failed to setup loop device for /home/<user>/.my_hypervisor/vms/<id>/rootfs.img. ``` `my_hypervisor create`/`start`/`stop` need root to set up loop devices and mount rootfs images. Setting file capabilities (`cap_sys_admin,cap_net_admin+ep`) on the binary did not unblock the loop-device setup on a stock Ubuntu host, so the PR wraps those invocations with `doas -n` instead. ### Additional changes in this PR **`crates/hero_compute_server/src/cloud/constants.rs`** - `HYPERVISOR_CMD_PREFIX_ENV = "HERO_COMPUTE_HYPERVISOR_CMD_PREFIX"` - `HYPERVISOR_CMD_PREFIX_DEFAULT = "doas -n "` **`crates/hero_compute_server/src/cloud/rpc.rs`** - New helper `hypervisor_cmd_prefix()` reads the env var and returns either the override or the default. The literal strings `"none"` or `""` disable the wrapper entirely (direct invocation). - All four `my_hypervisor` invocations in the cloud service (`deploy_vm`, `start_vm`, `stop_vm`, `restart_vm`) now prepend the prefix. The pre-flight probe still runs `my_hypervisor list` directly since `list` does not need elevated privileges. **`scripts/configure.sh`** - New `install_hypervisor_doas_rule` helper. When `doas` is installed and the script can write `/etc/doas.conf`, it appends a minimal rule: `permit nopass <service_user> cmd <path/to/my_hypervisor>`. Idempotent — it skips if the exact rule is already present. When `doas` is missing or the script is not running with write access, it prints guidance instead. - Called from both exit paths of `install_my_hypervisor`, alongside `chown_hypervisor_dirs`. ### End-to-end verification On the affected machine, after rebuilding, restarting the `hero_compute` service, and issuing the same curl from the original report: ``` state: running log_status: completed logs: [ ..., "Provisioning via hero_proc job 109", "Running health check...", "deploy_vm completed successfully" ] ``` Deploy completes. VM is reachable (Cloud Hypervisor started, TAP networking configured). ### Opt-out On hosts that grant `my_hypervisor` elevated privileges another way (e.g. systemd `AmbientCapabilities`, or a dedicated service user that is effectively root), set `HERO_COMPUTE_HYPERVISOR_CMD_PREFIX=none` (or the empty string) in hero_compute's environment to invoke `my_hypervisor` directly without a wrapper. ### Test results - `cargo test -p hero_compute_server`: 30 passed, 0 failed, 2 ignored (doctests). - `cargo build --workspace`: pass.

rawan commented 2026-04-21 21:47:59 +00:00

Author

Member

Copy link

Final fix — chown-back after doas-elevated calls

One more layer on top of the doas wrapper. Running my_hypervisor under
doas creates per-VM state directories (~/.my_hypervisor/vms/<id>/)
owned by root:root, which then breaks the post-deploy resolve_external_vm
step (runs as the service user and needs to read those directories) — the VM
ends up running but with hypervisor_id=None. Subsequent start/stop/
delete on that VM fail with VM has no hypervisor ID — it may not have been provisioned successfully. Delete and redeploy.

Change

crates/hero_compute_server/src/cloud/rpc.rs — new helper
wrap_with_chown_back(body) wraps a shell body so that, after the inner
commands finish, ownership of $HOME/.my_hypervisor is restored to the
calling user (chown -R "$(id -u):$(id -g)"). The wrapper preserves the
inner exit code. All four cloud-service invocations (deploy_vm,
start_vm, stop_vm, restart_vm) now go through it. When the privilege
wrapper is disabled (HERO_COMPUTE_HYPERVISOR_CMD_PREFIX=none), the helper
is a no-op and returns the command unchanged.

Verified end-to-end

After rebuilding and restarting the hero_compute service:

sid: 000n
state: running
log_status: completed
hypervisor_id: 6026f7a5f1a9
mycelium_ip: 569:a763:9776:7bbf:6470:eef8:cc98:fe11
logs: [
  ...,
  "Mycelium IP: 569:a763:9776:7bbf:6470:eef8:cc98:fe11",
  "Hostname set: test-vm-withid",
  "Health: Ping … FAILED | SSH port 22: OPEN (no banner)",
  "deploy_vm completed successfully"
]

Ownership of ~/.my_hypervisor/vms/* stays at rawan:rawan after the
deploy — no drift, no follow-up chown needed. delete_vm on the running VM
returned true. Pre-flight probe continues to pass.

Tests

• cargo test -p hero_compute_server: 30 passed, 0 failed, 2 ignored.
• cargo build --workspace: pass.

## Final fix — chown-back after doas-elevated calls One more layer on top of the doas wrapper. Running `my_hypervisor` under `doas` creates per-VM state directories (`~/.my_hypervisor/vms/<id>/`) owned by `root:root`, which then breaks the post-deploy `resolve_external_vm` step (runs as the service user and needs to read those directories) — the VM ends up `running` but with `hypervisor_id=None`. Subsequent `start`/`stop`/ `delete` on that VM fail with `VM has no hypervisor ID — it may not have been provisioned successfully. Delete and redeploy.` ### Change **`crates/hero_compute_server/src/cloud/rpc.rs`** — new helper `wrap_with_chown_back(body)` wraps a shell body so that, after the inner commands finish, ownership of `$HOME/.my_hypervisor` is restored to the calling user (`chown -R "$(id -u):$(id -g)"`). The wrapper preserves the inner exit code. All four cloud-service invocations (`deploy_vm`, `start_vm`, `stop_vm`, `restart_vm`) now go through it. When the privilege wrapper is disabled (`HERO_COMPUTE_HYPERVISOR_CMD_PREFIX=none`), the helper is a no-op and returns the command unchanged. ### Verified end-to-end After rebuilding and restarting the `hero_compute` service: ``` sid: 000n state: running log_status: completed hypervisor_id: 6026f7a5f1a9 mycelium_ip: 569:a763:9776:7bbf:6470:eef8:cc98:fe11 logs: [ ..., "Mycelium IP: 569:a763:9776:7bbf:6470:eef8:cc98:fe11", "Hostname set: test-vm-withid", "Health: Ping … FAILED | SSH port 22: OPEN (no banner)", "deploy_vm completed successfully" ] ``` Ownership of `~/.my_hypervisor/vms/*` stays at `rawan:rawan` after the deploy — no drift, no follow-up chown needed. `delete_vm` on the running VM returned `true`. Pre-flight probe continues to pass. ### Tests - `cargo test -p hero_compute_server`: 30 passed, 0 failed, 2 ignored. - `cargo build --workspace`: pass.

rawan referenced this issue 2026-04-21 22:08:17 +00:00

Fix: Linux deploy VM #99

rawan referenced this issue 2026-04-22 13:06:26 +00:00

Fix: Linux deploy VM #99

rawan closed this issue 2026-04-22 13:14:28 +00:00

Sign in to join this conversation.

No Branch/Tag specified

Branches Tags

development

feat/hypervisor-rpc-driver

development_mik02_heartbeat_preserve_ip

main

development_tfgrid

v0.1.8

v0.1.7

v0.1.6

v0.1.5

v0.1.4

v0.1.3

v0.1.2

v0.1.1

v0.1.0

Labels

Clear labels   

prio_critical

  

prio_low

  

type_bug

  

type_contact

  

type_issue

  

type_lead

  

type_question

  

type_story

  

type_task

No labels

prio_critical

prio_low

type_bug

type_contact

type_issue

type_lead

type_question

type_story

type_task

Milestone

Clear milestone

No items

No milestone

Projects

Clear projects

No items

No project

Assignees

Clear assignees

No assignees

rawan

1 participant

Notifications

Subscribe

Due date

The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference

lhumina_code/hero_compute#98

No description provided.