lhumina_code/home
0
0
Fork 0
Code Issues 37 Pull requests Projects Releases Packages Wiki Activity Actions

SUPER URGENT BASE PLATFORM #234

New issue
Open
opened 2026-05-16 17:33:56 +00:00 by despiegk · 5 comments
despiegk commented 2026-05-16 17:33:56 +00:00
Owner
Copy link

STEP 1: the bootstrap

read carefully the new bootstrap

  • simplify install of nushell & lab
  • test it on osx
  • test on linux (should work on ubuntu one)

STEP 2: simplify the paths used

  • we used to have code in home and code0 in chosen root, the path names were also not very nice and consistent
  • PATH_ROOT is the home where everything starts from, on OSX can be external disk, otherwise ${HOME}/hero (by default)
  • PATH_ROOT is set in ${HOME}/hero/cfg/init.sh
  • lab is used to set the env variables needed
  • see the bootstrap docs above to understand how it all works

STEP 3: rework lab to do above

  • implement all above
  • check docs
  • make sure everyone knows about it and is ok only using lab for future and above process

STEP 4: we have 2 agents now, soon 3 (hero_shrimp)

  • make sure lab agent ... works properly for calling agents unattended
  • lab skills sync: generates skills and puts on right location
  • lab skills sync: generates mcp points as needed (once builds done below)
  • lab skills sync: configures all which is needed for the agents to be operational
  • tripple check that agents are all configured properly
  • make sure all can use those so we work with same skills

STEP 5: build publish the core

  • build & publish, check main skills, fix where needed
    • hero_skills (to have labs)
    • hero_proc
    • hero_router
    • hero_db
    • hero_code
    • hero_logic

STEP 6a: review & fix hero_proc

  • the service start seems not to be good enough
  • get hero_proc to start db & router automatically as part of well recovering service
  • there is a large integration tester, it still finds bugs, need to resolve the errors
  • do good manual test for all features
  • check on all docu

STEP 6b: review ai_client_direct in herolib

  • there is a large refactor to make it much more easy to use AI, with good builder interfaces
  • review the client & its documentation, review, feedback to kristof then fix
  • in hero_lib_rhai properly wrap the ai client (now direct only, doesn't have to be called direct)
  • it has agent functions in, some from lab can be moved here, then attach to agent (make sure its logical)

STEP 7: review & fix hero_router

  • ...

STEP X:

  • get the client seamless to work with ai_broker
## STEP 1: the bootstrap > [read carefully the new bootstrap](https://forge.ourworld.tf/lhumina_code/hero_skills/src/branch/development/knowledge/bootstrap.md) - [x] simplify install of nushell & lab - [x] test it on osx - [ ] test on linux (should work on ubuntu one) ## STEP 2: simplify the paths used - we used to have code in home and code0 in chosen root, the path names were also not very nice and consistent - [x] PATH_ROOT is the home where everything starts from, on OSX can be external disk, otherwise ${HOME}/hero (by default) - [x] PATH_ROOT is set in ${HOME}/hero/cfg/init.sh - [x] lab is used to set the env variables needed - see the bootstrap docs above to understand how it all works ## STEP 3: rework lab to do above - [x] implement all above - [ ] check docs - [ ] make sure everyone knows about it and is ok only using lab for future and above process ## STEP 4: we have 2 agents now, soon 3 (hero_shrimp) - [ ] make sure lab agent ... works properly for calling agents unattended - [ ] lab skills sync: generates skills and puts on right location - [ ] lab skills sync: generates mcp points as needed (once builds done below) - [ ] lab skills sync: configures all which is needed for the agents to be operational - [ ] tripple check that agents are all configured properly - [ ] make sure all can use those so we work with same skills ## STEP 5: build publish the core - [ ] build & publish, check main skills, fix where needed - [ ] hero_skills (to have labs) - [ ] hero_proc - [ ] hero_router - [ ] hero_db - [ ] hero_code - [ ] hero_logic ## STEP 6a: review & fix hero_proc - [ ] the service start seems not to be good enough - [ ] get hero_proc to start db & router automatically as part of well recovering service - [ ] there is a large integration tester, it still finds bugs, need to resolve the errors - [ ] do good manual test for all features - [ ] check on all docu ## STEP 6b: review ai_client_direct in herolib - there is a large refactor to make it much more easy to use AI, with good builder interfaces - [ ] review the client & its documentation, review, feedback to kristof then fix - [ ] in hero_lib_rhai properly wrap the ai client (now direct only, doesn't have to be called direct) - [ ] it has agent functions in, some from lab can be moved here, then attach to agent (make sure its logical) ## STEP 7: review & fix hero_router - [ ] ... ## STEP X: - [ ] get the client seamless to work with ai_broker
sameh-farouk commented 2026-05-19 19:26:41 +00:00
Member
Copy link

Update — daemon singleton + this session's follow-throughs

Closing the loop on the hero_proc daemon-singleton work and a few related items from this week.

Daemon singleton enforcement in hero_proc (5abe664)
4-state PID machine (Inactive → Starting → Running → Success/Failed), 13 integration tests. A duplicate hero_proc_server now refuses to start with AcquireError::PeerAlive instead of silently coexisting and corrupting state. Per-host singleton is now a daemon-side concern.

Secrets preserved across schema upgrades (e7b9b1e)
Wired runs_model::auto_migrate_or_wipe into factory.rs::HeroProcDb::new. Schema bumps no longer destroy user secrets.

Follow-throughs this session (hero_skills):

  1. Republished hero_proc binaries on Forge. Until today the two fixes above were on development but not in the customer-install binary — release latest was 2+ days stale. New install now actually gets the singleton enforcement + secrets-preserve. Did it via lab build --upload — validates the lab-driven publish path for any Hero repo.

  2. Removed lab's pkill -f hero_proc prelude in start_hero_proc (4488d1f) — lab used to broad-substring-kill any process with "hero_proc" in argv before launching the daemon. With singleton enforcement landed, that prelude was redundant AND counter-productive: it masked the new AcquireError::PeerAlive diagnostic. Also was a serious user-side regression — killed colleagues' SSH sessions whose bash had cd hero_proc in argv. Linux now relies on the precise /proc/exe walk + the daemon-side singleton.

  3. hero_os_hosted step ordering (6840ebd) — bridge config moved before service start so per-user hero_router binds to its own mycelium IPv6 instead of colliding on 127.0.0.1:9988. Each hosted user now spawns its own hero_proc on its own UID with the singleton intact, no cross-user collisions.

  4. Mycelium opt-in setup + preflight (6829d09) — lab install mycelium --start for the host-level daemon; hero_os_hosted now preflights for it and dies loud with the exact install command if missing.

Verified end-to-end on kristof5: customer install → user provisioned → hero_proc + admin + router up with healthy sockets and singleton enforcement active.

Step 5 (Build & publish core) is current for lab (was 76 days stale), hero_router (7), hero_proc (2). Step 6a (Review hero_proc) advanced via the two fixes above landing in the customer binary.

**Update — daemon singleton + this session's follow-throughs** Closing the loop on the hero_proc daemon-singleton work and a few related items from this week. **Daemon singleton enforcement in hero_proc** ([`5abe664`](https://forge.ourworld.tf/lhumina_code/hero_proc/commit/5abe664)) 4-state PID machine (Inactive → Starting → Running → Success/Failed), 13 integration tests. A duplicate `hero_proc_server` now refuses to start with `AcquireError::PeerAlive` instead of silently coexisting and corrupting state. Per-host singleton is now a daemon-side concern. **Secrets preserved across schema upgrades** ([`e7b9b1e`](https://forge.ourworld.tf/lhumina_code/hero_proc/commit/e7b9b1e)) Wired `runs_model::auto_migrate_or_wipe` into `factory.rs::HeroProcDb::new`. Schema bumps no longer destroy user secrets. **Follow-throughs this session (hero_skills):** 1. **Republished hero_proc binaries on Forge.** Until today the two fixes above were on `development` but not in the customer-install binary — release `latest` was 2+ days stale. New install now actually gets the singleton enforcement + secrets-preserve. Did it via `lab build --upload` — validates the lab-driven publish path for any Hero repo. 2. **Removed lab's `pkill -f hero_proc` prelude in `start_hero_proc`** ([`4488d1f`](https://forge.ourworld.tf/lhumina_code/hero_skills/commit/4488d1f)) — lab used to broad-substring-kill any process with "hero_proc" in argv before launching the daemon. With singleton enforcement landed, that prelude was redundant AND counter-productive: it masked the new `AcquireError::PeerAlive` diagnostic. Also was a serious user-side regression — killed colleagues' SSH sessions whose bash had `cd hero_proc` in argv. Linux now relies on the precise `/proc/exe` walk + the daemon-side singleton. 3. **`hero_os_hosted` step ordering** ([`6840ebd`](https://forge.ourworld.tf/lhumina_code/hero_skills/commit/6840ebd)) — bridge config moved before service start so per-user hero_router binds to its own mycelium IPv6 instead of colliding on `127.0.0.1:9988`. Each hosted user now spawns its own hero_proc on its own UID with the singleton intact, no cross-user collisions. 4. **Mycelium opt-in setup + preflight** ([`6829d09`](https://forge.ourworld.tf/lhumina_code/hero_skills/commit/6829d09)) — `lab install mycelium --start` for the host-level daemon; `hero_os_hosted` now preflights for it and dies loud with the exact install command if missing. **Verified end-to-end on kristof5**: customer install → user provisioned → hero_proc + admin + router up with healthy sockets and singleton enforcement active. Step 5 (Build & publish core) is current for `lab` (was 76 days stale), `hero_router` (7), `hero_proc` (2). Step 6a (Review hero_proc) advanced via the two fixes above landing in the customer binary.
sameh-farouk commented 2026-05-21 11:09:15 +00:00
Member
Copy link

Update — hero_proc_test integration suite run + 13-issue triage (Step 6a items 1+3 progress)

Following up on Step 6a items 1 ("service start not good enough") + 3 ("integration tester still finds bugs"):

Step 6a item 3 — hero_proc_test run against hero_proc 719ba10

Ran --basic --functional against a freshly built hero_proc_server. 20 tests fail. Categorized:

Real defects in the daemon (need fixing):

  • uc09_failed_parent_blocks_child — child must NOT be succeeded when parent failed, got succeeded → hero_proc#64 reopened (DAG-gating logic in supervisor/mod.rs:660-742 looks right but doesn't actually cascade-cancel end-to-end)
  • uc07_retry_succeeds_on_second_attempt — expected exactly 2 attempts, got 1 → hero_proc#106 reopened (Kristof's 03e7ed8 retry fix not working end-to-end; likely related to HP-04 restart-counter reset)
  • uc31_action_cascade_delete — action delete should cascade to >=3 jobs, got 0 → relates to hero_proc#66 (cascade-delete unimplemented)
  • uc41_job_timeout_ms — job stuck in running after 15s → timeout enforcement not firing
  • uc32_process_stats_while_running / uc33_job_why_waiting — RPC -32100 "job not found" mid-test; race or DB lookup bug
  • uc12-16 (service lifecycle: start/stop/restart/remove/system-class) — multiple, need individual triage

Infra setup issues (not real bugs):

  • basic::daemon_singleton::* (5 tests) — try to spawn from /home/sameh/hero/bin/hero_proc_server (not installed via lab user init on the test host); not a defect of singleton itself.

Full output saved as artifacts in errors/ per test.

13-issue triage on hero_proc — 8 closed, 5 kept open

Closed with citation comments (verified empirically — code review + admin UI Playwright + unit tests):

  • #55 (PTY CPR strip — 10/10 unit tests pass)
  • #57 (services not shown properly — UI renders 41 rows)
  • #58, #59, #60, #62 (job/service/run/action detail views — all UI panels render with all fields)
  • #63 (cycle detection — explicitly reversed by SPECS.md:35)
  • #105 (D-10 closure runbook — own body declares "35/35 100%")

Kept open after empirical re-verification (with explanation comments):

  • #64 + #106 — code looks correct but tests uc09 and uc07 fail
  • #65 — code looks correct but no end-to-end test covers it
  • #77 — editAction and editSecret work, but editService is silently no-op
  • #111 — code FIXED on development and x86_64-Linux binary refreshed, but arm64+macOS assets still 4 days stale on latest (lab-publish.yaml is x86_64-only by design)

Step 6a item 1 status

hero_proc#92 (bind-race readiness await) still open — Step 6a item 1 verbatim. rpc/service.rs:317-385 returns synchronously after job-row insert with no readiness probe.

Other Step 6a items

  • Item 2 (auto-start db & router): hero_proc#95 open. Wire hero_db into server.rs::run() init phase.
  • Item 3 (integration tester finds bugs): see #64 / #106 / #66 / #12-16 above — that's exactly what the integration tester is finding.
  • Items 4-5 (manual test + check docu): ongoing operator activity.

Step 5 (Build & publish core) — hero_proc

lab-publish.yaml (Mik's f0ba646, 2026-05-19) is auto-republishing x86_64-Linux on every push to development. Latest x86_64 asset is from today 09:20 UTC. arm64+macOS need their own publish workflows.

**Update — `hero_proc_test` integration suite run + 13-issue triage (Step 6a items 1+3 progress)** Following up on Step 6a items 1 ("service start not good enough") + 3 ("integration tester still finds bugs"): ### Step 6a item 3 — `hero_proc_test` run against `hero_proc` 719ba10 Ran `--basic --functional` against a freshly built `hero_proc_server`. **20 tests fail.** Categorized: **Real defects in the daemon (need fixing):** - `uc09_failed_parent_blocks_child` — `child must NOT be succeeded when parent failed, got succeeded` → hero_proc#64 reopened (DAG-gating logic in `supervisor/mod.rs:660-742` looks right but doesn't actually cascade-cancel end-to-end) - `uc07_retry_succeeds_on_second_attempt` — `expected exactly 2 attempts, got 1` → hero_proc#106 reopened (Kristof's `03e7ed8` retry fix not working end-to-end; likely related to HP-04 restart-counter reset) - `uc31_action_cascade_delete` — `action delete should cascade to >=3 jobs, got 0` → relates to hero_proc#66 (cascade-delete unimplemented) - `uc41_job_timeout_ms` — `job stuck in running after 15s` → timeout enforcement not firing - `uc32_process_stats_while_running` / `uc33_job_why_waiting` — RPC -32100 "job not found" mid-test; race or DB lookup bug - `uc12-16` (service lifecycle: start/stop/restart/remove/system-class) — multiple, need individual triage **Infra setup issues (not real bugs):** - `basic::daemon_singleton::*` (5 tests) — try to spawn from `/home/sameh/hero/bin/hero_proc_server` (not installed via `lab user init` on the test host); not a defect of singleton itself. Full output saved as artifacts in `errors/` per test. ### 13-issue triage on hero_proc — 8 closed, 5 kept open Closed with citation comments (verified empirically — code review + admin UI Playwright + unit tests): - **#55** (PTY CPR strip — 10/10 unit tests pass) - **#57** (services not shown properly — UI renders 41 rows) - **#58, #59, #60, #62** (job/service/run/action detail views — all UI panels render with all fields) - **#63** (cycle detection — explicitly reversed by `SPECS.md:35`) - **#105** (D-10 closure runbook — own body declares "35/35 100%") Kept open after empirical re-verification (with explanation comments): - **#64** + **#106** — code looks correct but tests `uc09` and `uc07` fail - **#65** — code looks correct but no end-to-end test covers it - **#77** — `editAction` and `editSecret` work, but `editService` is silently no-op - **#111** — code FIXED on `development` and x86_64-Linux binary refreshed, but arm64+macOS assets still 4 days stale on `latest` (`lab-publish.yaml` is x86_64-only by design) ### Step 6a item 1 status `hero_proc#92` (bind-race readiness await) still open — Step 6a item 1 verbatim. `rpc/service.rs:317-385` returns synchronously after job-row insert with no readiness probe. ### Other Step 6a items - Item 2 (auto-start db & router): `hero_proc#95` open. Wire hero_db into `server.rs::run()` init phase. - Item 3 (integration tester finds bugs): see #64 / #106 / #66 / #12-16 above — that's exactly what the integration tester is finding. - Items 4-5 (manual test + check docu): ongoing operator activity. ### Step 5 (Build & publish core) — hero_proc `lab-publish.yaml` (Mik's `f0ba646`, 2026-05-19) is auto-republishing x86_64-Linux on every push to development. Latest x86_64 asset is from today 09:20 UTC. arm64+macOS need their own publish workflows.
sameh-farouk commented 2026-05-21 11:15:05 +00:00
Member
Copy link

Addendum to my previous comment — sandbox caveat on the hero_proc_test 20-failures finding:

I ran the test against a hero_proc_server I started by invoking ./target/release/hero_proc_server directly, NOT via lab service hero_proc --start --force --build (the canonical path each errors/*.md re-run instruction prints). My local box also isn't lab user init'd. So:

  • The 5 basic::daemon_singleton::* failures are confirmed-not-bugs (try to spawn from missing /home/sameh/hero/bin/hero_proc_server).
  • The 15 functional failures (uc07, uc09, uc12-16, uc31, uc32/33, uc41, etc.) I framed as "real defects" are provisional — some may be sandbox artifacts (canonical setup assumptions broken by my non-installed-binary, non-init.sh-sourced shell), and I haven't re-verified on a properly-bootstrapped box.

The Step 6a item 3 progress claim ("integration tester still finds bugs") is right in spirit — there are real bugs there — but the specific list of 15 needs re-run under lab service --start on kristof5 (or equivalent) before treating each as a separate ticket.

**Addendum to my previous comment — sandbox caveat on the `hero_proc_test` 20-failures finding:** I ran the test against a `hero_proc_server` I started by invoking `./target/release/hero_proc_server` directly, NOT via `lab service hero_proc --start --force --build` (the canonical path each `errors/*.md` re-run instruction prints). My local box also isn't `lab user init`'d. So: - The 5 `basic::daemon_singleton::*` failures are confirmed-not-bugs (try to spawn from missing `/home/sameh/hero/bin/hero_proc_server`). - The **15 functional failures** (uc07, uc09, uc12-16, uc31, uc32/33, uc41, etc.) I framed as "real defects" are **provisional** — some may be sandbox artifacts (canonical setup assumptions broken by my non-installed-binary, non-init.sh-sourced shell), and I haven't re-verified on a properly-bootstrapped box. The Step 6a item 3 progress claim ("integration tester still finds bugs") is right in spirit — there are real bugs there — but the specific list of 15 needs re-run under `lab service --start` on kristof5 (or equivalent) before treating each as a separate ticket.
sameh-farouk commented 2026-05-21 11:27:23 +00:00
Member
Copy link

Re-run on kristof5 under canonical setup — confirmed/corrected findings.

Per the addendum I posted earlier flagging the sandbox caveat, I re-ran hero_proc_test --basic --functional on kristof5 (PATH_ROOT=/home/sameh/hero, hero_proc auto-started via lab, latest binary from 2026-05-20T13:29). Result: 23 failures / 234 passes.

Confirmed real defects (fail on BOTH sandbox AND kristof5)

Test Failure
uc07_retry_succeeds_on_second_attempt expected exactly 2 attempts, got 1 → hero_proc#106
uc09_failed_parent_blocks_child child must NOT be succeeded when parent failed, got succeeded → hero_proc#64
uc12_define_service_start_stop service stuck running=false after 10s
uc13_restart_produces_new_job_ids service stuck running=false after 10s
uc14_stop_with_remove_jobs_deletes_records service stuck running=false after 10s
uc16_system_class_excluded_from_stop_all service stuck running=false after 10s
uc20_cron_scheduled_job_fires_within_75s cron job logs missing CRON_RAN
uc32_process_stats_while_running / uc33_job_why_waiting RPC -32100 "job not found"
uc41_job_timeout_ms job stuck in running after 15s — timeout not enforced
functional::runs::structured_logs_all_levels debug-level logs missing
functional::runs::logs_query_by_service_src_wildcard wildcard query returns 0 entries

That's 12 confirmed daemon-internal defects, several closely related — the uc12-16 cluster suggests services aren't transitioning to running state properly, which is a fundamental supervisor regression.

Big new finding — scheduler tests all fail on kristof5

10 functional::schedule::* tests fail on kristof5 with the same shape: timeout waiting for 1 jobs for action '<sched-...>'. None of these failed in my local sandbox. Looking at the affected tests:

  • test_interval_with_large_duration_works
  • test_time_window_within_range_allows_scheduling
  • test_time_window_transition_at_start_time + _end_time
  • test_nr_of_instances_default_is_one
  • test_scheduled_job_has_scheduled_tag
  • test_cron_scheduled_job_tagged_as_cron
  • test_scheduled_job_output_captured
  • test_schedule_policy_removal_triggers_cleanup
  • test_very_long_interval_works

The scheduler appears to be entirely non-functional on the published latest binary (2026-05-20). This is a production regression — customer scheduled jobs aren't firing. Worth filing as its own P0 issue separate from #92/#95/#106 (Step 6a items 1/2). My local sandbox (binary built today from 719ba10) passed many of these — fix may already exist on development, needs re-publish to verify.

Sandbox-only failures (fail locally, pass on kristof5)

These I correctly classified as sandbox artifacts earlier — kristof5 confirmation:

  • 4× basic::daemon_singleton::* (need installed lab binaries to spawn from)
  • 2× basic::logging::* (need installed lab paths)
  • uc31_action_cascade_delete — passes on kristof5 (so hero_proc#66 cascade-delete may already work in production)
  • uc19_interval_scheduled_job_fires_automatically — passes on kristof5

Updated triage on #64 and #106

Both addenda on those issues edited to confirm real-defect status (was provisional, now confirmed by kristof5 canonical run).

Revised recommendation for Step 6a

  1. P0 — scheduler regression: 10 schedule tests fail on latest. File as own issue; rebuild + republish to see if development already has the fix.
  2. P1 — uc12-16 cluster: service-state-transition regression (4 lifecycle tests all show services stuck running=false). Possibly the same root cause across all 4. Investigate first.
  3. P1 — #64 (uc09) + #106 (uc07): confirmed daemon-internal defects, both touch retry / dep-cancel paths. May share root cause with HP-04 (attempt counter).
  4. P1 — #92 (bind-race) + #95 (auto-start db): unchanged.
  5. Cleanup — close hero_proc#66 if cascade-delete passes on kristof5 (need second-pass to be sure it's not flaky).
**Re-run on kristof5 under canonical setup — confirmed/corrected findings.** Per the addendum I posted earlier flagging the sandbox caveat, I re-ran `hero_proc_test --basic --functional` on kristof5 (PATH_ROOT=/home/sameh/hero, hero_proc auto-started via lab, `latest` binary from 2026-05-20T13:29). Result: **23 failures / 234 passes.** ### Confirmed real defects (fail on BOTH sandbox AND kristof5) | Test | Failure | |---|---| | `uc07_retry_succeeds_on_second_attempt` | `expected exactly 2 attempts, got 1` → hero_proc#106 | | `uc09_failed_parent_blocks_child` | `child must NOT be succeeded when parent failed, got succeeded` → hero_proc#64 | | `uc12_define_service_start_stop` | service stuck `running=false` after 10s | | `uc13_restart_produces_new_job_ids` | service stuck `running=false` after 10s | | `uc14_stop_with_remove_jobs_deletes_records` | service stuck `running=false` after 10s | | `uc16_system_class_excluded_from_stop_all` | service stuck `running=false` after 10s | | `uc20_cron_scheduled_job_fires_within_75s` | cron job logs missing CRON_RAN | | `uc32_process_stats_while_running` / `uc33_job_why_waiting` | RPC -32100 "job not found" | | `uc41_job_timeout_ms` | job stuck in running after 15s — timeout not enforced | | `functional::runs::structured_logs_all_levels` | debug-level logs missing | | `functional::runs::logs_query_by_service_src_wildcard` | wildcard query returns 0 entries | That's **12 confirmed daemon-internal defects**, several closely related — the `uc12-16` cluster suggests services aren't transitioning to `running` state properly, which is a fundamental supervisor regression. ### Big new finding — scheduler tests all fail on kristof5 10 `functional::schedule::*` tests fail on kristof5 with the same shape: `timeout waiting for 1 jobs for action '<sched-...>'`. None of these failed in my local sandbox. Looking at the affected tests: - `test_interval_with_large_duration_works` - `test_time_window_within_range_allows_scheduling` - `test_time_window_transition_at_start_time` + `_end_time` - `test_nr_of_instances_default_is_one` - `test_scheduled_job_has_scheduled_tag` - `test_cron_scheduled_job_tagged_as_cron` - `test_scheduled_job_output_captured` - `test_schedule_policy_removal_triggers_cleanup` - `test_very_long_interval_works` **The scheduler appears to be entirely non-functional on the published `latest` binary** (2026-05-20). This is a production regression — customer scheduled jobs aren't firing. Worth filing as its own P0 issue separate from #92/#95/#106 (Step 6a items 1/2). My local sandbox (binary built today from 719ba10) passed many of these — fix may already exist on `development`, needs re-publish to verify. ### Sandbox-only failures (fail locally, pass on kristof5) These I correctly classified as sandbox artifacts earlier — kristof5 confirmation: - 4× `basic::daemon_singleton::*` (need installed lab binaries to spawn from) - 2× `basic::logging::*` (need installed lab paths) - `uc31_action_cascade_delete` — **passes on kristof5** (so hero_proc#66 cascade-delete may already work in production) - `uc19_interval_scheduled_job_fires_automatically` — passes on kristof5 ### Updated triage on #64 and #106 Both addenda on those issues edited to confirm real-defect status (was provisional, now confirmed by kristof5 canonical run). ### Revised recommendation for Step 6a 1. **P0** — scheduler regression: 10 schedule tests fail on `latest`. File as own issue; rebuild + republish to see if `development` already has the fix. 2. **P1** — `uc12-16` cluster: service-state-transition regression (4 lifecycle tests all show services stuck `running=false`). Possibly the same root cause across all 4. Investigate first. 3. **P1** — #64 (uc09) + #106 (uc07): confirmed daemon-internal defects, both touch retry / dep-cancel paths. May share root cause with HP-04 (attempt counter). 4. **P1** — #92 (bind-race) + #95 (auto-start db): unchanged. 5. **Cleanup** — close hero_proc#66 if cascade-delete passes on kristof5 (need second-pass to be sure it's not flaky).
sameh-farouk commented 2026-05-21 11:54:14 +00:00
Member
Copy link

Correction to my earlier scheduler-regression claim — published binary deployed on kristof5 + 3-way diff.

Sameh rightly pushed back on my "scheduler regression on latest binary" framing. I deployed today's published binary (hero_proc_server-linux-musl-x86_64 md5 11029c562b6ef3cb085548f820afd3f1, built from 719ba10, on latest since 2026-05-21T09:18) to kristof5, restarted hero_proc, and re-ran the full suite. Then I 3-way-diffed:

  • R1 local sandbox / clean DB / binary 719ba10 → 20 fails
  • R2 kristof5 / loaded DB / OLD May 20 binary → 23 fails (previous report)
  • R3 kristof5 / loaded DB / NEW 719ba10 binary → 29 fails

What the binary update actually accomplished
Fixed by new binary uc20_cron_scheduled_job_fires_within_75s, uc32_process_stats_while_running (failed R2, pass R3)
Introduced by new binary 0 real regressions. The 8 "new" failures are all schedule-cluster tests in the kristof5-only environmental set.

My scheduler-regression claim was wrong

The 10 functional::schedule::* tests that failed on kristof5 with the OLD binary STILL fail with the NEW binary on kristof5 — and additional schedule tests now fail too — but the SAME tests with the SAME new binary pass on clean local sandbox. So the failure cause is environmental on the loaded kristof5 box (2154 background jobs + supervisor under load → scheduler can't meet test tick deadlines), not a code regression. The scheduler works; it just times out under load.

11 stable real defects (fail in ALL 3 runs — independent of env or binary version)

basic::daemon_singleton::prepare_sockets_handles_zombie_socket
functional::runs::logs_query_by_service_src_wildcard
functional::runs::structured_logs_all_levels
functional::uc_06_07::uc07_retry_succeeds_on_second_attempt    → hero_proc#106
functional::uc_08_09::uc09_failed_parent_blocks_child           → hero_proc#64
functional::uc_12_16::uc12_define_service_start_stop
functional::uc_12_16::uc13_restart_produces_new_job_ids
functional::uc_12_16::uc14_stop_with_remove_jobs_deletes_records
functional::uc_12_16::uc16_system_class_excluded_from_stop_all
functional::uc_31_34::uc33_job_why_waiting
functional::uc_37_43::uc41_job_timeout_ms

The uc12-16 cluster (4 tests, all "service stuck running=false after 10s") is the most concerning shared-root-cause candidate. uc09+uc07 confirm hero_proc#64 and hero_proc#106 are real defects.

Other clarifications

  • uc31_action_cascade_delete is FLAKY, not a clean pass — failed R1+R3, passed R2. Cannot confirm hero_proc#66 either way; needs deeper investigation.
  • Step 6a Step 5 hero_proc bullet: customer-install x86_64 binary is current as of 09:20 UTC today, includes #106 + #64-65 + #55 + admin UI fixes. arm64 + macOS still 4 days stale.

Retracting my earlier "P0 scheduler regression" framing. Task list updated accordingly.

**Correction to my earlier scheduler-regression claim — published binary deployed on kristof5 + 3-way diff.** Sameh rightly pushed back on my "scheduler regression on `latest` binary" framing. I deployed today's published binary (`hero_proc_server-linux-musl-x86_64` md5 `11029c562b6ef3cb085548f820afd3f1`, built from 719ba10, on `latest` since 2026-05-21T09:18) to kristof5, restarted hero_proc, and re-ran the full suite. Then I 3-way-diffed: - **R1** local sandbox / clean DB / binary 719ba10 → 20 fails - **R2** kristof5 / loaded DB / OLD May 20 binary → 23 fails (previous report) - **R3** kristof5 / loaded DB / NEW 719ba10 binary → 29 fails ### What the binary update actually accomplished | | | |---|---| | **Fixed by new binary** | `uc20_cron_scheduled_job_fires_within_75s`, `uc32_process_stats_while_running` (failed R2, pass R3) | | **Introduced by new binary** | 0 real regressions. The 8 "new" failures are all schedule-cluster tests in the kristof5-only environmental set. | ### My scheduler-regression claim was wrong The 10 `functional::schedule::*` tests that failed on kristof5 with the OLD binary STILL fail with the NEW binary on kristof5 — and additional schedule tests now fail too — but the SAME tests with the SAME new binary **pass on clean local sandbox**. So the failure cause is **environmental on the loaded kristof5 box** (2154 background jobs + supervisor under load → scheduler can't meet test tick deadlines), not a code regression. The scheduler works; it just times out under load. ### 11 stable real defects (fail in ALL 3 runs — independent of env or binary version) ``` basic::daemon_singleton::prepare_sockets_handles_zombie_socket functional::runs::logs_query_by_service_src_wildcard functional::runs::structured_logs_all_levels functional::uc_06_07::uc07_retry_succeeds_on_second_attempt → hero_proc#106 functional::uc_08_09::uc09_failed_parent_blocks_child → hero_proc#64 functional::uc_12_16::uc12_define_service_start_stop functional::uc_12_16::uc13_restart_produces_new_job_ids functional::uc_12_16::uc14_stop_with_remove_jobs_deletes_records functional::uc_12_16::uc16_system_class_excluded_from_stop_all functional::uc_31_34::uc33_job_why_waiting functional::uc_37_43::uc41_job_timeout_ms ``` The `uc12-16` cluster (4 tests, all "service stuck running=false after 10s") is the most concerning shared-root-cause candidate. `uc09`+`uc07` confirm hero_proc#64 and hero_proc#106 are real defects. ### Other clarifications - **`uc31_action_cascade_delete` is FLAKY**, not a clean pass — failed R1+R3, passed R2. Cannot confirm hero_proc#66 either way; needs deeper investigation. - **Step 6a Step 5 hero_proc bullet**: customer-install x86_64 binary is current as of 09:20 UTC today, includes #106 + #64-65 + #55 + admin UI fixes. arm64 + macOS still 4 days stale. Retracting my earlier "P0 scheduler regression" framing. Task list updated accordingly.
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
main
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
Assignees
Clear assignees
No assignees
2 participants
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/home#234
No description provided.