lhumina_code/home
22
0
Fork 0
Code Issues 8 Pull requests Projects Releases Packages Wiki Activity Actions

Smoke Tests, Production Deployment & Cleanup #13

New issue
Closed
opened 2026-03-11 01:59:26 +00:00 by mik-tf · 2 comments
mik-tf commented 2026-03-11 01:59:26 +00:00
Owner
Copy link

Smoke Tests, Production Deployment & Cleanup

Follow-up from #12 (Infrastructure Sync — completed).

herodev and herodemo are deployed and working (31 services each, HTTP 200). This issue covers smoke testing, service bug fixes, and the remaining cleanup before production.

Build Pipeline

Issue #12 replaced the old Dockerfile.dev (SSH git cloning inside Docker, 40-60 min) with the local build pipeline:

cd hero_services
make deploy        # dist → pack → push → deploy to herodev
Step Command What it does
1 make dist docker/build-local.sh — compiles all service repos inside rust:1.93-bookworm containers with lhumina_code/ and geomind_code/ volume-mounted. Whatever is checked out on disk gets built. Persistent cargo caches: 1-3 min incremental, 10-15 min cold.
2 make pack docker build -f Dockerfile.pack — copies pre-built dist/ into thin debian:bookworm-slim image. No compilation.
3 make push Pushes hero_zero:dev to forge.ourworld.tf/lhumina_code/hero_zero
4 make update ENV=herodev SSHes into VM, pulls new image, restarts container

Promotion: make demo tags :dev → :demo, pushes, deploys to herodemo.

Stale files to delete: Dockerfile (old hero_zero) and Dockerfile.prod (SSH-based BuildKit) — both replaced by Dockerfile.pack + docker/build-local.sh.

Smoke Tests

Three bash/curl test suites verify the live gateway remotely. ~110 tests, ~30 seconds, zero dependencies beyond curl/bash/python3.

cd deploy/single-vm
make test ENV=herodev     # or ENV=herodemo
Suite Tests Coverage
smoke_gateway.sh 47 SPA routing, auth flow (challenge→login→validate_session), inspector discovery (33 services, 12 with methods), RPC discovery (OSIS 509 methods, Books 19, Inspector 18), MCP gateway, WebDAV (fossil), SSE, 16 health checks, 10 UI HTML pages
smoke_test.sh 52 All UI health endpoints (12 services), page content verification (cloud, books, fossil/HeroFoundry, redis, inspector, os, etc.), RPC calls (OSIS, Books, Inspector, AIBroker, Embedder), WASM island RPC, 7 server-side health checks, socat-bridged services (forge, shrimp)
smoke_theme.sh 14 hero:theme postMessage listener present in all 14 iframe-embedded services

How each test works — just curl:

  • Health: curl /hero_redis_ui/health → expect HTTP 200
  • HTML pages: curl /hero_cloud_ui/ → expect HTTP 200 + content-type: text/html
  • JSON-RPC: curl -X POST /hero_osis_ui/rpc/root with JSON body → expect "jsonrpc" in response
  • RPC discovery: call rpc.discover, count methods ≥ threshold
  • Auth flow: full challenge → login → validate_session cycle with admin/admin
  • Inspector: call inspector.services, check 33 services found, 12 have methods
  • WebDAV: curl -X OPTIONS /hero_fossil_server/webdav/ → expect 204
  • SSE: curl --max-time 3 → check text/event-stream header
  • MCP: POST JSON-RPC to /hero_inspector_ui/mcp/hero_redis_server
  • Theme: check HTML contains hero:theme listener

The smoke tests are the safety gate for :demo promotion. When make test ENV=herodev passes, all services are healthy, auth works, RPC responds, UI renders, themes sync. The fix→deploy→test cycle is 3-5 minutes.

Test Bug Fixes (completed)

6 bugs in the test scripts themselves (not service bugs):

File Bug Fix
smoke_gateway.sh Inspector method services.list doesn't exist → inspector.services
smoke_gateway.sh Inspector field methods doesn't exist → methods_count
smoke_gateway.sh hero_indexer_ui in health list but no socket Removed
smoke_test.sh Checks base href="/zinit_ui/" but zinit uses <meta name="base-path"> Fixed match
smoke_test.sh Embedder rpc.discover not implemented → health method
smoke_test.sh Fossil RPC test but fossil uses WebDAV, not JSON-RPC Removed

Coverage extended: gateway 20→47 tests, service 35→52 tests. All 15 services from services/user/*.toml now covered.

Service Bugs Found

Smoke tests against herodev.gent02.grid.tf (2026-03-11) found 3 genuine service bugs:

Repo Bug Impact
hero_redis hero_redis_ui HTTP 500: Template 'login.html' not found Page completely broken
hero_indexer / hero_indexer_ui hero_indexer_ui HTTP 404 — no UI socket registered with proxy Service unreachable via gateway
zinit (geomind_code) zinit_ui missing hero:theme postMessage listener Theme sync broken when embedded in iframe

Results before fixes:

smoke_gateway.sh:  46 passed, 0 failed, 1 skipped
smoke_test.sh:     44 passed, 1 failed (hero_redis_ui 500)
smoke_theme.sh:    11 passed, 3 failed (redis 500, indexer 404, zinit no theme)

DevOps Workflow

Branching & PRs

One issue = one commit on development. The PR squash merge handles this automatically.

development_{name}     ← work freely, many commits, push often
    ↓ PR with "closes #N"
    ↓ Forgejo "Squash commit" merge
development            ← one clean commit per issue

Flow:

  1. Create development_{name} branch from development (same name across all affected repos)
  2. Work freely — commit as often as you want, push after every test cycle
  3. The branch is your scratch pad — full history preserved in the PR for review
  4. When done: create PR with closes #13 in the description
  5. Merge using "Squash commit" on Forgejo
  6. Result: one clean commit on development with the issue URL as message

No manual squashing, no force pushing, no rewriting history. Forgejo does it.

Commit message format (set by Forgejo on squash): https://forge.ourworld.tf/lhumina_code/home/issues/13 — Smoke Tests, Production Deployment & Cleanup

Rules:

  • Never rebase — merge development INTO feature branch if needed
  • Never commit directly to main — releases only via PR from development
  • PRs into development: Squash commit
  • PRs into main: Create merge commit (preserves release boundary)

11-Step Pipeline (3 human gates)

Step Action Who Gate
1 Fix code on development_{name} in each repo AI —
2 make deploy (1-3 min incremental) AI Must compile
3 make test ENV=herodev (30 sec, ~110 tests) AI —
— Iterate steps 1-3 until green. Push freely. AI —
4 Human verifies herodev Human ✓ Must confirm
5 Create PRs: development_{name} → development with closes #N AI Only after step 4
6 Squash merge PRs on Forgejo AI/Human —
7 git checkout development && git pull all repos, make deploy AI Clean build
8 Human verifies herodev (clean merged code) Human ✓ Must confirm
9 make demo (tag :dev→:demo, deploy herodemo) AI Only after step 8
10 Human verifies herodemo Human ✓ Must confirm
11 Update issue log with session, commits, image SHA AI —

Why two builds (steps 2 and 7): Step 2 builds from local feature branch (fast iteration). Step 7 builds from clean merged development (proves the pushed code works).

Critical rules:

  • Never push code before human confirms herodev (step 4 gate)
  • Never tag :demo before human confirms clean build (step 8 gate)
  • herodev is the development playground; herodemo is protected
  • No hot-swapping WASM — always full make deploy

Tasks

1. Smoke Tests

  • Run existing tests against herodev, fix test bugs
  • Extend coverage to all 15 services (~110 tests)
  • Add make test target running all 3 suites
  • Fix 3 service bugs (redis template, indexer socket, zinit theme)
  • All ~110 tests green on herodev
  • Human confirms herodev
  • Promote to herodemo, all tests green

2. Infrastructure Cleanup

  • Delete stale Dockerfile and Dockerfile.prod
  • Fix herodemo terraform state (points to unreachable VM, actual container on herodev VM)
  • Fix dist/ WASM root-owned files in build-local.sh

3. Branch Cleanup

  • Merge hero_aibroker development_theme_sync → development
  • Delete stale development_mik5 branches across repos
  • Run cargo test on zinit

4. Production Deployment (after smoke tests green)

  • Tag :dev as :prod, push
  • Deploy heroprod (needs terraform provisioning or reuse existing VM)
  • Human approval gate — NO auto-deploy to prod

5. Build Pipeline CI (nice-to-have)

  • Forgejo Actions workflow: on push to development, run make dist + make pack
  • Publish dist tarball as release artifact

Environments

Tier Gateway Port Image Container
dev herodev.gent02.grid.tf 8805 hero_zero:dev herodev
demo herodemo.gent02.grid.tf 8806 hero_zero:demo herodemo
prod TBD TBD hero_zero:prod heroprod

VM: Both containers on same VM at Mycelium IP 495:72fa:8ec3:9264:ff0f:c0a8:abad:234c
Registry: forge.ourworld.tf/lhumina_code/hero_zero

Current Status

Step 3 complete — smoke tests assessed, test script bugs fixed, 3 service bugs identified. Ready for step 1 (implement service fixes).

# Smoke Tests, Production Deployment & Cleanup Follow-up from [#12](https://forge.ourworld.tf/lhumina_code/home/issues/12) (Infrastructure Sync — completed). herodev and herodemo are deployed and working (31 services each, HTTP 200). This issue covers smoke testing, service bug fixes, and the remaining cleanup before production. --- ## Build Pipeline Issue #12 replaced the old Dockerfile.dev (SSH git cloning inside Docker, 40-60 min) with the local build pipeline: ```bash cd hero_services make deploy # dist → pack → push → deploy to herodev ``` | Step | Command | What it does | |------|---------|--------------| | 1 | `make dist` | `docker/build-local.sh` — compiles all service repos inside `rust:1.93-bookworm` containers with `lhumina_code/` and `geomind_code/` **volume-mounted**. Whatever is checked out on disk gets built. Persistent cargo caches: 1-3 min incremental, 10-15 min cold. | | 2 | `make pack` | `docker build -f Dockerfile.pack` — copies pre-built `dist/` into thin `debian:bookworm-slim` image. No compilation. | | 3 | `make push` | Pushes `hero_zero:dev` to `forge.ourworld.tf/lhumina_code/hero_zero` | | 4 | `make update ENV=herodev` | SSHes into VM, pulls new image, restarts container | Promotion: `make demo` tags `:dev` → `:demo`, pushes, deploys to herodemo. Stale files to delete: `Dockerfile` (old hero_zero) and `Dockerfile.prod` (SSH-based BuildKit) — both replaced by `Dockerfile.pack` + `docker/build-local.sh`. --- ## Smoke Tests Three bash/curl test suites verify the live gateway remotely. ~110 tests, ~30 seconds, zero dependencies beyond curl/bash/python3. ```bash cd deploy/single-vm make test ENV=herodev # or ENV=herodemo ``` | Suite | Tests | Coverage | |-------|-------|----------| | `smoke_gateway.sh` | 47 | SPA routing, auth flow (challenge→login→validate_session), inspector discovery (33 services, 12 with methods), RPC discovery (OSIS 509 methods, Books 19, Inspector 18), MCP gateway, WebDAV (fossil), SSE, 16 health checks, 10 UI HTML pages | | `smoke_test.sh` | 52 | All UI health endpoints (12 services), page content verification (cloud, books, fossil/HeroFoundry, redis, inspector, os, etc.), RPC calls (OSIS, Books, Inspector, AIBroker, Embedder), WASM island RPC, 7 server-side health checks, socat-bridged services (forge, shrimp) | | `smoke_theme.sh` | 14 | `hero:theme` postMessage listener present in all 14 iframe-embedded services | How each test works — just curl: - **Health:** `curl /hero_redis_ui/health` → expect HTTP 200 - **HTML pages:** `curl /hero_cloud_ui/` → expect HTTP 200 + `content-type: text/html` - **JSON-RPC:** `curl -X POST /hero_osis_ui/rpc/root` with JSON body → expect `"jsonrpc"` in response - **RPC discovery:** call `rpc.discover`, count methods ≥ threshold - **Auth flow:** full challenge → login → validate_session cycle with admin/admin - **Inspector:** call `inspector.services`, check 33 services found, 12 have methods - **WebDAV:** `curl -X OPTIONS /hero_fossil_server/webdav/` → expect 204 - **SSE:** `curl --max-time 3` → check `text/event-stream` header - **MCP:** POST JSON-RPC to `/hero_inspector_ui/mcp/hero_redis_server` - **Theme:** check HTML contains `hero:theme` listener **The smoke tests are the safety gate for `:demo` promotion.** When `make test ENV=herodev` passes, all services are healthy, auth works, RPC responds, UI renders, themes sync. The fix→deploy→test cycle is 3-5 minutes. --- ## Test Bug Fixes (completed) 6 bugs in the test scripts themselves (not service bugs): | File | Bug | Fix | |------|-----|-----| | `smoke_gateway.sh` | Inspector method `services.list` doesn't exist | → `inspector.services` | | `smoke_gateway.sh` | Inspector field `methods` doesn't exist | → `methods_count` | | `smoke_gateway.sh` | `hero_indexer_ui` in health list but no socket | Removed | | `smoke_test.sh` | Checks `base href="/zinit_ui/"` but zinit uses `<meta name="base-path">` | Fixed match | | `smoke_test.sh` | Embedder `rpc.discover` not implemented | → `health` method | | `smoke_test.sh` | Fossil RPC test but fossil uses WebDAV, not JSON-RPC | Removed | Coverage extended: gateway 20→47 tests, service 35→52 tests. All 15 services from `services/user/*.toml` now covered. --- ## Service Bugs Found Smoke tests against herodev.gent02.grid.tf (2026-03-11) found 3 genuine service bugs: | Repo | Bug | Impact | |------|-----|--------| | **hero_redis** | hero_redis_ui HTTP 500: `Template 'login.html' not found` | Page completely broken | | **hero_indexer / hero_indexer_ui** | hero_indexer_ui HTTP 404 — no UI socket registered with proxy | Service unreachable via gateway | | **zinit** (geomind_code) | zinit_ui missing `hero:theme` postMessage listener | Theme sync broken when embedded in iframe | Results before fixes: ``` smoke_gateway.sh: 46 passed, 0 failed, 1 skipped smoke_test.sh: 44 passed, 1 failed (hero_redis_ui 500) smoke_theme.sh: 11 passed, 3 failed (redis 500, indexer 404, zinit no theme) ``` --- ## DevOps Workflow ### Branching & PRs **One issue = one commit on `development`.** The PR squash merge handles this automatically. ``` development_{name} ← work freely, many commits, push often ↓ PR with "closes #N" ↓ Forgejo "Squash commit" merge development ← one clean commit per issue ``` Flow: 1. Create `development_{name}` branch from `development` (same name across all affected repos) 2. Work freely — commit as often as you want, push after every test cycle 3. The branch is your scratch pad — full history preserved in the PR for review 4. When done: create PR with `closes #13` in the description 5. Merge using **"Squash commit"** on Forgejo 6. Result: one clean commit on `development` with the issue URL as message No manual squashing, no force pushing, no rewriting history. Forgejo does it. Commit message format (set by Forgejo on squash): `https://forge.ourworld.tf/lhumina_code/home/issues/13 — Smoke Tests, Production Deployment & Cleanup` **Rules:** - Never rebase — merge `development` INTO feature branch if needed - Never commit directly to `main` — releases only via PR from `development` - PRs into `development`: **Squash commit** - PRs into `main`: **Create merge commit** (preserves release boundary) ### 11-Step Pipeline (3 human gates) | Step | Action | Who | Gate | |------|--------|-----|------| | 1 | Fix code on `development_{name}` in each repo | AI | — | | 2 | `make deploy` (1-3 min incremental) | AI | Must compile | | 3 | `make test ENV=herodev` (30 sec, ~110 tests) | AI | — | | — | _Iterate steps 1-3 until green. Push freely._ | AI | — | | 4 | **Human verifies herodev** | Human | ✓ Must confirm | | 5 | Create PRs: `development_{name}` → `development` with `closes #N` | AI | Only after step 4 | | 6 | **Squash merge** PRs on Forgejo | AI/Human | — | | 7 | `git checkout development && git pull` all repos, `make deploy` | AI | Clean build | | 8 | **Human verifies herodev** (clean merged code) | Human | ✓ Must confirm | | 9 | `make demo` (tag :dev→:demo, deploy herodemo) | AI | Only after step 8 | | 10 | **Human verifies herodemo** | Human | ✓ Must confirm | | 11 | Update issue log with session, commits, image SHA | AI | — | **Why two builds (steps 2 and 7):** Step 2 builds from local feature branch (fast iteration). Step 7 builds from clean merged `development` (proves the pushed code works). **Critical rules:** - Never push code before human confirms herodev (step 4 gate) - Never tag `:demo` before human confirms clean build (step 8 gate) - herodev is the development playground; herodemo is protected - No hot-swapping WASM — always full `make deploy` --- ## Tasks ### 1. Smoke Tests - [x] Run existing tests against herodev, fix test bugs - [x] Extend coverage to all 15 services (~110 tests) - [x] Add `make test` target running all 3 suites - [ ] Fix 3 service bugs (redis template, indexer socket, zinit theme) - [ ] All ~110 tests green on herodev - [ ] Human confirms herodev - [ ] Promote to herodemo, all tests green ### 2. Infrastructure Cleanup - [ ] Delete stale `Dockerfile` and `Dockerfile.prod` - [ ] Fix herodemo terraform state (points to unreachable VM, actual container on herodev VM) - [ ] Fix `dist/` WASM root-owned files in `build-local.sh` ### 3. Branch Cleanup - [ ] Merge hero_aibroker `development_theme_sync` → `development` - [ ] Delete stale `development_mik5` branches across repos - [ ] Run `cargo test` on zinit ### 4. Production Deployment (after smoke tests green) - [ ] Tag `:dev` as `:prod`, push - [ ] Deploy heroprod (needs terraform provisioning or reuse existing VM) - [ ] Human approval gate — NO auto-deploy to prod ### 5. Build Pipeline CI (nice-to-have) - [ ] Forgejo Actions workflow: on push to `development`, run `make dist` + `make pack` - [ ] Publish dist tarball as release artifact --- ## Environments | Tier | Gateway | Port | Image | Container | |------|---------|------|-------|-----------| | dev | `herodev.gent02.grid.tf` | 8805 | `hero_zero:dev` | `herodev` | | demo | `herodemo.gent02.grid.tf` | 8806 | `hero_zero:demo` | `herodemo` | | prod | TBD | TBD | `hero_zero:prod` | `heroprod` | **VM:** Both containers on same VM at Mycelium IP `495:72fa:8ec3:9264:ff0f:c0a8:abad:234c` **Registry:** `forge.ourworld.tf/lhumina_code/hero_zero` --- ## Current Status **Step 3 complete** — smoke tests assessed, test script bugs fixed, 3 service bugs identified. Ready for step 1 (implement service fixes).
mik-tf added this to the ACTIVE project 2026-03-11 02:11:59 +00:00
mik-tf commented 2026-03-11 03:41:17 +00:00
Author
Owner
Copy link

Step 5 complete — PRs created (branch: development_mik6)

3 service bugs fixed:

Bug Root Cause Fix PR
hero_redis_ui HTTP 500 Dockerfile.pack template COPY paths wrong (/build/... vs /src/...) Fix paths to match CARGO_MANIFEST_DIR hero_services#48
hero_indexer_ui 404 Default bind TCP 127.0.0.1:9753, no Unix socket created Default to ~/hero/var/sockets/hero_indexer_ui.sock hero_indexer_ui#1
zinit_ui theme broken Missing hero:theme postMessage listener Add listener to base.html zinit#52

Smoke test results (107 passed, 0 failed):

  • smoke_gateway.sh: 47 passed, 1 skipped (SSE needs auth)
  • smoke_test.sh: 46 passed
  • smoke_theme.sh: 14 passed

Pipeline status:

  • Step 1: Implement on development_mik6
  • Step 2: make deploy
  • Step 3: make test ENV=herodev — all green
  • Step 4: Human verified herodev
  • Step 5: PRs created
  • Step 6: Merge PRs into development
  • Step 7: Clean make deploy from merged development
  • Step 8: Human verifies herodev
  • Step 9: make demo
  • Step 10: Human verifies herodemo
  • Step 11: Update issue log
## Step 5 complete — PRs created (branch: `development_mik6`) ### 3 service bugs fixed: | Bug | Root Cause | Fix | PR | |-----|-----------|-----|----| | hero_redis_ui HTTP 500 | Dockerfile.pack template COPY paths wrong (`/build/...` vs `/src/...`) | Fix paths to match `CARGO_MANIFEST_DIR` | [hero_services#48](https://forge.ourworld.tf/lhumina_code/hero_services/pulls/48) | | hero_indexer_ui 404 | Default bind TCP `127.0.0.1:9753`, no Unix socket created | Default to `~/hero/var/sockets/hero_indexer_ui.sock` | [hero_indexer_ui#1](https://forge.ourworld.tf/lhumina_code/hero_indexer_ui/pulls/1) | | zinit_ui theme broken | Missing `hero:theme` postMessage listener | Add listener to `base.html` | [zinit#52](https://forge.ourworld.tf/geomind_code/zinit/pulls/52) | ### Smoke test results (107 passed, 0 failed): - `smoke_gateway.sh`: 47 passed, 1 skipped (SSE needs auth) - `smoke_test.sh`: 46 passed - `smoke_theme.sh`: 14 passed ### Pipeline status: - [x] Step 1: Implement on `development_mik6` - [x] Step 2: `make deploy` - [x] Step 3: `make test ENV=herodev` — all green - [x] Step 4: Human verified herodev - [x] Step 5: PRs created - [ ] Step 6: Merge PRs into `development` - [ ] Step 7: Clean `make deploy` from merged `development` - [ ] Step 8: Human verifies herodev - [ ] Step 9: `make demo` - [ ] Step 10: Human verifies herodemo - [ ] Step 11: Update issue log
mik-tf commented 2026-03-11 03:59:21 +00:00
Author
Owner
Copy link

Pipeline complete — all 11 steps done

Final results

Env URL Status
herodev https://herodev.gent02.grid.tf ✅ 107 tests green
herodemo https://herodemo.gent02.grid.tf ✅ Verified by human

Steps completed

  • 1. Implement on development_mik6
  • 2. make deploy
  • 3. make test ENV=herodev — 107 passed, 0 failed
  • 4. Human verified herodev
  • 5. PRs created and merged
  • 6. Merge PRs into development
  • 7. Clean make deploy from merged development — all green
  • 8. Human verified herodev (clean build)
  • 9. make demo — :dev tagged as :demo, deployed
  • 10. Human verified herodemo
  • 11. Issue log updated

Additional fix

  • Fixed herodemo deploy: added SSH_HOST/GATEWAY_FQDN override in app.env so make update ENV=herodemo works (herodemo shares the herodev VM, stale terraform state pointed to unreachable IP)

Merged PRs

## Pipeline complete — all 11 steps done ### Final results | Env | URL | Status | |-----|-----|--------| | herodev | https://herodev.gent02.grid.tf | ✅ 107 tests green | | herodemo | https://herodemo.gent02.grid.tf | ✅ Verified by human | ### Steps completed - [x] 1. Implement on `development_mik6` - [x] 2. `make deploy` - [x] 3. `make test ENV=herodev` — 107 passed, 0 failed - [x] 4. Human verified herodev - [x] 5. PRs created and merged - [x] 6. Merge PRs into `development` - [x] 7. Clean `make deploy` from merged `development` — all green - [x] 8. Human verified herodev (clean build) - [x] 9. `make demo` — `:dev` tagged as `:demo`, deployed - [x] 10. Human verified herodemo - [x] 11. Issue log updated ### Additional fix - Fixed herodemo deploy: added `SSH_HOST`/`GATEWAY_FQDN` override in `app.env` so `make update ENV=herodemo` works (herodemo shares the herodev VM, stale terraform state pointed to unreachable IP) ### Merged PRs - [hero_services#48](https://forge.ourworld.tf/lhumina_code/hero_services/pulls/48) — Dockerfile.pack template paths + smoke tests - [hero_indexer_ui#1](https://forge.ourworld.tf/lhumina_code/hero_indexer_ui/pulls/1) — Unix socket default - [zinit#52](https://forge.ourworld.tf/geomind_code/zinit/pulls/52) — theme listener
Sign in to join this conversation.
No Branch/Tag specified
Branches Tags
development
No results found.
Labels
Clear labels
No items
No labels
Milestone
Clear milestone
No items
No milestone
Projects
Clear projects
No items
No project
ACTIVE
Assignees
Clear assignees
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
lhumina_code/home#13
No description provided.