[nu-demo] hero_aibroker modelsconfig.yml ships Groq-only models with no fallback chain #137

New issue

Closed

opened 2026-04-23 23:16:20 +00:00 by mik-tf · 1 comment

mik-tf commented

2026-04-23 23:16:20 +00:00

Owner

Symptom

When the user's Groq account hit a spend-limit during the demo, agent.chat routed to groq-primary models and failed completely with 401/400 (no fallback). 10 of 42 bundled models in modelsconfig.yml were groq-only or groq-priority-1 with no secondary provider.

Root cause

The bundled hero_aibroker/modelsconfig.yml includes:

7 groq-only models: llama-3.3-70b-versatile, llama-3.1-8b-instant, and 5 others.
3 multi-provider models where groq is priority-1 but no openrouter fallback is listed.

The auto-routing logic does not degrade to the next model on 401/403/429 from the primary — it fails the whole request.

Demo workaround (applied 2026-04-23)

Edited modelsconfig.yml to drop all groq backends (both in the repo copy AND the var/ copy — see companion issue on reseeding clobbering edits). Chat went through openrouter only after that.

Proper fix

(a) Every multi-provider model should list openrouter as a priority-2 fallback in modelsconfig.yml.
(b) hero_aibroker's auto-routing should automatically degrade to the next available model (or next provider on the same model) when the primary returns a retryable-like error (401/403/429).
(c) Consider removing groq-only models from the bundled default, or gating them behind a groq-enabled=true marker that's only on when a valid Groq key is present at startup.

Filed 2026-04-23 nu-shell demo bring-up. Signed-off-by: mik-tf

## Symptom When the user's Groq account hit a spend-limit during the demo, `agent.chat` routed to groq-primary models and failed completely with 401/400 (no fallback). 10 of 42 bundled models in `modelsconfig.yml` were groq-only or groq-priority-1 with no secondary provider. ## Root cause The bundled `hero_aibroker/modelsconfig.yml` includes: - 7 groq-only models: `llama-3.3-70b-versatile`, `llama-3.1-8b-instant`, and 5 others. - 3 multi-provider models where groq is priority-1 but no openrouter fallback is listed. The auto-routing logic does not degrade to the next model on 401/403/429 from the primary — it fails the whole request. ## Demo workaround (applied 2026-04-23) Edited `modelsconfig.yml` to drop all groq backends (both in the repo copy AND the var/ copy — see companion issue on reseeding clobbering edits). Chat went through openrouter only after that. ## Proper fix (a) Every multi-provider model should list openrouter as a priority-2 fallback in `modelsconfig.yml`. (b) hero_aibroker's auto-routing should automatically degrade to the next available model (or next provider on the same model) when the primary returns a retryable-like error (401/403/429). (c) Consider removing groq-only models from the bundled default, or gating them behind a `groq-enabled=true` marker that's only on when a valid Groq key is present at startup. Filed 2026-04-23 nu-shell demo bring-up. Signed-off-by: mik-tf