llmbroker

A lightweight LLM request broker with an OpenAI-compatible API. It routes requests to various providers (OpenAI, Groq, SambaNova, OpenRouter) with cost-aware strategies, and includes an MCP broker for aggregating tools from multiple sources.

Features

• OpenAI Compatible: Drop-in replacement for Chat, TTS, STT, and Embeddings.
• Smart Routing: Supports "cheapest" routing strategy to save costs.
• Provider Support: OpenAI, Groq, SambaNova, and OpenRouter.
• YAML Model Config: Define virtual model names with multiple backend providers.
• Automatic Model Mapping: Maps standard models (e.g., whisper-1, tts-1) to provider-specific optimized models.
• Observability: Built-in Prometheus metrics and structured logging.
• MCP Broker: Aggregates tools from multiple Model Context Protocol (MCP) servers into a single endpoint.
• Built-in MCP Servers: Includes search and scraping tools (SerpAPI, Serper, Exa, ScraperAPI, Scrapfly).

Quick Start

1. Configure: Copy .env.example to .env and add your API keys.
2. Run Server:

   make run
   

3. Use CLI:

   make cli
   

API Endpoints

Endpoint	Description
POST /v1/chat/completions	OpenAI-compatible chat completions
POST /v1/embeddings	Text embeddings
POST /v1/audio/speech	Text-to-speech
POST /v1/audio/transcriptions	Speech-to-text
GET /v1/models	List available models
GET /mcp/sse	MCP SSE endpoint for tool access
GET /mcp/tools	List aggregated MCP tools

MCP Broker

The MCP Broker aggregates tools from multiple MCP servers into a single endpoint. This allows LLM clients (like Claude Desktop, VS Code Kilo) to access multiple toolsets through one connection.

Built-in MCP Servers

llmbroker includes several MCP servers for search and web scraping:

Server	Command	Tools	Required Env Var
SerpAPI	go run ./cmd/mcp-serpapi	serpapi_search - Multi-engine search (Google, Bing, YouTube, eBay)	SERPAPI_API_KEY
Serper	go run ./cmd/mcp-serper	serper_search, serper_images, serper_news, serper_videos, serper_places	SERPER_API_KEY
Exa	go run ./cmd/mcp-exa	exa_search, exa_find_similar, exa_get_contents, exa_answer	EXA_API_KEY
ScraperAPI	go run ./cmd/mcp-scraperapi	scraperapi_scrape, scraperapi_post, scraperapi_account	SCRAPERAPI_API_KEY
Scrapfly	go run ./cmd/mcp-scrapfly	scrapfly_scrape, scrapfly_extract, scrapfly_screenshot	SCRAPFLY_API_KEY

Configuration

Create mcp_servers.json to configure which MCP servers to aggregate:

{
  "mcpServers": [
    {
      "name": "serpapi",
      "command": "go",
      "args": ["run", "./cmd/mcp-serpapi"],
      "env": {}
    },
    {
      "name": "serper",
      "command": "go",
      "args": ["run", "./cmd/mcp-serper"],
      "env": {}
    },
    {
      "name": "exa",
      "command": "go",
      "args": ["run", "./cmd/mcp-exa"],
      "env": {}
    },
    {
      "name": "scraperapi",
      "command": "go",
      "args": ["run", "./cmd/mcp-scraperapi"],
      "env": {}
    },
    {
      "name": "scrapfly",
      "command": "go",
      "args": ["run", "./cmd/mcp-scrapfly"],
      "env": {}
    }
  ]
}

Connecting Clients

VS Code (Kilo MCP Extension)

"mcp.servers": {
  "llmbroker": {
    "url": "http://localhost:8080/mcp/sse",
    "transport": "sse"
  }
}

Trae

{
  "mcpServers": {
    "llmbroker": {
      "url": "http://localhost:8080/mcp/sse",
      "transport": "sse"
    }
  }
}

See the MCP Broker Guide for more details.

Model Configuration

Models are configured in modelsconfig.yml. Each model can have multiple backend providers:

models:
  gpt4o:
    display_name: "GPT-4o"
    tier: premium
    capabilities: [tool_calling, vision]
    context_window: 128000
    backends:
      - provider: openrouter
        model_id: openai/gpt-4o
        priority: 1
        input_cost: 2.5
        output_cost: 10.0

The broker automatically selects the best available backend based on configured providers and routing strategy.

Environment Variables

Variable	Description
PORT	Server port (default: 8080)
OPENROUTER_API_KEY	OpenRouter API key
OPENAI_API_KEY	OpenAI API key
GROQ_API_KEY	Groq API key
SAMBANOVA_API_KEY	SambaNova API key
SERPAPI_API_KEY	SerpAPI key (for MCP server)
SERPER_API_KEY	Serper.dev key (for MCP server)
EXA_API_KEY	Exa API key (for MCP server)
SCRAPERAPI_API_KEY	ScraperAPI key (for MCP server)
SCRAPFLY_API_KEY	Scrapfly key (for MCP server)
MODELS_CONFIG_PATH	Path to models config (default: modelsconfig.yml)
MCP_SERVERS_PATH	Path to MCP servers config (default: mcp_servers.json)
ROUTING_STRATEGY	Routing strategy: "cheapest" or "best"

Documentation

• Examples: How to use the API with code examples.
• Development: Project structure, adding providers, and technical details.
• Data Flow: Detailed life cycle and transformation of data.
• MCP Broker Guide: Detailed guide for the MCP Broker.

License

MIT