Hermes CLI + litellm でカスタムエンドポイントを設定する方法を紹介しました
Hermes CLI 自体は Google AI Studio に対応しているので使えます
設定しているカスタムエンドポイント側の gemini が function calling や web_search に対応していなかったりリクエストやレスポンスの形式が少し異なる場合にはそのままでは使えません
services:litellm:image: docker.litellm.ai/berriai/litellm:latest
container_name: litellm
environment:- AI_SERVICE_API_KEY=${AI_SERVICE_API_KEY}ports:-"4000:4000"volumes:- ./litellm_config.yaml:/app/config.yaml
restart: unless-stopped
command:["--config","/app/config.yaml"]gemini-proxy:# Python proxy for gemini CLI (native Gemini API format).# Replaces nginx because request body JSON transformation is required:# strips unsupported 'id' field from function_call/function_response,# rewrites :streamGenerateContent?alt=sse → :streamGenerateContentSse,# and injects AI_SERVICE_API_KEY as x-goog-api-key header.# Configure gemini CLI to use port 4001 instead of 4000.build:context: .
dockerfile: Dockerfile.proxy
container_name: gemini-proxy
environment:- AI_SERVICE_API_KEY=${AI_SERVICE_API_KEY}ports:-"4001:4001"restart: unless-stopped
proxy.py
API を互換させるためのプロキシです
先ほども紹介しましたが custom endpoint が完全互換の場合は不要です
#!/usr/bin/env python3"""
Proxy for gemini CLI → Your custom Gemini endpoint.
Handles:
- URL mapping: :streamGenerateContent?alt=sse → :streamGenerateContentSse
- Request body: strip unsupported 'id' field from function_call / function_response
- Auth: replace incoming key with AI_SERVICE_API_KEY as x-goog-api-key header
"""import json
import os
import httpx
from fastapi import FastAPI, Request
from fastapi.responses import Response, StreamingResponse
CUSTOM_BASE = os.environ.get("CUSTOM_BASE","https://your-gemini-custom-endpoint",)
API_KEY = os.environ["AI_SERVICE_API_KEY"]
app = FastAPI()defstrip_unsupported_fields(body:dict)->dict:"""Strip 'id' from function_call/function_response — field not supported by this endpoint."""for content in body.get("contents",[]):for part in content.get("parts",[]):for key in("functionCall","function_call"):ifisinstance(part.get(key),dict):
part[key].pop("id",None)for key in("functionResponse","function_response"):ifisinstance(part.get(key),dict):
part[key].pop("id",None)return body
@app.post("/v1beta/models/{model}:{method}")asyncdefproxy(request: Request, model:str, method:str):
is_streaming = method =="streamGenerateContent"# Custom endpoint uses a dedicated SSE endpoint instead of ?alt=sse query param
target_method ="streamGenerateContentSse"if is_streaming else method
target_url = f"{CUSTOM_BASE}/{model}:{target_method}"
body_bytes =await request.body()try:
body = strip_unsupported_fields(json.loads(body_bytes))
body_bytes = json.dumps(body, ensure_ascii=False).encode()except(json.JSONDecodeError, AttributeError):pass
upstream_headers ={"Content-Type":"application/json","x-goog-api-key": API_KEY}if is_streaming:asyncdefgenerate():asyncwith httpx.AsyncClient(timeout=120.0)as client:asyncwith client.stream("POST", target_url, content=body_bytes, headers=upstream_headers
)as resp:asyncfor chunk in resp.aiter_bytes():yield chunk
return StreamingResponse(
generate(),
media_type="text/event-stream",
headers={"Cache-Control":"no-cache","X-Accel-Buffering":"no"},)else:asyncwith httpx.AsyncClient(timeout=120.0)as client:
resp =await client.post(
target_url, content=body_bytes, headers=upstream_headers
)return Response(
content=resp.content,
status_code=resp.status_code,
media_type=resp.headers.get("content-type","application/json"),)
Dockerfile.proxy
先程の proxy.py をビルドしてイメージを作成する dockerfile です
FROM python:3.12-slim
WORKDIR /app
RUN pip install --no-cache-dir fastapi httpx uvicorn
COPY proxy.py .
CMD["uvicorn","proxy:app","--host","0.0.0.0","--port","4001"]
model_list:-model_name: gemini-3.5-flash
litellm_params:# gemini/ prefix is required to use API key auth instead of Vertex AI ADC.# Model name must match model_name above to avoid LiteLLM response model override error# that corrupts streaming responses. drop_params: true handles unsupported fields.model: gemini/gemini-3.5-flash
api_base: https://your-gemini-custom-endpoint:generateContent
api_key: os.environ/AI_SERVICE_API_KEY
max_tokens:4096temperature:0.2rpm:30tpm:60000-model_name: gemini-3.1-flash-lite
litellm_params:# gemini/ prefix is required to use API key auth instead of Vertex AI ADC.model: gemini/gemini-3.5-flash
api_base: https://your-gemini-custom-endpoint:generateContent
api_key: os.environ/AI_SERVICE_API_KEY
litellm_settings:drop_params:truegeneral_settings:global_max_parallel_requests:1max_parallel_requests:1# Pass-through: forward native Gemini API requests (/v1beta/models/...) directly# to the upstream endpoint, bypassing LiteLLM's model routing and SDK layer.# This avoids the "cannot override response model" bug caused by GenerateContentResponse# not having a 'model' attribute in LiteLLM's google_genai provider path.pass_through_endpoints:-path:"/v1beta/models/{path:path}"target:"https://your-gemini-custom-endpoint/gemini/{path}"headers:x-goog-api-key: os.environ/AI_SERVICE_API_KEY
router_settings:num_retries:0retry_after:20allowed_fails:1cooldown_time:180timeout:120stream_timeout:120server_settings:port:4000
CONTAINER ID IMAGE COMMAND CREATED STATUS PORTS NAMES
385829c1f469 mcp/time "mcp-server-time" About a minute ago Up About a minute beautiful_bassi
26cb3f21bde5 ghcr.io/open-webui/open-webui:main "bash start.sh" 22 minutes ago Up 22 minutes (healthy) 0.0.0.0:3000->8080/tcp, [::]:3000->8080/tcp open-webui
e38fe5175159 ghcr.io/berriai/litellm:main-latest "docker/prod_entrypo…" 22 minutes ago Up 22 minutes 0.0.0.0:4000->4000/tcp, :::4000->4000/tcp litellm
OpenWebUI 上で MCPO サーバと接続設定をする
あとは open-webui 上から MCPO サーバに接続する設定をすれば OK です
設定からツールで MCPO を新規ツールとして接続します
ブラウザから直接アクセスするので localhost や local IP ではなくブラウザからアクセスできる IP アドレスを指定しましょう
また Bearer トークンに先ほど設定したシークレット情報を入力します
くるくるマークを押すと接続テストができるのでここでちゃんと接続できるか確認しておきましょう
動作確認
あとはチャットに戻り MCPO 経由で質問できるか確認します
まずチャット画面上でツールが有効になっているか確認します
ツールの数が 1 になっていれば OK です
また詳細を確認すると実際に MCPO で使えるコマンドが表示されます
あとは質問してみましょう
こんな感じでちゃんと MCPO 経由で回答が来ていることが確認できます
MCPO 側にも以下のようなログがあることが確認できると思います
INFO: 192.168.100.2:30848 - "POST /time/get_current_time HTTP/1.1" 200 OK
{"id":"chatcmpl-2062ec16-2314-4772-a264-6f987407689d","created":1745811058,"model":"gemini-flash","object":"chat.completion","system_fingerprint":null,"choices":[{"finish_reason":"stop","index":0,"message":{"content":"I am a large language model, trained by Google.\n","role":"assistant","tool_calls":null,"function_call":null}}],"usage":{"completion_tokens":12,"prompt_tokens":5,"total_tokens":17,"completion_tokens_details":null,"prompt_tokens_details":{"audio_tokens":null,"cached_tokens":null}},"vertex_ai_grounding_metadata":[],"vertex_ai_safety_results":[],"vertex_ai_citation_metadata":[]}
最後に
LiteLLM Proxy で Gemini flash を動作させる方法を紹介しました
これで Open WebUI のモデルを簡単に変更することができます
{"id":"chatcmpl-BOZAENZRJ1j3WErgssAmUj41gcl16","created":1745194742,"model":"gpt-4o-2024-08-06","object":"chat.completion","system_fingerprint":"fp_ee1d74bde0","choices":[{"finish_reason":"stop","index":0,"message":{"content":"I am based on OpenAI's GPT-4 architecture, which is a type of large language model (LLM) designed for understanding and generating human-like text.","role":"assistant","tool_calls":null,"function_call":null}}],"usage":{"completion_tokens":34,"prompt_tokens":12,"total_tokens":46,"completion_tokens_details":{"accepted_prediction_tokens":0,"audio_tokens":0,"reasoning_tokens":0,"rejected_prediction_tokens":0},"prompt_tokens_details":{"audio_tokens":0,"cached_tokens":0}},"service_tier":null,"prompt_filter_results":[{"prompt_index":0,"content_filter_results":{"hate":{"filtered":false,"severity":"safe"},"self_harm":{"filtered":false,"severity":"safe"},"sexual":{"filtered":false,"severity":"safe"},"violence":{"filtered":false,"severity":"safe"}}}]}
