-
Stefy Lanza (nextime / spora ) authored
Two additions, one goal: no text model is stuck on this box. 1. codai/backends/remote_openai.py — set `service_url` on a model entry and coderai proxies that model to any OpenAI-compatible endpoint (another coderai, llama-server, vLLM, SGLang, a rented pod) instead of loading it. `served_model` renames it for the remote, `api_key` adds a bearer token. It reuses RunPod's request/SSE machinery and only replaces endpoint resolution, so streaming and usage accounting are the same code path. This is what makes the GGUF/llama.cpp catalogue remotizable: those models load in-process through llama-cpp-python, so there is no service to redirect — pointing the *model* at a URL is the only redirect available. The manager checks it first, ahead of every engine route, and `_model_is_runpod` grew into `_model_is_remote` so such models are never downloaded, cached, or charged local VRAM (old name kept as an alias). 2. RunPod pods pick their server per model. A GGUF cannot simply go to vLLM: its GGUF loader is experimental — single file only, a limited architecture list, and it still wants the original repo's tokenizer. So `engine: auto` sends an HF repo to vLLM and a GGUF to llama.cpp's own server image (ghcr.io/ggml-org/llama.cpp:server-cuda), which downloads the weights itself from `hf_gguf` ("user/repo:Q4_K_M") — a local /AI/….gguf path means nothing on a rented machine, and we don't upload multi-GB weights. `engine: vllm|llamacpp` overrides. Both servers answer /v1/models, so the readiness probe is unchanged. Tested against fake servers: routing, /v1 normalisation, auth header, chat, SSE streaming, usage, and each engine-selection branch. Co-Authored-By:Claude Opus 5 (1M context) <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
6c6646c4
| Name |
Last commit
|
Last update |
|---|---|---|
| .. | ||
| __init__.py | ||
| base.py | ||
| colibri.py | ||
| colibri_families.py | ||
| cuda.py | ||
| ds4.py | ||
| k3.py | ||
| ktransformers.py | ||
| remote_openai.py | ||
| runpod.py | ||
| vllm.py | ||
| vulkan.py |