• Stefy Lanza (nextime / spora )'s avatar
    llm: make every text model remotizable, GGUF included · 6c6646c4
    Stefy Lanza (nextime / spora ) authored
    Two additions, one goal: no text model is stuck on this box.
    
    1. codai/backends/remote_openai.py — set `service_url` on a model entry
       and coderai proxies that model to any OpenAI-compatible endpoint
       (another coderai, llama-server, vLLM, SGLang, a rented pod) instead of
       loading it. `served_model` renames it for the remote, `api_key` adds a
       bearer token. It reuses RunPod's request/SSE machinery and only
       replaces endpoint resolution, so streaming and usage accounting are
       the same code path.
    
       This is what makes the GGUF/llama.cpp catalogue remotizable: those
       models load in-process through llama-cpp-python, so there is no
       service to redirect — pointing the *model* at a URL is the only
       redirect available. The manager checks it first, ahead of every engine
       route, and `_model_is_runpod` grew into `_model_is_remote` so such
       models are never downloaded, cached, or charged local VRAM (old name
       kept as an alias).
    
    2. RunPod pods pick their server per model. A GGUF cannot simply go to
       vLLM: its GGUF loader is experimental — single file only, a limited
       architecture list, and it still wants the original repo's tokenizer.
       So `engine: auto` sends an HF repo to vLLM and a GGUF to llama.cpp's
       own server image (ghcr.io/ggml-org/llama.cpp:server-cuda), which
       downloads the weights itself from `hf_gguf` ("user/repo:Q4_K_M") —
       a local /AI/….gguf path means nothing on a rented machine, and we
       don't upload multi-GB weights. `engine: vllm|llamacpp` overrides.
       Both servers answer /v1/models, so the readiness probe is unchanged.
    
    Tested against fake servers: routing, /v1 normalisation, auth header,
    chat, SSE streaming, usage, and each engine-selection branch.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    6c6646c4
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...