• Stefy Lanza (nextime / spora )'s avatar
    llm: make every text model remotizable, GGUF included · 6c6646c4
    Stefy Lanza (nextime / spora ) authored
    Two additions, one goal: no text model is stuck on this box.
    
    1. codai/backends/remote_openai.py — set `service_url` on a model entry
       and coderai proxies that model to any OpenAI-compatible endpoint
       (another coderai, llama-server, vLLM, SGLang, a rented pod) instead of
       loading it. `served_model` renames it for the remote, `api_key` adds a
       bearer token. It reuses RunPod's request/SSE machinery and only
       replaces endpoint resolution, so streaming and usage accounting are
       the same code path.
    
       This is what makes the GGUF/llama.cpp catalogue remotizable: those
       models load in-process through llama-cpp-python, so there is no
       service to redirect — pointing the *model* at a URL is the only
       redirect available. The manager checks it first, ahead of every engine
       route, and `_model_is_runpod` grew into `_model_is_remote` so such
       models are never downloaded, cached, or charged local VRAM (old name
       kept as an alias).
    
    2. RunPod pods pick their server per model. A GGUF cannot simply go to
       vLLM: its GGUF loader is experimental — single file only, a limited
       architecture list, and it still wants the original repo's tokenizer.
       So `engine: auto` sends an HF repo to vLLM and a GGUF to llama.cpp's
       own server image (ghcr.io/ggml-org/llama.cpp:server-cuda), which
       downloads the weights itself from `hf_gguf` ("user/repo:Q4_K_M") —
       a local /AI/….gguf path means nothing on a rented machine, and we
       don't upload multi-GB weights. `engine: vllm|llamacpp` overrides.
       Both servers answer /v1/models, so the readiness probe is unchanged.
    
    Tested against fake servers: routing, /v1 normalisation, auth header,
    chat, SSE streaming, usage, and each engine-selection branch.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    6c6646c4
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
gpu_lock.py Loading commit data...
gpu_query.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
peft_compat.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...