• Stefy Lanza (nextime / spora )'s avatar
    llm: make every text model remotizable, GGUF included · 6c6646c4
    Stefy Lanza (nextime / spora ) authored
    Two additions, one goal: no text model is stuck on this box.
    
    1. codai/backends/remote_openai.py — set `service_url` on a model entry
       and coderai proxies that model to any OpenAI-compatible endpoint
       (another coderai, llama-server, vLLM, SGLang, a rented pod) instead of
       loading it. `served_model` renames it for the remote, `api_key` adds a
       bearer token. It reuses RunPod's request/SSE machinery and only
       replaces endpoint resolution, so streaming and usage accounting are
       the same code path.
    
       This is what makes the GGUF/llama.cpp catalogue remotizable: those
       models load in-process through llama-cpp-python, so there is no
       service to redirect — pointing the *model* at a URL is the only
       redirect available. The manager checks it first, ahead of every engine
       route, and `_model_is_runpod` grew into `_model_is_remote` so such
       models are never downloaded, cached, or charged local VRAM (old name
       kept as an alias).
    
    2. RunPod pods pick their server per model. A GGUF cannot simply go to
       vLLM: its GGUF loader is experimental — single file only, a limited
       architecture list, and it still wants the original repo's tokenizer.
       So `engine: auto` sends an HF repo to vLLM and a GGUF to llama.cpp's
       own server image (ghcr.io/ggml-org/llama.cpp:server-cuda), which
       downloads the weights itself from `hf_gguf` ("user/repo:Q4_K_M") —
       a local /AI/….gguf path means nothing on a rented machine, and we
       don't upload multi-GB weights. `engine: vllm|llamacpp` overrides.
       Both servers answer /v1/models, so the readiness probe is unchanged.
    
    Tested against fake servers: routing, /v1 normalisation, auth header,
    chat, SSE streaming, usage, and each engine-selection branch.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0182DhR1eFNPQyGmDrpHbedY
    6c6646c4
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...