• Stefy Lanza (nextime / spora )'s avatar
    cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang... · f5ca2d82
    Stefy Lanza (nextime / spora ) authored
    cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form
    
    Several coderai installs as one:
    
    - Cluster nodes (cluster.nodes): another install is an engine of this front —
      polled on its /cluster/state, assigned models, pinned by name, routed by
      capability and load; entries pushed to /cluster/reload-config and localized
      there; Load/Unload reach it through token-guarded twins. Live add/remove.
    - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints
      through ctypes and gives every load an explicit device list (local cards,
      then the model's rpc_servers); the split and the VRAM pool see remote
      memory. cluster.rpc_servers runs this machine's rpc-server processes;
      packaging/build-rpc-server.sh builds it from the vendored llama.cpp;
      build.sh and the images build llama-cpp-python with GGML_RPC.
    - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model
      vllm/kt blocks over the global config, node start/stop commands, ray head
      and worker mode in boot.sh.
    - Pools: several URLs per capability remote and several machines per host
      model, with failover.
    - Models page: engine/backend selects narrowed live to compatible pairs,
      impossible pairs refused; Settings → Cluster; /admin/cluster page.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz
    f5ca2d82
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...