• Stefy Lanza (nextime / spora )'s avatar
    cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang... · f5ca2d82
    Stefy Lanza (nextime / spora ) authored
    cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form
    
    Several coderai installs as one:
    
    - Cluster nodes (cluster.nodes): another install is an engine of this front —
      polled on its /cluster/state, assigned models, pinned by name, routed by
      capability and load; entries pushed to /cluster/reload-config and localized
      there; Load/Unload reach it through token-guarded twins. Live add/remove.
    - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints
      through ctypes and gives every load an explicit device list (local cards,
      then the model's rpc_servers); the split and the VRAM pool see remote
      memory. cluster.rpc_servers runs this machine's rpc-server processes;
      packaging/build-rpc-server.sh builds it from the vendored llama.cpp;
      build.sh and the images build llama-cpp-python with GGML_RPC.
    - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model
      vllm/kt blocks over the global config, node start/stop commands, ray head
      and worker mode in boot.sh.
    - Pools: several URLs per capability remote and several machines per host
      model, with failover.
    - Models page: engine/backend selects narrowed live to compatible pairs,
      impossible pairs refused; Settings → Cluster; /admin/cluster page.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz
    f5ca2d82
ui_pages.py 7.63 KB