• Stefy Lanza (nextime / spora )'s avatar
    cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang... · f5ca2d82
    Stefy Lanza (nextime / spora ) authored
    cluster: nodes as engines, one GGUF over llama.cpp RPC, vLLM/SGLang multi-node, pools, compat-filtered model form
    
    Several coderai installs as one:
    
    - Cluster nodes (cluster.nodes): another install is an engine of this front —
      polled on its /cluster/state, assigned models, pinned by name, routed by
      capability and load; entries pushed to /cluster/reload-config and localized
      there; Load/Unload reach it through token-guarded twins. Live add/remove.
    - llama.cpp RPC: codai/backends/ggml_rpc.py registers rpc-server endpoints
      through ctypes and gives every load an explicit device list (local cards,
      then the model's rpc_servers); the split and the VRAM pool see remote
      memory. cluster.rpc_servers runs this machine's rpc-server processes;
      packaging/build-rpc-server.sh builds it from the vendored llama.cpp;
      build.sh and the images build llama-cpp-python with GGML_RPC.
    - vLLM on Ray (pipeline parallel across nodes) and SGLang --nnodes: per-model
      vllm/kt blocks over the global config, node start/stop commands, ray head
      and worker mode in boot.sh.
    - Pools: several URLs per capability remote and several machines per host
      model, with failover.
    - Models page: engine/backend selects narrowed live to compatible pairs,
      impossible pairs refused; Settings → Cluster; /admin/cluster page.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz
    f5ca2d82
Name
Last commit
Last update
..
__init__.py Loading commit data...
_film_net.py Loading commit data...
_rife_ifnet.py Loading commit data...
app.py Loading commit data...
archive.py Loading commit data...
audio_backends.py Loading commit data...
audio_clean.py Loading commit data...
audio_gen.py Loading commit data...
audio_stems.py Loading commit data...
audiocraft_worker.py Loading commit data...
canary_worker.py Loading commit data...
characters.py Loading commit data...
colibri_worker.py Loading commit data...
crisperwhisper_worker.py Loading commit data...
custom_pipelines.py Loading commit data...
diarization.py Loading commit data...
ds4_kv_janitor.py Loading commit data...
ds4_worker.py Loading commit data...
embeddings.py Loading commit data...
environments.py Loading commit data...
faceswap.py Loading commit data...
fastlog.py Loading commit data...
h3_worker.py Loading commit data...
host_worker.py Loading commit data...
hub_compat.py Loading commit data...
images.py Loading commit data...
k3_worker.py Loading commit data...
kt_worker.py Loading commit data...
lipsync.py Loading commit data...
log.py Loading commit data...
lora_archs.py Loading commit data...
loras.py Loading commit data...
melotts_worker.py Loading commit data...
model_test.py Loading commit data...
models_transfer.py Loading commit data...
mux_remote.py Loading commit data...
ocr.py Loading commit data...
parler_worker.py Loading commit data...
pipelines.py Loading commit data...
pod_http.py Loading commit data...
pod_tls.py Loading commit data...
prompt_cache.py Loading commit data...
pyannote_worker.py Loading commit data...
ratelimit.py Loading commit data...
remote_gateway.py Loading commit data...
rerank.py Loading commit data...
runpod_client.py Loading commit data...
runpod_ledger.py Loading commit data...
runpod_worker.py Loading commit data...
spatial.py Loading commit data...
speaker_embeddings.py Loading commit data...
speaker_registry.py Loading commit data...
state.py Loading commit data...
stt_backends.py Loading commit data...
text.py Loading commit data...
transcriptions.py Loading commit data...
tts.py Loading commit data...
tts_backends.py Loading commit data...
urlutils.py Loading commit data...
video.py Loading commit data...
vllm_worker.py Loading commit data...
voice_clone.py Loading commit data...
voice_convert.py Loading commit data...
vpr_salad_server.py Loading commit data...
xtts_worker.py Loading commit data...