• Stefy Lanza (nextime / spora )'s avatar
    cluster: mDNS discovery with a shared token, prefix-cache routing, Prometheus... · 0d4a338b
    Stefy Lanza (nextime / spora ) authored
    cluster: mDNS discovery with a shared token, prefix-cache routing, Prometheus /metrics + per-key usage, node logs and re-push on return, GGUF row split, cosign-signed images
    
    - codai/cluster/discovery.py: _coderai._tcp announce + browse (zeroconf), HMAC
      fingerprint of cluster.token as membership; members auto-join as nodes,
      listed nodes with a blank key get the shared token; nodes and engines accept
      the cluster token (front /cluster/*, engine BearerAuth); env overrides for
      capability images (CODERAI_CLUSTER_TOKEN/DISCOVERY/NODE_NAME/ADVERTISE_HOST);
      coderai-docker --host-network
    - frontproxy/affinity.py + router prefer=: a conversation goes back to the
      engine/node that served it when the model is resident in several places;
      engine_for_model now picks the least busy holder
    - frontproxy/metrics.py: /metrics (requests per key/model/kind/status/engine,
      engines/nodes up, VRAM, in-flight, RPC, discovery, RunPod spend), usage
      tables on the Cluster page, server.metrics_public
    - node logs: /cluster/logs on the node, /admin/api/cluster/logs on the head,
      log viewer on the Cluster page; a node that comes back gets its assigned
      models pushed again
    - per-model split_mode=row (llama.cpp tensor parallel over local + RPC devices);
      vLLM TP may span nodes (documented)
    - packaging/sign-images.sh + packaging/cosign.pub: all 21 GHCR images signed
    - docs/cluster.md §§ discovery, 7 prefix routing, 9 operating, 10 accelerators
      evaluation; tests/test_cluster_ops.py (11)
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01FKm7Z56iKjHgNkHc6VhWKz
    0d4a338b
Name
Last commit
Last update
..
__init__.py Loading commit data...
compat.py Loading commit data...
components.py Loading commit data...
ddp.py Loading commit data...
discovery.py Loading commit data...
fanout.py Loading commit data...
multinode.py Loading commit data...
nodes.py Loading commit data...
overview.py Loading commit data...
rpc.py Loading commit data...