-
Stefy Lanza (nextime / spora ) authored
vLLM integrated exactly like ds4/ktransformers — a managed external engine in an ISOLATED venv (it pins torch 2.13/cu13, conflicting with the main venv), proxied over its OpenAI HTTP API, selected per-model via a `backend: vllm` pin or the vllm.model_id alias (never auto-claimed). Continuous batching for high aggregate throughput. - config.py: VllmConfig (+ Config field, from_dict, to_dict) - codai/api/vllm_worker.py: managed vllm.entrypoints.openai.api_server subprocess in the isolated venv (venv resolver: config>/opt/coderai/vllm_venv>/cache>~/.coderai), /v1/models health gate, auto_build - codai/backends/vllm.py: VllmBackend OpenAI proxy (mirrors ktransformers) - manager: get_active_vllm_config, "vllm" in _ENGINE_BACKENDS, _vllm_name_claims=False, vllm_should_handle, load branch, text-accept, /v1/models surfacing - front-proxy: required_capability (vllm pin+alias), _DEFAULT_CAPS vllm on GPU nodes, assignment/engine_superviso...
a5d9fe82
Loading