Skip to content
  • Stefy Lanza (nextime / spora )'s avatar
    Add vLLM as a first-class managed engine backend (LLM + OCR Surya2) · a5d9fe82
    Stefy Lanza (nextime / spora ) authored
    vLLM integrated exactly like ds4/ktransformers — a managed external engine in an
    ISOLATED venv (it pins torch 2.13/cu13, conflicting with the main venv), proxied over
    its OpenAI HTTP API, selected per-model via a `backend: vllm` pin or the vllm.model_id
    alias (never auto-claimed). Continuous batching for high aggregate throughput.
    
    - config.py: VllmConfig (+ Config field, from_dict, to_dict)
    - codai/api/vllm_worker.py: managed vllm.entrypoints.openai.api_server subprocess in the
      isolated venv (venv resolver: config>/opt/coderai/vllm_venv>/cache>~/.coderai), /v1/models
      health gate, auto_build
    - codai/backends/vllm.py: VllmBackend OpenAI proxy (mirrors ktransformers)
    - manager: get_active_vllm_config, "vllm" in _ENGINE_BACKENDS, _vllm_name_claims=False,
      vllm_should_handle, load branch, text-accept, /v1/models surfacing
    - front-proxy: required_capability (vllm pin+alias), _DEFAULT_CAPS vllm on GPU nodes,
      assignment/engine_superviso...
    a5d9fe82