• Stefy Lanza (nextime / spora )'s avatar
    Add vLLM as a first-class managed engine backend (LLM + OCR Surya2) · a5d9fe82
    Stefy Lanza (nextime / spora ) authored
    vLLM integrated exactly like ds4/ktransformers — a managed external engine in an
    ISOLATED venv (it pins torch 2.13/cu13, conflicting with the main venv), proxied over
    its OpenAI HTTP API, selected per-model via a `backend: vllm` pin or the vllm.model_id
    alias (never auto-claimed). Continuous batching for high aggregate throughput.
    
    - config.py: VllmConfig (+ Config field, from_dict, to_dict)
    - codai/api/vllm_worker.py: managed vllm.entrypoints.openai.api_server subprocess in the
      isolated venv (venv resolver: config>/opt/coderai/vllm_venv>/cache>~/.coderai), /v1/models
      health gate, auto_build
    - codai/backends/vllm.py: VllmBackend OpenAI proxy (mirrors ktransformers)
    - manager: get_active_vllm_config, "vllm" in _ENGINE_BACKENDS, _vllm_name_claims=False,
      vllm_should_handle, load branch, text-accept, /v1/models surfacing
    - front-proxy: required_capability (vllm pin+alias), _DEFAULT_CAPS vllm on GPU nodes,
      assignment/engine_supervisor/app threading + reload list
    - admin: routes get/set, settings.html vLLM card, models.html dropdown option
    - requirements-vllm.txt (vllm==0.27.1); docs/vllm.md marked implemented; __version__ 0.1.86
    
    OCR Surya2-via-vLLM: ocr.surya_serve = local|vllm|llamacpp. In vllm mode the Surya engine
    serves surya_model (datalab-to/surya-ocr-2) through the vLLM backend and attaches via
    SURYA_INFERENCE_URL — the correct path for the latest "Surya2" VLM (llama-cpp-python's
    server hit a recurrent/hybrid KV-slot bug on surya-2).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01Mw2KQiswmD69T45fTfjKwW
    a5d9fe82