• Stefy Lanza (nextime / spora )'s avatar
    Add vLLM as a first-class managed engine backend (LLM + OCR Surya2) · a5d9fe82
    Stefy Lanza (nextime / spora ) authored
    vLLM integrated exactly like ds4/ktransformers — a managed external engine in an
    ISOLATED venv (it pins torch 2.13/cu13, conflicting with the main venv), proxied over
    its OpenAI HTTP API, selected per-model via a `backend: vllm` pin or the vllm.model_id
    alias (never auto-claimed). Continuous batching for high aggregate throughput.
    
    - config.py: VllmConfig (+ Config field, from_dict, to_dict)
    - codai/api/vllm_worker.py: managed vllm.entrypoints.openai.api_server subprocess in the
      isolated venv (venv resolver: config>/opt/coderai/vllm_venv>/cache>~/.coderai), /v1/models
      health gate, auto_build
    - codai/backends/vllm.py: VllmBackend OpenAI proxy (mirrors ktransformers)
    - manager: get_active_vllm_config, "vllm" in _ENGINE_BACKENDS, _vllm_name_claims=False,
      vllm_should_handle, load branch, text-accept, /v1/models surfacing
    - front-proxy: required_capability (vllm pin+alias), _DEFAULT_CAPS vllm on GPU nodes,
      assignment/engine_supervisor/app threading + reload list
    - admin: routes get/set, settings.html vLLM card, models.html dropdown option
    - requirements-vllm.txt (vllm==0.27.1); docs/vllm.md marked implemented; __version__ 0.1.86
    
    OCR Surya2-via-vLLM: ocr.surya_serve = local|vllm|llamacpp. In vllm mode the Surya engine
    serves surya_model (datalab-to/surya-ocr-2) through the vLLM backend and attaches via
    SURYA_INFERENCE_URL — the correct path for the latest "Surya2" VLM (llama-cpp-python's
    server hit a recurrent/hybrid KV-slot bug on surya-2).
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01Mw2KQiswmD69T45fTfjKwW
    a5d9fe82
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...