• Stefy Lanza (nextime / spora )'s avatar
    runpod: an embedding model on a capability pod had no capability (v0.3.77) · e3dc237b
    Stefy Lanza (nextime / spora ) authored
    Switching a model's `runpod.engine` from `coderai-vllm` to `coderai` -- the
    correct setting for a capability pod -- silently removed its capability.
    `_CODERAI_SERVER_ENGINES["coderai-vllm"]` had been supplying the classification;
    plain `coderai` supplies none, and `model_capability()` returned "".
    
    Both halves of the failure follow from that:
      * the front could not classify the request, so the model was loaded LOCALLY
        on a GPU-less orchestrator;
      * default_capability_image("") is "", so a blank image field raises
        "set `image` on the runpod block" at provisioning time.
    
    It presented as "tries to load on the wrong machine", which is a long way from
    "this entry has no capability".
    
    The cause: model_capability() receives a BARE DICT, and _model_entry_for()
    iterates md.values() -- it matches inside `embedding_models` and discards the
    section key. Nothing had ever told it that an entry in `embedding_models` is an
    embeddings model. Now it asks: by object identity first (the dict the loader
    holds), falling back to the id so a copied or rebuilt entry still resolves, and
    returning False on any failure because a capability GUESS is never worth an
    exception in a placement path. An explicit `capability` field still wins, which
    is what that field is for.
    
    Only `embedding_models` is inferred, deliberately. The other sections work
    today, and remapping them all to fix one would change placement for every model
    kind -- a blast radius this does not need. A test asserts the inference does not
    reach into text_models, image_models, video_models or tts_models.
    
    Found by a user report ("broken -- front loads bge-m3 locally on the GPU-less
    host; not in /v1/models") after I made exactly that config change. I had already
    noted this gap when the capability came back empty during the switch, said it
    was worth raising rather than working around, and then worked around it in the
    config anyway -- which is how it reached production.
    
    11 tests. Suite at the 25 pre-existing failures.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    e3dc237b
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...