• Stefy Lanza (nextime / spora )'s avatar
    shared pools: a name is not a path (v0.3.86) · 48fbcf76
    Stefy Lanza (nextime / spora ) authored
    A shared pool is one pod serving several names over one copy of the weights —
    a base model and its adapters. Live, every request naming the adapter got
    
        404 Client Error: Not Found for url: https://…/v1/chat/completions
    
    and kept getting it: the pool keeps the pod warm, so nothing replaced it. The
    pod, asked directly, listed only the base and reported {"loras": []}. Which of
    the two names worked depended on which model happened to rent the pod, so it
    looked intermittent rather than wrong.
    
    Five places decided "same model?" by comparing `path`, and a LoRA's path IS its
    base's path:
    
    * the orchestrator seeded the other pool members with no served name, so they
      arrived with no alias and were registered under their path;
    * the pod's boot seeder skipped the second entry as already known;
    * register_model_runtime did the same, which made teaching a running pod a
      no-op that answered "already known";
    * the engine-side seed filter in main.py did it a third time;
    * and vLLM's service key included the served name, so had the adapter got
      through it would have started a SECOND copy of a 27B on the same card.
    
    Identity is now (weights, name) in all five, and one engine serves one set of
    weights under every name configured for it: the base under its own,
    each adapter through --lora-modules, which --served-model-name must not repeat.
    
    Adoption checks the image, the GPU class and the bearer token — none of which
    say anything about the catalogue. So a pod that cannot serve what the pool
    promises is now TAUGHT: the adapter bytes are pushed, the entry is registered,
    and the same request is retried on the same pod. Renting a second card to serve
    a name the running one can hold is the answer this avoids. Once per pod per
    model; a pod that still refuses is a real failure.
    
    An engine is also relaunched when it does not serve the name asked of it: vLLM
    registers adapters at LAUNCH, and a pod is taught after it boots. Only an
    ANSWERED probe that omits the name forces that — restarting a loaded 27B
    because one probe timed out would be worse than the doubt.
    
    Finally, raise_for_status() discarded the pod's body, which is the whole
    message: it names the models the pod actually has. The caller saw only the
    status line, which reads like a routing fault. The pod's own words now travel.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    48fbcf76
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...