• Stefy Lanza (nextime / spora )'s avatar
    shared pools: a name is not a path (v0.3.86) · 48fbcf76
    Stefy Lanza (nextime / spora ) authored
    A shared pool is one pod serving several names over one copy of the weights —
    a base model and its adapters. Live, every request naming the adapter got
    
        404 Client Error: Not Found for url: https://…/v1/chat/completions
    
    and kept getting it: the pool keeps the pod warm, so nothing replaced it. The
    pod, asked directly, listed only the base and reported {"loras": []}. Which of
    the two names worked depended on which model happened to rent the pod, so it
    looked intermittent rather than wrong.
    
    Five places decided "same model?" by comparing `path`, and a LoRA's path IS its
    base's path:
    
    * the orchestrator seeded the other pool members with no served name, so they
      arrived with no alias and were registered under their path;
    * the pod's boot seeder skipped the second entry as already known;
    * register_model_runtime did the same, which made teaching a running pod a
      no-op that answered "already known";
    * the engine-side seed filter in main.py did it a third time;
    * and vLLM's service key included the served name, so had the adapter got
      through it would have started a SECOND copy of a 27B on the same card.
    
    Identity is now (weights, name) in all five, and one engine serves one set of
    weights under every name configured for it: the base under its own,
    each adapter through --lora-modules, which --served-model-name must not repeat.
    
    Adoption checks the image, the GPU class and the bearer token — none of which
    say anything about the catalogue. So a pod that cannot serve what the pool
    promises is now TAUGHT: the adapter bytes are pushed, the entry is registered,
    and the same request is retried on the same pod. Renting a second card to serve
    a name the running one can hold is the answer this avoids. Once per pod per
    model; a pod that still refuses is a real failure.
    
    An engine is also relaunched when it does not serve the name asked of it: vLLM
    registers adapters at LAUNCH, and a pod is taught after it boots. Only an
    ANSWERED probe that omits the name forces that — restarting a loaded 27B
    because one probe timed out would be worse than the doubt.
    
    Finally, raise_for_status() discarded the pod's body, which is the whole
    message: it names the models the pod actually has. The caller saw only the
    status line, which reads like a routing fault. The pod's own words now travel.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    48fbcf76
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
cluster Loading commit data...
compose Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
ocr Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
util Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...