• Stefy Lanza (nextime / spora )'s avatar
    shared pools: a name is not a path (v0.3.86) · 48fbcf76
    Stefy Lanza (nextime / spora ) authored
    A shared pool is one pod serving several names over one copy of the weights —
    a base model and its adapters. Live, every request naming the adapter got
    
        404 Client Error: Not Found for url: https://…/v1/chat/completions
    
    and kept getting it: the pool keeps the pod warm, so nothing replaced it. The
    pod, asked directly, listed only the base and reported {"loras": []}. Which of
    the two names worked depended on which model happened to rent the pod, so it
    looked intermittent rather than wrong.
    
    Five places decided "same model?" by comparing `path`, and a LoRA's path IS its
    base's path:
    
    * the orchestrator seeded the other pool members with no served name, so they
      arrived with no alias and were registered under their path;
    * the pod's boot seeder skipped the second entry as already known;
    * register_model_runtime did the same, which made teaching a running pod a
      no-op that answered "already known";
    * the engine-side seed filter in main.py did it a third time;
    * and vLLM's service key included the served name, so had the adapter got
      through it would have started a SECOND copy of a 27B on the same card.
    
    Identity is now (weights, name) in all five, and one engine serves one set of
    weights under every name configured for it: the base under its own,
    each adapter through --lora-modules, which --served-model-name must not repeat.
    
    Adoption checks the image, the GPU class and the bearer token — none of which
    say anything about the catalogue. So a pod that cannot serve what the pool
    promises is now TAUGHT: the adapter bytes are pushed, the entry is registered,
    and the same request is retried on the same pod. Renting a second card to serve
    a name the running one can hold is the answer this avoids. Once per pod per
    model; a pod that still refuses is a real failure.
    
    An engine is also relaunched when it does not serve the name asked of it: vLLM
    registers adapters at LAUNCH, and a pod is taught after it boots. Only an
    ANSWERED probe that omits the name forces that — restarting a loaded 27B
    because one probe timed out would be worse than the doubt.
    
    Finally, raise_for_status() discarded the pod's body, which is the whole
    message: it names the models the pod actually has. The caller saw only the
    status line, which reads like a routing fault. The pod's own words now travel.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    48fbcf76
test_shared_pool_adapter_identity.py 11.2 KB