• Stefy Lanza (nextime / spora )'s avatar
    a pool pod keeps the LoRA door open (v0.3.89) · 5fb825ca
    Stefy Lanza (nextime / spora ) authored
    v0.3.88 made the pod FIND the adapter. The live pod then said:
    
        [lora] 'diogene-alpha': adapter sha256:2e820c… is not in this machine's
               blob store — serving without it
        [vllm] launching: … --served-model-name cyankiwi/Qwen3.8-27B-AWQ-INT4 qwen38-awq
    
    Found, and dropped. A seed names an adapter by CONTENT ID — computed here so
    the pod can be told about it before it exists — and the bytes travel
    separately. They were pushed on the first `acquire`, but the pool's WARM-UP
    request loads the model seconds after boot, long before any acquire. So vLLM
    started without the adapter, and `--enable-lora` cannot be turned on
    afterwards.
    
    Two changes, both gated on the model declaring a `pool`, because a pool is the
    explicit statement that this engine serves several names over one copy of the
    weights:
    
      * a pool engine launches with `--enable-lora` and the capacity flags even
        with no adapter resolvable yet. The flags that must be decided at LAUNCH
        are the capacity ones — the machinery, the slot size, how many adapters may
        be live in one batch — not the list. Opening the door costs a buffer
        allocation; keeping it shut costs a full reload of a 27B on a card billed
        by the hour, the first time an adapter arrives.
      * an adapter is then handed to the RUNNING engine
        (`/v1/load_lora_adapter`, vLLM >= 0.9 behind VLLM_ALLOW_RUNTIME_LORA_UPDATING,
        which the launch sets whenever it enabled LoRA) instead of restarting it.
    
    The restart stays as the fallback and is still reached on every path that
    cannot use the API — an engine started without `--enable-lora`, an older vLLM,
    an adapter whose rank exceeds the slot size. A pod rented an hour ago may be
    running an older image than the one we would launch today, so "it should be
    supported" is not something this can assume.
    
    What this buys: one rented card, one copy of the base model, and QLoRAs added
    or swapped per request by name with no reload.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    5fb825ca
runpod_worker.py 279 KB