-
Stefy Lanza (nextime / spora ) authored
v0.3.88 made the pod FIND the adapter. The live pod then said: [lora] 'diogene-alpha': adapter sha256:2e820c… is not in this machine's blob store — serving without it [vllm] launching: … --served-model-name cyankiwi/Qwen3.8-27B-AWQ-INT4 qwen38-awq Found, and dropped. A seed names an adapter by CONTENT ID — computed here so the pod can be told about it before it exists — and the bytes travel separately. They were pushed on the first `acquire`, but the pool's WARM-UP request loads the model seconds after boot, long before any acquire. So vLLM started without the adapter, and `--enable-lora` cannot be turned on afterwards. Two changes, both gated on the model declaring a `pool`, because a pool is the explicit statement that this engine serves several names over one copy of the weights: * a pool engine launches with `--enable-lora` and the capacity flags even with no adapter resolvable yet. The flags that must be decided at LAUNCH are the capacity ones — the machinery, the slot size, how many adapters may be live in one batch — not the list. Opening the door costs a buffer allocation; keeping it shut costs a full reload of a 27B on a card billed by the hour, the first time an adapter arrives. * an adapter is then handed to the RUNNING engine (`/v1/load_lora_adapter`, vLLM >= 0.9 behind VLLM_ALLOW_RUNTIME_LORA_UPDATING, which the launch sets whenever it enabled LoRA) instead of restarting it. The restart stays as the fallback and is still reached on every path that cannot use the API — an engine started without `--enable-lora`, an older vLLM, an adapter whose rank exceeds the slot size. A pod rented an hour ago may be running an older image than the one we would launch today, so "it should be supported" is not something this can assume. What this buys: one rented card, one copy of the base model, and QLoRAs added or swapped per request by name with no reload. Co-Authored-By:Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
5fb825ca