• Stefy Lanza (nextime / spora )'s avatar
    seed an offloaded pipeline, and stop advertising a direction a model cannot go · e0a3c0f8
    Stefy Lanza (nextime / spora ) authored
    **A seeded request to an offloaded pipeline failed as a load error.** Sequential
    CPU offload leaves every module on the META device until its own forward pulls
    it in, and a pipeline whose modules are all on meta reports meta as its device.
    torch.Generator(device=pipeline.device) then reached torch's accelerator hooks
    with a meta device:
    
        RuntimeError: META device type not an accelerator.
    
    The image route reported that as "Failed to load image model
    'Tongyi-MAI/Z-Image-Turbo'" -- naming the load, which had in fact succeeded
    (footprint measured, 1.56 GB offloaded to host RAM) -- so the model read as
    unloadable. And only a SEEDED request failed, so it looked broken to anyone who
    passes a seed and perfectly fine to anyone who does not.
    
    diffusers already knows the answer: _execution_device is where an offloaded
    pipeline actually computes. models.utils.generator_device() prefers it, rejects
    meta and lazy, and falls back to cuda/cpu. Four sites constructed a Generator
    from a pipeline's own device -- text-to-image, img2img, edit, and audio
    generation -- and fixing one would have left the same failure reachable by three
    other routes, so all four go through it.
    
    **Four working edit models were reported as broken because of their capability
    list.** The Qwen-Image-Edit siblings carried image_generation and modes: null,
    while diffusers maps no text-to-image class for the family and the request path
    says so outright:
    
        501 QwenImageEditPlusPipeline cannot generate from a prompt alone:
            diffusers offers no text-to-image pipeline for this family.
    
    The runtime was right and the capability list was wrong, which is the wrong way
    round. The instruction-edit branch of the heuristic lumped two different kinds
    of model together: FLUX.2 is unified -- one checkpoint that generates from text
    AND edits -- while Qwen-Image-Edit, FLUX.1 Kontext, Step1X-Edit and the
    Rapid-AIO fine-tunes are edit-only. They are split now, so only a unified family
    claims image_generation.
    
    And the listing publishes the directions. `modes` is how a client learns a model
    has no text-to-image path without sending one and reading a 501, and most cards
    never set it; when a card says nothing, the modes are now inferred from the
    capabilities the model already publishes (t2i/i2i, t2v/i2v/v2v). Inferred, never
    invented: every mode is backed by a flag that was already there, and a card
    that states its own modes always wins.
    
    Verified live while testing this: Wan2.2-I2V DOES serve a prompt-only request
    (it returned an mp4), so that row was collateral from the VRAM contention
    rather than a missing path.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_014T3rZozi1Ehk96NG1WesgC
    e0a3c0f8
test_seeded_generator_device.py 3.92 KB