• Stefy Lanza (nextime / spora )'s avatar
    edit an image from an instruction, and load a GGUF component at all (v0.4.96) · 02eaca42
    Stefy Lanza (nextime / spora ) authored
    Requested for a client that edits photos by instruction -- "make the dress
    red", "add a necklace" -- which plain img2img cannot do at any strength,
    because `strength` is the fraction of the source it destroys before
    re-imagining it. Serving Qwen-Image-Edit / FLUX.1-Kontext class models turned
    up two latent bugs on the way.
    
    The instruction path. /v1/images/edits now shapes its call to what the
    pipeline accepts, from the signature, rather than branching on the model:
    
      QwenImageEditPlus / QwenImageEdit / FluxKontext   image ✓  strength ✗
      SDXL-Img2Img / QwenImageImg2Img / FluxImg2Img     image ✓  strength ✓
    
    Taking an `image` with no `strength` IS the distinction, and it generalises,
    so no list of class names has to be maintained. `strength` is dropped where it
    does not exist -- a caller cannot know which kind of model it is addressing,
    and these pipelines carry no **kwargs to swallow it, so passing it through
    would be a hard TypeError. `guidance_scale` becomes `true_cfg_scale`, the knob
    they actually read. Steps default to 4 at cfg 1.0 for them instead of
    inheriting img2img's 30 at 7.5, which would waste a minute an image and
    overcook the result. Up to three images may be given (`image` plus `images`)
    for a composed edit, conditioned at one shape; more is a 400.
    
    `instruction_edit` is reported in /v1/models so a client can hide the strength
    slider and treat the prompt as an instruction. Verified to classify the four
    Qwen/Kontext/FireRed ids and to leave Pony, Z-Image and Illustrious as img2img.
    
    component_models is new: a map from a component to ANOTHER repo, loaded and
    handed to the pipeline as a prebuilt object. Several Qwen-Image-Edit
    derivatives ship a transformer and nothing else -- the base supplies the VAE,
    the text encoder and the scheduler -- and there was previously no way to
    express that, so they could not be served at all. Its entry in
    component_quantization still applies, so a transformer too large for the card
    is quantized as it loads.
    
    Two latent bugs, both of which anyone using GGUF image components would have
    hit:
    
      * `gguf` was never a declared dependency. build_gguf_pipeline_components has
        existed through several releases and could not have worked: diffusers needs
        gguf>=0.10.0 for GGUFQuantizationConfig and only says so mid-load, as
        "Unable to load weights from checkpoint file". Now in requirements.txt,
        which build.sh installs.
      * from_single_file was called with no `config`, so it INFERRED the
        architecture -- and its fallback is Stable Diffusion 1.5. A Qwen-Image,
        Flux or Wan transformer therefore failed with "stable-diffusion-v1-5 does
        not appear to have a file named config.json". It now passes the base
        pipeline's own component subfolder, which was already at hand.
    
    Measured on an RTX 3090 after the fixes: the Q4_K_M transformer binds as
    QwenImageTransformer2DModel (13.23B params, 85s) and the assembled
    QwenImageEditPlusPipeline is 22.68 GiB resident with a 4-bit text encoder --
    which fits 24 GB but leaves 1.32 GiB and OOMs on 390 MiB of activations. That
    is a deployment setting, not a code one, and it is why the figure is measured
    here rather than asserted.
    
    Full suite 4558 passed with the same 15 pre-existing failures as 0.4.91 (diff
    empty).
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    02eaca42
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...