• Stefy Lanza (nextime / spora )'s avatar
    speak the prompt syntax every Stable-Diffusion client already uses (v0.4.89) · 777406dc
    Stefy Lanza (nextime / spora ) authored
    A CLIP text encoder reads 77 tokens, two of them markers. Handing a longer
    prompt to a diffusers pipeline does not fail: the tokenizer truncates it and
    the model renders what fits. The booru-tag fine-tune families (Pony,
    Illustrious, NoobAI) open with a run of quality tags, so the part that got
    dropped was the subject -- and the request answered 200 with an image, which
    is why it was easy to miss.
    
    Image prompts are now encoded here rather than by the pipeline, which is what
    makes the rest possible:
    
      * as many 75-token windows as the prompt needs, concatenated -- no limit
      * the A1111 weight grammar: (tag:1.3), [tag], bare (tag) = 1.1, nesting
        multiplies, \( escapes. Per-token scaling with the window mean restored,
        so emphasis does not drag the brightness along with it
      * BREAK, which closes the current window instead of blending tags meant
        for different subjects
      * clip_skip, mapped to diffusers' own convention -- SDXL counts from the
        penultimate layer, SD 1.x from the final output and needs the layer norm
        applied by hand. Getting that backwards shifts every image
    
    Three places were dropping the negative prompt outright. The sd.cpp path
    hardcoded negative_prompt='', so for every GGUF image model whatever the
    caller asked to avoid stopped at the API -- half the idiom, since the quality
    tags have a negative half (score_4, score_5, score_6). /v1/images/edits and
    /v1/images/inpaint had no field for one at all, so an edit could only ever be
    steered positively. All three now pass it, and take clip_skip and
    prompt_weighting with it.
    
    clip_skip and prompt_weighting are also per-model and server-wide settings,
    because a client speaking the plain OpenAI image API has no field to ask for
    clip_skip and a model trained at 2 needs it on every request. Request beats
    model beats global.
    
    Weighting defaults on, so a literal '(' in prose now weighs 1.1 -- the
    ecosystem's convention, with \( and prompt_weighting:false as the way out.
    What that cannot be allowed to change is prompts that already worked: against
    a real CLIPTextModel a short unweighted prompt encodes bit-identically to what
    diffusers produced, because CLIP pads with its own end-of-text token, so
    [bos]+toks+[eos]+pad... and [bos]+toks+pad...+[eos] are the same ids. The test
    records that reasoning, so a tokenizer where it stops holding is caught.
    
    Flux, SD3 and Z-Image carry a T5 with its own limit and conventions and keep
    their pipeline's native text path.
    
    49 tests; full suite 4410 passed with the same 15 pre-existing failures as
    0.4.88 (diff empty).
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66
    777406dc
main.py 80.9 KB