speak the prompt syntax every Stable-Diffusion client already uses (v0.4.89)

A CLIP text encoder reads 77 tokens, two of them markers. Handing a longer
prompt to a diffusers pipeline does not fail: the tokenizer truncates it and
the model renders what fits. The booru-tag fine-tune families (Pony,
Illustrious, NoobAI) open with a run of quality tags, so the part that got
dropped was the subject -- and the request answered 200 with an image, which
is why it was easy to miss.

Image prompts are now encoded here rather than by the pipeline, which is what
makes the rest possible:

  * as many 75-token windows as the prompt needs, concatenated -- no limit
  * the A1111 weight grammar: (tag:1.3), [tag], bare (tag) = 1.1, nesting
    multiplies, \( escapes. Per-token scaling with the window mean restored,
    so emphasis does not drag the brightness along with it
  * BREAK, which closes the current window instead of blending tags meant
    for different subjects
  * clip_skip, mapped to diffusers' own convention -- SDXL counts from the
    penultimate layer, SD 1.x from the final output and needs the layer norm
    applied by hand. Getting that backwards shifts every image

Three places were dropping the negative prompt outright. The sd.cpp path
hardcoded negative_prompt='', so for every GGUF image model whatever the
caller asked to avoid stopped at the API -- half the idiom, since the quality
tags have a negative half (score_4, score_5, score_6). /v1/images/edits and
/v1/images/inpaint had no field for one at all, so an edit could only ever be
steered positively. All three now pass it, and take clip_skip and
prompt_weighting with it.

clip_skip and prompt_weighting are also per-model and server-wide settings,
because a client speaking the plain OpenAI image API has no field to ask for
clip_skip and a model trained at 2 needs it on every request. Request beats
model beats global.

Weighting defaults on, so a literal '(' in prose now weighs 1.1 -- the
ecosystem's convention, with \( and prompt_weighting:false as the way out.
What that cannot be allowed to change is prompts that already worked: against
a real CLIPTextModel a short unweighted prompt encodes bit-identically to what
diffusers produced, because CLIP pads with its own end-of-text token, so
[bos]+toks+[eos]+pad... and [bos]+toks+pad...+[eos] are the same ids. The test
records that reasoning, so a tokenizer where it stops holding is caught.

Flux, SD3 and Z-Image carry a T5 with its own limit and conventions and keep
their pipeline's native text path.

49 tests; full suite 4410 passed with the same 15 pre-existing failures as
0.4.88 (diff empty).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01S8LAP3ueNzQJtwzhgVbq66

README.md

@@ -838,6 +838,11 @@ persisted per `config_id`, so the configs don't overwrite each other.
 | `POST /v1/images/from3d` | 3D model → rendered 2D image |
 | `GET /v1/images/progress` | Current generation progress (step/total) |
 
+Generations, edits and inpaint take `negative_prompt`, `clip_skip` and the
+`(tag:1.3)` / `[tag]` / `BREAK` prompt grammar, and encode prompts of any
+length (CLIP's 77-token limit does not apply) — see
+[docs/image-prompts.md](docs/image-prompts.md).
+
 ### Video
 
 | Endpoint | Description |

codai/__init__.py

@@ -16,7 +16,7 @@
 
 # Canonical product version for CoderAI — single source of truth. Both the API
 # metadata and the admin web UI read from here.
-__version__ = "0.4.88"
+__version__ = "0.4.89"
 __tagline__ = "Complete orchestration, distribution and escalation of remotizable advanced inference."
 
 # Configure the CUDA caching allocator BEFORE torch is imported anywhere.

codai/admin/routes.py

@@ -2927,6 +2927,11 @@ async def api_model_configure(request: Request, username: str = Depends(require_
                 "lora_scale",
                 "lora_train_base_model", "lora_train_nodes",
                 "max_vram", "sdcpp_flash_attn", "sdcpp_diffusion_flash_attn", "vae_tiling",
+                # Image prompt handling: which CLIP layer to read, and whether
+                # the A1111 weight grammar is parsed. Per model because the
+                # booru-tag fine-tunes need clip_skip 2 whatever the caller
+                # sends, and an OpenAI-API client has no field to send it in.
+                "clip_skip", "prompt_weighting",
                 "component_quantization", "output_crf", "force_vram_update",
                 "balanced_gpu_percent", "acceleration",
                 "cache_type_k", "cache_type_v", "kv_offload", "n_batch", "n_ubatch", "n_seq_max",

codai/admin/templates/models.html

@@ -1541,6 +1541,18 @@ window.__DEFAULT_WHISPER_SERVER_PATH__ = {{ default_whisper_server_path|tojson }
         <label style="display:flex;align-items:center;gap:.5rem;cursor:pointer;font-size:13px"><input type="checkbox" id="cfg-vae-tiling"> VAE Tiling <span class="muted">(reduces VRAM at decode)</span></label>
       </div>
 
+      <!-- Prompt handling — image models (SD 1.x / SDXL text encoders) -->
+      <div class="card-title" style="margin-top:1.25rem">Prompt Handling <span class="muted">(image models)</span></div>
+      <div class="form-row" style="margin-bottom:.5rem;max-width:340px">
+        <label class="form-label">CLIP skip <span class="muted">(blank = the pipeline's own)</span></label>
+        <input type="number" id="cfg-clip-skip" class="form-input" min="1" max="12" step="1" placeholder="e.g. 2">
+        <span class="form-hint">Takes the text embedding this many layers from the end of CLIP. The booru-tag SDXL fine-tune families — Pony, Illustrious, NoobAI and their derivatives — are trained at <b>2</b> and look washed out and off-prompt without it. Set it here because a client speaking the OpenAI image API has no field to ask for it; a request's own <code>clip_skip</code> still wins.</span>
+      </div>
+      <div style="display:flex;gap:1.5rem;flex-wrap:wrap;margin-top:.5rem">
+        <label style="display:flex;align-items:center;gap:.5rem;cursor:pointer;font-size:13px"><input type="checkbox" id="cfg-prompt-weighting" checked> Prompt weighting <span class="muted">((tag:1.3), [tag], BREAK)</span></label>
+      </div>
+      <span class="form-hint">On, the A1111 grammar is parsed: <code>(tag:1.3)</code> emphasises, <code>[tag]</code> attenuates, bare <code>(tag)</code> is 1.1, nesting multiplies, <code>\(</code> is a literal bracket, and <code>BREAK</code> starts a new 75-token window. Off, brackets reach the model as text. Either way the prompt is encoded in as many windows as it needs, so nothing is cut at 77 tokens.</span>
+
       <!-- video output quality (CRF) — only relevant for video models -->
       <div class="form-row" style="margin-top:.75rem;max-width:340px">
         <label class="form-label">Video output quality — CRF
@@ -4291,6 +4303,8 @@ function openCfgModal(idx, cfgIdx){
   document.getElementById('cfg-sdcpp-flash').checked = !!s.sdcpp_flash_attn;
   document.getElementById('cfg-sdcpp-diff-flash').checked = !!s.sdcpp_diffusion_flash_attn;
   document.getElementById('cfg-vae-tiling').checked = !!s.vae_tiling;
+  document.getElementById('cfg-clip-skip').value = (s.clip_skip != null ? s.clip_skip : '');
+  document.getElementById('cfg-prompt-weighting').checked = (s.prompt_weighting !== false);
   document.getElementById('cfg-output-crf').value = (s.output_crf != null ? s.output_crf : '');
   document.getElementById('cfg-balanced-gpu-pct').value = (s.balanced_gpu_percent != null ? s.balanced_gpu_percent : '');
   _populateCompSelects();
@@ -5393,6 +5407,9 @@ async function saveModelConfig(){
                           ? null : parseFloat(document.getElementById('cfg-balanced-gpu-pct').value)),
     acceleration:      _collectAccel(),
     turboquant:        _collectTurboQuant(),
+    clip_skip:         (document.getElementById('cfg-clip-skip').value.trim() === ''
+                          ? null : parseInt(document.getElementById('cfg-clip-skip').value, 10)),
+    prompt_weighting:  document.getElementById('cfg-prompt-weighting').checked,
   };
   try{
     const r = await fetch(ROOT_PATH + '/admin/api/model-configure',{

codai/config.py

@@ -303,6 +303,13 @@ class ImageConfig:
     seed: Optional[int] = None
     vae_tiling: bool = False
     clip_on_cpu: bool = False
+    # Which CLIP layer the text embedding comes from, counted from the end.
+    # None = the pipeline's own choice. The booru-tag SDXL fine-tunes (Pony,
+    # Illustrious, NoobAI) are trained at 2; a model can override this.
+    clip_skip: Optional[int] = None
+    # Parse the A1111 prompt grammar -- (tag:1.3), [tag], BREAK. Off keeps
+    # brackets literal; the prompt is windowed past 77 tokens either way.
+    prompt_weighting: bool = True
 
 
 @dataclass
@@ -2128,7 +2135,9 @@ class ConfigManager:
                 "cpu_offload": self.config.image.cpu_offload,
                 "seed": self.config.image.seed,
                 "vae_tiling": self.config.image.vae_tiling,
-                "clip_on_cpu": self.config.image.clip_on_cpu
+                "clip_on_cpu": self.config.image.clip_on_cpu,
+                "clip_skip": self.config.image.clip_skip,
+                "prompt_weighting": self.config.image.prompt_weighting
             },
             "archive": {
                 "enabled": self.config.archive.enabled,

codai/main.py

@@ -1225,6 +1225,8 @@ def main():
     global_args.image_seed = config.image.seed
     global_args.vae_tiling = config.image.vae_tiling
     global_args.clip_on_cpu = config.image.clip_on_cpu
+    global_args.image_clip_skip = config.image.clip_skip
+    global_args.image_prompt_weighting = config.image.prompt_weighting
     global_args.system_prompt = config.system_prompt
     global_args.tools_closer_prompt = config.tools_closer_prompt
     global_args.grammar_guided_gen = config.grammar_guided

codai/pydantic/imagerequest.py

@@ -53,6 +53,15 @@ class ImageGenerationRequest(BaseModel):
         "Null out the diffusers safety_checker so uncensored fine-tunes are not blocked. "
         "Only affects SD 1.x/2.x (SDXL/Flux ship no checker)."))
     negative_prompt: Optional[str] = Field(None, description="What to avoid in the output.")
+    clip_skip: Optional[int] = Field(None, description=(
+        "Take the text embedding this many CLIP layers from the end instead of the last. "
+        "The booru-tag SDXL fine-tune families (Pony, Illustrious, NoobAI) are trained at 2. "
+        "Omitted uses the model's configured default, then the pipeline's own."))
+    prompt_weighting: Optional[bool] = Field(None, description=(
+        "Parse the A1111 prompt grammar: '(tag:1.3)' to emphasise, '[tag]' to attenuate, "
+        "'(tag)' for 1.1, nesting multiplies, '\\\\(' escapes, and BREAK starts a new "
+        "75-token window. False keeps brackets literal. Either way the prompt is encoded "
+        "in as many windows as it needs, so nothing is truncated at 77 tokens."))
 
     # Per-request component overrides
     vae_model: Optional[str] = Field(None, description="Override the VAE for this request.")

docs/README.md

@@ -28,6 +28,7 @@ reference; these are the deep dives.
 | [ocr.md](ocr.md) | The `/v1/ocr` document-transcription subsystem |
 | [chunking.md](chunking.md) | `POST /v1/chunk`: splitting a document for a retrieval index, with the structure decided per request by a template |
 | [expressive-tts.md](expressive-tts.md) | Expressive text-to-speech |
+| [image-prompts.md](image-prompts.md) | Image prompts: the `(tag:1.3)` / `[tag]` / `BREAK` grammar, prompts past CLIP's 77 tokens, `clip_skip`, negative prompts |
 | [zimage-lora-training.md](zimage-lora-training.md) | LoRA training for image models |
 | [dtype-auto-selection.md](dtype-auto-selection.md) | How model dtype is chosen automatically |
 

docs/image-prompts.md

+# Image prompts: weights, BREAK, long prompts and CLIP skip
+
+A CLIP text encoder reads 77 tokens, two of which are the start and end
+markers. That limit is the reason the Stable Diffusion ecosystem grew a prompt
+syntax of its own, and the reason a server that ignores it renders something
+other than what was asked for — without failing, which is the awkward part.
+
+CoderAI encodes image prompts itself rather than handing the text to the
+pipeline. Three things follow from that, on every CLIP-family image model
+(SD 1.x, SD 2.x, SDXL and their fine-tunes):
+
+* **No 77-token limit.** The prompt is split into as many 75-token windows as
+  it needs, each encoded and concatenated. Nothing is truncated and nothing is
+  dropped quietly.
+* **Weights.** `(tag:1.3)` emphasises, `[tag]` attenuates, bare `(tag)` is
+  1.1, nesting multiplies, and `\(` is a literal bracket.
+* **BREAK.** Starts the next window, so tags describing different subjects are
+  not blended by sharing one.
+
+And `clip_skip`, which decides how deep in CLIP the embedding is read from.
+
+## Why it matters
+
+The booru-tag fine-tune families — Pony Diffusion, Illustrious, NoobAI and the
+hundreds of models derived from them — are trained on prompts that open with a
+run of quality tags and are read at `clip_skip` 2:
+
+```
+score_9, score_8_up, score_7_up, score_6_up, rating_safe, source_anime,
+(masterpiece:1.2), 1girl, red hair, standing in a field of wheat, golden hour,
+(detailed background:1.3) BREAK
+wide shot, depth of field, film grain
+```
+
+Against a server that passes that text straight to a pipeline, three separate
+things go wrong at once: the prompt is cut off partway through the quality tags
+so the subject is never seen; the brackets and colons arrive as literal tokens,
+i.e. as noise; and the embedding is read from the wrong layer, which washes the
+output out. All three answer `200` with an image, which is why they are easy to
+miss and worth stating.
+
+## Requests
+
+`POST /v1/images/generations`, `/v1/images/edits` and `/v1/images/inpaint` all
+take:
+
+| Field | Meaning |
+|---|---|
+| `negative_prompt` | What to avoid. The counterpart of the quality tags (`score_4, score_5, score_6, worst quality`) and as load-bearing as the prompt itself. |
+| `clip_skip` | Read the embedding this many CLIP layers from the end. Omitted uses the model's configured value, then the pipeline's own default. |
+| `prompt_weighting` | `false` keeps brackets literal, for a caller sending prose. The windowing stays on either way, so turning it off never costs tokens. |
+
+```json
+{
+  "model": "pony-xl",
+  "prompt": "score_9, score_8_up, (masterpiece:1.2), 1girl, red hair BREAK wide shot",
+  "negative_prompt": "score_4, score_5, score_6, worst quality, blurry",
+  "clip_skip": 2,
+  "size": "832x1216"
+}
+```
+
+## Per model, for clients that cannot ask
+
+A client speaking the plain OpenAI image API has no `clip_skip` field to send,
+and a model that needs 2 needs it on every request. So both settings are part
+of the model's own configuration — **Models → Prompt Handling** in the admin
+UI, or `clip_skip` / `prompt_weighting` on the model's entry in `models.json`.
+A request that sends either one still wins; the config is the default for
+everyone who does not.
+
+There is also a server-wide default under `image` in `config.json`
+(`clip_skip`, `prompt_weighting`), which the model's entry overrides in turn.
+
+## GGUF models (sd.cpp)
+
+sd.cpp parses the weight grammar and manages the token windows itself, inside
+the engine, so a prompt reaches it as written and `clip_skip` is passed
+through. Until v0.4.89 the negative prompt was not: the call hardcoded an empty
+string, so for every GGUF image model whatever the caller asked to avoid was
+discarded between the API and the engine.
+
+## What is not covered
+
+Flux, SD3 and Z-Image carry a T5 encoder with a much longer limit and
+conventions of its own. CoderAI leaves those on their pipeline's native text
+path — the weight grammar is a CLIP-era convention and a T5 reads it as text,
+which is what those models expect. Negative prompts work there as the pipeline
+supports them.
+
+Textual-inversion embeddings (`easynegative` and friends) are not loaded by
+name yet; a LoRA is the supported way to add learned tokens
+(see [zimage-lora-training.md](zimage-lora-training.md)).
+
+## Implementation
+
+`codai/models/prompt_weighting.py` holds the parser, the windowing and the
+encoder, and documents how `clip_skip` maps to a hidden-state index — SDXL
+takes the penultimate layer by default and counts from there, SD 1.x takes the
+final output and needs the layer norm applied by hand. Getting that mapping
+wrong shifts every image, so it mirrors diffusers exactly.
+`tests/test_prompt_weighting_and_clip_skip.py` pins the grammar table, the
+window arithmetic, both layer conventions and the three defects above.