text: stop flattening image content to a placeholder (fixes vision over the API)

ChatMessage.convert_content_array_to_string ran at parse time (mode=before) and
unconditionally joined any multipart content list into a string, replacing each
image_url part with the literal text "[image_url content]". That destroyed the
image before text.py's vision pipeline (_vision_ok / _normalize_vision_content,
which the backend's mmproj/MTMDChatHandler consume) ever saw it — so a vision
model (e.g. lisa = Gemma-4 + mmproj) received only text and answered "no image
attached", and agents looped retrying. text.py's end-to-end vision handling was
effectively dead code because content was always pre-stringified.

Now flatten only TEXT-ONLY multipart arrays (the KiloCode case); preserve the
list whenever it carries an image_url (or any non-text part) so the multimodal
backend receives the image. Non-vision models still degrade to the placeholder
downstream (text.py:1439), and text-only/plain-string paths are unchanged.
Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
parent 758d948f
...@@ -44,21 +44,32 @@ class ChatMessage(BaseModel): ...@@ -44,21 +44,32 @@ class ChatMessage(BaseModel):
@field_validator('content', mode='before') @field_validator('content', mode='before')
@classmethod @classmethod
def convert_content_array_to_string(cls, v): def convert_content_array_to_string(cls, v):
"""Convert multipart content array to string for compatibility.""" """Flatten a TEXT-ONLY multipart content array to a string for clients that
send text as parts (e.g. KiloCode). Multimodal arrays — anything with an
``image_url`` (or other non-text part) — are PRESERVED as a list so the
vision backend (mmproj/CLIP) actually receives the image. Flattening those
replaced the image with a "[image_url content]" placeholder, so a vision
model saw only text and answered "no image attached"."""
if v is None: if v is None:
return None return None
if isinstance(v, str): if isinstance(v, str):
return v return v
if isinstance(v, list): if isinstance(v, list):
# Handle multipart content array format (e.g., from KiloCode) # Preserve the array untouched when it carries any non-text part — the
# Format: [{"type": "text", "text": "..."}, {"type": "text", "text": "..."}] # downstream chat path (vulkan _fold_system + MTMDChatHandler) handles
# list content and feeds image_url parts to the vision projector.
has_non_text = any(
isinstance(item, dict) and item.get('type') not in (None, 'text')
for item in v)
if has_non_text:
return v
# Pure text parts → join to a single string (the KiloCode case).
parts = [] parts = []
for item in v: for item in v:
if isinstance(item, dict): if isinstance(item, dict):
if item.get('type') == 'text' and 'text' in item: if item.get('type') == 'text' and 'text' in item:
parts.append(item['text']) parts.append(item['text'])
else: else:
# Handle other content types (image_url, etc.) by converting to placeholder
parts.append(f"[{item.get('type', 'unknown')} content]") parts.append(f"[{item.get('type', 'unknown')} content]")
else: else:
parts.append(str(item)) parts.append(str(item))
......
Markdown is supported
0% or
You are about to add 0 people to the discussion. Proceed with caution.
Finish editing this message first!
Please register or to comment