• Stefy Lanza (nextime / spora )'s avatar
    text: stop streaming tool calls as message content (gemma <|tool_call> leak) · 06d7fbad
    Stefy Lanza (nextime / spora ) authored
    Gemma emits tool calls as text — <|tool_call>call:NAME{…} — which the first-pass
    streaming loop forwarded straight to the client as assistant content. So the call
    showed up as a visible message and only got parsed/executed at end-of-stream.
    
    Add _ToolCallStreamGate: as chunks stream, emit visible content only up to the
    first tool-call marker (gemma's <|tool_call>/<|tool_response> plus the common tag
    formats), holding back a small tail so a marker split across chunk boundaries
    isn't leaked; once a marker appears, emit nothing further. The full text still
    accumulates for the existing end-of-stream tool extraction (GemmaParser), which
    emits a proper tool_calls delta. The second pass is already buffered+extracted, so
    it never leaked.
    
    Also add gemma's "<|tool_call>" to the reasoning/tool split pattern lists.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    06d7fbad
Name
Last commit
Last update
..
cache Loading commit data...
__init__.py Loading commit data...
acceleration.py Loading commit data...
capabilities.py Loading commit data...
grammar.py Loading commit data...
hf_loading.py Loading commit data...
manager.py Loading commit data...
parser.py Loading commit data...
pipeline_cache.py Loading commit data...
quant.py Loading commit data...
ram_monitor.py Loading commit data...
templates.py Loading commit data...
thermal.py Loading commit data...
tmp_janitor.py Loading commit data...
tool_call_grammar.gbnf Loading commit data...
turboquant.py Loading commit data...
utils.py Loading commit data...