• Stefy Lanza (nextime / spora )'s avatar
    broker/text: move model-load status out of the engine; gate by load_status_updates · bb8ebcf0
    Stefy Lanza (nextime / spora ) authored
    The engine-side "Waiting for model reply..." content injection ran inside the
    GIL-blocked engine handler during model load, so it never streamed live -- it
    only got concatenated into the final reply and polluted the output. Removed it.
    
    Load-status now lives in the responsive front-proxy/broker layer:
    - SSE: broker_execute_stream yields a NON-content chunk (empty delta.content +
      x_queue_info status) during each not-ready wait (engine down / starting /
      model loading), so a watching client sees progress without reply pollution.
    - out-of-band: the existing broker `pending` keepalives are kept.
    
    Both are gated by a single flag, resolved by _load_status_enabled(model):
    - global models.load_status_updates (default True)
    - per-model "load_status_updates" override in models.json (unset = inherit)
    The same gate drives the SSE side and, via client.status_gate, the out-of-band
    `pending` keepalives. Unknown model defaults to on so the relay deadline stays
    protected.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01RdMufYvtTbtGDWsiZVoXce
    bb8ebcf0
client.py 22 KB