• Stefy Lanza (nextime / spora )'s avatar
    broker: stream chat responses token-by-token instead of buffering · 6998bb25
    Stefy Lanza (nextime / spora ) authored
    The front's broker path buffered the entire SSE response and sent one envelope, so
    broker clients (lisa, and any OpenAI client via AISBF) got the whole reply at once.
    Now the front streams it:
    
    - streaming.py: stream_chunk_envelope uses event="chunk" with payload.chunk (the
      shape the AISBF relay consumes), finalize_stream uses event="done".
    - dispatcher.py: extracted resolve_broker_request() (op routing + body decode +
      validation) shared by the buffered dispatch and the new streaming path, so they
      can't diverge; added BrokerDispatchError.
    - app.py: broker_execute_stream() proxies to the engine with stream=True and yields
      each SSE chunk (sharing the per-model queue + in-flight tracking); start_broker
      wires client.stream_dispatcher.
    - client.py: handle_message, for stream=true requests, relays each yielded chunk as
      a `chunk` envelope and ends with a `done` (instead of the single buffered reply).
    
    Requires the matching AISBF change (send_request returns early for streaming and
    the relay drains the chunk queue) — committed in the aisbf repo.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
    6998bb25
dispatcher.py 7.87 KB