• Stefy Lanza (nextime / spora )'s avatar
    broker: stream chat responses token-by-token instead of buffering · 6998bb25
    Stefy Lanza (nextime / spora ) authored
    The front's broker path buffered the entire SSE response and sent one envelope, so
    broker clients (lisa, and any OpenAI client via AISBF) got the whole reply at once.
    Now the front streams it:
    
    - streaming.py: stream_chunk_envelope uses event="chunk" with payload.chunk (the
      shape the AISBF relay consumes), finalize_stream uses event="done".
    - dispatcher.py: extracted resolve_broker_request() (op routing + body decode +
      validation) shared by the buffered dispatch and the new streaming path, so they
      can't diverge; added BrokerDispatchError.
    - app.py: broker_execute_stream() proxies to the engine with stream=True and yields
      each SSE chunk (sharing the per-model queue + in-flight tracking); start_broker
      wires client.stream_dispatcher.
    - client.py: handle_message, for stream=true requests, relays each yielded chunk as
      a `chunk` envelope and ends with a `done` (instead of the single buffered reply).
    
    Requires the matching AISBF change (send_request returns early for streaming and
    the relay drains the chunk queue) — committed in the aisbf repo.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
    6998bb25
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...
video_editor.config.json Loading commit data...