• Your Name's avatar
    feat: implement smart request batching (v0.8.0) · 709b6f80
    Your Name authored
    - Add aisbf/batching.py module with RequestBatcher class
    - Implement time-based (100ms window) and size-based batching
    - Add provider-specific batching configurations (OpenAI: 10, Anthropic: 5)
    - Integrate batching with BaseProviderHandler
    - Add batching configuration to config/aisbf.json
    - Initialize batching system in main.py startup
    - Update version to 0.8.0 in setup.py and pyproject.toml
    - Add batching.py to setup.py data_files
    - Update README.md and TODO.md documentation
    - Expected benefit: 15-25% latency reduction
    
    Features:
    - Automatic batch formation and processing
    - Response splitting and distribution
    - Statistics tracking (batches formed, requests batched, avg batch size)
    - Graceful error handling and fallback
    - Non-blocking async queue management
    - Streaming request bypass (batching disabled for streams)
    709b6f80
setup.py 5.09 KB