• Stefy Lanza (nextime / spora )'s avatar
    vram: learn+persist stable offload split; front: graceful crash + task cancel routing · 223b46f7
    Stefy Lanza (nextime / spora ) authored
    manager (record_vram_delta): record the FULL footprint after a load — including
    offloaded loads (no longer skipped) — so the next load skips the OOM→move-to-RAM
    retry dance:
      * measured_vram_gb      — GPU VRAM delta + reserve (GPU portion; eviction uses it)
      * measured_ram_gb       — host-RAM delta (the CPU-offloaded layers)   [new]
      * measured_n_gpu_layers — the layer split llama.cpp settled on         [new]
      total need = vram + ram. All gated by force_vram_update (else only when
      used_vram_gb unset). New _learned_n_gpu_layers(): when n_gpu_layers is auto/-1,
      reuse the learned split so a reload jumps straight to the config that fit. Both
      load sites snapshot host RAM and read the backend's settled n_gpu_layers.
    
    frontproxy:
      * graceful streaming proxy: when an engine dies mid-stream (e.g. a CUDA
        ggml_abort SIGABRT on a VRAM-OOM decode), end the relayed stream cleanly
        instead of throwing an unhandled ASGI exception (noisy traceback).
      * task actions (cancel/interrupt/pause/resume/restart/DELETE) now route to the
        engine that OWNS the task — fan out to every live engine (each validates the
        same session cookie), first success wins — fixing 'Task not found' for tasks on
        a non-primary engine; front-only synthetic ids handled locally.
    
    Bump version to 0.1.7.
    223b46f7
Name
Last commit
Last update
..
admin Loading commit data...
api Loading commit data...
backends Loading commit data...
broker Loading commit data...
frontproxy Loading commit data...
models Loading commit data...
openai Loading commit data...
pydantic Loading commit data...
queue Loading commit data...
tasks Loading commit data...
__init__.py Loading commit data...
cli.py Loading commit data...
config.py Loading commit data...
main.py Loading commit data...
platform_paths.py Loading commit data...
system_app.py Loading commit data...