• Stefy Lanza (nextime / spora )'s avatar
    vram: learn+persist stable offload split; front: graceful crash + task cancel routing · 223b46f7
    Stefy Lanza (nextime / spora ) authored
    manager (record_vram_delta): record the FULL footprint after a load — including
    offloaded loads (no longer skipped) — so the next load skips the OOM→move-to-RAM
    retry dance:
      * measured_vram_gb      — GPU VRAM delta + reserve (GPU portion; eviction uses it)
      * measured_ram_gb       — host-RAM delta (the CPU-offloaded layers)   [new]
      * measured_n_gpu_layers — the layer split llama.cpp settled on         [new]
      total need = vram + ram. All gated by force_vram_update (else only when
      used_vram_gb unset). New _learned_n_gpu_layers(): when n_gpu_layers is auto/-1,
      reuse the learned split so a reload jumps straight to the config that fit. Both
      load sites snapshot host RAM and read the backend's settled n_gpu_layers.
    
    frontproxy:
      * graceful streaming proxy: when an engine dies mid-stream (e.g. a CUDA
        ggml_abort SIGABRT on a VRAM-OOM decode), end the relayed stream cleanly
        instead of throwing an unhandled ASGI exception (noisy traceback).
      * task actions (cancel/interrupt/pause/resume/restart/DELETE) now route to the
        engine that OWNS the task — fan out to every live engine (each validates the
        same session cookie), first success wins — fixing 'Task not found' for tasks on
        a non-primary engine; front-only synthetic ids handled locally.
    
    Bump version to 0.1.7.
    223b46f7
__init__.py 4.11 KB