• Stefy Lanza (nextime / spora )'s avatar
    perf: cache the cache-scans + gpu-stats so the models page paints instantly · dc3800fe
    Stefy Lanza (nextime / spora ) authored
    The models page was slow because the work itself is slow, not because of routing:
    cached-models walks the HF/GGUF cache doing per-model capability detection, and
    cache-stats walks every file summing sizes — many seconds each on a large cache.
    
    - routes.py: cached-models + cache-stats are now stale-while-revalidate (TTL 25s).
      First call computes; afterwards they return the last result instantly and refresh
      in a background thread. Mutating ops (clear-cache, delete cached model, free-disk)
      force the next refresh. The coderai-system worker pre-warms both scans on startup,
      so even the first page load is fast.
    - gpu_detect.gpu_stats(): short 1.5s TTL cache — the Tasks page polls it ~every
      second and nvidia-smi is slow on a saturated GPU, so collapse repeats.
    
    Verified: _cached returns stale instantly and refreshes in the background
    (2 underlying calls across first+expire); all modules compile.
    Co-Authored-By: 's avatarClaude Opus 4.8 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_011DDv7BchtZQWsnPG6Jm49m
    dc3800fe
gpu_detect.py 18.6 KB