• Stefy Lanza (nextime / spora )'s avatar
    a pod was sent its OCR concurrency and never read it (v0.4.31) · 9632e662
    Stefy Lanza (nextime / spora ) authored
    A rented pod has no config.json and no models.json -- it creates defaults and
    environment is the only channel it has. CODERAI_OCR_CHANDRA_INSTANCES and
    CODERAI_OCR_OLMOCR_INSTANCES were sent to every pod and read by nobody, so
    those engines fell back to the dataclass default of ONE instance however many
    the operator configured, while the admin UI showed the configured value.
    
    Measured on one A40 with chandra_instances=128 and 32 pages in flight:
    
      [ocr] engine 'chandra': 1 instance(s) ready
      vLLM: Running: 1 reqs, Waiting: 0 reqs, GPU KV cache usage: 1.6%
            Avg generation throughput: 54.5 tokens/s
    
    against surya's ~2000 tokens/s and ~32 concurrent sequences on the same card
    and the same pages. It read as "chandra is 36x slower than surya"; it was one
    page at a time. Surya worked only because its own env rows had been mapped --
    the gap was invisible for exactly as long as surya was the engine being tested.
    
    Adds the two instance counts, plus chandra's and olmOCR's page budgets
    (MAX_TOKENS, MAX_CONTINUATIONS, MAX_RETRIES, DPI), which were sent and
    unread for the same reason: seeding an engine's wiring but not its tuning
    gives a pod running that engine's library defaults, which is
    indistinguishable from the outside from the wrong pod image.
    
    The test is deliberately generic -- for each engine and each tuning knob, if
    the orchestrator sends it the pod must read it. A list of expected names would
    have passed before this fix on surya's rows alone and said nothing about the
    other two engines, which is how this survived.
    
    NOTE: config.py runs ON the pod, so a pod only gets this once its image
    carries it. coderai-vllm-ocr must be rebuilt before chandra serves more than
    one page at a time.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    9632e662