Skip to content
Commit 4025664d authored by Stefy Lanza (nextime / spora )'s avatar Stefy Lanza (nextime / spora )
Browse files

runpod: vLLM pods need ninja too, the dep after g++



Triton and torch.inductor JIT-compile vLLM's kernels at run time and need a full
build toolchain for the stubs they emit. Each piece was found on a live olmOCR
pod, one layer at a time, because from outside the failure is identical every
time: vLLM dies before registering and the pod answers 502.

    gcc          the C compiler
    g++          the launcher Triton builds is C++, so gcc alone is not enough
    ninja-build  inductor shells out to ninja for its extensions

Only olmOCR hits this, and the reason is worth recording: it is FP8, so vLLM's
quantisation kernels go down the Triton path. surya-2 is dense bf16 and served
fine on images that had none of these, which is why "the image lacks a compiler"
looked for a while like it could not be the problem.

Applied to vllm.apt AND vllm-ocr.apt, which is copied from it.

Co-Authored-By: default avatarClaude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
parent 81235f2e
Loading
Loading
Loading
Loading
0% Loading or .
You are about to add 0 people to the discussion. Proceed with caution.
Please register or to comment