• Stefy Lanza (nextime / spora )'s avatar
    longcat: block-sparse attention as an option, and a cp_split_hw that works (v0.2.69) · 3532636c
    Stefy Lanza (nextime / spora ) authored
    Attention is the expensive part at video token counts — ~37k tokens for 93
    frames at 480x832, and it is O(N^2) in that. The checkpoint already carries
    bsa_params with sparsity 0.9375, roughly a 16x cut, and upstream's
    implementation is pure Triton, so none of it needs the flash-attn source build.
    
    So it is now a setting: longcat.bsa server-wide, "bsa" per model, --bsa on the
    service. Off by default because it is an approximation and video shows
    approximation as temporal flicker, not one soft frame. "auto" stays off when
    triton is missing; "on" says so instead, because someone who asked for it would
    otherwise be left wondering why nothing got faster. BSA replaces the dense
    backend rather than joining it, since Attention tests enable_bsa first.
    
    And cp_split_hw, found while wiring this: the DiT's forward opens with
    `if self.cp_split_hw[0] * self.cp_split_hw[1] > 1:` and has no None guard, but
    dit/config.json ships "cp_split_hw": null — so a DiT built from the config as
    published raises TypeError on its first forward. Our only assignment made it
    worse by storing cp_size, a process count, where a [h, w] pair belongs: an int
    is not subscriptable either. It defaults to [1, 1] now, derives [1, cp_size]
    under context parallelism, and takes an explicit tile that is checked against
    the group size.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_0185zEBiy3KWB37xNGSrxL7w
    3532636c
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...