• Stefy Lanza (nextime / spora )'s avatar
    longcat: INT8 weights that can be built AND loaded (v0.2.66) · 999efe6c
    Stefy Lanza (nextime / spora ) authored
    Building the INT8 DiT needed ~42 GB because it materialised the bf16 model
    first; loading it needed ~55 GB for the same reason, plus every shard
    accumulated into one dict and then copied in. Both took the host down.
    
    Both now stream. The quantiser reads tensors one at a time through safe_open
    and flushes 4 GB shards, enumerating the Linear layers on a meta-device
    skeleton instead of a real model. _load_quantized_base_dit builds its skeleton
    on meta too, then assigns each shard into it as it is read.
    
    Measured on the 13.6B DiT: 51 GB bf16 -> 13.6 GB INT8, ~4 GB peak to write,
    and the load is 20 s at 0.6 GB RSS (safetensors mmap) with all 437 quantised
    Linears present and nothing stranded on meta.
    
    A checkpoint that leaves tensors on the meta device is now an error naming
    them, rather than a model that fails later somewhere unrelated.
    Co-Authored-By: 's avatarClaude Opus 5 (1M context) <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01QXD9yKKFZHAB9t1LfWQWwz
    999efe6c
Name
Last commit
Last update
..
test_admin_auth.py Loading commit data...
test_admin_remote_ui.py Loading commit data...
test_audio_ml_backends.py Loading commit data...
test_audio_ml_endpoints.py Loading commit data...
test_broker_config.py Loading commit data...
test_broker_dispatch.py Loading commit data...
test_broker_protocol.py Loading commit data...
test_broker_streaming.py Loading commit data...
test_burst_placement.py Loading commit data...
test_character_studio_fetch.py Loading commit data...
test_compose.py Loading commit data...
test_front_ocr_swap_gate.py Loading commit data...
test_image_has_c_compiler.py Loading commit data...
test_longcat_cancellable.py Loading commit data...
test_longcat_config_ui.py Loading commit data...
test_longcat_external_training.py Loading commit data...
test_longcat_int8_load.py Loading commit data...
test_longcat_lora.py Loading commit data...
test_longcat_orchestration.py Loading commit data...
test_longcat_packaging.py Loading commit data...
test_longcat_quantize.py Loading commit data...
test_longcat_scale.py Loading commit data...
test_longcat_tasks.py Loading commit data...
test_longcat_variants.py Loading commit data...
test_longcat_vendored_source.py Loading commit data...
test_longcat_venv_env.py Loading commit data...
test_longcat_worker.py Loading commit data...
test_manual_multimodal_test_client.py Loading commit data...
test_model_scan_cache.py Loading commit data...
test_ocr_olmocr.py Loading commit data...
test_ocr_vram_share_and_fallback.py Loading commit data...
test_remote_execution.py Loading commit data...
test_runpod_ocr_remotization.py Loading commit data...
test_studio_composed_surfaces.py Loading commit data...
test_township_api_key.py Loading commit data...
test_township_longcat.py Loading commit data...
test_township_page_js.py Loading commit data...
test_township_templates.py Loading commit data...
test_township_video_dataset.py Loading commit data...
test_upgrade_launcher.py Loading commit data...
test_vllm_share_self_tuning.py Loading commit data...
test_vram_releasers.py Loading commit data...
test_weight_stores.py Loading commit data...
test_whisper_server_local_models.py Loading commit data...