• Stefy Lanza (nextime / spora )'s avatar
    runpod: hours nobody ledgered, and a flag the ledger threw away (v0.3.63) · d8984250
    Stefy Lanza (nextime / spora ) authored
    Two gaps I left open last night, both about the ledger telling the truth.
    
    **A pod that went away without a pool noticing was never billed to anyone.** A
    pool closes its own pods' bills — `_terminate` for one it retires, `forget` for
    one RunPod says is gone — and neither runs for a pod only the REGISTRY knew
    about: one orphaned by a restart, or terminated from the console or straight
    through the RunPod API. Observed: pod lgdgfs749chkib, killed by hand, never
    appeared in the ledger while two pods that went through coderai did.
    `close_bill_from_registry()` now computes the hours from the registry's own
    `created` — the only record of when a pod started once no handle exists — and
    records them against the pool it belonged to. Both reaper paths use it, so the
    "terminated as stale" and "vanished from the account" cases cannot drift apart,
    and a pod a pool still holds is deliberately skipped: `forget` ledgers that one,
    and counting it here as well would double-count every pod RunPod reclaims.
    
    **`record_spend` dropped the `reaped` flag it was being handed.** It keeps a
    deliberate allow-list, which is right — the file is persisted and pruned for
    every pod — but the list has to carry what distinguishes one kind of charge from
    another. A pod the reaper killed looked in the file exactly like one a pool
    retired on purpose. `reaped` and `reason` are kept now.
    
    Writing the test for the first fix found a third thing, and it was mine: v0.3.56
    made a stale registry entry stop counting as "known" so the reaper could kill the
    pod it named — but the reconcile was only ever asked about known ids, so that
    same change stopped stale entries being RECONCILED. An entry for a pod that no
    longer exists was then never pruned and its hours never attributed. The reaper
    now reconciles against the whole registry; `known` keeps its one job, which is
    deciding what is shielded from reaping.
    
    9 tests in tests/test_ledger_out_of_band.py, including the double-count guard
    and the pruning regression. test_runpod_client_scale's source anchor moves with
    the change and says why.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW
    d8984250
Name
Last commit
Last update
..
conftest.py Loading commit data...
ocr_lane_stub_worker.py Loading commit data...
test_admin_auth.py Loading commit data...
test_admin_forward_auth.py Loading commit data...
test_admin_remote_ui.py Loading commit data...
test_adoption_matches_image_and_card.py Loading commit data...
test_audio_ml_backends.py Loading commit data...
test_audio_ml_endpoints.py Loading commit data...
test_autoupgrade.py Loading commit data...
test_autoupgrade_recovers_a_down_server.py Loading commit data...
test_autoupgrade_sees_training.py Loading commit data...
test_broker_config.py Loading commit data...
test_broker_dispatch.py Loading commit data...
test_broker_protocol.py Loading commit data...
test_broker_shutdown.py Loading commit data...
test_broker_streaming.py Loading commit data...
test_burst_placement.py Loading commit data...
test_character_studio_fetch.py Loading commit data...
test_chat_passthrough.py Loading commit data...
test_cluster.py Loading commit data...
test_compose.py Loading commit data...
test_forward_auth_peer_trust.py Loading commit data...
test_front_api_auth.py Loading commit data...
test_front_inflight_leak.py Loading commit data...
test_front_ocr_swap_gate.py Loading commit data...
test_image_has_c_compiler.py Loading commit data...
test_ledger_out_of_band.py Loading commit data...
test_longcat_avatar_layout.py Loading commit data...
test_longcat_bsa.py Loading commit data...
test_longcat_cancellable.py Loading commit data...
test_longcat_checkpoint_resolve.py Loading commit data...
test_longcat_component_quant.py Loading commit data...
test_longcat_config_ui.py Loading commit data...
test_longcat_external_training.py Loading commit data...
test_longcat_int8_load.py Loading commit data...
test_longcat_lora.py Loading commit data...
test_longcat_orchestration.py Loading commit data...
test_longcat_packaging.py Loading commit data...
test_longcat_placement.py Loading commit data...
test_longcat_progress.py Loading commit data...
test_longcat_prompt_cache.py Loading commit data...
test_longcat_quantize.py Loading commit data...
test_longcat_scale.py Loading commit data...
test_longcat_tasks.py Loading commit data...
test_longcat_variants.py Loading commit data...
test_longcat_vendored_source.py Loading commit data...
test_longcat_venv_env.py Loading commit data...
test_longcat_vram_estimate.py Loading commit data...
test_longcat_worker.py Loading commit data...
test_manual_multimodal_test_client.py Loading commit data...
test_model_scan_cache.py Loading commit data...
test_models_context_window.py Loading commit data...
test_ocr_accent_repair.py Loading commit data...
test_ocr_blank_page.py Loading commit data...
test_ocr_chandra.py Loading commit data...
test_ocr_confidence.py Loading commit data...
test_ocr_fanout.py Loading commit data...
test_ocr_language.py Loading commit data...
test_ocr_olmocr.py Loading commit data...
test_ocr_request_budgets.py Loading commit data...
test_ocr_resume_engines.py Loading commit data...
test_ocr_resume_shared.py Loading commit data...
test_ocr_served_model.py Loading commit data...
test_ocr_token_trace.py Loading commit data...
test_ocr_unreadable_and_dpi.py Loading commit data...
test_ocr_vram_share_and_fallback.py Loading commit data...
test_ocr_window_published.py Loading commit data...
test_ocr_worker_lanes.py Loading commit data...
test_olmocr_pod_tls.py Loading commit data...
test_oom_priority.py Loading commit data...
test_pod_log_drain.py Loading commit data...
test_pod_token_adoption_race.py Loading commit data...
test_pool_growth_adopts.py Loading commit data...
test_proxy_logprobs.py Loading commit data...
test_remote_execution.py Loading commit data...
test_runpod_client_scale.py Loading commit data...
test_runpod_fleet_status.py Loading commit data...
test_runpod_ocr_remotization.py Loading commit data...
test_runpod_placement.py Loading commit data...
test_runpod_pod_extra_args.py Loading commit data...
test_runpod_reaper_grace.py Loading commit data...
test_runpod_regional_volumes.py Loading commit data...
test_runpod_registry_heartbeat.py Loading commit data...
test_runpod_request_gpu_choice.py Loading commit data...
test_runpod_scale_down.py Loading commit data...
test_runpod_scale_out.py Loading commit data...
test_runpod_schedule_region_spend.py Loading commit data...
test_runpod_warm_floor_rerent.py Loading commit data...
test_runpod_warmup.py Loading commit data...
test_studio_composed_surfaces.py Loading commit data...
test_surya_item_cap.py Loading commit data...
test_surya_retry_budget.py Loading commit data...
test_surya_runpod_fixes.py Loading commit data...
test_surya_thread_boundary.py Loading commit data...
test_text_lora_to_pod.py Loading commit data...
test_township_api_key.py Loading commit data...
test_township_clip_geometry.py Loading commit data...
test_township_longcat.py Loading commit data...
test_township_page_js.py Loading commit data...
test_township_templates.py Loading commit data...
test_township_video_dataset.py Loading commit data...
test_upgrade_launcher.py Loading commit data...
test_video_as_b64_scope.py Loading commit data...
test_video_geometry.py Loading commit data...
test_vllm_share_self_tuning.py Loading commit data...
test_vram_releasers.py Loading commit data...
test_weight_stores.py Loading commit data...
test_whisper_server_local_models.py Loading commit data...