• Stefy Lanza (nextime / spora )'s avatar
    a page forward is a wait on a socket, not a thread (v0.4.9) · b044ed11
    Stefy Lanza (nextime / spora ) authored
    0.4.8 moved the forward off the default 32-thread pool and the measured result
    on the live orchestrator was 33 -> 483 pages in flight, 1.65-2.2 -> 7.99
    pages/s, and a control-plane request that used to time out at 40 s answering in
    12 ms. But a thread is still the wrong unit for the wait: ~60 s per page
    against a real corpus means 15 documents/second at ~3.5 pages each is ~3,200
    outstanding forwards, against a pod-pool ceiling of 2048. Threads cannot get
    there, and should not try.
    
    So the hot path is a coroutine. `pod_http` gains `apost`/`aget` on httpx --
    already in the stack (codai/cluster/nodes.py, codai/main.py), so no new
    dependency -- with everything the sync path learned kept intact:
    
    * the pinned TLS context, unchanged: a direct-TCP pod serves HTTPS with a
      certificate signed by THIS install's CA at an ip the certificate cannot name,
      so the CA is trusted and the hostname check is off. NOT verify=False, which
      would accept anybody;
    * one client per event loop AND per pinned-ness, because a client belongs to
      the loop its sockets were opened on and the two trust settings must never
      share one;
    * connection limits from the same `_pool_size()` the sync session uses;
    * a proxied pod (http://, or https:// to a name) stays an ordinary request.
    
    `_apost_page_to_pod` and `_apod_still_alive` mirror the sync twins, and both
    twins now share ONE request builder and ONE response reader -- two copies of
    that field list is how an engine name, a dpi or a token budget comes to apply
    on one transport and not the other. Every distinction 0.4.2-0.4.6 established
    is preserved and tested on the new path: an ANSWER (even 503) is a verdict
    about the pod and belongs on the refusal path, nothing-came-back is
    PodUnreachable, a dead pooled socket is retried on a fresh one, and a timeout
    is never retried -- resending a page the pod is still working on is a second
    card doing the same work and a double bill.
    
    `ocr.pod_http_async` (default on, admin API + settings control) puts the
    forward back on the pod thread pool. That path is kept working deliberately,
    not as dead code: this is a new transport on the hottest path in the
    subsystem, and an operator who hits something odd should change one setting
    rather than wait for a release.
    
    Four existing test files had to be repaired, and the reason is worth keeping:
    they stubbed only the SYNC twin, so the moment the async default landed they
    stopped testing their own stubs and began dialling 10.0.0.1 for real -- 300 s
    hangs where there should have been failures. Their helpers now stub both twins
    via `_stub_page_post`, and the policy tests say `pod_http_async: False` out
    loud because they stub `pod_http.post` specifically. A test that quietly
    measures something else is worse than one that breaks, and this change
    produced four of them.
    
    21 new transport tests. 3345 passed, 13 pre-existing failures (broker, studio,
    manual client) untouched.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    b044ed11
Name
Last commit
Last update
..
conftest.py Loading commit data...
ocr_lane_stub_worker.py Loading commit data...
test_admin_auth.py Loading commit data...
test_admin_forward_auth.py Loading commit data...
test_admin_remote_ui.py Loading commit data...
test_adoption_matches_image_and_card.py Loading commit data...
test_audio_ml_backends.py Loading commit data...
test_audio_ml_endpoints.py Loading commit data...
test_autoupgrade.py Loading commit data...
test_autoupgrade_force_options.py Loading commit data...
test_autoupgrade_midnight_force.py Loading commit data...
test_autoupgrade_recovers_a_down_server.py Loading commit data...
test_autoupgrade_sees_training.py Loading commit data...
test_broker_config.py Loading commit data...
test_broker_dispatch.py Loading commit data...
test_broker_protocol.py Loading commit data...
test_broker_shutdown.py Loading commit data...
test_broker_streaming.py Loading commit data...
test_burst_placement.py Loading commit data...
test_character_studio_fetch.py Loading commit data...
test_chat_passthrough.py Loading commit data...
test_chunking.py Loading commit data...
test_cluster.py Loading commit data...
test_compose.py Loading commit data...
test_embed_long_input.py Loading commit data...
test_embedding_capability.py Loading commit data...
test_embedding_repo_file_detection.py Loading commit data...
test_field_report_fixes.py Loading commit data...
test_forward_auth_peer_trust.py Loading commit data...
test_front_api_auth.py Loading commit data...
test_front_inflight_leak.py Loading commit data...
test_front_ocr_swap_gate.py Loading commit data...
test_image_has_c_compiler.py Loading commit data...
test_ledger_out_of_band.py Loading commit data...
test_longcat_avatar_layout.py Loading commit data...
test_longcat_bsa.py Loading commit data...
test_longcat_cancellable.py Loading commit data...
test_longcat_checkpoint_resolve.py Loading commit data...
test_longcat_component_quant.py Loading commit data...
test_longcat_config_ui.py Loading commit data...
test_longcat_external_training.py Loading commit data...
test_longcat_int8_load.py Loading commit data...
test_longcat_lora.py Loading commit data...
test_longcat_orchestration.py Loading commit data...
test_longcat_packaging.py Loading commit data...
test_longcat_placement.py Loading commit data...
test_longcat_progress.py Loading commit data...
test_longcat_prompt_cache.py Loading commit data...
test_longcat_quantize.py Loading commit data...
test_longcat_scale.py Loading commit data...
test_longcat_tasks.py Loading commit data...
test_longcat_variants.py Loading commit data...
test_longcat_vendored_source.py Loading commit data...
test_longcat_venv_env.py Loading commit data...
test_longcat_vram_estimate.py Loading commit data...
test_longcat_worker.py Loading commit data...
test_manual_multimodal_test_client.py Loading commit data...
test_model_scan_cache.py Loading commit data...
test_models_context_window.py Loading commit data...
test_ocr_accent_repair.py Loading commit data...
test_ocr_blank_page.py Loading commit data...
test_ocr_chandra.py Loading commit data...
test_ocr_confidence.py Loading commit data...
test_ocr_degraded_retry.py Loading commit data...
test_ocr_detector_venv.py Loading commit data...
test_ocr_dual_mode.py Loading commit data...
test_ocr_fanout.py Loading commit data...
test_ocr_language.py Loading commit data...
test_ocr_max_pages.py Loading commit data...
test_ocr_olmocr.py Loading commit data...
test_ocr_page_image.py Loading commit data...
test_ocr_pod_async_transport.py Loading commit data...
test_ocr_pod_pipeline_gate.py Loading commit data...
test_ocr_pod_pipeline_policy.py Loading commit data...
test_ocr_pod_pool_width.py Loading commit data...
test_ocr_raster_pool.py Loading commit data...
test_ocr_request_budgets.py Loading commit data...
test_ocr_resident_budget.py Loading commit data...
test_ocr_resume_engines.py Loading commit data...
test_ocr_resume_shared.py Loading commit data...
test_ocr_served_model.py Loading commit data...
test_ocr_token_trace.py Loading commit data...
test_ocr_unreadable_and_dpi.py Loading commit data...
test_ocr_vram_share_and_fallback.py Loading commit data...
test_ocr_window_published.py Loading commit data...
test_ocr_worker_lanes.py Loading commit data...
test_olmocr_pod_tls.py Loading commit data...
test_oom_priority.py Loading commit data...
test_placement_declines.py Loading commit data...
test_pod_http_pool.py Loading commit data...
test_pod_log_drain.py Loading commit data...
test_pod_loopback_tls.py Loading commit data...
test_pod_seed_model_type.py Loading commit data...
test_pod_teach_on_unknown_model.py Loading commit data...
test_pod_token_adoption_race.py Loading commit data...
test_pool_growth_adopts.py Loading commit data...
test_proxy_logprobs.py Loading commit data...
test_remote_execution.py Loading commit data...
test_remote_gateway_gate.py Loading commit data...
test_runpod_client_scale.py Loading commit data...
test_runpod_fleet_status.py Loading commit data...
test_runpod_grow_reason.py Loading commit data...
test_runpod_ocr_remotization.py Loading commit data...
test_runpod_placement.py Loading commit data...
test_runpod_pod_extra_args.py Loading commit data...
test_runpod_reaper_grace.py Loading commit data...
test_runpod_regional_volumes.py Loading commit data...
test_runpod_registry_heartbeat.py Loading commit data...
test_runpod_release_and_boot_cancel.py Loading commit data...
test_runpod_request_gpu_choice.py Loading commit data...
test_runpod_restart_orphans.py Loading commit data...
test_runpod_restart_reclaim.py Loading commit data...
test_runpod_scale_down.py Loading commit data...
test_runpod_scale_out.py Loading commit data...
test_runpod_schedule_region_spend.py Loading commit data...
test_runpod_transient_api.py Loading commit data...
test_runpod_warm_floor_parallel.py Loading commit data...
test_runpod_warm_floor_rerent.py Loading commit data...
test_runpod_warmup.py Loading commit data...
test_shared_pool_adapter_identity.py Loading commit data...
test_studio_composed_surfaces.py Loading commit data...
test_surya_item_cap.py Loading commit data...
test_surya_loop_reported.py Loading commit data...
test_surya_retry_budget.py Loading commit data...
test_surya_runpod_fixes.py Loading commit data...
test_surya_thread_boundary.py Loading commit data...
test_text_lora_to_pod.py Loading commit data...
test_thread_pool_sizing.py Loading commit data...
test_township_api_key.py Loading commit data...
test_township_clip_geometry.py Loading commit data...
test_township_longcat.py Loading commit data...
test_township_page_js.py Loading commit data...
test_township_templates.py Loading commit data...
test_township_video_dataset.py Loading commit data...
test_upgrade_launcher.py Loading commit data...
test_video_as_b64_scope.py Loading commit data...
test_video_geometry.py Loading commit data...
test_vllm_share_self_tuning.py Loading commit data...
test_vram_releasers.py Loading commit data...
test_weight_stores.py Loading commit data...
test_whisper_server_local_models.py Loading commit data...