• Stefy Lanza (nextime / spora )'s avatar
    a page forward is a wait on a socket, not a thread (v0.4.9) · b044ed11
    Stefy Lanza (nextime / spora ) authored
    0.4.8 moved the forward off the default 32-thread pool and the measured result
    on the live orchestrator was 33 -> 483 pages in flight, 1.65-2.2 -> 7.99
    pages/s, and a control-plane request that used to time out at 40 s answering in
    12 ms. But a thread is still the wrong unit for the wait: ~60 s per page
    against a real corpus means 15 documents/second at ~3.5 pages each is ~3,200
    outstanding forwards, against a pod-pool ceiling of 2048. Threads cannot get
    there, and should not try.
    
    So the hot path is a coroutine. `pod_http` gains `apost`/`aget` on httpx --
    already in the stack (codai/cluster/nodes.py, codai/main.py), so no new
    dependency -- with everything the sync path learned kept intact:
    
    * the pinned TLS context, unchanged: a direct-TCP pod serves HTTPS with a
      certificate signed by THIS install's CA at an ip the certificate cannot name,
      so the CA is trusted and the hostname check is off. NOT verify=False, which
      would accept anybody;
    * one client per event loop AND per pinned-ness, because a client belongs to
      the loop its sockets were opened on and the two trust settings must never
      share one;
    * connection limits from the same `_pool_size()` the sync session uses;
    * a proxied pod (http://, or https:// to a name) stays an ordinary request.
    
    `_apost_page_to_pod` and `_apod_still_alive` mirror the sync twins, and both
    twins now share ONE request builder and ONE response reader -- two copies of
    that field list is how an engine name, a dpi or a token budget comes to apply
    on one transport and not the other. Every distinction 0.4.2-0.4.6 established
    is preserved and tested on the new path: an ANSWER (even 503) is a verdict
    about the pod and belongs on the refusal path, nothing-came-back is
    PodUnreachable, a dead pooled socket is retried on a fresh one, and a timeout
    is never retried -- resending a page the pod is still working on is a second
    card doing the same work and a double bill.
    
    `ocr.pod_http_async` (default on, admin API + settings control) puts the
    forward back on the pod thread pool. That path is kept working deliberately,
    not as dead code: this is a new transport on the hottest path in the
    subsystem, and an operator who hits something odd should change one setting
    rather than wait for a release.
    
    Four existing test files had to be repaired, and the reason is worth keeping:
    they stubbed only the SYNC twin, so the moment the async default landed they
    stopped testing their own stubs and began dialling 10.0.0.1 for real -- 300 s
    hangs where there should have been failures. Their helpers now stub both twins
    via `_stub_page_post`, and the policy tests say `pod_http_async: False` out
    loud because they stub `pod_http.post` specifically. A test that quietly
    measures something else is worse than one that breaks, and this change
    produced four of them.
    
    21 new transport tests. 3345 passed, 13 pre-existing failures (broker, studio,
    manual client) untouched.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM
    b044ed11
test_ocr_pod_pipeline_policy.py 20.8 KB