The previous revision told a client to retry 404 and 502 indefinitely.
That is wrong and would spin forever in two real cases: a document that
reliably 502s never succeeds and blocks the corpus behind it, and a 404
that is not a restart (wrong URL, route removed) never clears. Both end
with pods billing and nothing being produced -- the exact failure the
advice was written to prevent.

It was an over-correction. On 2026-10-19 a restart answered 618 requests
with 404 over ~40 seconds, the client treated 404 as fatal and stopped,
and nine pods sat ready at $5.31/hr until a human noticed. The fix for
that is "do not treat 404 as fatal immediately", not "never give up".

What separates the codes is what each says about the server:

  429  healthy and full, and it stated when to return -> honour
       Retry-After, no attempt cap, alert if 15 min pass with no page
       completing.
  404  restarting -> retry with backoff for up to 5 minutes, then STOP
       and alert. A route that is gone does not come back by being
       asked again, and a real "no such endpoint" would have answered
       404 to the first request rather than the ten-thousandth.
  502  the pod failed this page -> 5 attempts, then PARK the page and
       move on. ~3-5% of pages 502 at least once; one unparkable
       document must not stop a million-page run.
  400/413  the document is the problem -> park, never retry.

Plus a circuit breaker, which matters more than any per-page rule: stop
feeding if more than 20% of the last 200 pages ended parked or erroring.
No per-page policy can see "the fleet is costing money and producing
nothing"; only the aggregate can.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01KnfkxBpcZ1v6ypRFFyhrVM