`ocr.surya_token_trace` (off by default) reports the per-token bytes at every
place a surya page lost a character. It exists to answer one question that the
finished text cannot: whose fault is it?

  invalid-utf8   the reassembled stream does not decode -- a lone UTF-8 lead
                 byte with nothing to complete it. The MODEL emitted a broken
                 sequence; the text faithfully renders a wrong answer.
  model-emitted  the stream decodes and a token IS the replacement character.
                 The model chose it.
  no findings    the bytes were sound and the markers appeared anyway, so the
                 loss happened AFTER them -- OURS, in the serving stack.

That last case is why an empty trace is a result, not a failure.

CORRECTION, and the reason this was needed. v0.3.21 claimed U+FFFD could not be
a byte-level artefact, on the grounds that `a`-grave was a single token so one
marker had to mean the model chose it. That was wrong. Measured:

  'a-grave' utf8=[195, 160]  ids=[2131, 2258]  pieces=['A-tilde', 'l-bar']
  decode([2131])       = U+FFFD    <- a lone lead byte, exactly ONE marker
  decode([2131, 2104]) = 'e-grave'

surya-ocr-2's vocabulary is BYTE-level, so one marker per accented letter is
precisely the signature of a byte-level failure -- the thing it was claimed to
rule out. The vocab lookup was misread too: `vocab['a-grave'] = 2160` does not
mean the letter is token 2160; in a byte-mapped vocabulary that key is the glyph
standing for byte 0xA0, which alone also decodes to one marker. The
"one marker vs two" discriminator does not exist. Digesta's charset instinct was
closer to the mark than the rebuttal. base.UNREADABLE and docs/ocr.md now carry
the correction rather than the refuted argument.

What IS established, measured on the Aruba install at the same DPI through the
same transport on perfect synthetic Italian (crisp vector glyphs, 34pt, 250 DPI):

    surya    29 U+FFFD markers, zero accents surviving
    chandra   0 markers, all 12 accents intact

So the document is exonerated. Also notable: the em dash survived surya intact
while every two-byte accent died, and surya emitted U+1EEA (Vietnamese) in place
of one -- no decoding bug can fabricate a character that was never in the input.

Implementation notes:

- surya DROPS this data. Its recognition predictor does pass it on
  (`BlockOCRResult(..., raw_logprobs=out.logprobs)`) but the model never declares
  the field and pydantic defaults to extra="ignore", so it is constructed and
  silently discarded -- their own comment calls it "phase 2 use". Verified by
  construction: the attribute does not exist on the result. So the value is taken
  one layer earlier, wrapping `_generate_one`, which still has it. The wrapper
  only reads and swallows every error: a diagnostic that could break a page would
  be worse than no diagnostic.
- The scan reassembles the WHOLE byte stream and walks it as UTF-8 rather than
  judging tokens pairwise. Per token is meaningless here -- one character
  legitimately arrives as two or three tokens, each "invalid" alone -- and the
  first version of this did judge them pairwise and flagged every accent and
  every em dash, the exact false positive that would make it worthless.
- Capped at 40 findings per page, two tokens of context each side, because a lead
  byte is only wrong in the light of what followed it.
- Clean pages are byte-identical to before: the key appears only when the text
  actually contains a marker, and it is reported ONCE document-wide under
  `unreadable.tokens` rather than repeated in every page's meta.
- The flag is in _pool_params, because the hook patches surya's backend at LOAD
  -- without that, toggling it would appear to save and take effect never.

A third discriminator needing no code: serve the same weights through llama.cpp
(`datalab-to/surya-ocr-2-gguf`, `surya_serve: "llamacpp"`). A different
detokenizer on identical weights would convict vLLM outright.

tests/test_ocr_token_trace.py: 30 tests. 818 passed across the OCR/config/admin/
runpod selection with the same 10 pre-existing failures.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01PxYHWbCAjjjD7A1rhFKtaW