• Stefy Lanza (nextime / spora )'s avatar
    chunk: byte-identity with the implementation this ports (v0.3.78) · 06b7b2e8
    Stefy Lanza (nextime / spora ) authored
    Ran both side by side on the same documents instead of reading the spec,
    and the spec's prose turned out to allow several readings the port took
    differently. Seven divergences in reference extraction, one in sectioning:
    
    * `artt. 163, 164 e 167 c.p.c.` extracted nothing -- the code had to sit
      immediately after the number, so an enumeration lost the whole citation.
      It now resolves to its first article, which is what the indexes hold.
    * `art. 2055-bis c.c.` extracted nothing (only a space was tolerated before
      the suffix); the hyphen is now kept in the key, a space still squeezed out.
    * `art. 2043 cod. civ.` extracted nothing -- the code spelled out was in the
      normalisation table but not in the pattern.
    * `articolo 2043 c.c.` and `art. 2043 della c.c.` extracted a reference the
      original does not. Looking like an improvement does not make it one: it
      puts keys in the field that the corpus it will be matched against lacks.
    * `legge 241 del 1990` extracted nothing; `l 241/1990` without the dot
      extracted one. Both inverted.
    * `art. 360` with no code came back as `art.360`, and `d.p.r. 115/02` as
      nothing -- the two already reported.
    
    And the off-by-one, which was none of the things it looked like. Section
    bodies were not stripped, so merging two adjacent same-label sections (a
    filing opening its conclusions twice) joined a body ending in a blank line
    to the next, turning a line break into a paragraph break -- packed as
    `\n\n` rather than riding inside one piece. Exactly one character, on
    exactly the documents that merge. Empty sections were also kept, spending
    a `parent` id and shifting every id after it.
    
    Verified by diffing chunk-for-chunk over 63 generated documents covering
    CRLF, repeated headers, enumerations, two-digit years and degenerate input:
    zero divergences in text, section, parent and norm_refs.
    
    The one deliberate difference is `**FATTO**`, which the original does not
    match and `it-legal` does. `it-legal-strict` is added for re-chunking a
    corpus indexed with the original, where a boundary that moves makes the new
    index incomparable with the old.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    06b7b2e8
Name
Last commit
Last update
..
superpowers Loading commit data...
README.md Loading commit data...
admin-forward-auth.md Loading commit data...
chunking.md Loading commit data...
cluster.md Loading commit data...
deepseek-ds4.md Loading commit data...
dtype-auto-selection.md Loading commit data...
expressive-tts.md Loading commit data...
frontend-engine-split.md Loading commit data...
gguf-process-isolation.md Loading commit data...
glm-colibri.md Loading commit data...
install-from-packages.md Loading commit data...
kimi-k3.md Loading commit data...
ktransformers.md Loading commit data...
longcat-video.md Loading commit data...
minimax-h3.md Loading commit data...
modelli-digesta-e-container.md Loading commit data...
ocr.md Loading commit data...
orchestrator-runpod-guide.md Loading commit data...
piano-inferenza-su-coderai-prod.md Loading commit data...
process-isolation-plans.md Loading commit data...
reference-projects.md Loading commit data...
remote-execution.md Loading commit data...
reverse-proxy-nginx.md Loading commit data...
runpod-regional-storage-plan.md Loading commit data...
runpod.md Loading commit data...
tools-as-services.md Loading commit data...
unlimited-ocr-plan.md Loading commit data...
video-compose.md Loading commit data...
vllm.md Loading commit data...
zimage-lora-training.md Loading commit data...