• Stefy Lanza (nextime / spora )'s avatar
    chunk: byte-identity with the implementation this ports (v0.3.78) · 06b7b2e8
    Stefy Lanza (nextime / spora ) authored
    Ran both side by side on the same documents instead of reading the spec,
    and the spec's prose turned out to allow several readings the port took
    differently. Seven divergences in reference extraction, one in sectioning:
    
    * `artt. 163, 164 e 167 c.p.c.` extracted nothing -- the code had to sit
      immediately after the number, so an enumeration lost the whole citation.
      It now resolves to its first article, which is what the indexes hold.
    * `art. 2055-bis c.c.` extracted nothing (only a space was tolerated before
      the suffix); the hyphen is now kept in the key, a space still squeezed out.
    * `art. 2043 cod. civ.` extracted nothing -- the code spelled out was in the
      normalisation table but not in the pattern.
    * `articolo 2043 c.c.` and `art. 2043 della c.c.` extracted a reference the
      original does not. Looking like an improvement does not make it one: it
      puts keys in the field that the corpus it will be matched against lacks.
    * `legge 241 del 1990` extracted nothing; `l 241/1990` without the dot
      extracted one. Both inverted.
    * `art. 360` with no code came back as `art.360`, and `d.p.r. 115/02` as
      nothing -- the two already reported.
    
    And the off-by-one, which was none of the things it looked like. Section
    bodies were not stripped, so merging two adjacent same-label sections (a
    filing opening its conclusions twice) joined a body ending in a blank line
    to the next, turning a line break into a paragraph break -- packed as
    `\n\n` rather than riding inside one piece. Exactly one character, on
    exactly the documents that merge. Empty sections were also kept, spending
    a `parent` id and shifting every id after it.
    
    Verified by diffing chunk-for-chunk over 63 generated documents covering
    CRLF, repeated headers, enumerations, two-digit years and degenerate input:
    zero divergences in text, section, parent and norm_refs.
    
    The one deliberate difference is `**FATTO**`, which the original does not
    match and `it-legal` does. `it-legal-strict` is added for re-chunking a
    corpus indexed with the original, where a boundary that moves makes the new
    index incomparable with the old.
    Co-Authored-By: 's avatarClaude Opus 5 <noreply@anthropic.com>
    Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3
    06b7b2e8
Name
Last commit
Last update
codai Loading commit data...
docs Loading commit data...
packaging Loading commit data...
samples Loading commit data...
tests Loading commit data...
third_party Loading commit data...
tools Loading commit data...
.dockerignore Loading commit data...
.gitignore Loading commit data...
AI.PROMPT Loading commit data...
CODERAI_API_DOCUMENTATION.md Loading commit data...
CoderAI.gif Loading commit data...
DISTRIBUTION.md Loading commit data...
LICENSE.md Loading commit data...
MULTIMODAL_CAPABILITIES.md Loading commit data...
MULTIMODAL_UI_EXAMPLES.md Loading commit data...
README.md Loading commit data...
REQUEST-geoclip-embeddings.md Loading commit data...
REQUEST-vpr-embeddings.md Loading commit data...
amdgpu-coredump-05000.bin Loading commit data...
build-oci.sh Loading commit data...
build.ps1 Loading commit data...
build.sh Loading commit data...
coderai Loading commit data...
coderai-broker-implementation-reference.md Loading commit data...
coderai-integration.md Loading commit data...
commands Loading commit data...
osxbuild.sh Loading commit data...
package-oci.sh Loading commit data...
package-tarball.sh Loading commit data...
requirements-crisperwhisper.txt Loading commit data...
requirements-h3.txt Loading commit data...
requirements-longcat-train.txt Loading commit data...
requirements-longcat.txt Loading commit data...
requirements-melotts.txt Loading commit data...
requirements-nemo.txt Loading commit data...
requirements-nvidia.txt Loading commit data...
requirements-ocr-paddle.txt Loading commit data...
requirements-ocr.txt Loading commit data...
requirements-pyannote.txt Loading commit data...
requirements-surya.txt Loading commit data...
requirements-vllm.txt Loading commit data...
requirements-vulkan.txt Loading commit data...
requirements.txt Loading commit data...
run-oci.sh Loading commit data...
smoke-test-oci.sh Loading commit data...
todo.md Loading commit data...