chunk: byte-identity with the implementation this ports (v0.3.78)

Ran both side by side on the same documents instead of reading the spec,
and the spec's prose turned out to allow several readings the port took
differently. Seven divergences in reference extraction, one in sectioning:

* `artt. 163, 164 e 167 c.p.c.` extracted nothing -- the code had to sit
  immediately after the number, so an enumeration lost the whole citation.
  It now resolves to its first article, which is what the indexes hold.
* `art. 2055-bis c.c.` extracted nothing (only a space was tolerated before
  the suffix); the hyphen is now kept in the key, a space still squeezed out.
* `art. 2043 cod. civ.` extracted nothing -- the code spelled out was in the
  normalisation table but not in the pattern.
* `articolo 2043 c.c.` and `art. 2043 della c.c.` extracted a reference the
  original does not. Looking like an improvement does not make it one: it
  puts keys in the field that the corpus it will be matched against lacks.
* `legge 241 del 1990` extracted nothing; `l 241/1990` without the dot
  extracted one. Both inverted.
* `art. 360` with no code came back as `art.360`, and `d.p.r. 115/02` as
  nothing -- the two already reported.

And the off-by-one, which was none of the things it looked like. Section
bodies were not stripped, so merging two adjacent same-label sections (a
filing opening its conclusions twice) joined a body ending in a blank line
to the next, turning a line break into a paragraph break -- packed as
`\n\n` rather than riding inside one piece. Exactly one character, on
exactly the documents that merge. Empty sections were also kept, spending
a `parent` id and shifting every id after it.

Verified by diffing chunk-for-chunk over 63 generated documents covering
CRLF, repeated headers, enumerations, two-digit years and degenerate input:
zero divergences in text, section, parent and norm_refs.

The one deliberate difference is `**FATTO**`, which the original does not
match and `it-legal` does. `it-legal-strict` is added for re-chunking a
corpus indexed with the original, where a boundary that moves makes the new
index incomparable with the old.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3

codai/api/chunking.py

@@ -115,26 +115,57 @@ IT_LEGAL: Dict[str, Any] = {
     ],
 }
 
-#: DEVIATION FROM docs/chunking.md, deliberate and recorded here because it
-#: affects byte-identity with the implementation this ports.
+#: Byte-identical to the implementation this ports: the anchors are the plain
+#: `^\s*fatto\s*$` of the original, so `**FATTO**` falls through and the
+#: document is not sectioned at all.
 #:
-#: The doc gives the patterns verbatim and asks for a fidelity case with a
-#: markdown-decorated header (`**FATTO**`). Those two requirements contradict
-#: each other: the verbatim `^\s*fatto\s*$` cannot match `**FATTO**`. The
-#: anchors here are widened to `^[\s*#>_]*fatto[\s*#>_]*$` so the documented
-#: case works.
+#: It exists for ONE job: re-chunking a corpus that was indexed with the
+#: original, where a boundary that moves makes the new index incomparable with
+#: the old. Anyone starting fresh wants `it-legal`.
+IT_LEGAL_STRICT: Dict[str, Any] = dict(
+    IT_LEGAL, name="it-legal-strict",
+    sections=[
+        {"label": "premessa",
+         "pattern": r"tanto\s+premesso|ci[oò]\s+premesso|^\s*[*#>_ ]*premesso\s+che\b|"
+                    r"^\s*premess[ae]\s*$"},
+        {"label": "fatto",
+         "pattern": r"svolgimento\s+del\s+processo|ragioni\s+di\s+fatto|in\s+fatto\b|"
+                    r"fatto\s+e\s+diritto|^\s*fatto\s*$|esposizione\s+del\s+fatto"},
+        {"label": "motivazione",
+         "pattern": r"motivi\s+della\s+decisione|ragioni\s+della\s+decisione|"
+                    r"motivazione|considerato\s+in\s+diritto|in\s+diritto\b|"
+                    r"motivi\s+in\s+diritto|ragioni\s+in\s+diritto"},
+        {"label": "dispositivo",
+         "pattern": r"p\.?\s*q\.?\s*m\.?|p\.?\s*t\.?\s*m\.?|per\s+questi\s+motivi|"
+                    r"^\s*dispositivo\s*$|così\s+deciso|così\s+provvede"},
+        {"label": "conclusioni",
+         "pattern": r"^\s*[*#>_ ]*conclusion[ei]\b|\bconclusioni\s*:|"
+                    r"voglia\s+(?:l[\'’]ill\.?mo\s+)?(?:il\s+)?(?:tribunale|giudice|giudicante)|"
+                    r"^\s*[*#>_ ]*chiede\b|salvis\s+iuribus"},
+    ])
+
+#: THE ONE DELIBERATE DIVERGENCE from the implementation this ports, and it is
+#: here rather than in a commit message because it is the only thing that can
+#: move a chunk boundary.
+#:
+#: Verified by running both against the same documents: the original does NOT
+#: match `**FATTO**` -- its `^\s*fatto\s*$` cannot, since the `**` is consumed
+#: before the line anchor is tested. A document with emphasised headers is
+#: therefore not sectioned at all by the original; `it-legal` widens the
+#: anchors to `^[\s*#>_]*fatto[\s*#>_]*$` so it is.
 #:
 #: The widening is purely additive -- no line that matched before stops
 #: matching, so no existing match position moves. What changes is which
 #: documents are sectioned AT ALL: one whose headers were emphasised used to
-#: fall back to flat chunking and now gets labels. Those documents' chunks
-#: differ from an index built before this and must be re-chunked; documents
-#: already sectioned are identical.
+#: fall back to flat chunking and now gets labels. On everything else the two
+#: templates agree byte for byte.
 #:
-#: Worth confirming against the production implementation: if it DOES match
-#: `**FATTO**`, its patterns differ from the table in the doc, and that
-#: difference is what matters for identity rather than this note.
-BUILTIN_TEMPLATES: Dict[str, Dict[str, Any]] = {"it-legal": IT_LEGAL}
+#: `it-legal-strict` keeps the original behaviour for re-chunking an index that
+#: was built with it.
+BUILTIN_TEMPLATES: Dict[str, Dict[str, Any]] = {
+    "it-legal": IT_LEGAL,
+    "it-legal-strict": IT_LEGAL_STRICT,
+}
 
 
 def _flags_from(spec: str) -> int:
@@ -267,25 +298,35 @@ def split_sections(text: str, tpl: Optional[dict]) -> Tuple[List[dict], bool, Op
     ``keep_header_line`` is set, so the header stays inside its own section.
     """
     if tpl is None:
-        return [{"label": "", "text": text}], False, None
+        return [{"label": "", "text": text.strip()}], False, None
     rx, labels, opts = compile_template(tpl)
     marks = [(m.start(), labels[int(m.lastgroup[1:])])
              for m in rx.finditer(text) if m.lastgroup]
     if not marks:
         # No header anywhere: flat, and SAID so, rather than inventing one
         # section per paragraph or pretending the template applied.
-        return [{"label": "", "text": text}], False, opts["name"]
+        return [{"label": "", "text": text.strip()}], False, opts["name"]
     out: List[dict] = []
-    head = text[:marks[0][0]]
-    if head.strip() and opts["head_label"] is not None:
+    # Every section body is STRIPPED, and an empty one is dropped entirely.
+    # Both matter beyond tidiness: an unstripped body ends with the blank line
+    # before the next header, so merging two of them would insert a PARAGRAPH
+    # break where there should be a line break -- one character, on every
+    # merged section, which is enough to make a re-chunked index incomparable
+    # with the one it replaces. An empty section would spend a `parent` id and
+    # shift every id after it.
+    head = text[:marks[0][0]].strip()
+    if head and opts["head_label"] is not None:
         out.append({"label": str(opts["head_label"] or ""), "text": head})
     for i, (pos, label) in enumerate(marks):
         end = marks[i + 1][0] if i + 1 < len(marks) else len(text)
-        body = text[pos:end] if opts["keep_header_line"] else text[pos:end]
+        body = (text[pos:end] if opts["keep_header_line"] else text[pos:end]).strip()
+        if not body:
+            continue
         if opts["merge_adjacent"] and out and out[-1]["label"] == label:
             # Filings routinely open the conclusions twice (CONCLUSIONI, then
             # "Voglia il Tribunale..."). Left separate, one block becomes two
             # and the parent ids multiply, defeating the point of `parent`.
+            # Joined with ONE newline, so the two halves stay one paragraph.
             out[-1]["text"] = out[-1]["text"] + "\n" + body
         else:
             out.append({"label": label, "text": body})
@@ -363,21 +404,37 @@ def pack(pieces: List[str], target: int, overlap: int) -> List[str]:
 # ---------------------------------------------------------------------------
 # Reference extraction
 # ---------------------------------------------------------------------------
-_ART_SUFFIX = r"(?:\s*(?:bis|ter|quater|quinquies|sexies))?"
-#: The code suffix is MANDATORY, not optional. A bare `art. 360` is ambiguous
-#: -- every procedural code has one -- so it is deliberately NOT a reference:
-#: this field exists to be exact-matched, and an unqualified number would put
-#: three different articles under one key.
+#: Ported VERBATIM from the implementation these fixtures were cut against.
+#: Every liberty taken with these patterns -- `articolo` as well as `art.`,
+#: `della` as well as `del`, a dotless `l` -- moves a reference into or out of
+#: a chunk's filter field, and an index built with one cannot be compared with
+#: an index built with the other.
+_ART_SUFFIX = r"(?:[- ]?(?:bis|ter|quater|quinquies|sexies))?"
+#: `cod. civ.` spelled out, as well as every dotted abbreviation. Order matters:
+#: `c.p.c.` must be offered before `c.p.`, or the longer code loses its tail.
+_CODE = (r"(?:c\.?c\.?|cod\.?\s*civ\.?|c\.?p\.?c\.?|c\.?p\.?p\.?"
+         r"|c\.?p\.?|cost\.?)")
+#: The code suffix is MANDATORY. A bare `art. 360` is ambiguous -- every
+#: procedural code has one -- so it is deliberately NOT a reference: this field
+#: exists to be exact-matched, and an unqualified number would put three
+#: different articles under a single key.
+#:
+#: The middle group swallows an ENUMERATION (`artt. 163, 164 e 167 c.p.c.`)
+#: without capturing it, so the citation resolves to its FIRST article and the
+#: code at the end is still reached. Only the first is emitted -- that is the
+#: behaviour the existing indexes were built with.
 _ART_RX = re.compile(
-    r"\bart(?:t|icol[oi])?\.?\s*(\d+" + _ART_SUFFIX + r")"
-    r"\s*(?:del\s+|della\s+)?(codciv|c\.?c\.?|c\.?p\.?c\.?|c\.?p\.?p\.?|c\.?p\.?|cost\.?)",
-    re.IGNORECASE)
-#: The year is kept VERBATIM, two digits included: `d.p.r. 115/02` normalises
-#: to `d.p.r.115/02`, never to `/2002`. Expanding it would silently split the
-#: same act across two keys depending on how the source wrote it.
+    r"artt?\.?\s*(\d{1,4}" + _ART_SUFFIX + r")"
+    r"(?:\s*,?\s*(?:e|ed|,)\s*\d{1,4})*"
+    r"\s*(?:del\s+)?(" + _CODE + r")", re.IGNORECASE)
+#: The year is two to four digits and is kept VERBATIM: `d.p.r. 115/02`
+#: normalises to `d.p.r.115/02`, never to `/2002`. Expanding it would file the
+#: same act under two keys depending on how the source wrote it. `del` is a
+#: separator as well as `/`, because `legge 241 del 1990` is how it is written
+#: in running prose.
 _ACT_RX = re.compile(
-    r"\b(d\.?\s*lgs\.?|d\.?\s*l\.?|d\.?\s*p\.?\s*r\.?|legge|l\.?)\s*"
-    r"(?:n\.?\s*)?(\d+)\s*/\s*(\d{2,4})", re.IGNORECASE)
+    r"(d\.?\s*lgs\.?|d\.?\s*l\.?|d\.?\s*p\.?\s*r\.?|legge|l\.)\s*"
+    r"(?:n\.?\s*)?(\d{1,4})\s*(?:/|\s+del\s+)\s*(\d{2,4})", re.IGNORECASE)
 _CODE_MAP = {"codciv": "c.c.", "cc": "c.c.", "cpc": "c.p.c.", "cpp": "c.p.p.",
              "cp": "c.p.", "cost": "cost."}
 _KIND_MAP = {"dlgs": "d.lgs.", "dl": "d.l.", "dpr": "d.p.r.", "legge": "l.",
@@ -393,9 +450,12 @@ def extract_refs(text: str) -> List[str]:
     """
     seen: Dict[str, None] = {}
     for m in _ART_RX.finditer(text):
-        num = re.sub(r"\s+", "", m.group(1) or "").lower()
+        # Spaces only: `2055-bis` keeps its hyphen, `2055 bis` loses its space.
+        num = re.sub(r"\s", "", m.group(1) or "").lower()
         code_raw = re.sub(r"[.\s]", "", (m.group(2) or "")).lower()
-        code = _CODE_MAP.get(code_raw, "")
+        # An unmapped code is kept as written rather than dropped -- the suffix
+        # was matched, so it IS a code, just not one the table knows.
+        code = _CODE_MAP.get(code_raw, code_raw)
         seen.setdefault(f"art.{num} {code}".strip(), None)
     for m in _ACT_RX.finditer(text):
         kind_raw = re.sub(r"[.\s]", "", (m.group(1) or "")).lower()

docs/chunking.md

@@ -254,11 +254,31 @@ extracts article and act citations, normalises them, and de-duplicates while
   The code is **required**: a bare `art. 360` is not emitted, because every
   procedural code has one and the field exists to be exact-matched — an
   unqualified number files three different articles under a single key.
+  Only `art`/`artt` open a citation (not `articolo`), only `del` links it to
+  its code (not `della`), the code may be spelled `cod. civ.`, and a suffix may
+  be hyphenated (`art.2055-bis c.c.`) — the hyphen is kept, a space is not.
+  An **enumeration** (`artt. 163, 164 e 167 c.p.c.`) resolves to its first
+  article only: `art.163 c.p.c.`
 - Acts → `<kind><num>/<year>`, e.g. `d.lgs.209/2005`, with
   `dlgs → d.lgs.`, `dl → d.l.`, `dpr → d.p.r.`, `legge|l → l.`
   The year is two to four digits and is kept **verbatim**: `d.p.r. 115/02`
   normalises to `d.p.r.115/02`, never to `/2002`, so the same act does not
-  split across two keys depending on how the source wrote it.
+  split across two keys depending on how the source wrote it. `del` separates
+  as well as `/` (`legge 241 del 1990`), and the bare `l` needs its dot.
+
+### `it-legal-strict`
+
+`it-legal` widens the line-anchored headers to tolerate markdown emphasis, so
+`**FATTO**` opens a section. The implementation this ports does not: its
+`^\s*fatto\s*$` cannot match once the `**` is consumed, and such a document is
+not sectioned at all. That is the **only** deliberate behavioural difference
+between the two, and it is purely additive — no line that matched before stops
+matching, and on an undecorated document the two templates agree byte for byte,
+including sections, parents and refs.
+
+`it-legal-strict` keeps the original anchors. Use it to re-chunk a corpus that
+was indexed with the original, where a boundary that moves makes the new index
+incomparable with the old. Anyone starting fresh wants `it-legal`.
 
 These are a filter field and an exact-match signal for hybrid search — the thing
 dense embeddings blur, since `art. 2043` and `art. 2054` sit close in vector

tests/test_chunking.py

@@ -370,6 +370,40 @@ def test_acts_normalise_to_kind_number_year():
     assert ch.extract_refs("legge 24/2017") == ["l.24/2017"]
 
 
+# Cases taken one by one from a diff against the implementation this ports,
+# run side by side on the same documents. Each was a real divergence.
+def test_an_article_enumeration_resolves_to_its_first_article():
+    """`artt. 163, 164 e 167 c.p.c.` must reach the code at the end and emit
+    the FIRST article only. Requiring the code to sit immediately after the
+    number dropped the whole citation instead."""
+    assert ch.extract_refs("artt. 115 e 116 c.p.c.") == ["art.115 c.p.c."]
+    assert ch.extract_refs("artt. 163, 164 e 167 c.p.c.") == ["art.163 c.p.c."]
+
+
+def test_an_article_suffix_may_be_hyphenated():
+    """`art. 2055-bis c.c.` keeps its hyphen; `art. 2055 bis c.c.` loses its
+    space. Only whitespace is squeezed out of the number."""
+    assert ch.extract_refs("art. 2055-bis c.c.") == ["art.2055-bis c.c."]
+    assert ch.extract_refs("art. 2055 bis c.c.") == ["art.2055bis c.c."]
+
+
+def test_the_code_may_be_spelled_out():
+    assert ch.extract_refs("art. 2043 cod. civ.") == ["art.2043 c.c."]
+
+
+def test_articolo_spelled_out_is_not_an_article_citation():
+    """`art`/`artt` only. Accepting `articolo` adds references the corpus this
+    was measured against does not have, which is a divergence even though it
+    looks like an improvement."""
+    assert ch.extract_refs("articolo 2043 c.c.") == []
+    assert ch.extract_refs("visto l'art. 2043 della c.c.") == []
+
+
+def test_an_act_may_be_written_with_del_instead_of_a_slash():
+    assert ch.extract_refs("legge 241 del 1990") == ["l.241/1990"]
+    assert ch.extract_refs("l 241/1990") == []        # the dot is required
+
+
 def test_a_bare_article_number_is_not_a_reference():
     """Reported from production: `art. 380` and `art. 360` came back as
     `['art.380']` -- an unqualified number is ambiguous (every code has an
@@ -377,20 +411,70 @@ def test_a_bare_article_number_is_not_a_reference():
     key. The code suffix is required, and a citation without one is dropped."""
     assert ch.extract_refs("si veda l'art. 360 n. 3") == []
     assert ch.extract_refs("art.380, art.360, art.115, art.395") == []
-    # ...but the same numbers WITH a code are still references.
     assert ch.extract_refs("violazione dell'art. 380 c.p.c.") == ["art.380 c.p.c."]
 
 
 def test_an_act_year_may_be_two_digits_and_is_kept_verbatim():
-    """`d.p.r. 115/02` was extracted as nothing at all, because the year was
-    pinned to four digits. It is kept as written: expanding `02` to `2002`
-    would file the same act under two keys depending on the source's style."""
+    """`d.p.r. 115/02` was extracted as nothing, because the year was pinned to
+    four digits. It is kept as written: expanding `02` to `2002` would file the
+    same act under two keys depending on the source's style."""
     assert ch.extract_refs("ai sensi del d.p.r. 115/02") == ["d.p.r.115/02"]
     assert ch.extract_refs(
         "del d.p.r. 115/02 e del d.lgs. 446/1998") == ["d.p.r.115/02",
                                                        "d.lgs.446/1998"]
 
 
+# ------------------------------------------------- section bodies are strip #
+def test_two_adjacent_sections_merge_into_one_paragraph_not_two():
+    """THE off-by-one. An unstripped section body ends with the blank line
+    before the next header, so merging two of them inserted a PARAGRAPH break
+    where the original puts a line break -- and a paragraph break is packed as
+    `\n\n` while a line break rides along inside one piece. One character per
+    merged section, which is enough to make a re-chunked index incomparable."""
+    txt = ("CONCLUSIONI\n\nchiede il rigetto.\n\n"
+           "Voglia il Tribunale\n\naccogliere le domande.\n")
+    secs, applied, _ = ch.split_sections(txt, ch.IT_LEGAL)
+    assert applied
+    merged = [s for s in secs if s["label"] == "conclusioni"]
+    assert len(merged) == 1, [s["label"] for s in secs]
+    body = merged[0]["text"]
+    assert not body.startswith(("\n", " ")) and not body.endswith(("\n", " "))
+    assert "chiede il rigetto.\nVoglia il Tribunale" in body
+
+
+def test_an_empty_section_does_not_spend_a_parent_id():
+    """Two headers back to back leave an empty body between them. Kept, it
+    would take a `parent` and shift every id after it."""
+    txt = "testa.\n\nSVOLGIMENTO DEL PROCESSO\nP.Q.M.\n\ncondanna."
+    secs, _, _ = ch.split_sections(txt, ch.IT_LEGAL)
+    assert all(s["text"].strip() for s in secs)
+
+
+# -------------------------------------------------------- strict template #
+def test_strict_template_keeps_the_original_anchors():
+    """`it-legal` matches a markdown-emphasised header; the implementation this
+    ports does not, so a corpus indexed with it must be re-chunked the old way
+    or not at all. `it-legal-strict` is that old way, and it is the ONLY
+    deliberate behavioural difference between the two."""
+    txt = "testa.\n\n**FATTO**\n\ncorpo.\n\n**IN DIRITTO**\n\nmotivi."
+    wide = ch.chunk_document(txt, structure="it-legal", **DEFAULTS)
+    strict = ch.chunk_document(txt, structure="it-legal-strict", **DEFAULTS)
+    assert [c["section"] for c in wide["chunks"]] == ["intestazione", "fatto",
+                                                      "motivazione"]
+    assert [c["section"] for c in strict["chunks"]] == ["intestazione",
+                                                        "motivazione"]
+    assert strict["structure"] == "it-legal-strict"
+
+
+def test_strict_and_wide_agree_on_an_undecorated_document():
+    """The widening is purely additive: a document whose headers carry no
+    markdown is sectioned identically by both."""
+    wide = ch.chunk_document(RULING, structure="it-legal", **DEFAULTS)
+    strict = ch.chunk_document(RULING, structure="it-legal-strict", **DEFAULTS)
+    assert [(c["text"], c["section"], c["parent"]) for c in wide["chunks"]] == \
+           [(c["text"], c["section"], c["parent"]) for c in strict["chunks"]]
+
+
 def test_references_are_deduplicated_in_first_seen_order():
     txt = "art. 2054 c.c., poi art. 2043 c.c., di nuovo art. 2054 c.c."
     assert ch.extract_refs(txt) == ["art.2054 c.c.", "art.2043 c.c."]