chunk: byte-identity with the implementation this ports (v0.3.78)

Ran both side by side on the same documents instead of reading the spec,
and the spec's prose turned out to allow several readings the port took
differently. Seven divergences in reference extraction, one in sectioning:

* `artt. 163, 164 e 167 c.p.c.` extracted nothing -- the code had to sit
  immediately after the number, so an enumeration lost the whole citation.
  It now resolves to its first article, which is what the indexes hold.
* `art. 2055-bis c.c.` extracted nothing (only a space was tolerated before
  the suffix); the hyphen is now kept in the key, a space still squeezed out.
* `art. 2043 cod. civ.` extracted nothing -- the code spelled out was in the
  normalisation table but not in the pattern.
* `articolo 2043 c.c.` and `art. 2043 della c.c.` extracted a reference the
  original does not. Looking like an improvement does not make it one: it
  puts keys in the field that the corpus it will be matched against lacks.
* `legge 241 del 1990` extracted nothing; `l 241/1990` without the dot
  extracted one. Both inverted.
* `art. 360` with no code came back as `art.360`, and `d.p.r. 115/02` as
  nothing -- the two already reported.

And the off-by-one, which was none of the things it looked like. Section
bodies were not stripped, so merging two adjacent same-label sections (a
filing opening its conclusions twice) joined a body ending in a blank line
to the next, turning a line break into a paragraph break -- packed as
`\n\n` rather than riding inside one piece. Exactly one character, on
exactly the documents that merge. Empty sections were also kept, spending
a `parent` id and shifting every id after it.

Verified by diffing chunk-for-chunk over 63 generated documents covering
CRLF, repeated headers, enumerations, two-digit years and degenerate input:
zero divergences in text, section, parent and norm_refs.

The one deliberate difference is `**FATTO**`, which the original does not
match and `it-legal` does. `it-legal-strict` is added for re-chunking a
corpus indexed with the original, where a boundary that moves makes the new
index incomparable with the old.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01HXRMe9ZPUQCCCtBai9uqt3

docs/chunking.md

@@ -254,11 +254,31 @@ extracts article and act citations, normalises them, and de-duplicates while
   The code is **required**: a bare `art. 360` is not emitted, because every
   procedural code has one and the field exists to be exact-matched — an
   unqualified number files three different articles under a single key.
+  Only `art`/`artt` open a citation (not `articolo`), only `del` links it to
+  its code (not `della`), the code may be spelled `cod. civ.`, and a suffix may
+  be hyphenated (`art.2055-bis c.c.`) — the hyphen is kept, a space is not.
+  An **enumeration** (`artt. 163, 164 e 167 c.p.c.`) resolves to its first
+  article only: `art.163 c.p.c.`
 - Acts → `<kind><num>/<year>`, e.g. `d.lgs.209/2005`, with
   `dlgs → d.lgs.`, `dl → d.l.`, `dpr → d.p.r.`, `legge|l → l.`
   The year is two to four digits and is kept **verbatim**: `d.p.r. 115/02`
   normalises to `d.p.r.115/02`, never to `/2002`, so the same act does not
-  split across two keys depending on how the source wrote it.
+  split across two keys depending on how the source wrote it. `del` separates
+  as well as `/` (`legge 241 del 1990`), and the bare `l` needs its dot.
+
+### `it-legal-strict`
+
+`it-legal` widens the line-anchored headers to tolerate markdown emphasis, so
+`**FATTO**` opens a section. The implementation this ports does not: its
+`^\s*fatto\s*$` cannot match once the `**` is consumed, and such a document is
+not sectioned at all. That is the **only** deliberate behavioural difference
+between the two, and it is purely additive — no line that matched before stops
+matching, and on an undecorated document the two templates agree byte for byte,
+including sections, parents and refs.
+
+`it-legal-strict` keeps the original anchors. Use it to re-chunk a corpus that
+was indexed with the original, where a boundary that moves makes the new index
+incomparable with the old. Anyone starting fresh wants `it-legal`.
 
 These are a filter field and an exact-match signal for hybrid search — the thing
 dense embeddings blur, since `art. 2043` and `art. 2054` sit close in vector

tests/test_chunking.py

@@ -370,6 +370,40 @@ def test_acts_normalise_to_kind_number_year():
     assert ch.extract_refs("legge 24/2017") == ["l.24/2017"]
 
 
+# Cases taken one by one from a diff against the implementation this ports,
+# run side by side on the same documents. Each was a real divergence.
+def test_an_article_enumeration_resolves_to_its_first_article():
+    """`artt. 163, 164 e 167 c.p.c.` must reach the code at the end and emit
+    the FIRST article only. Requiring the code to sit immediately after the
+    number dropped the whole citation instead."""
+    assert ch.extract_refs("artt. 115 e 116 c.p.c.") == ["art.115 c.p.c."]
+    assert ch.extract_refs("artt. 163, 164 e 167 c.p.c.") == ["art.163 c.p.c."]
+
+
+def test_an_article_suffix_may_be_hyphenated():
+    """`art. 2055-bis c.c.` keeps its hyphen; `art. 2055 bis c.c.` loses its
+    space. Only whitespace is squeezed out of the number."""
+    assert ch.extract_refs("art. 2055-bis c.c.") == ["art.2055-bis c.c."]
+    assert ch.extract_refs("art. 2055 bis c.c.") == ["art.2055bis c.c."]
+
+
+def test_the_code_may_be_spelled_out():
+    assert ch.extract_refs("art. 2043 cod. civ.") == ["art.2043 c.c."]
+
+
+def test_articolo_spelled_out_is_not_an_article_citation():
+    """`art`/`artt` only. Accepting `articolo` adds references the corpus this
+    was measured against does not have, which is a divergence even though it
+    looks like an improvement."""
+    assert ch.extract_refs("articolo 2043 c.c.") == []
+    assert ch.extract_refs("visto l'art. 2043 della c.c.") == []
+
+
+def test_an_act_may_be_written_with_del_instead_of_a_slash():
+    assert ch.extract_refs("legge 241 del 1990") == ["l.241/1990"]
+    assert ch.extract_refs("l 241/1990") == []        # the dot is required
+
+
 def test_a_bare_article_number_is_not_a_reference():
     """Reported from production: `art. 380` and `art. 360` came back as
     `['art.380']` -- an unqualified number is ambiguous (every code has an
@@ -377,20 +411,70 @@ def test_a_bare_article_number_is_not_a_reference():
     key. The code suffix is required, and a citation without one is dropped."""
     assert ch.extract_refs("si veda l'art. 360 n. 3") == []
     assert ch.extract_refs("art.380, art.360, art.115, art.395") == []
-    # ...but the same numbers WITH a code are still references.
     assert ch.extract_refs("violazione dell'art. 380 c.p.c.") == ["art.380 c.p.c."]
 
 
 def test_an_act_year_may_be_two_digits_and_is_kept_verbatim():
-    """`d.p.r. 115/02` was extracted as nothing at all, because the year was
-    pinned to four digits. It is kept as written: expanding `02` to `2002`
-    would file the same act under two keys depending on the source's style."""
+    """`d.p.r. 115/02` was extracted as nothing, because the year was pinned to
+    four digits. It is kept as written: expanding `02` to `2002` would file the
+    same act under two keys depending on the source's style."""
     assert ch.extract_refs("ai sensi del d.p.r. 115/02") == ["d.p.r.115/02"]
     assert ch.extract_refs(
         "del d.p.r. 115/02 e del d.lgs. 446/1998") == ["d.p.r.115/02",
                                                        "d.lgs.446/1998"]
 
 
+# ------------------------------------------------- section bodies are strip #
+def test_two_adjacent_sections_merge_into_one_paragraph_not_two():
+    """THE off-by-one. An unstripped section body ends with the blank line
+    before the next header, so merging two of them inserted a PARAGRAPH break
+    where the original puts a line break -- and a paragraph break is packed as
+    `\n\n` while a line break rides along inside one piece. One character per
+    merged section, which is enough to make a re-chunked index incomparable."""
+    txt = ("CONCLUSIONI\n\nchiede il rigetto.\n\n"
+           "Voglia il Tribunale\n\naccogliere le domande.\n")
+    secs, applied, _ = ch.split_sections(txt, ch.IT_LEGAL)
+    assert applied
+    merged = [s for s in secs if s["label"] == "conclusioni"]
+    assert len(merged) == 1, [s["label"] for s in secs]
+    body = merged[0]["text"]
+    assert not body.startswith(("\n", " ")) and not body.endswith(("\n", " "))
+    assert "chiede il rigetto.\nVoglia il Tribunale" in body
+
+
+def test_an_empty_section_does_not_spend_a_parent_id():
+    """Two headers back to back leave an empty body between them. Kept, it
+    would take a `parent` and shift every id after it."""
+    txt = "testa.\n\nSVOLGIMENTO DEL PROCESSO\nP.Q.M.\n\ncondanna."
+    secs, _, _ = ch.split_sections(txt, ch.IT_LEGAL)
+    assert all(s["text"].strip() for s in secs)
+
+
+# -------------------------------------------------------- strict template #
+def test_strict_template_keeps_the_original_anchors():
+    """`it-legal` matches a markdown-emphasised header; the implementation this
+    ports does not, so a corpus indexed with it must be re-chunked the old way
+    or not at all. `it-legal-strict` is that old way, and it is the ONLY
+    deliberate behavioural difference between the two."""
+    txt = "testa.\n\n**FATTO**\n\ncorpo.\n\n**IN DIRITTO**\n\nmotivi."
+    wide = ch.chunk_document(txt, structure="it-legal", **DEFAULTS)
+    strict = ch.chunk_document(txt, structure="it-legal-strict", **DEFAULTS)
+    assert [c["section"] for c in wide["chunks"]] == ["intestazione", "fatto",
+                                                      "motivazione"]
+    assert [c["section"] for c in strict["chunks"]] == ["intestazione",
+                                                        "motivazione"]
+    assert strict["structure"] == "it-legal-strict"
+
+
+def test_strict_and_wide_agree_on_an_undecorated_document():
+    """The widening is purely additive: a document whose headers carry no
+    markdown is sectioned identically by both."""
+    wide = ch.chunk_document(RULING, structure="it-legal", **DEFAULTS)
+    strict = ch.chunk_document(RULING, structure="it-legal-strict", **DEFAULTS)
+    assert [(c["text"], c["section"], c["parent"]) for c in wide["chunks"]] == \
+           [(c["text"], c["section"], c["parent"]) for c in strict["chunks"]]
+
+
 def test_references_are_deduplicated_in_first_seen_order():
     txt = "art. 2054 c.c., poi art. 2043 c.c., di nuovo art. 2054 c.c."
     assert ch.extract_refs(txt) == ["art.2054 c.c.", "art.2043 c.c."]