fix(ownership,search): per-transaction footnote attribution + table snippet

Two follow-up fixes from the #862/#863 review.

edgartools-t043 — Form 4/5 per-transaction footnote attribution:
Footnote references attach to many transaction sub-elements (security
title, date, shares, price, post-transaction amounts), but the extractor
collected them only from <transactionCoding>, so TransactionActivity.
footnote_ids/.footnotes_text came through empty for most filings (e.g. a
10b5-1 disclosure on <securityTitle>). The extractor now gathers footnote
IDs from the whole transaction; get_footnotes dedupes (order-preserving)
so a footnote referenced twice in one transaction isn't resolved twice.
This makes per-transaction is_10b5_1_plan work directly rather than via
the filing-wide fallback added in #863.

edgartools-fi4h — table-search snippet over-highlight:
The semantic table:<term> search set end_offset to the full table length
against a context truncated to ~200 chars, so SearchResult.snippet wrapped
the entire truncated context in ** ** for any table >200 chars. The match
is now located within the table and a correct context window is produced
via _get_context_with_offsets, consistent with text/regex search.

Adds fast (no-network) regression tests for both using the 374Water
fixture and an HTML table; updates the #863 AAPL end-to-end test to assert
the now-correct per-transaction attribution.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

Author: dgunning

CHANGELOG.md

@@ -13,6 +13,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 
+- **Form 4/5 transaction footnotes are now attributed per transaction** — footnote references attach to many transaction sub-elements (security title, transaction date, shares, price, post-transaction amounts), but the extractor previously collected them only from `<transactionCoding>`, so `TransactionActivity.footnote_ids` / `.footnotes_text` came through empty for most filings. The extractor now gathers footnote IDs from the whole transaction (deduplicated), so per-transaction footnote reasoning — including `is_10b5_1_plan` — works directly rather than relying on the filing-wide fallback.
+- **Table search snippets highlight the match, not the whole table** — the semantic `table:<term>` search built a result whose end offset was the full table length against a context truncated to ~200 chars, so `SearchResult.snippet` wrapped the entire truncated context in `**…**` for any table longer than 200 characters. The match is now located within the table and a correct context window is produced, consistent with text and regex search.
 - **Rule 10b5-1 plan detection now uses the official Form 4/5 checkbox when present** — ownership summaries honor the structured `aff10b5One` value before falling back to footnote text. When the checkbox is absent (e.g. pre-2023 filings), `has_10b5_1_plan` now scans the filing's full footnote set instead of relying solely on per-transaction footnote attribution, which is frequently empty. The fallback matcher no longer confuses the separate anti-fraud Rule 10b-5 with Rule 10b5-1 trading plans while recognizing common spacing and dash variants. ([#863](https://github.com/dgunning/edgartools/issues/863))
 - **`edgar` package logger no longer leaks log output in unconfigured applications** — the package-root logger had no `NullHandler`, so when an application had not configured logging itself, edgartools warnings fell back to Python's `logging.lastResort` handler and were written to stderr. This was especially harmful in MCP / stdio environments, where stray stderr output corrupts the protocol stream. A `NullHandler` is now attached to the `edgar` logger per the Python logging HOWTO for libraries, keeping edgartools silent until the application opts into logging. ([#856](https://github.com/dgunning/edgartools/issues/856))
 - **BDC data sets download again after SEC URL move** — the SEC relocated the DERA BDC data-set files from `/files/structureddata/data/` to `/files/datastandardsinnovation/data/`, which made `fetch_bdc_dataset()` and related functions fail with 404; the base URL now points to the new location.

edgar/documents/search.py

@@ -287,14 +287,23 @@ def _semantic_search(self, query: str) -> List[SearchResult]:
                 if isinstance(node, TableNode):
                     # Search in table content
                     table_text = node.text()
-                    if table_text and search_text.lower() in table_text.lower():
-                        results.append(SearchResult(
-                            node=node,
-                            text=f"Table: {node.caption or 'Untitled'}",
-                            start_offset=0,
-                            end_offset=len(table_text),
-                            context=table_text[:200] + "..." if len(table_text) > 200 else table_text
-                        ))
+                    if table_text:
+                        match_pos = table_text.lower().find(search_text.lower())
+                        if match_pos != -1:
+                            # Build a context window around the actual match with
+                            # offsets relative to that window, so snippet highlights
+                            # the match rather than the whole truncated table text
+                            # (edgartools-fi4h).
+                            context, adj_start, adj_end = self._get_context_with_offsets(
+                                table_text, match_pos, match_pos + len(search_text)
+                            )
+                            results.append(SearchResult(
+                                node=node,
+                                text=f"Table: {node.caption or 'Untitled'}",
+                                start_offset=adj_start,
+                                end_offset=adj_end,
+                                context=context
+                            ))
 
         elif search_type == 'section':
             # Search sections

edgar/ownership/core.py

@@ -75,7 +75,11 @@ def transaction_footnote_id(tag: Tag) -> Tuple[str, Optional[str]]:
 
 
 def get_footnotes(tag: Tag) -> str:
-    footnotes = []
+    # A single transaction can reference the same footnote id from several
+    # sub-elements (e.g. securityTitle and shares), so dedupe while preserving
+    # first-seen order to avoid resolving the same footnote text twice.
+    footnotes: list[str] = []
+    seen = set()
     for el in tag.find_all("footnoteId"):
         if isinstance(el, Tag):
             footnote_id = el.attrs.get('id')
@@ -84,7 +88,10 @@ def get_footnotes(tag: Tag) -> str:
                 if isinstance(footnote_id, list):
                     footnote_id = footnote_id[0] if footnote_id else None
                 if footnote_id:
-                    footnotes.append(str(footnote_id))
+                    footnote_id = str(footnote_id)
+                    if footnote_id not in seen:
+                        seen.add(footnote_id)
+                        footnotes.append(footnote_id)
     return '\n'.join(footnotes)
 
 

edgar/ownership/table_containers.py

@@ -155,6 +155,10 @@ def extract_transactions(table: Tag) -> NonDerivativeTransactions:
                     ('AcquiredDisposed', child_text(transaction_amt_tag, 'transactionAcquiredDisposedCode')),
                     ('DirectIndirect', child_text(ownership_nature_tag, 'directOrIndirectOwnership')),
                     ('NatureOfOwnership', child_text(ownership_nature_tag, 'natureOfOwnership')),
+                    # Collect footnoteId references from the whole transaction, not just
+                    # <transactionCoding>: footnotes attach to securityTitle, transaction
+                    # date, shares, price, etc. (edgartools-t043).
+                    ('footnotes', get_footnotes(transaction_tag)),
                 ]
             )
             transaction_coding_tag = transaction_tag.find("transactionCoding")
@@ -164,7 +168,6 @@ def extract_transactions(table: Tag) -> NonDerivativeTransactions:
                         ('form', child_text(transaction_coding_tag, 'transactionFormType')),
                         ('Code', child_text(transaction_coding_tag, 'transactionCode')),
                         ('EquitySwap', get_bool(child_text(transaction_coding_tag, 'equitySwapInvolved'))),
-                        ('footnotes', get_footnotes(transaction_coding_tag))
                     ]
                 )
                 transaction.update(transaction_coding)
@@ -276,6 +279,9 @@ def extract_transactions(table: Tag) -> DerivativeTransactions:
                     ('AcquiredDisposed', child_text(transaction_amt_tag, 'transactionAcquiredDisposedCode')),
                     ('Date', child_value(transaction_tag, 'transactionDate')),
                     ('Remaining', child_text(post_transaction_tag, 'sharesOwnedFollowingTransaction')),
+                    # Collect footnoteId references from the whole transaction, not just
+                    # <transactionCoding> (edgartools-t043).
+                    ('footnotes', get_footnotes(transaction_tag)),
                 ]
             )
 
@@ -287,7 +293,6 @@ def extract_transactions(table: Tag) -> DerivativeTransactions:
                         ('form', child_text(transaction_coding_tag, 'transactionFormType')),
                         ('Code', child_text(transaction_coding_tag, 'transactionCode')),
                         ('EquitySwap', get_bool(child_text(transaction_coding_tag, 'equitySwapInvolved'))),
-                        ('footnotes', get_footnotes(transaction_coding_tag))
                     ]
                 )
                 transaction.update(transaction_coding)

tests/issues/regression/test_issue_863_10b5_plan_detection.py

@@ -134,14 +134,19 @@ def test_aff10b5_one_checkbox_flows_through_xml_parsing_end_to_end():
 
 @pytest.mark.network
 @pytest.mark.vcr
-def test_problem2_full_footnote_fallback_on_real_pre_checkbox_filing():
+def test_problem2_real_pre_checkbox_filing_resolves_via_per_transaction_footnotes():
     """End-to-end on the reported Problem 2 filing: AAPL insider Form 4 (2022).
 
     Pinned by accession (not ``.latest()``) and VCR-backed for determinism. This
-    is a pre-2023-04 Form 4 that carries NO ``<aff10b5One>`` checkbox, and whose
-    per-transaction footnote attribution is empty, yet whose filing-level
-    footnotes describe a Rule 10b5-1 plan. Before the fix, ``has_10b5_1_plan``
-    returned ``None``; the full-footnote-set fallback now returns ``True``.
+    is a pre-2023-04 Form 4 that carries NO ``<aff10b5One>`` checkbox; its 10b5-1
+    footnote is attached to ``<securityTitle>`` (not ``<transactionCoding>``).
+
+    Originally (#863 Problem 2) per-transaction footnote attribution came through
+    empty, so ``has_10b5_1_plan`` returned ``None`` and only the full-footnote
+    fallback could recover it. After edgartools-t043 fixed the attribution, each
+    transaction carries its real footnote IDs and per-transaction detection works
+    directly. (The full-footnote fallback remains a safety net, covered by the
+    synthetic tests above.)
 
     Reported in https://github.com/dgunning/edgartools/issues/863 (Problem 2).
     """
@@ -151,10 +156,10 @@ def test_problem2_full_footnote_fallback_on_real_pre_checkbox_filing():
                     filing_date='2022-05-06', accession_no='0000320193-22-000061')
     summary = filing.obj().get_ownership_summary()
 
-    # No structured checkbox on this pre-2023 filing — the fallback is what fires.
+    # No structured checkbox on this pre-2023 filing.
     assert summary.aff10b5_one is None
-    # Per-transaction footnote attribution is empty in the summary path...
-    assert all(not t.footnotes_text for t in summary.transactions)
-    # ...but the filing-level footnotes mention the plan, so the scan finds it.
-    assert '10b5-1' in summary.all_footnotes_text
+    # Per-transaction footnote attribution is now populated (edgartools-t043)...
+    assert all(t.footnote_ids for t in summary.transactions)
+    # ...and at least one transaction's own footnotes describe the 10b5-1 plan.
+    assert any(t.is_10b5_1_plan is True for t in summary.transactions)
     assert summary.has_10b5_1_plan is True

tests/issues/regression/test_issue_fi4h_table_snippet.py

@@ -0,0 +1,55 @@
+"""Regression test for edgartools-fi4h: table-search snippet over-highlight.
+
+The semantic ``table:<term>`` search path built a SearchResult with
+``start_offset=0`` and ``end_offset=len(table_text)`` against a context
+truncated to ~200 chars. ``SearchResult.snippet`` highlights
+``context[start_offset:end_offset]``, so for any table longer than 200 chars the
+out-of-range end made it wrap the *entire* truncated context in ``** **`` rather
+than the matched term.
+
+The fix locates the match within the table text and produces a context window
+with offsets relative to that window (same mechanism as text/regex search).
+
+Discovered while reviewing PRs #862 / #860 (the text/regex offset fix), which
+did not touch the table path.
+"""
+
+from edgar.documents import parse_html
+from edgar.documents.nodes import NodeType
+from edgar.documents.search import DocumentSearch, SearchMode
+
+
+def _doc_with_long_table(target: str = "REVENUE"):
+    """A table whose text() exceeds 200 chars, with `target` appearing late."""
+    cells = "".join(
+        f"<tr><td>filler cell {i} with several words here</td></tr>" for i in range(20)
+    )
+    html = (
+        f"<html><body><table>{cells}"
+        f"<tr><td>{target} total 12345</td></tr></table></body></html>"
+    )
+    return parse_html(html)
+
+
+def test_table_snippet_highlights_match_not_whole_table():
+    doc = _doc_with_long_table()
+    tables = [n for n in doc.root.walk() if n.type == NodeType.TABLE]
+    assert len(tables[0].text()) > 200  # precondition: context would be truncated
+
+    result = DocumentSearch(doc).search("table:REVENUE", mode=SearchMode.SEMANTIC)[0]
+
+    # Offsets stay inside the context (the over-highlight bug ran end_offset past it).
+    assert result.end_offset <= len(result.context)
+    # The highlighted span is the matched term, not the entire table.
+    assert result.context[result.start_offset:result.end_offset].lower() == "revenue"
+    assert "**REVENUE**" in result.snippet
+    # And the snippet does not wrap the whole context.
+    assert not result.snippet.startswith("**...")
+
+
+def test_table_search_no_match_returns_no_result():
+    doc = _doc_with_long_table()
+
+    results = DocumentSearch(doc).search("table:NONEXISTENT", mode=SearchMode.SEMANTIC)
+
+    assert results == []

tests/issues/regression/test_issue_t043_footnote_attribution.py

@@ -0,0 +1,71 @@
+"""Regression test for edgartools-t043: per-transaction footnote attribution.
+
+Footnote references in a Form 4/5 transaction attach to many sub-elements
+(securityTitle, transactionDate, transactionShares, transactionPricePerShare,
+postTransactionAmounts, ...), not only <transactionCoding>. The extractor
+previously collected footnoteIds from <transactionCoding> alone, so a footnote
+attached elsewhere (e.g. a 10b5-1 disclosure on <securityTitle>) was dropped and
+TransactionActivity.footnote_ids / .footnotes_text came through empty.
+
+Follow-up to GitHub #863 (Problem 2): the summary-level has_10b5_1_plan had a
+full-footnote fallback, but per-transaction footnote reasoning was still blind.
+
+The 374Water Form 4 fixture is a real filing whose only transaction footnote
+(F1) is attached to <securityTitle> with nothing under <transactionCoding> —
+exactly the dropped-attribution case — so this runs offline.
+"""
+
+from pathlib import Path
+
+import pytest
+
+from edgar.ownership.core import get_footnotes
+from edgar.ownership.forms import Form4
+
+FIXTURE = Path('data/ownership/374WaterForm4.xml')
+
+pytestmark = pytest.mark.skipif(not FIXTURE.exists(), reason='fixture not available')
+
+
+def _summary():
+    return Form4.parse_xml(FIXTURE.read_text()).get_ownership_summary()
+
+
+def test_transaction_footnote_ids_are_populated_from_whole_transaction():
+    """Each transaction surfaces the footnote attached to <securityTitle>."""
+    summary = _summary()
+
+    assert summary.transactions, "fixture should produce transactions"
+    # Every transaction in this filing references footnote F1 (vesting terms),
+    # attached to <securityTitle> — not <transactionCoding>.
+    for t in summary.transactions:
+        assert 'F1' in (t.footnote_ids or ''), f"missing F1 on {t.footnote_ids!r}"
+        assert t.footnotes_text, "footnote text should resolve from the IDs"
+        assert 'Vesting commencement' in t.footnotes_text
+
+
+def test_footnote_ids_are_deduped_within_a_transaction():
+    """A footnote referenced twice in one transaction is not resolved twice."""
+    summary = _summary()
+
+    for t in summary.transactions:
+        ids = [i for i in (t.footnote_ids or '').split('\n') if i]
+        assert len(ids) == len(set(ids)), f"duplicate footnote ids: {ids!r}"
+        # Resolved text should not repeat the same footnote sentence.
+        assert t.footnotes_text.count('Vesting commencement date') == 1
+
+
+def test_get_footnotes_dedupes_preserving_order():
+    """Unit check on the shared helper used by table extraction."""
+    from bs4 import BeautifulSoup
+
+    xml = (
+        "<root>"
+        "<securityTitle><footnoteId id='F1'/></securityTitle>"
+        "<transactionShares><footnoteId id='F2'/></transactionShares>"
+        "<transactionPricePerShare><footnoteId id='F1'/></transactionPricePerShare>"
+        "</root>"
+    )
+    tag = BeautifulSoup(xml, 'xml').find('root')
+
+    assert get_footnotes(tag) == 'F1\nF2'