Skip to content

feat(artifacts): search a document's structure pages - #2319

Open
pirhoo wants to merge 18 commits into
mainfrom
feat/artifacts-structure-search
Open

feat(artifacts): search a document's structure pages#2319
pirhoo wants to merge 18 commits into
mainfrom
feat/artifacts-structure-search

Conversation

@pirhoo

@pirhoo pirhoo commented Aug 12, 2026

Copy link
Copy Markdown
Member

Adds an in-document search over a document's persisted structure pages, counting a query's occurrences per page the way /documents/searchContent counts them over the indexed content.

  • feat: add GET /:project/artifacts/structure/search/:id, answering {count, pages, scanned, hits}
  • feat: port the searchOccurrences.painless.java folding rules to Java, with a test pinning the script so the two counters cannot drift apart
  • fix: render structure pages with the OCR the document was indexed with, and record it in the artifact fingerprint
  • fix: bound the page total a manifest can advertise, so a total written by another producer cannot become an unbounded loop
  • fix: count an unreadable page as unscanned rather than failing the whole search, and log it once per scan instead of once per page
  • chore: the OCR fingerprint gains ocr and ocrStrategy, so every existing structure artifact is stale and the next ARTIFACT run re-extracts

How it works, and what it does not do

The scan reads one page-N.md at a time and folds it with the rules ported from searchOccurrences.painless.java, so the endpoint and /documents/searchContent fold text identically. Occurrences are counted per page with indexOf stepping by the raw query length, and summed. Work is linear in pages, capped by a 10s budget: scanned below pages means the count is a floor, not a total.

Known limits, all deliberate:

  • Counts the stored Markdown, not the indexed content, so flexmark escaping, table pipes and link URLs are part of the searched text. XHTML is rejected for this reason.
  • A match straddling a page break counts zero here and one in Elasticsearch, which scans the concatenated content field.
  • Original language only: structure pages are rendered from the source bytes, so there is no targetLanguage.
  • A scanned PDF renders empty unless the run passes --ocrStrategy, which is the same condition under which INDEX extracted nothing from it.

@pirhoo
pirhoo requested a review from a team August 12, 2026 09:21
@pirhoo pirhoo self-assigned this Aug 12, 2026
@pirhoo pirhoo added this to Sprint 49 Aug 12, 2026
@pirhoo pirhoo moved this to In Review in Sprint 49 Aug 12, 2026
@pirhoo
pirhoo force-pushed the feat/artifacts-structure-search branch from 2ef70a3 to e5160f1 Compare August 14, 2026 16:54

@caro3801 caro3801 left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Great feature and solid implementation easy to follow through
There are the two blocking correctness issues before merge:

  • the page-count cap currently affects all artifact routes,
  • and narrowed read-error handling may turn some 404 cases into 500s.

I also put a question about storing the OCR strategy used at index time to avoid future content/rendering mismatches. The rest is mostly cleanup, clarification, and small structure-search efficiency improvements.

Comment thread datashare-app/src/main/java/org/icij/datashare/web/ArtifactResource.java Outdated
/** Beyond this a total is a corrupt manifest, not a long document. It matters because {@link
* StructureSearch} turns a total written by another producer into a loop bound, and unbounded,
* {@code Integer.MAX_VALUE} never terminates at all: the counter wraps to MIN_VALUE, still in range. */
public static final int MAX_SERVABLE_PAGES = 100_000;

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

important: MAX_SERVABLE_PAGES is enforced here in servableTotal(), which every artifact route
goes through (manifest, single-page fetch, and structure search) - but the doc comment on the
constant says its only purpose is bounding StructureSearch's scan loop from running away on a
corrupt total.

Could we move the cap into StructureSearch.scan() instead (clamp its own loop
bound to min(total, MAX_SERVABLE_PAGES)) and let servableTotal() accept any positive total? As
written, a legitimately huge document (a merged archive, a bulk-exported log) with more than
100,000 pages now 404s on every artifact route, not just search.

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Moved in 414807e: servableTotal() now takes any positive total, and StructureSearch clamps its own loop to min(total, MAX_SCANNED_PAGES). pages still reports the manifest total, so past the cap scanned < pages marks the count as a floor.

public Hits search(String query) throws IOException {
ManifestEntry entry = reader.servableEntry(docArtifactDir, TYPE);
Integer total = entry == null ? null : reader.servableTotal(docArtifactDir, TYPE, entry);
// Without the formats probe, a document whose page-N.md files are gone would answer "no

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion(perf): search()'s formats() probe stats page 1 to confirm the format exists, and then
scan()'s first loop iteration reads that same page 1 file again a few lines later.
This is the exact stat-then-read round trip the sibling ArtifactReader.page() change in this PR was written to eliminate, just reintroduced once per search request.

Could we have the probe read page 1's bytes directly (instead of just checking readability) and hand them to scan() so the first iteration reuses them instead of re-reading?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fixed in aba5c50, one step further: the probe is gone. search() counts page 1 through ArtifactReader.page() and seeds the scan, which starts at page 2. One consequence: under byte-range pagination a malformed range 1 now 404s the search, a scheme no structure artifact uses.

pirhoo and others added 5 commits August 18, 2026 14:01
Co-authored-by: Caroline Desprat <cdesprat@icij.org>
Co-authored-by: Caroline Desprat <cdesprat@icij.org>
Co-authored-by: Caroline Desprat <cdesprat@icij.org>
Co-authored-by: Caroline Desprat <cdesprat@icij.org>
Co-authored-by: Caroline Desprat <cdesprat@icij.org>
@pirhoo
pirhoo requested a review from caro3801 August 18, 2026 14:39
Comment on lines +93 to +100
private PDFParserConfig.OCR_STRATEGY pdfOcrStrategy() {
try {
return PDFParserConfig.OCR_STRATEGY.valueOf(
propertiesProvider.get(OCR_STRATEGY_OPT).orElse("NO_OCR").toUpperCase(Locale.ROOT));
} catch (IllegalArgumentException unknown) {
return PDFParserConfig.OCR_STRATEGY.NO_OCR;
}
}

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

hint: do you want to reuse the same enum parsing/mapping as in datashare cli and map it into Tika's so the two don't drift independently if a member is ever renamed?

    private PDFParserConfig.OCR_STRATEGY pdfOcrStrategy() {
        try {
            OcrStrategy strategy = OcrStrategy.valueOf(
                    propertiesProvider.get(OCR_STRATEGY_OPT).orElse("NO_OCR").toUpperCase(Locale.ROOT));
            return PDFParserConfig.OCR_STRATEGY.valueOf(strategy.name());
        } catch (IllegalArgumentException unknown) {
            return PDFParserConfig.OCR_STRATEGY.NO_OCR;
        }
    }

// Half-open [start, end). A range outside the file means manifest and payload disagree, which
// is a 404 for that page rather than a truncated body.
private byte[] slice(Path content, List<long[]> ranges, int page, ArtifactType type, int total) throws IOException {
if (!Files.isReadable(content)) {

@caro3801 caro3801 Aug 18, 2026

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

suggestion: slice() does a Files.isReadable() presence check (stat) before opening the file, the exact stat-then-read pattern this same PR removed from the sibling filesystem-page branch a few lines up, with a comment calling it out as a doubled round trip on a shared/network artifactDir.
Could we drop the pre-check here too and read-then-catch instead, the way page()'s filesystem branch now does?

remark(TOCTOU): this pattern can be sometimes racy, especially on network/shared filesystems. It’s often better to just try the read/open directly and handle IOException, because the file could change between the check and the read anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: In Review

Development

Successfully merging this pull request may close these issues.

2 participants