fix: reuse a single ZipFile per database during indexing instead of one per LOB - #637
Open
luis100 wants to merge 1 commit into
Open
fix: reuse a single ZipFile per database during indexing instead of one per LOB#637luis100 wants to merge 1 commit into
luis100 wants to merge 1 commit into
Conversation
…ne per LOB detectMimeType() and getCLOBValue() opened a new ZipFile (re-reading the whole ZIP central directory) for every LOB cell, making indexing O(N^2) in row count. DbvtkExportModule now opens the SIARD ZipFile once in initDatabase(), threads it through getRow/getCells/getCell into detectMimeType/getCLOBValue for reuse, and closes it in finishDatabase(). Falls back to opening a ZipFile per LOB when no shared instance is available (e.g. SIARD-DK, which isn't a ZIP file). Fixes #636 Co-Authored-By: Claude Sonnet 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01EqF9bUgRmnwnuSFCFjic1p
dosubot
Bot
added
size:L
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
ToolkitStructure2ViewerStructure.detectMimeType()andgetCLOBValue()opened a brand newZipFile(re-reading the entire ZIP central directory) for every LOB cell in every row, making indexing O(N²) in row count for tables with LOB columns.DbvtkExportModulenow opens the SIARDZipFileonce ininitDatabase(), threads it throughgetRow→getCells→getCellintodetectMimeType/getCLOBValue, and closes it once infinishDatabase(). If it can't be opened as a ZIP (e.g. SIARD-DK, which isn't ZIP-based), it falls back to the previous per-accessZipFilebehavior so nothing regresses for those formats.detectMimeType's "blank file extension" fallback path (the previously-openedInputStreamwas never closed before being reassigned) and moved cleanup into afinallyblock so LOB streams/handles aren't left open when a sharedZipFileis now living across many calls.Why
Fixes #636 — reported by @seso-kdrs: indexing a SIARD file with 25M rows and one LOB column (with ~800K ZIP entries) ran at ~10 rows/sec because the ZIP central directory was parsed millions of times. Reusing one
ZipFileper database (instead of one per LOB) removes that redundant work while keeping MIME type auto-detection enabled.Test plan
mvn test-compilesucceeds