Can JPlag Support for Incremental Comparison with a Persistent Reference Repository (e.g., Pre-built Fingerprint Index stored on disk)?

[ElgebarOri]

Feature Description

Hi JPlag team,

First of all, thank you for developing and maintaining such a powerful and widely-used code similarity detection tool!

I'm currently using JPlag to detect code reuse or potential plagiarism in a large codebase (e.g., a course with many student submissions over multiple semesters). I have a feature request / question regarding incremental comparison.

Currently, JPlag requires all submissions (both historical and new) to be provided together in a single run for comparison. This works well for batch processing, but becomes inefficient when:

The historical codebase (e.g., past submissions) is very large.
New submissions arrive incrementally (e.g., weekly assignments).
We want to avoid re-parsing and re-processing the entire historical dataset every time.
🚀 Feature Request:
Is it possible to support a mode where:

A persistent reference repository or fingerprint index can be pre-built from a large codebase and stored on local disk.
For future checks, users can submit only new code files, and JPlag compares them against the pre-built index on disk without requiring the full original set.
This would significantly improve performance and usability in long-term or large-scale deployments.

Use Case

🔍 Alternatives / Workarounds:
I understand this may require substantial changes to the current architecture. As a workaround, I'm currently merging new submissions with the full historical set before running JPlag — but this becomes slow and resource-intensive over time.

Are there any plans or existing tools/plugins that support such a capability? Or would you consider this as a future enhancement?

Thank you for your time and consideration!

Best regards,
ElgebarOri

[robinmaisch]

Hi ElgebarOri,

Thanks for this detailed description! While we do not offer special handling for this use case so far, I can imagine how your archive grows over time and becomes a challenge to handle.
In fact, the efficiency problem caused by the necessity to re-run analyses for large sets of submissions might be remedied by features that we plan to implement in the long run. The same feature bundle should also help with issue #2541. This might still take some time to release, however. We will keep this in mind and hope you hold on in the meantime!

Best regards,
Robin
Co-Maintainer