Is there an existing issue for this?
Is your feature request related to a problem? Please describe the problem.
Our current Lucene.Net.Analysis.OpenNLP library uses IKVM and a MavenReference to bring in OpenNLP. This works, but has a few drawbacks:
- We are currently blocked from upgrading IKVM to the latest version because it has ballooned in size to several GB in newer releases, which makes it take up too much disk space on Azure DevOps hosted agents.
- We cannot support
netstandard2.0 or net462 targets; we only target net472, net8.0, and net10.0. While the lack of net462 is something I very much can live without, it might be nice to be able to support netstandard2.0 while we still support it in other libraries. At a minimum, this requires a hacky TargetFrameworks workaround that diverges from our other projects.
- We cannot upgrade to the newest versions of OpenNLP, as OpenNLP 2.x requires JDK 11 or later, and IKVM maxes out at JDK 8. OpenNLP is also currently working towards a 3.0 release with further breaking changes.
- Because we expose some OpenNLP APIs, this requires exposing the IKVM foundations our MavenReference is built upon. It would be nice if this were just an implementation detail instead, but we can't avoid that with our current approach.
- Performance is quite worse than a native .NET approach.
Describe the solution you'd like
I had started work on porting just the parts of OpenNLP that we needed a while back, and forgot about that until recently. I just had Claude Code finish up my port to get it building and passing tests. It hasn't been fully reviewed or vetted yet, but initial results are promising enough to propose it as a real solution:
- All existing tests pass.
- The IKVM dependency has been removed, which removed several GB from the restore (and resolves the issue with the Azure DevOps agent disk space, and upgrade blocker)
- We can now target
netstandard2.0 and net462, and my prototype already adds these targets for full target parity with Lucene.NET.
dotnet build/restore times are now greatly improved; the OpenNLP library cold restore time alone went from 34.12s to 1.67s on my machine, and disk usage reduced from 3,226 MB to 529 MB. Clean build time reduced from 16.6s to 8.7s. Note that the restore time will also impact any users of our library, since they'll currently have to pull in IKVM and won't with this change.- Performance gains are huge (see numbers below) - typically 6+ times faster for common use
- Allocation reductions are also huge (see numbers below) - typically over 50% reduction
- We will have the flexibility to upgrade OpenNLP to take advantage of new features and keep up with Lucene post-4.8 (note that Lucene now uses OpenNLP 2.5.11 and they work to keep this up to date).
- Minor benefit, but we can use proper .NET idioms and naming, instead of inheriting the Java-ness of OpenNLP's API.
Additional context
Benchmarks (all times in µs):
| Benchmark |
Current (IKVM) |
Prototype |
Speedup |
Alloc cut |
| LoadSentenceModel |
1,115.9 |
41.4 |
27.0x |
97% |
| LoadPOSModel |
2,185.1 |
273.8 |
8.0x |
79% |
| SplitSentences |
19.7 |
1.9 |
10.3x |
0% |
| Tokenize |
36.4 |
4.8 |
7.6x |
52% |
| POSTag |
108.6 |
14.6 |
7.5x |
42% |
| Chunk |
576.0 |
84.1 |
6.9x |
58% |
| FullPipeline |
3,643.3 |
576.7 |
6.3x |
57% |
FullPipeline represents what an analyzer actually does per document: split sentences, tokenize each, POS-tag, and chunk, so it is the more real-world benchmark.
Note that those benchmarks were based on a first pass at porting the code and have not been analyzed or performance tuned at all. There are additional gains to be had here for both raw speed and allocations. A quick, initial pass by Claude Code surfaced several opportunities (BeamSearch inner loop, Context generators allocation per token, Cache on the hot path, GISModel.Eval allocation per call, etc.).
It would be an open question as to where this code would live, and how it would be maintained and either vendored or depended upon. That can be settled without affecting the merits of this issue; there is enough here to warrant consideration of this regardless. No matter which way we go, upstream licensing should be preserved to avoid possible issues.
In my opinion, we should consider this for beta 19 if possible, due to the public API breaking change it would involve.
Alternatives Explored
sergey-tihon/OpenNLP.NET uses IKVM, so that doesn't buy us anything. It is also currently blocked from upgrading to 2.x.
AlexPoint/OpenNLP has not seen a commit in 5 years, and does not target modern .NET.
Fixing issues with IKVM size/performance and upgrading it to support JDK 11 - likely too large of an effort.
Prototype Code
My* prototype code is here: https://github.com/paulirwin/lucene.net/commits/prototype/opennlp/
(* latest commit developed with Claude Code, Opus 5. Initial WIP commit was partially done with Claude Code, but don't remember the model at the time. That first commit was also partially done by hand, and also with my JavaToCSharp tool.)
Is there an existing issue for this?
Is your feature request related to a problem? Please describe the problem.
Our current Lucene.Net.Analysis.OpenNLP library uses IKVM and a MavenReference to bring in OpenNLP. This works, but has a few drawbacks:
netstandard2.0ornet462targets; we only targetnet472,net8.0, andnet10.0. While the lack ofnet462is something I very much can live without, it might be nice to be able to supportnetstandard2.0while we still support it in other libraries. At a minimum, this requires a hacky TargetFrameworks workaround that diverges from our other projects.Describe the solution you'd like
I had started work on porting just the parts of OpenNLP that we needed a while back, and forgot about that until recently. I just had Claude Code finish up my port to get it building and passing tests. It hasn't been fully reviewed or vetted yet, but initial results are promising enough to propose it as a real solution:
netstandard2.0andnet462, and my prototype already adds these targets for full target parity with Lucene.NET.dotnetbuild/restore times are now greatly improved; the OpenNLP library cold restore time alone went from 34.12s to 1.67s on my machine, and disk usage reduced from 3,226 MB to 529 MB. Clean build time reduced from 16.6s to 8.7s. Note that the restore time will also impact any users of our library, since they'll currently have to pull in IKVM and won't with this change.Additional context
Benchmarks (all times in µs):
FullPipeline represents what an analyzer actually does per document: split sentences, tokenize each, POS-tag, and chunk, so it is the more real-world benchmark.
Note that those benchmarks were based on a first pass at porting the code and have not been analyzed or performance tuned at all. There are additional gains to be had here for both raw speed and allocations. A quick, initial pass by Claude Code surfaced several opportunities (BeamSearch inner loop, Context generators allocation per token, Cache on the hot path, GISModel.Eval allocation per call, etc.).
It would be an open question as to where this code would live, and how it would be maintained and either vendored or depended upon. That can be settled without affecting the merits of this issue; there is enough here to warrant consideration of this regardless. No matter which way we go, upstream licensing should be preserved to avoid possible issues.
In my opinion, we should consider this for beta 19 if possible, due to the public API breaking change it would involve.
Alternatives Explored
sergey-tihon/OpenNLP.NET uses IKVM, so that doesn't buy us anything. It is also currently blocked from upgrading to 2.x.
AlexPoint/OpenNLP has not seen a commit in 5 years, and does not target modern .NET.
Fixing issues with IKVM size/performance and upgrading it to support JDK 11 - likely too large of an effort.
Prototype Code
My* prototype code is here: https://github.com/paulirwin/lucene.net/commits/prototype/opennlp/
(* latest commit developed with Claude Code, Opus 5. Initial WIP commit was partially done with Claude Code, but don't remember the model at the time. That first commit was also partially done by hand, and also with my JavaToCSharp tool.)