[TEXT-235] Add Damerau-Levenshtein distance

[LorgeN]

Adds Damerau-Levenshtein distance as a measure. Implementation based on original Levenshtein distance implementation, and Wikipedia. Unit tests written by comparing various strings manually, and by comparing to the value outputted by RapidFuzz.

Relevant Jira ticket; https://issues.apache.org/jira/browse/TEXT-235

[garydgregory]

Hello @LorgeN

Thank you for your PR.

It looks like this PR decreases the code coverage. You can run mvn clean package site and look at the JaCoCo report in target\site and go from there to increase the coverage.

You can also run mvn (solo) to run all build checks.

[garydgregory]

I don't know if we can accept this PR due to possible provenance incompatibility with "Claude 4.0 Sonnet" and the Apache License. If this were a clean room PR, there would be no issue of course.

FTR https://issues.apache.org/jira/browse/LEGAL-709

[LorgeN]

I don't know if we can accept this PR due to possible provenance incompatibility with "Claude 4.0 Sonnet" and the Apache License. If this were a clean room PR, there would be no issue of course.

FTR https://issues.apache.org/jira/browse/LEGAL-709

Understood! I suppose we will see what they say. The usage here is minimal, but appreciate that the lines are somewhat blurred.

If that is not acceptable, what would be the best path forward? Rewrite manually? Shouldn't really take that long, the implementation is straightforward enough.

[garydgregory]

Hi @LorgeN

Yes, I think the best path forward is a clean room implementation.

Beyond that, I think it would help to have test data from known sources documented, even if it's just from Wikipedia. Maybe there is an original paper that has examples?

Ty!

[LorgeN]

Hi @LorgeN

Yes, I think the best path forward is a clean room implementation.

Beyond that, I thibk it would help to have test data from known sources documented, even if it's just from Wikipedia. Maybe there is an original paper that has examples?

Ty!

Ack, makes sense. Will see what I can find!

[LorgeN]

Hey again @garydgregory! Thanks for all your help here.

I've rewritten the implementation without using any LLMs, and increased the test coverage. I was however not able to find any good examples of values anywhere. Wikipedia has nothing, and the papers all quote aggregate statistics (or use generated samples where they insert "mistakes" using a dictionary of words).

[garydgregory]

@LorgeN

Thank you for your updates. Looks reasonable to me aside from the builds failing 😉

You MUST run mvn by itself (no args) before pushing to catch all build failures

TY.

@ppkarwasz Any thoughts?

[LorgeN]

Hey again @garydgregory! Apologies for the delay here. Everything runs successfully locally now :)

[garydgregory]

LGTM.

@ppkarwasz Any thoughts?

[LorgeN]

Hey! Just checking in, @garydgregory. Anything you need from me to get this merged?

Appreciate the help here :)

[garydgregory]

Hello @LorgeN

I am looking for advice in https://issues.apache.org/jira/browse/LEGAL-709

[LorgeN]

Hey @garydgregory, just wanted to check in again to see if there is anything I can do to help speed this up?

[garydgregory]

Hello @LorgeN

Thank you for your patience.

As you can see this comment from a couple of weeks ago, I asked for another review but got zero feedback. I'll merge the PR very soon, when I sit down for my next FOSS session.

[garydgregory]

Hello @LorgeN

Merged 🚀

[Jiehong]

Too late to ask, but were there any consideration to have a SIMD implementation of the (Damerau-)Levenshtein distance?

(thinking of https://github.com/Turnerj/Quickenshtein/blob/main/src/Quickenshtein/Levenshtein.Intrinsics.cs for example)

[garydgregory]

Hello @Jiehong

That's a question for @LorgeN, but feel free to open a pull request.