Skip to content

package.md

Latest commit

 

History

History
135 lines (110 loc) · 7.55 KB

package.md

File metadata and controls

135 lines (110 loc) · 7.55 KB
Raw
Error in user YAML: Alias parsing is not enabled.
---
uid: Lucene.Net.Collation
summary: *content
---

Unicode collation support. Collation converts each token into its binary System.Globalization.SortKey using the provided System.Globalization.CompareInfo (the .NET platform collator), allowing it to be stored as an index term.

Use Cases

  • Efficient sorting of terms in languages that use non-Unicode character orderings. (Lucene Sort using a Locale can be very slow.)

  • Efficient range queries over fields that contain terms in languages that use non-Unicode character orderings. (Range queries using a Locale can be very slow.)

  • Effective Locale-specific normalization (case differences, diacritics, etc.). (xref:Lucene.Net.Analysis.Core.LowerCaseFilter and xref:Lucene.Net.Analysis.Miscellaneous.ASCIIFoldingFilter provide these services in a generic way that doesn't take into account locale-specific needs.)

Example Usages

Farsi Range Queries

CompareInfo collator = CompareInfo.GetCompareInfo("fa");
CollationKeyAnalyzer analyzer = new CollationKeyAnalyzer(LuceneVersion.LUCENE_48, collator);
Store.Directory ramDir = new RAMDirectory();
using (IndexWriter writer = new IndexWriter(ramDir, new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer)))
{
    Document doc = new Document();
    doc.Add(new TextField("content", "ساب", Field.Store.YES));
    writer.AddDocument(doc);
}
using IndexReader ir = DirectoryReader.Open(ramDir);
IndexSearcher searcher = new IndexSearcher(ir);

QueryParser aqp = new QueryParser(LuceneVersion.LUCENE_48, "content", analyzer);
aqp.AnalyzeRangeTerms = true;

// Unicode order would include U+0633 in [ U+062F - U+0698 ], but Farsi
// orders the U+0698 character before the U+0633 character, so the single
// indexed Term above should NOT be returned by a TermRangeQuery with a
// Farsi Collator.
ScoreDoc[] result = searcher.Search(aqp.Parse("[ د TO ژ ]"), null, 1000).ScoreDocs;
assertEquals("The index Term should not be included.", 0, result.Length);

Danish Sorting

Analyzer analyzer = new CollationKeyAnalyzer(LuceneVersion.LUCENE_48, CompareInfo.GetCompareInfo("da-DK"));
Store.Directory indexStore = new RAMDirectory();
using (IndexWriter writer = new IndexWriter(indexStore, new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer)))
{
    string[] tracer = new string[] { "A", "B", "C", "D", "E" };
    string[] data = new string[] { "HAT", "HUT", "HÅT", "HØT", "HOT" };
    for (int i = 0; i < data.Length; ++i)
    {
        Document doc = new Document();
        doc.Add(new StoredField("tracer", tracer[i]));
        doc.Add(new TextField("contents", data[i], Field.Store.NO));
        writer.AddDocument(doc);
    }
}
string[] sortedTracerOrder = new string[] { "A", "E", "B", "D", "C" };
using IndexReader ir = DirectoryReader.Open(indexStore);
IndexSearcher searcher = new IndexSearcher(ir);
Sort sort = new Sort();
sort.SetSort(new SortField("contents", SortFieldType.STRING));
Query query = new MatchAllDocsQuery();
ScoreDoc[] result = searcher.Search(query, null, 1000, sort).ScoreDocs;
for (int i = 0; i < result.Length; ++i)
{
    Document doc = searcher.Doc(result[i].Doc);
    assertEquals(sortedTracerOrder[i], doc.GetValues("tracer")[0]);
}

Turkish Case Normalization

// Primary collation strength is approximated with CompareOptions on the platform collator.
CompareInfo collator = CompareInfo.GetCompareInfo("tr-TR");
Analyzer analyzer = new CollationKeyAnalyzer(LuceneVersion.LUCENE_48, collator,
    CompareOptions.IgnoreCase | CompareOptions.IgnoreNonSpace);
Store.Directory ramDir = new RAMDirectory();
using (IndexWriter writer = new IndexWriter(ramDir, new IndexWriterConfig(LuceneVersion.LUCENE_48, analyzer)))
{
    Document doc = new Document();
    doc.Add(new TextField("contents", "DIGY", Field.Store.NO));
    writer.AddDocument(doc);
}
using IndexReader ir = DirectoryReader.Open(ramDir);
IndexSearcher searcher = new IndexSearcher(ir);
QueryParser parser = new QueryParser(LuceneVersion.LUCENE_48, "contents", analyzer);
Query query = parser.Parse("dıgy");   // U+0131: dotless i
ScoreDoc[] result = searcher.Search(query, null, 1000).ScoreDocs;
assertEquals("The index Term should be included.", 1, result.Length);

Caveats and Comparisons

WARNING: Make sure you use exactly the same collator (System.Globalization.CompareInfo and System.Globalization.CompareOptions) at index and query time -- System.Globalization.SortKeys are only comparable when produced by the same collator. Since the platform collator is not independently versioned, it is unsafe to search against stored System.Globalization.SortKeys unless the following are exactly the same (best practice is to store this information with the index and check that they remain the same at query time):

  1. The .NET runtime version, and the active globalization backend. .NET Framework uses Windows NLS, while .NET 5+ uses ICU by default; the two produce different sort keys and orderings. You can detect the active backend at runtime (see Globalization and ICU) and store that value with the index to verify it matches at query time.

  2. The language (and country and variant, if specified) of the culture used when obtaining the collator via System.Globalization.CompareInfo.GetCompareInfo.

  3. The System.Globalization.CompareOptions used - which approximate the collation strength (for example IgnoreCase | IgnoreNonSpace for primary strength) and Unicode normalization (decomposition).

Note

Unlike Lucene's java.text.RuleBasedCollator, the .NET System.Globalization.CompareInfo does not support tailored (custom) collation rules, so xref:Lucene.Net.Collation.CollationKeyFilterFactory does not accept a custom ruleset. Before generating a sort key, each term is normalized to Unicode Normalization Form C (NFC) so that decomposed and precomposed input collate consistently across globalization backends.

The ICUCollationKeyAnalyzer, available in the Lucene.Net.ICU package, uses ICU4N's Collator, which makes its version available, thus allowing collation to be versioned independently from the .NET runtime. ICUCollationKeyAnalyzer also generates significantly shorter keys than CollationKeyAnalyzer, and supports tailored (custom) rulesets. See http://site.icu-project.org/charts/collation-icu4j-sun for key generation timing and key length comparisons between ICU4J and java.text.Collator over several languages.

System.Globalization.SortKeys generated by the platform collator are not compatible with those generated by ICU Collators. Specifically, if you use CollationKeyAnalyzer to generate index terms, do not use ICUCollationKeyAnalyzer on the query side, or vice versa.