SOLR-13681: make Lucene's index sorting directly configurable in Solr #313

cpoerschke · 2021-09-28T13:15:23Z

https://issues.apache.org/jira/browse/SOLR-13681

see also https://issues.apache.org/jira/browse/SOLR-17310 scribbles w.r.t. potentially combining "within segment sorting" (e.g. this PR) and "between segments sorting" (e.g. @weiwang19's #2477 PR) in one configuration element

Resolved Conflicts: solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc

janhoy

Great.

Add a CHANGES entry. I may try to re-format the RefGuide edits from the patch later...

solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java

janhoy · 2021-09-28T18:38:47Z

solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

@@ -146,6 +146,7 @@ public void testSortingMPSolrIndexConfigCreation() throws Exception {
    final Sort expected = new Sort(new SortField(expectedFieldName, expectedFieldType, expectedFieldSortDescending));
    final Sort actual = sortingMergePolicy.getSort();
    assertEquals("SortingMergePolicy.getSort", expected, actual);
+    assertEquals("indexSort", expected, iwc.getIndexSort());


Let's create a new test method for the new config option, which will survive removal of SMP. Instead of creating yet another solrconfig-xyz.xml I think it should be possible to let the test invoke config-api to set the new setting.

Yes, I would also favour not creating yet another solrconfig-xyz.xml file. Have used the config-api in tests before but from my research done so far it seems indexConfig is not yet supported by it, or rather 'get' works but 'set' does not?

Also interestingly V2 'get' works but V1 does not, for indexConfig but not for (say) requestHandler

both work

curl "http://localhost:8983/solr/gettingstarted/config/requestHandler" curl "http://localhost:8983/api/collections/gettingstarted/config/requestHandler"

latter works, former does not

curl "http://localhost:8983/solr/gettingstarted/config/indexConfig" curl "http://localhost:8983/api/collections/gettingstarted/config/indexConfig"

https://solr.apache.org/guide/8_10/config-api.html#retrieving-the-config lists indexConfig as available top-level section.

$ curl "http://localhost:8983/solr/gettingstarted/config/indexConfig" <html> <head> <meta http-equiv="Content-Type" content="text/html;charset=utf-8"/> <title>Error 404 Not Found</title> </head> <body><h2>HTTP ERROR 404 Not Found</h2> <table> <tr><th>URI:</th><td>/solr/gettingstarted/config/indexConfig</td></tr> <tr><th>STATUS:</th><td>404</td></tr> <tr><th>MESSAGE:</th><td>Not Found</td></tr> <tr><th>SERVLET:</th><td>default</td></tr> </table> </body> </html>

Ok, perhaps they are R/O because it is not supported to change these after collection creation?
In that case, a new xml file is probably better anyway.

Are you planning to validate that index sorting is working in the test, or simply that the config is set? I think perhaps a new method in the existing IndexConfig test would be enough?

Yes, and perhaps changing some indexConfig parameters after collection creation would be fine but not so for others. Also the merge policy (factory) configuration is nested, not sure if config-api would easily support that.

Added a new method, modelled on the existing sorting-merge-policy one that will go away with sorting merge policy. There is existing TestSegmentSorting coverage w.r.t. index sorting working in a test, so it might make sense to tap into that somehow?

This reverts commit 8171b27.

…ter-part to testSortingMPSolrIndexConfigCreation

…ig-indexSort.xml dual test coverage (latter still failing)

This reverts commit 7514d34.

cpoerschke · 2021-09-29T12:39:19Z

solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java

    if (cmd.getSegmentTerminateEarly()) {
      final Sort cmdSort = cmd.getSort();
      final int cmdLen = cmd.getLen();
-      final Sort mergeSort = core.getSolrCoreState().getMergePolicySort();
+      final Sort mergeSort = core.getSolrCoreState().getMergePolicySort(); // TODO: need this account for indexSort also?


https://issues.apache.org/jira/browse/SOLR-15390 is about deprecating the segmentTerminateEarly flag.

solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java

…igured

cpoerschke · 2021-09-29T12:48:21Z

Great.

Add a CHANGES entry. I may try to re-format the RefGuide edits from the patch later...

Thanks for attaching SOLR-13681-refguide-skel.patch to the JIRA ticket!

If you wish, Allow edits and access to secrets by maintainers is enabled for this pull request, feel free to add documentation or code or test change commits directly to the pull request branch here.

…estSegmentSorting pass

janhoy · 2021-10-22T07:25:55Z

Sorry for going silent on this, I'll put it on my radar for next week.

Resolved Conflicts: solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java

…er origin/main merge

Resolved Conflicts: solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

atris · 2024-05-09T17:09:50Z

@cpoerschke I am about to pick this up and start a fresh PR since this one has gotten stale. Is it ok if I proceed? I will ensure that the PR is marked for your review and you are credited as co author.

Resolved Conflicts: solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

cpoerschke · 2024-05-10T08:47:56Z

@cpoerschke I am about to pick this up and start a fresh PR since this one has gotten stale. Is it ok if I proceed? I will ensure that the PR is marked for your review and you are credited as co author.

@atris - wonderful, yes please proceed. Also I've just resolved the (trivial) merge conflict on this PR and "allow edits by maintainers" is enabled for the PR i.e. as an alternative to a fresh PR you'd also be welcome to add commits to the PR's branch if you like.

atris · 2024-06-13T14:34:16Z

Still working on this - have a PR in process. Will publish by weekend and tag you for review

cpoerschke added 3 commits September 28, 2021 14:03

git apply SOLR-13681.patch

b37f45c

Merge remote-tracking branch 'origin/main' into SOLR-13681

564a66d

Resolved Conflicts: solr/solr-ref-guide/src/indexconfig-in-solrconfig.adoc

make it compile

bc39779

janhoy reviewed Sep 28, 2021

View reviewed changes

cpoerschke added 7 commits September 29, 2021 09:08

code review feedback: logging tweak

e8ecd26

add (skeleton) SolrIndexConfigIndexSortTest class

7514d34

add indexConfig fragments to TestSolrConfigHandler

8171b27

Revert "add indexConfig fragments to TestSolrConfigHandler"

33a722e

This reverts commit 8171b27.

SolrIndexConfigTest: add testIndexSortSolrIndexConfigCreation as coun…

2247d79

…ter-part to testSortingMPSolrIndexConfigCreation

TestSegmentSorting: solrconfig-sortingmergepolicyfactory.xml/solrconf…

1478be7

…ig-indexSort.xml dual test coverage (latter still failing)

Revert "add (skeleton) SolrIndexConfigIndexSortTest class"

14b8c49

This reverts commit 7514d34.

cpoerschke commented Sep 29, 2021

View reviewed changes

solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java Outdated Show resolved Hide resolved

SolrIndexConfigTest: assert that indexSort is not present if not conf…

7feca02

…igured

cpoerschke added 2 commits September 30, 2021 12:50

address TODO in SolrIndexSearcher.buildAndRunCollectorChain to make T…

736a75b

…estSegmentSorting pass

Merge remote-tracking branch 'origin/main' into SOLR-13681

3920768

cpoerschke added 4 commits December 22, 2021 13:29

Merge remote-tracking branch 'origin/main' into SOLR-13681

97b1206

Resolved Conflicts: solr/core/src/java/org/apache/solr/update/SolrIndexConfig.java

adjust SolrIndexConfigTest.testIndexSortSolrIndexConfigCreation() aft…

f0889c0

…er origin/main merge

Merge remote-tracking branch 'origin/main' into SOLR-13681

0fd2c9f

Resolved Conflicts: solr/core/src/java/org/apache/solr/search/SolrIndexSearcher.java solr/core/src/test/org/apache/solr/cloud/TestSegmentSorting.java solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

./gradlew spotlessApply

eae8527

janhoy requested a review from atris May 7, 2024 12:14

Merge remote-tracking branch 'origin/main' into SOLR-13681

94190d5

Resolved Conflicts: solr/core/src/test/org/apache/solr/update/SolrIndexConfigTest.java

github-actions bot added tests cat:search cat:cloud cat:index labels May 10, 2024

Merge remote-tracking branch 'origin/main' into SOLR-13681

eee971c

epugh mentioned this pull request Feb 2, 2025

SOLR-17310: Configurable LeafSorter to customize segment search order #2477

Closed

7 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

SOLR-13681: make Lucene's index sorting directly configurable in Solr #313

SOLR-13681: make Lucene's index sorting directly configurable in Solr #313

cpoerschke commented Sep 28, 2021 •

edited

Loading

janhoy left a comment

janhoy Sep 28, 2021

cpoerschke Sep 29, 2021

cpoerschke Sep 29, 2021

janhoy Sep 29, 2021

cpoerschke Sep 29, 2021

cpoerschke Sep 29, 2021

cpoerschke commented Sep 29, 2021

janhoy commented Oct 22, 2021

atris commented May 9, 2024

cpoerschke commented May 10, 2024

atris commented Jun 13, 2024

SOLR-13681: make Lucene's index sorting directly configurable in Solr #313

Are you sure you want to change the base?

SOLR-13681: make Lucene's index sorting directly configurable in Solr #313

Conversation

cpoerschke commented Sep 28, 2021 • edited Loading

janhoy left a comment

Choose a reason for hiding this comment

janhoy Sep 28, 2021

Choose a reason for hiding this comment

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

janhoy Sep 29, 2021

Choose a reason for hiding this comment

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

cpoerschke Sep 29, 2021

Choose a reason for hiding this comment

cpoerschke commented Sep 29, 2021

janhoy commented Oct 22, 2021

atris commented May 9, 2024

cpoerschke commented May 10, 2024

atris commented Jun 13, 2024

cpoerschke commented Sep 28, 2021 •

edited

Loading