Fix extraction in Solr >= 8.

[andresmrm]

Related to django-haystack/django-haystack#2000

Didn't run the test suite...

[cclauss]

@DhavalGojiya Your review, please.

[DhavalGojiya]

The difference between Solr ≤ 7 and Solr ≥ 8 is clear:

However, since the conditional logic was added in this PR at pysolr.py, the same logic also needs to be added in the test cases. CC: @andresmrm

Otherwise, the test case will start to break. We are running tests only against Solr 4.10.4, so it's not an issue right now, but I recommend updating this as well.

1- tests.test_client.SolrTestCase.test_extract

pysolr/tests/test_client.py

Line 1070 in 9b35283

m = extracted["metadata"]

The updated code should be:

m = extracted["metadata"]

if "file" in m["stream_name"]:
    # For Solr >= 8.
    self.assertEqual(["file"], m["stream_name"])
else:
    # For Solr <= 7.
    self.assertEqual([fake_f.name], m["stream_name"])

2- tests.test_client.SolrTestCase.test_extract_special_char_in_filename

pysolr/tests/test_client.py

Line 1113 in 9b35283

m = extracted["metadata"]

Updated code:

m = extracted["metadata"]

if "file" in m["stream_name"]:
    # For Solr >= 8.
    self.assertEqual(["file"], m["stream_name"])
else:
    # For Solr <= 7.
    self.assertEqual([quote(fake_f.name.encode("utf-8"))], m["stream_name"])

---

My Local test results

dhaval@lg-ubuntu:~/Open-Source/pysolr$ uv run python -m unittest -v tests.test_client.SolrTestCase.test_extract
test_extract (tests.test_client.SolrTestCase.test_extract) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.539s

OK
dhaval@lg-ubuntu:~/Open-Source/pysolr$ uv run python -m unittest -v tests.test_client.SolrTestCase.test_extract_special_char_in_filename
test_extract_special_char_in_filename (tests.test_client.SolrTestCase.test_extract_special_char_in_filename) ... ok

----------------------------------------------------------------------
Ran 1 test in 0.499s

OK

[DhavalGojiya]

This script for starting Solr servers at ./start-solr-test-server.sh does not work with newer versions of Solr (it is currently hard-coded with SOLR_VERSION=4.10.4).

If you temporarily need to test and verify changes related to this PR, I have created a complete Docker based solution.

Here is my docker-compose.yaml:

services:
  solr:
    image: solr:8.9.0  # Update this tag to switch between Solr <= 7 and Solr >= 8
    container_name: solr-standalone
    ports:
      - "8983:8983"
    volumes:
      - solrdata:/var/solr
    environment:
      - SOLR_CORE0=core0
      - SOLR_CORE1=core1
      - CONFIGSET=/opt/solr/server/solr/configsets/sample_techproducts_configs
    command: >
      bash -c "
        precreate-core $$SOLR_CORE0 $$CONFIGSET &&
        precreate-core $$SOLR_CORE1 $$CONFIGSET &&
        exec solr-foreground
      "
volumes:
  solrdata:

sudo docker compose up

and then

• uv run python -m unittest -v tests.test_client.SolrTestCase.test_extract
• uv run python -m unittest -v tests.test_client.SolrTestCase.test_extract_special_char_in_filename

[DhavalGojiya]

@cclauss | @acdha

I have quite good experience with working Apache Solr at Docker, Docker compose and server level.

Do we want to consider replacing the current ./start-solr-test-server.sh script with a Docker compose based setup in the future? Currently it's very hard to test newer solr versions with this script

Also, the current script is hard-coded to Solr version 4.10.4, and it breaks when we try to use newer versions of solr.

[cclauss]

Yes, please to Solr in Docker in GitHub Actions.

Given Solr versions, I doubt that Solr < v7 is worth worrying about. Perhaps even Solr v7 is not worth worrying about.

• https://solr.apache.org/downloads.html

[DhavalGojiya]

• https://solr.apache.org/downloads.html

Solr 8.x is already EOL (End Of Life) as we prepare for releasing version 10

I think we can directly jump to Solr 9.x.x versions.
I also checked some of the test cases in this package, and a few of them never pass with newer Solr versions. This is because the tests use assertions like assertIn on the response, and the responses have changed slightly in newer versions. We need to update some of these test cases, and add new ones if necessary.

[acdha]

This looks reasonable but I'm also wondering whether we should simply let this be a breaking change in a non-patch release since the Solr project themselves are phasing out Solr 8 support as its end of life approaches:

https://solr.apache.org/downloads.html

[DhavalGojiya]

This looks reasonable but I'm also wondering whether we should simply let this be a breaking change in a non-patch release since the Solr project themselves are phasing out Solr 8 support as its end of life approaches:

https://solr.apache.org/downloads.html

Does pysolr follow semver?
Isn't according to semver the patch release and minor release bump should be backwards compatible. But Nowdays people follow mixed practices.

[acdha]

This looks reasonable but I'm also wondering whether we should simply let this be a breaking change in a non-patch release since the Solr project themselves are phasing out Solr 8 support as its end of life approaches:
https://solr.apache.org/downloads.html

Does pysolr follow semver? Isn't according to semver the patch release and minor release bump should be backwards compatible. But Nowdays people follow mixed practices.

I think this comes down to what you think of as a breaking change: if this required you to update your Python code, it'd definitely become 4.0.0 but I'm not sure dropping support for version of Solr which hasn't been supported in years is the same.

[DhavalGojiya]

I think this comes down to what you think of as a breaking change: if this required you to update your Python code, it'd definitely become 4.0.0 but I'm not sure dropping support for version of Solr which hasn't been supported in years is the same.

Yeah, we are going to change/update/add only the test case and assertions with new Solr versions after dropping old Solr version support. The actual Pysolr package source code remains unchanged, so a new minor bump to pysolr==3.12.0 will still work without any breaking changes.