Add solr

[cwaddingham]

Problem

We need to add first‐class Solr support to VSB while ensuring robustness, configurability, and code quality.

• Solr queries were returning zero recall due to metadata flattening and type mismatches.
• Long‐running test runs (100k queries) needed a way to be limited for quick iteration.
• Solr core creation/deletion and index schema setup were brittle, leading to 503 errors and stale cores.
• Performance of Solr k-NN queries was unacceptably slow out-of-the-box.
• The codebase had accumulated linter violations and inconsistent formatting.

Solution

• Integrate Solr as a supported backend
  • Implemented SolrClient & SolrNamespace in vsb/databases/solr/solr.py with robust retry, backoff, schema management, core lifecycle, and metadata flattening that promotes fields to top-level.
  • Added --query-limit CLI option (vsb/cmdline_args.py) and plumbed it through workload builders (locustfile.py, ParquetWorkload) to allow small test runs.
  • Cast both actual & expected IDs to strings in the search client and metrics (vsb/metrics.py) to guarantee correct recall.
• Resilient index creation & cleanup
  • Enhanced core unload/create logic with deleteIndex, deleteDataDir, deleteInstanceDir, proper polling, and Docker volume adjustments to avoid stale data.
  • Wrapped all errors in StopUser for clean Locust shutdown.
• Performance tuning
  • Exposed Solr JVM heap (SOLR_JAVA_MEM), G1GC flags, ulimits, CPU/memory quotas in docker/solr/docker-compose.yml.
  • Mounted custom solrconfig.xml & managed-schema.xml with tuned filterCache, queryResultCache, docValuesCache, thread pools, and HNSW parameters (ef, beamWidth) via bind mounts.
• Code quality & infrastructure
  • Ran Black (v24.4.2) across the repo, fixed all formatting violations.
  • Updated tests in tests/integration for Solr mirroring the existing common test framework.
  • Added documentation to README and vsb/databases/solr/README.md on configset seeding and Docker usage.

Type of Change

• New feature (adds Solr integration and related improvements)
• Bug fix (metadata flattening, ID type casting, core teardown)
• Infrastructure change (Docker Compose, solrconfig.xml, cache tuning)
• Documentation update (README additions, configset instructions)

Test Plan

1. Unit & integration tests
   • pytest tests/unit and pytest tests/integration --db solr pass with query-limit and recall > 0.
2. Manual Solr validation
   • Use curl to confirm k-NN queries with ef, fq parameters return expected docs.
3. Locust workload
   • Run locust -f vsb/locustfile.py --query-limit 100 --database solr and verify correct op/s and recall metrics.
4. Performance check
   • Benchmark before/after cache and JVM tuning; ensure p95 latency drops under target (e.g. <500 ms).
5. Linter & formatting
   • black . --check returns no changes.
6. Docker restart
   • Confirm Solr container picks up custom solrconfig.xml without deleting /var/solr/data.

[gitguardian]

⚠️ GitGuardian has uncovered 1 secret following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secret in your pull request

GitGuardian id	GitGuardian status	Secret	Commit	Filename
20844949	Triggered	Generic Password	3c4f073	tests/integration/test_solr.py	View secret

🛠 Guidelines to remediate hardcoded secrets

1. Understand the implications of revoking this secret by investigating where it is used in your code.
2. Replace and store your secret safely. Learn here the best practices.
3. Revoke and rotate this secret.
4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider

• following these best practices for managing and storing secrets including API keys and other credentials
• install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.

---

^{🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.}

[vkrishna1084]

reviewed and merging the changes