Search: Adds re-try mechanism while re-indexing

[Tschuppi81]

Search: Adds re-try mechanism while re-indexing

A failed re-indexing will surface as an error now

TYPE: Feature
LINK: pro-1568

[codecov]

Codecov Report

❌ Patch coverage is 85.71429% with 3 lines in your changes missing coverage. Please review.
✅ Project coverage is 85.50%. Comparing base (4711862) to head (6f0d4f7).
⚠️ Report is 2 commits behind head on master.
✅ All tests successful. No failed tests found.

Files with missing lines	Patch %	Lines
src/onegov/search/integration.py	85.71%	3 Missing ⚠️

Additional details and impacted files

Files with missing lines	Coverage Δ
src/onegov/search/indexer.py	87.09% <ø> (ø)
src/onegov/search/integration.py	93.20% <85.71%> (+0.02%)	⬆️

... and 2 files with indirect coverage changes

---

Continue to review full report in Codecov by Harness.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4711862...6f0d4f7. Read the comment docs.

🚀 New features to boost your workflow:

• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[Daverball]

I don't think the proposed fix will actually work, when a real serialization failure occurs, since it will put the database transaction into a different state, so the retries will not actually execute.

We could take an entirely different approach and make indexer serialization failures impossible by introducing a per-table lock that's shared between all processes, so the application indexers skip writing changes to that table, while the full reindex is in progress for that same table. Although it's probably possible for the skipped entries to become desynced that way, so it's probably better that those serialization failures happen. If we delay indexing instead of skipping it, we could probably cause the serialization failure to always happen in a regular request, instead of during indexing, but it may slow down some requests significantly while indexing is in progress and we potentially risk the lock getting stuck if reindexing crashes.

[Daverball]

I'm not quite sure what will happen to the raw connection here with a real database error, since we no longer use a session.begin_nested in the indexer.

I think it will probably be in an invalid state and will refuse to execute any further queries. So this fix wouldn't actually help right now. I'm not sure if doing a session.execute('ROLLBACK') here would be safe, since we're on a different thread we should be getting our own session, but I'm not sure if that means we're necessarily also getting our own raw connection.

Maybe manually doing tx = session.begin_nested() before processing the indexer tasks and then calling tx.rollback() in this branch would work, but I'm not sure.

Your mocked test case doesn't exercise the details here properly, since you're completely bypassing the indexer, when you emit an exception, so to the database it still looks like you only emitted the query once, and that it succeeded.

I'm not sure if there is a robust way to test this. You might be able to trigger a real serialization failure reliably by using two threads and some events for synchronization.

[Daverball]

Savepoints/Nested transactions will probably not work, since the start of the main transaction will be the same, so the database will still consider the retry a serialization failure. We probably need to hope here that SQLAlchemy properly isolates the connections in the pool and that a manual session.execute('ROLLBACK') is safe.

[Tschuppi81]

I like we will automatically get error if indexing issues occur e.g. after a rollout. A developer could easily miss that while manually observing reindexing..