Document fields disappear in Elasticsearch after upgrading MongoDB from 3.6 to 7.0 #739
Description
We are using Monstache v6.7.17 to sync data between MongoDB and Elasticsearch 8.3.2. Everything worked fine with MongoDB 3.6. However, after upgrading to MongoDB 7.0, we started experiencing issues: when updating some fields in a MongoDB document, the other fields seem to disappear from the document in Elasticsearch. The document itself still exists in Elasticsearch by _id, but I can't see indexed fields due to _source: disabled.
Here is my monstache configuration:
gzip = true
replay = false
resume = true
resume-name = 'models'
resume-write-unsafe = false
resume-from-earliest-timestamp = true
namespace-regex = "^cars\\.(models|\\$cmd)$"
delete-strategy = 1
dropped-databases = true
dropped-collections = true
verbose = false
enable-http-server = true
http-server-addr = ":8081"
stats = false
stats-duration = "60s"
fail-fast = false
enable-oplog = true
Could this be related to changes in the Oplog format or behavior in MongoDB 7.0?
- We tried updating a MongoDB document using standard update operations ($set) and expected Monstache to sync only the updated fields to Elasticsearch, preserving all other fields in the document.
- However, after the upgrade to MongoDB 7.0, Monstache seems to only send the updated fields to Elasticsearch, resulting in the removal of other fields in the document.
- Checked Monstache logs (verbose = true) and observed that the Oplog updates were being processed, but only partial updates (diff) were being sent to Elasticsearch.
- Verified the data in MongoDB, which was correct, but Elasticsearch documents were missing fields.