Explore migrating from pymongo to motor as an async python libary for mongodb

[mkshah605]

Given that there is now "official" guidance from MongoDB Inc. on using their official async python library for mongodb, i.e. motor, with FastAPI, explore pros/cons of migrating from using the older, more-stable-but-not-async pymongo library in order to take full performance advantage of FastAPI's async support.

The implication of this would be that a single-container deployment of the runtime would:

• support a higher rate of requests (avoid having to run multiple containers)
• in turn, potentially help avoid having to consider race conditions wrt multiple runtime containers accessing the same backing mongodb instance

Goal: take some time to compare both approaches to better inform a path forward

[dwinston]

Given that there is now "official" guidance from MongoDB Inc. on using their official async python library for mongodb, i.e. motor, with FastAPI

i.e. https://github.com/mongodb-developer/farm-stack-to-do-app

[eecavanna]

Hello, @mkshah605 — welcome to the project!

@mkshah605 and @dwinston — While working on migrating nmdc-runtime from pip-tools to Poetry (still very much a work in progress, here), I came across both motor and pymongo in the dependency tree. Based on the digging I did at the time, I (to my surprise) came away thinking that the Runtime was actually already using motor and it was only some miscellaneous experiments that were using pymongo.

nmdc-runtime/nmdc_runtime/api/db/mongo.py

Line 15 in d7da63f

from motor.motor_asyncio import AsyncIOMotorClient, AsyncIOMotorDatabase

So, today, I was surprised when I noticed this ticket in the repo.

[eecavanna]

Oops... correction: it is using both (and I think it's using pymongo for most things—as opposed to the opposite, which I suggested in my previous message).

nmdc-runtime/nmdc_runtime/api/db/mongo.py

Lines 14 to 15 in d7da63f

	from pymongo.errors import OperationFailure, AutoReconnect
	from motor.motor_asyncio import AsyncIOMotorClient, AsyncIOMotorDatabase

nmdc-runtime/nmdc_runtime/api/db/mongo.py

Lines 27 to 28 in d7da63f

	from pymongo import MongoClient, ReplaceOne
	from pymongo.database import Database as MongoDatabase

Maybe someone was experimenting with motor and left their experimentation in the code base.

[dwinston]

@eecavanna yes, @elais was leading the experimentation with motor before he was eaten by JAWS... 🦈

[mkshah605]

Thank you @eecavanna !

Yes, after looking into this, it seems like motor is the way to go. It is functionally equivalent to pymongo, and is designed for asynchronous frameworks like asyncio. Especially since we already have some code that uses motor, it might be worth doing a full refactor for consistency.

Ticket for this linked here.