Project status, short-term road-map (Week 1, May 2021)

[brainless]

Hello everyone, @rgxai, @Utkarshbhimte and myself had a follow-up call after the call yesterday and chats here. Here is the summary:

1. We prefer staying serverless and focus on features since we do have time limitations
2. Twitter consumer is the project by Viksit and we add/update it (@apoorv should have access by now, please check GitHub emails)
3. We can add more consumers, each being serverless
4. We can create an aggregation layer when needed witch a K/V storage on serverless (CloudFlare provides this among others)
5. Focus on the frontend app making a better experience
6. Possibly help other orgs consume this APIs

@SwarnimWalavalkar please share your suggested infrastructure setup.

Let us break the work into GitHub issues and take it forward.

[SwarnimWalavalkar]

Proposed Infrastructure Setup

• NextJS Client deployed on Vercel
• Use Vercel Serverless Functions for the API, database operations, and other auxiliary scripts
• Upstash managed Redis database

[arn4v]

Since we'll be using viksit/covidconnect to query the Twitter API I think we should focus on moving the business logic of the backend to NextJS API routes.

We can explore setting up our own Redis server but I believe Upstash should be good enough considering their GraphQL API.

I have an initial setup going here that you can refer to: covidarmy/frontend@e36ccb6

[Utkarshbhimte]

The backend code, unlike say frontend code, can control connection pool

@brainless are you suggesting, that we have a FE code, this BE server, and Redis DB? So we are just discussing should we replace mongo with Redis or not?

What're your thoughts on the GraphQL API they provide? I think it can be very useful

@SwarnimWalavalkar Graphql would be helpful if we had a lot of different collections in our database. I would bring any difference in DX in near future.

[brainless]

@Utkarshbhimte no I did not mean that sorry. When I read the comment from @SwarnimWalavalkar about NextJS issues with Redis I simply pointed out we are in backend. There is no NextJS.

[arn4v]

@SwarnimWalavalkar Graphql would be helpful if we had a lot of different collections in our database. I would bring any difference in DX in near future.

Swarnim's referring to the GraphQL API Upstash exposes to execute Redis commands. This approach would entirely avoid the connection overloading issue we have experienced with Mongo.

[brainless]

@arn4v, @SwarnimWalavalkar I think we are over complicating. Please correct me if I am wrong, but the Mongo connection issues were from earlier frontend code, am I right? This is the backend project. Connection pools should be available.

[SwarnimWalavalkar]

Finalized Infrastructure setup

• Frontend NextJS app & Node server backend to be deployed on Google Cloud Run respectively.
• And use Google Memorystore with Redis as the managed database

Decision parameters

Must have's

• A fully managed, scalable, and highly available cloud infrastructure
• A productive and hassle-free developer experience
• Be compatible with future feature needs and use-cases

Related Issues

Migrate from MongoDB to Redis
Deploy the Node backend to Cloud Run
Setup Backend CI/CD with Cloud Run
Deploy NextJS app to Cloud Run
Setup Frontend CI/CD with Cloud Run

[rgxai]

Adding to the Must have's list which led to this decision (just for future context)

• Low latency (that's why we discarded Vercel + Upstash option due to Upstash Redis cache being in Tokyo)
• Easy Availability of developer credits so that we can reach out as a not for profit open source initiative (this seemed most easily available on Google and not so much on Heroku, Upstash or Cloudflare)