[Docs] Project Requirements

[yusufaine]

THIS ISSUE SERVES AS A DRAFT FOR PROJECT REPORT.

Decisions

Tech stack

• Backend
  • NestJS
  • NestJS/passport
  • nodemailer
  • redis
  • socket.io
  • yjs
  • zod
  • Prisma + PlanetScale
• Frontend
  • Vite
  • Tailwind
  • React-query
  • zustand
  • y-socket.io
  • simple-peer
  • monaco-editor
• Data
  • NodeJS
  • Prisma + PlanetScale

---

Deployment

For the development and demonstration of the application, we would focus on running the service locally. Once all the features are up and running nearer and as we approach the deadline, we would ideally dockerise some of the microservices and deploy it as follows:

Frontend

	Vercel	Google App Engine
Price	Free for personal repositories.	Free for the first 28 instance hours of "F" instances
Availability	Available 24/7, also provides readable URLs when deployed	Available 24/7, default URLs are not very readable
Scaling	Unable to find, but should not be an issue	Handled by Google for free, within the deployed instance class (F1 - F4), different increasing memory and CPU limits.
CI/CD Difficulty	Vercel has their own CICD pipeline	To find out more on, but currently thinking of manually integration and deployment

Leaning towards Vercel if we can integrate it with our backend.

---

Backend

	Google Cloud Run	Google App Engine
Price	Free 180000 vCPU seconds per month	Free for the first 28 instance hours of "F" instances
Availability	Pay-per-use, first request would is slow as it needs to "spin up"	Available 24/7, does not require "spinning up"
Scaling	Handled by Google for free	Handled by Google for free, within the deployed instance class (F1 - F4), different increasing memory and CPU limits.
CI/CD Difficulty	Relatively simple, can be done with Github Actions (example)	To find out more on, but currently thinking of manually integration and deployment

• Why GCP?
  1. We have some credits that we can spend
  2. Closer to how industry deploys their services
  3. We intend on letting the service run a little while longer after the module has ended (Heroku free tier no longer available in November)

Leaning towards Google App Engine.

---

User service

Planetscale -- Hosted SQL database.

1. Reasonable free tier
2. Chosen over NoSQL service as the data in our service can be relational and we would like the ability to work with that.

---

Matching service

Utilise Redis' namespace as the different types of "queues" (PubSub topics).

---

Question service

Planetscale to allow for relational queries/mappings, Redis for quick queries.

---

Design

We will prioritise functionality first before user experience. Simply put, the team would work on getting a functional version up first before improving on the UI elements that would improve the user experience.

---

Diagrams

All of our diagrams are stored in this Gist.

Architecture

Simplified

Specifics

• Deployment is subjected to change but we are currently leaning towards the following:
  • Backend: Google App Engine
  • Frontend: Vercel

User flow

Specifics

1. At any page after Dashboard users can navigate access the pages that are immediately a child to Dashboard as there would be a navigation bar.
   • Locked room page with past session can access Matching page directly without needing to go back to Dashboard.
2. At any page, a user may choose to log out and would be directed back to the start of the user flow.

[yusufaine]

User Service

Authentication

Functional Requirements

1. Allow users to create an account with username and password.
2. Ensure that every account created has a unique username.
3. Allow users to sign in by entering their username and password.
4. Allow users to sign out of their account.
5. Allow users to delete their own account.
6. Allow users to change their password.
7. Allow users to reset their password.

Non-functional Requirements

1. Write operations of an account should only be able to be carried out by the user themselves, or the admin
2. Read operations of an account should only display the relevant information
   • Users should not be able to see each other's passwords
   • Users should be allowed to see other non-private fields related to the service (historical data/sessions) -- low priority, depends on deployment of History Service
3. Resetting of account should only be done to the appropriate user.
   • If Alice resets the password, only Alice should be allowed to set the new password.
4. User service should be able to support at least 100 users.
5. Users passwords should be hashed and salted before storing in the database.

Progress

• Unique account creation, salt and hash password
  • Backend
  • Frontend
• User login (username, password), prevent logging in of same account(?)
  • Backend
  • Frontend
• User password change
  • Backend
  • Frontend
• User reset password
  • Backend
  • Frontend

History (Stretch Goal)

Functional Requirements

• Users should be able to view their past attempts to see how they did approached the question previously and compare if they've made any improvements.

Non-functional Requirements

• System should be able to get all of the user's past historical data since feature inception.

Progress

• Backend
  • r-w to DB
  • API
• Frontend
  • History page

Diagrams

Authentication

History

[yusufaine]

Matching Service

Functional Requirements

1. Allow users to select the difficulty level of the questions they wish to attempt.
2. Match 2 waiting users with similar difficulty levels and put them in the same room.
3. System should pair matching users within 30 seconds. (Pair within 30 seconds)
4. Inform users that no match is available if a match cannot be found within 30 seconds.
5. Users should be able to explicitly leave a room once matched.

Non-functional Requirements

1. Matching service should allow at least 100 concurrent users to be in the queue

Progress

• User matches based on difficulty
  • Backend
  • Frontend
• Matching times out if no match found in 30 seconds
  • Backend
  • Frontend

Stretch Goals

• Potential to introduce more matching criteria
  • Topic
  • Question of the Day
• User to explicitly accept match (allows viewing of question summary)

Diagram(s)

Finding a match

No match found, joining the queue

Match found, joining room

User is already in a room

[yusufaine]

Question Service

The service is linked to backend and utilises the following:

1. PlanetScale -- Hosted MySQL database (also used by User Service)
2. Prisma -- ORM for PlanetScale
3. Firebase/GCP Cloud Functions
   • HTTP-trigger to manually seed,
   • Event-triggered (PubSub + Cloud Scheduler) to simulate cron job.

Functional Requirements

1. Matched users should be served the same question based on their matching criteria.
2. System should be able to get questions by difficulty and topic.
3. Users should be able to see the latest details of the question at a glance.

Non-functional Requirements

1. Questions, summary and details, should be served to users within 3 seconds.

Detailed Design

Addressing FR and NFR

1. Store the question details in a database -- resolve availabilty issues.
   • This is to prevent the service from being strongly coupled to (depend on) Leetcode's endpoint.
   • If Leetcode becomes unavailable -- goes down, blocks public access to data -- or changes the endpoint, the service would not be able to serve the users anything.
   • The only time the service would not be able to serve questions is if the database goes down.
     • Nice-to-have: Automate backups in the event the database crashes and loses all data. Handled by PlanetScale, free daily backup. 👍
2. Users will be served updated data as the database gets updated every hour and checks for

Potential Issues

1. Issue: Exceeding runtime of Cloud Function
   • Cloud Functions runtime is limited to 540 seconds (see here), and Prisma does not support createMany and establish relations, this has to be done one-by-one.
   • There is no known Leetcode GraphQL endpoint that returns the QuestionContent of all questions and it has to be done manually.
   • With ~2000 free Leetcode questions and querying each content takes ~300ms, querying all the questions would take ~600 seconds, prior to accounting for updating QuestionSummary, retries and writing to the database and would overrun the limitation of Cloud Function.

To resolve this:

• Separate updating the QuestionSummary and QuestionContent, and execute them sequentially in that order (content relies on summary).

2. Issue: Leetcode rate-limiting queries and potential network I/O errors
   • As mentioned in the point above, at worst, the service would need to query all question contents. This does happen when the database is fresh or if LeetCode adds a lot of new questions at once.

To resolve this:

• Process LC queries by chunks rather than sending all the requests at once (Promise.all())
  • Seeding the database: This does not fully resolve the issue but yields better results (more HTTP status 200)
  • Updating the database: This should not prove to be an issue as it is unlikely that LC adds >10 questions in a day.

The following were implemented but only (2) is deployed:

1. Creating a HTTP-triggered function and repeat it until there are no leftovers, or
2. Increase the frequency of the event-triggered function.

The service implements both to seed and update by the hour every hour to update the summary values, daily quetion and resolve and leftover missing content, if any.

Schemas

Updated as of: 25th Septemeber 2022

This is based of what LeetCode returns, for the actual schema that was applied, refer to prisma/schema.prisma which also covers the relations such as the one-to-many relation that was mentioned in the first point of the previous section.

QuestionSummary

Column Name	Data Type	Description
acRate	number	Solution acceptance rate (%)
difficulty	string	Difficulty level of the question
paidOnly	boolean	Is the question only available to paid users?
title	string	Question title
titleSlug	string	Slug associated to question
topicTags	string[]	Topics that are associated to the question
createdAt	timestamp	When the question was seeded
updatedAt	timestamp	When the question was last updated

QuestionContent

Column Name	Data Type	Description
content	string	Question context and details.
hints	string[]	Hints to solving the question, if any.
topicTags	string[]	Topics that are associated to the question
discussionLink	string	Link to the question's discussion page on Leetcode
createdAt	timestamp	When the question was seeded
updatedAt	timestamp	When the question was last updated

GraphQL Queries

Updated as of: 25th Septemeber 2022

All questions

query psetQuestionList {
  psetQuestionList: questionList(
    categorySlug: ""
    limit: -1
    skip: 0
    filters: {}
  ) {
    total: totalNum
    questions: data {
      acRate
      difficulty
      paidOnly: isPaidOnly
      title
      titleSlug
      topicTags {
        slug
      }
    }
  }
}

Question of the day

query questionOfToday {
  activeDailyCodingChallengeQuestion {
    question {
      acRate
      difficulty
      paidOnly: isPaidOnly
      title
      titleSlug
      topicTags {
        slug
      }
      hasSolution
      hasVideoSolution
    }
  }
}

Get question from slug

query getQuestionDetail($titleSlug: String!) {
  question(titleSlug: $titleSlug) {
    content
    hints
    topicTags {
      name
    }
  }
}

# var
# {
#  "titleSlug": "two-sum"
# }

Potential upgrades

• Allow users to browse questions -- "solo mode"
• Display summary of question before entering room

Progress

• Serverless function CRUD -- Recess Week
• Serverless function scheduling -- Recess Week
• Backend -- Recess Week
• Frontend -- Week 7

Diagram

[yusufaine]

COLLABORATION SERVICE MERGED UNDER ROOM SERVICE

Hidden

Collaboration Service

Functional Requirements

• Users would need to be able to live-edit a text document and changes should be synchronised across both parties.

Non-functional Requirements

• Latency should be < 1000 milliseconds.

Progress

• Working collaborative editor (Week 7)

[yusufaine]

User Interface

Design Decisions

We prioritise XYZ and as such did the following:

1. X
2. Y
3. Z

Hidden

Functional Requirements

Non-functional Requirements

Progress

• TO UPDATE, IDEALLY LINK TO PR USING [text](link_to_pr)
• AND MARK UPDATED ACCORDINGLY

[yusufaine]

COMMUNICATION SERVICE MERGED UNDER ROOM SERVICE

Hidden

Communication Service

Functional Requirements

• Users should be able to orally communicate with their match.

Non-functional Requirements

• Communication needs to be as close to real time as possible.

Progress

• Voice and video data through websockets
• UI

[yusufaine]

HISTORY SERVICE MERGED UNDER USER SERVICE

Hidden

Learning Pathway/History Service

Functional Requirements

• Users may want to view they past attempts to see how they did it before and compare if they've made any improvements.

Non-functional Requirements

• System should be able to get all of the user's past historical data since feature inception.

Progress

Backend:

• Add table to DB associating attempts to users, should only contain userID, title slug, submission attempt, and when it was created at
• API endpoint to GET all submissions ("summary"), get contents of specific submission, and POST to add new submissions

Frontend:

• "Submit" button in room service

[yusufaine]

Room Service

Room

Rooms serve the purpose of containing the question content from the question service, and host the collaboration and communication service.

Functional Requirements

1. Users should be able to explicitly leave the room.
2. Matched users should have the same questions synced in their view.

Non-functional requirements

• NIL

Stretch goal

• Integration with history service, ability to save/submit questions
  • Associate to user, room ID, and title slug
  • Explicit saving

Collaboration Service

This service mainly covers the ability to work collaboratively on the same document as the partner's.

Functional Requirements

1. Changes made by to the document by one user needs to be reflected on the other's.
2. The shared document needs to persist in cases of leaving the room accidentally.
3. Document needs to be cleared when users explicitly disconnect/leave.

Non-functional Requirements

1. The system needs to be as close to real time as possible (<1000 milliseconds).

Communication Service

While working on the same document, users would need a way to communicate their ideas/suggestions across to each other. Collaboration service can function in place of a chat service through comment blocks but does not simulate a proper interview/discussion and is no where near as interactive.

Functional Requirements

1. Users should be able to orally communicate with their match.

Non-functional Requirements

1. Communication needs to be as close to real time as possible

Progress

• Room UI
• Collaboration Service:
  • Live text editor
  • UI
• Communication Service:
  • Websocket transmit voice/video data
  • UI