[Bug] Unhandled GitHub errors can cause unintended drifts #583
Description
Describe the bug
The CMS currently does not handle all GitHub errors, which can cause an inconsistent state in the GitHub repository for the site, which can cause confusion for the end user.
This was first reported in the moe-ast-moehc repository, where the Isomer army member attempted to move a collection of over 500 files at once.
- This commit was the final commit made from the move: isomerpages/moe-ast-moehc@695193e
- This page lists the commits that were happening, which involved deleting and recreating the files (essentially a move done in 2 commits), notice that "Hong Kah Secondary School" is missing: https://github.com/isomerpages/moe-ast-moehc/commits/ddda8526a08a6f1c4fb51194529ead7dda1b5a8c?after=ddda8526a08a6f1c4fb51194529ead7dda1b5a8c+1364&branch=ddda8526a08a6f1c4fb51194529ead7dda1b5a8c&path%5B%5D=_school-histories&qualified_name=ddda8526a08a6f1c4fb51194529ead7dda1b5a8c
- This folder shows the files that were missed in the move: https://github.com/isomerpages/moe-ast-moehc/tree/695193eaced67b635038dbde798d01e2be2a2f35/_school-histories/School%20Histories
- In the CloudWatch logs for the period of 18:00:00 to 18:30:00 (UTC+8) on 9 November 2022, there were a number of
FileNotFounderrors thrown by the GitHubService. The timestamps of those logs matches roughly when the files should have been moved. This can be done by comparing with the timestamps of the neighbouring commits in the page on the previous point.
This is an excerpt of the error received:
Nov 9 10:08:41 ip-172-31-51-8 web: 2022-11-09 18:08:41 Wed Nov 09 2022 10:08:41 GMT+0000 (Coordinated Universal Time): {
"name": "NotFoundError",
"status": 404,
"message": "File does not exist",
"isIsomerError": true,
"stack": "NotFoundError: File does not exist\n at GitHubService.read (/var/app/current/src/services/db/GitHubService.js:86:36)\n at runMicrotasks (<anonymous>)\n at processTicksAndRejections (internal/process/task_queues.js:95:5)\n at async SubcollectionPageService.read (/var/app/current/src/services/fileServices/MdPageServices/SubcollectionPageService.js:53:42)\n at async SubcollectionPageService.updateSubcollection (/var/app/current/src/services/fileServices/MdPageServices/SubcollectionPageService.js:154:9)\n at async SubcollectionDirectoryService.renameDirectory (/var/app/current/src/services/directoryServices/SubcollectionDirectoryService.js:107:7)\n at async CollectionsRouter.renameCollectionDirectory (/var/app/current/src/routes/v2/authenticatedSites/collections.js:102:7)"
}
Possible solutions
2 possible solutions come to mind, but will require more investigation:
- Before updating the collection.yml file, a final sanity check should be performed to ensure that all the files are actually moved and the old folder is no longer existent. If there are still files in the old folder, the move should be retried.
- More generally, the
NotFoundErrorthrown here is uncaught, and encountering this error should be an indicator that there is an issue somewhere that should be resolved. In this case, we should catch the error and retry the move.
Additional context
Relevant thread on Slack