[BE] Can we improve the Programming languages sync script

[alexanmtz]

✏️ Summary

Improve and optimize our sync for programming languages

📋 Description

We have a script for syncing the GitHub programming languages for a project used for the bounty filters: https://github.com/worknenjoy/gitpay/blob/master/scripts/update_projects_programming_languages.js

We want to optimize this script so we avoid the Github API limit exceeded and have a better handle of the programming languages. Currently, we are clearing all and re-associating again.

You could maybe use a header to be smarter about these verifications and know if this was already synced so we don't hit the limit.

A good extra would be when the GitHub API limit is exceeded. We would get the response for how long we will be blocked, and rerun the script after that interval.

You should write automated tests for it.

🔍 Steps to Reproduce (or trigger)

1. Check out the project repository: https://github.com/worknenjoy/gitpay
2. Follow our README file to setup the project locally
3. Import at least one issue (if you import many, you will probably see the limit when executing the script)
4. Run the script node ./scripts/update_projects_programming_languages.js

✅ Expected Behavior

1. The script should sync programming languages with the issues imported to Gitpay and display on our filters
2. The script should not make unnecessary calls or clear the Programming languages and associate again, it should check
3. It would be nice if we hit the API limit, and we get from the response the blocked time and rerun the script when we can call the API again

🛠️ Stack Affected

Node.js

📜 Relevant Logs / Payload

API limit message

https://github.com/worknenjoy/gitpay/blob/master/test/data/github.issue.error.js

    { "message": "API rate limit exceeded for 87.52.110.50. (But here's the good news: Authenticated requests get a higher rate limit. Check out the documentation for more details.) If you reach out to GitHub Support for help, please include the request ID FF0B:15FEB:9CD277E:9D9D116:66193B8E.", "documentation_url": "https://docs.github.com/rest/overview/rate-limits-for-the-rest-api" }

https://docs.github.com/en/rest/using-the-rest-api/rate-limits-for-the-rest-api?apiVersion=2022-11-28#checking-the-status-of-your-rate-limit

The headers will contain

x-ratelimit-reset - The time at which the current rate limit window resets, in UTC epoch seconds

💰 Is there a bounty for this issue?

No

💵 Bounty Amount (if any)

No bounty for this issue

📜 Bounty Details (if applicable)

There's no Bounty for this issue.

This issue is part of the Only Dust Hackathon: https://onlydust.notion.site/May-28th-Open-Source-Hackathon-1f244b682a80801890e5fda15b779746

[dosubot]

Here’s a summary of how you can improve and optimize the programming languages sync script based on your requirements and the current implementation:

---

1. Avoid Unnecessary API Calls

The script currently clears all programming language associations and re-fetches them for every project, regardless of whether there have been changes. To optimize:

• Track Last Sync: Store a lastSyncedAt timestamp or a hash of the languages for each project. Before making an API call, compare the current state with the last known state. Only sync if there are changes.
• Conditional Requests: Use ETags or the If-Modified-Since header with the GitHub API if supported, to avoid fetching data that hasn’t changed.

---

2. Smarter Verification Using Headers

• Leverage Response Headers: The GitHub API provides headers like x-ratelimit-remaining and x-ratelimit-reset. Before each request, check if you are close to the rate limit and pause or skip syncing if needed.
• Short-Circuit on No Change: If you have a way to determine that the languages haven’t changed (e.g., via commit SHA or last updated timestamp), skip the API call and association logic.

---

3. Handle API Rate Limits Gracefully

• Detect Rate Limit Exceeded: When you receive a rate limit error, parse the x-ratelimit-reset header from the response. This tells you when you can resume making requests.
• Auto-Retry Logic: Implement logic to pause the script until the reset time, then automatically resume. This can be done by calculating the wait time from the current time to the reset epoch and using setTimeout or similar.
• Exponential Backoff: For robustness, consider exponential backoff for repeated failures.

The current script does not handle rate limit errors or use the x-ratelimit-reset header for retry logic. You can see the current error handling is just a simple catch and log statement (source).

---

4. Automated Testing

The project uses Mocha, Chai, Supertest, and Nock for testing (source). For this script, you can:

• Mock GitHub API Responses: Use Nock to simulate GitHub API responses, including normal responses, unchanged data, and rate limit errors.
• Test Sync Logic: Write tests to ensure that:
  • The script does not make unnecessary API calls when data hasn’t changed.
  • The script correctly updates associations when languages change.
  • The script handles rate limit errors and retries after the correct interval.
• Edge Cases: Test scenarios where the API returns errors, empty responses, or unexpected data.

---

References

• Current script implementation
• Mocha/Chai/Nock usage example
• GitHub API Rate Limit Docs

---

If you need more specific code examples or have questions about implementing any of these suggestions, let me know!

To reply, just mention my name @dosu.

---

^{How did I do? Good | Irrelevant | Incorrect | Verbose | Hallucination | Report 🐛 | Other}   

[Chubbi-Stephen]

I’d love to take this on and work on optimizing the sync script to handle rate limits more gracefully and avoid unnecessary operations. Just wanted to confirm if I have the go-ahead to start working on this.

Let me know if there’s anything else I should keep in mind before diving in! 🚀

[Chubbi-Stephen]

I just got assigned this issue, I'll be taking it on right now, I will be done within 24 hours.

[Chubbi-Stephen]

I just made a pull request now. 🚀🚀

[Chubbi-Stephen]

Hello please canI get a review on my PR