Build fails with getCurationSketches when OP API inaccessible

[ksen0]

Most appropriate sections of the p5.js website?

Other (specify if possible)

What is your operating system?

Other (specify if possible)

Web browser and version

No response

Actual Behavior

Build is failing for multiple PRs because of getCurationSketches break. That code contains pollng the OpenProcessing API.

Expected Behavior

Should be possible to fetch OP. Should have a fallback: if OP fetched data already exists it should not be re-fetched at every new CI run.

Steps to reproduce

See failed builds for:

• https://github.com/processing/p5.js-website/actions/runs/22182341582/job/64146661413
• Add Hindi translation for conditionals-and-interactivity tutorial #1183
• chore(deps): bump the npm_and_yarn group across 2 directories with 10 updates #1184
• added standard hindi translated words table #1186

Would you like to work on the issue?

I can but if someone else wants to help investigate and fix they are welcome!

[kajal-jotwani]

Hi @ksen0!
I checked the failing CI logs and it looks like getCurationSketches throws when OP returns 500/429, which then breaks the static build.
I think we could add a persistent fallback for example, saving the fetched OP data to a JSON file after a successful request and reusing it in future CI runs instead of re-fetching each time, along with handling API failures more gracefully.
If that sounds okay, can I work on this issue?

[priyaa-j]

Hi @ksen0 and @kajal-jotwani! I encountered this exact getCurationSketches 403/500 error while working on the broken asset cleanup in #1189. I've been using a local bypass to keep my environment functional, but I’d love to assist in implementing a permanent fix.
@kajal-jotwani, I'm happy to help test your fallback logic or assist with the JSON data structure to ensure it handles these API failures gracefully. I would love to help verify the fix on my end or assist in any way. Thanks!

[ksen0]

Thank you both @kajal-jotwani @priyaa-j ! Since you commented first @kajal-jotwani I'll assign you. But thanks a lot for your input @priyaa-j I hope you also support with review.

[ksen0]

May be useful to review previous discussion when this issue happeend before, #955, where @davepagurek suggested:

Based on the code in https://github.com/processing/p5.js-website/pull/838/files, it looks like it should only hit that error if we're trying to load a sketch that's not in one of our default curations which we load in bulk once. It looks like this could happen if:
- We are calling it with different params so it's not cached. Looks like we always call with the same params though.
- Something about `memoize` isn't working, e.g. we're building pages in parallel where it used to be serial and the separate processes aren't using the same cache. We could try having a manual build script that pulls the data and writes the result to a json file that we check into git, so that we aren't constantly querying OP on every build
- We're requesting a sketch that isn't in a curation (not sure where this would come from in our source code though)
- Something changed on OP's side. Do we see the failures starting after a change on our end?

I don't know what the specific issue is, but the second bullet would probably address it regardless if we make querying OP a manual step. If we aren't sure what the cause is, that could be a way forward? e.g.:
- Make a script that pulls all curation sketches + their source code into a big JSON file, e.g. `npm run build:sketches`
- Check in initial results in git
- Update code to read from that file rather than from a `fetch` result
- Update our docs to say that when we want to update content from OP, we have to rerun the script

[aashu2006]

As @davepagurek and I were discussing earlier about the OpenProcessing build issues, I took a closer look at the implementation.

From debugging this locally, it seems the core issue is the live OpenProcessing API call during SSR. Whether it fails with 403 (cloudflare) or 429 (rate limiting), the problem is that we are depending on a non essential external API during build.

Looking at the current implementation (getCurationSketches and the related size check logic), there seem to be two main directions:

1. Move curated sketch rendering fully client-side so the API is not used at build time.

2. Cache minimal metadata (title + fixed-size info) in a local file and avoid live requests during SSR.

Given the current structure of the site, the caching approach feels slightly more aligned, but both options seem reasonable depending on the trade-offs.

[ksen0]

Adding here a link to @clairep94 's previous hotfix: #838 and there's other discussion there on limiting use of OP endpoints. Also includes some discussion with maintainer of OP about this topic.

[ksen0]

@aashu2006 I do think it makes sense to keep the fetch logic at build time, rather than moving to client, but non-breaking / fallback behavior is definitely essential, this is the second or third time this happened.

@kajal-jotwani Since this is a little bit more of a time-sensitive issue than others, could you let us know if you need any help, and if you would be able to work on this in the next 1-4 days? Totally ok if not, but if you're too busy then it might make sense to open to other volunteers? Let me know what you think!

[aashu2006]

Yes @ksen0 . That makes sense, keeping it at build time with proper fallback feels like the right direction. Let me know if I can help with testing or reviewing once there’s a draft.

[priyaa-j]

Hi @ksen0 @aashu2006! I did some testing on the OpenProcessing API endpoints to better understand the issue.

I created a simple test script that tried multiple endpoints,

What I found: All of them return 403 Forbidden with Cloudflare's "Just a moment..." HTML page. No JSON data comes through at all during automated requests.

This confirms what @aashu2006 mentioned about the 403 errors the API is actively blocking build-time requests.

Given this, the caching approach (manual fetch + JSON file) seems like the right way forward since we can't rely on the API during builds. I'll be happy to assist in testing or reviewing or however I can, Thanks!

[kajal-jotwani]

@kajal-jotwani Since this is a little bit more of a time-sensitive issue than others, could you let us know if you need any help, and if you would be able to work on this in the next 1-4 days? Totally ok if not, but if you're too busy then it might make sense to open to other volunteers? Let me know what you think!

@ksen0 sure! I’ll start working on this today, review the previous discussions, and aim to put up a draft PR soon.

If I run into any blockers, I’ll update here. Thanks everyone for the support!