Discuss state of r2 migration on Build WG meeting

[MoLow]

Opening this issue just to track this in the meeting.
we are pretty much done with getting https://r2.nodejs.org/dist/ working.
we want to make sure we did not miss anything and discuss the next steps - mainly when we should pull the plug on the migration
the Cloudflare team asked for an estimated date so they can prepare (they are depending on https://nodejs.org/dist for installing node on their infra :), they want to avoid circular reference issue )

CC @flakey5 @ovflowd

[richardlau]

Opening this issue just to track this in the meeting.

FYI unless #3503 changes the meeting times, the next scheduled Build WG meeting is in the inconvenient-for-everybody slot on Wednesday 18th October. The next meeting after that, in the earlier slot, is on Tuesday 7th November.

I guess one question is whether we want to migrate before or after Node.js 21 -- usually a semver major release results in a spike in traffic.

[ovflowd]

Opening this issue just to track this in the meeting.

FYI unless #3503 changes the meeting times, the next scheduled Build WG meeting is in the inconvenient-for-everybody slot on Wednesday 18th October. The next meeting after that, in the earlier slot, is on Tuesday 7th November.

I guess one question is whether we want to migrate before or after Node.js 21 -- usually a semver major release results in a spike in traffic.

Oh yeah waiting till November is definitely a no go IMO. It would make sense to discuss about R2 way before that.

[mhdawson]

I just got an error when trying to hit the link mentioned

[mhdawson]

Does it make any sense to switch over a portion of our downloads first to validate on a smaller subset versus everything. Is that possible/worth any additional work.

[mhdawson]

When I hit the page again, it now seems to work instead of the error

[ovflowd]

I just got an error when trying to hit the link mentioned

@flakey5 ☝️

[flakey5]

I just got an error when trying to hit the link mentioned

Looking into it now to see what happened

Does it make any sense to switch over a portion of our downloads first to validate on a smaller subset versus everything. Is that possible/worth any additional work.

As in, for example, requests going to /dist/... hit the worker and all requests going to /download/... hit the origin? In that case it's definitely possible and would just be up to the routing config. For deeper paths like /dist/latest hitting the worker while /dist/vX.X.X hits the origin it should be possible, but I'm not 100% sure on that.

[flakey5]

Believe I know what the error is, should be fixed in nodejs/release-cloudflare-worker#44

[mhdawson]

As in, for example, requests going to /dist/... hit the worker and all requests going to /download/... hit the origin?

Something like that. Roll out in smaller steps than a big bang. Do you have a suggestion for a first good step?

[richardlau]

I'm running into errors trying to promote Node.js 18.18.1 😞:

# Creating SHASUMS256.txt ...
upload failed: nodejs/release/v18.18.1/docs/api/all.html to s3://dist-prod/nodejs/release/v18.18.1/docs/api/all.html SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/docs/api/all.html EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-aix-ppc64.tar.gz?uploadId=AK5en5fhbO8ZsJyjv_fBucFKF_nxOZWmnYVx8Q3garrFPWRyq2q8drTOSKCzgKUhPajKU0Amnxyb2bP2ibtu23YtoUzUWurgo-eyOrTCGb7-r84FCEeUXsm9nK2fPct0NOWqy39ojE_p7kl2lGHDPl76k0bmcmHag7Rz-srSGElHAqv-EdcbuQZaXdZQaYDidov2W1Oi3W1FvPMrrfF1Jw6aqMaNspg1MH6ESNXsyVdovcRmlDS89m2oSX2-zr_CiirdKWSA8o0X0bWPNWS2zoXujri0NFD7yT39wKmrGCbGG20JkHj5a1C7WnTmtFoJNuV398LIyXsQQbeCdxxNKgI&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.gz?uploadId=AH2Q3Vic8_76OWmwnrMacUFjhcSDZH4Q5BH-k9Ox65vWGdaedlgo0IMLJra90QGpY-VFwr6MzXIRaganfleHJK_-vfcD-KxuBW3Pz3M3edpEqJaE-2qc3Y_inkMdP8i_gvN4atAFEybBvGyYEaw0z999v5miTeym7udzACkhiJGGJcv97e4LrFL9K6vZ3KjCyidIZDNrTgcUfGJAWYY9JmgMveCwjJ6OuZkZYYxCPHVpgNqLeAuFr7gTBFU8AvrQB5DGoVaKve-ElV1zqPIUDggDeyNv5prkhIPvEEv-gOT4dduZSxO3VbqTjvXrGYrSHtn7bjW4QZkmLgCaDQbnJHo&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-arm64.tar.xz An error occurred (InternalError) when calling the UploadPart operation (reached max retries: 2): We encountered an internal connectivity issue. Please try again.
upload failed: nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-darwin-x64.tar.xz?uploadId=AHpycjrfMp3Ou8gBMPUA88nOYF1d23N-rQtud7r-4Q2nvFp9Y2HDYhBzCl76vriPZRJx4bIr--HetvrX_pyEBKaq2Md1QsDUvmH69wAYRnajN9I-v5I6sQrZbX5vv_FUxwA8Oj2qWTAL6nKZz-KFH5SCpSvtGe_i9YALGX2m7aata16Up0NtaDynsL_vaT_FQZxh6M9aOpO1mv9ucbxsOsFwXZhZOC6ZJhscmyGPUH47jwvJDwXZVGJTQAG8QvKjB4uZQPZyn4aJ5_EtupYx8K8X6yKK-3cTgyraNlFX9SWtn022WQiq8vfoJQWmqkwad2_qaee2m3NCIFLK2YXtBh4&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.gz?uploadId=AD0two2TSNT8hnMWVnQOf4M5G-xs2gouFzsq33f5bMnQKRgg0pKP0jF54WSjAvZ3a0vbyTGzDmVPFpuzhWzqc84hzrL6BlGn3yJa8qkxUNlPQA2TZ9b23KjSlcIixVEGpyWKtkADiZAWEpyYjOzTySL181clydXJAMMa_ewH155cVd_m6ysMSIYN5ltYOhsSfxYh080uL_AhymJCVlDfm7SBRLIRFOOXeOpmYGsHkCmLgfWZLNCPqAtIFqNd9YKIeo1GgQnPiDyDw4Fs4XjALtQu1hyk-qqkLvI7hW_XmpXmhm136441N6XhlSfRW0tmvV6yhtk81uLo8GtIEzseOiU&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-headers.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-headers.tar.gz An error occurred (502) when calling the UploadPart operation (reached max retries: 2): Bad Gateway
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-arm64.tar.xz?uploadId=ABMnDebWVxhBc1k04Wcg99_ZuWBywvH307mvd1RGbzBQ-Nl9Cd8WcOFPgqrceIebh0o8OfNfLQxbWSEgf0FcSJYFZwl9RBG4VaAOaz_kUCvpqyWh6rjc9ovLQZ7XwBgHxW8qk-aKrdjAK5FuAu7h6Eo8Kh5Q8MdtRcd9xaTTBdAMWiDNrUnGFdnCjhw53jW9tUEt1lMevfOKlvjkAI9_kT833WFcCwdacQBF1xAsKC9ugXPRhWQbyeXrOCxZvTQzxFbYVRaCFgrWvR6-wCaaae4amq7cYqL0EiP7DIUIRLvWvYZhKsvx5xDXdgLA_z20tr0pjedEN1TK6xeQQqu5UIw&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-armv7l.tar.gz?uploadId=APNla1tU948J1WlUYujXN074miBtOynCvXW_pEHKPz33APXaSBXYPv68C5aYgawgnL4EV_T_ziGWXPLnMPD8fLS8vPnxStqmIUI8HR9_d62NbT1oXH0-sRSm3s7HVYnf8QlKK2xmQPr4IFqspoWUgG-vbEegPG1f3qrYajeSx6oOmGu9H5mTi893neyUdcqIkI--VWUtxTCfazc9lSBeIlT3qFKGBlLd_lnj2iPCrbACchTTE46RI5OxDQzfL9Hy7Wb9ZGe-lg8TnTEg7w9NKMkywUTGa_d5PulEEq0wdSNKOtXd_GfXdpXyTwbziuR6BI3vhPk5gAe67X27b9Hl-Jo&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.gz?uploadId=AIwdPt9ASHPR_MWigdIMmSVw177O9XCiEOxWvAFjsr_4EipKvxjx46ylZXRo7WL4m-ubBC1vBAo3V33m8WVu7C4_Yd7WCkMEltL23-FMyiVJg_jWmY2wOwBpFoeXn2VlTio_4O2JS2s72miIlWFdRZE6D_DrzBzQFJ65i29ubGX1Xl31Nf8ZDex0_E31pajt2IuO6ZR-8fygJS5Q7qGCjNzM7U2uBTt3Y4gcxZPFH_uE-LTHBVCzsA4-8bWqBWDpLzkDBCG-05GjlUfsvRQyku5vXNTvYTVQ2vypFuSnfQYACXHHb2w7zed8Gq1tzXqX6nAp0hqI89_Wl_XnRd8xvM8&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.gz?uploadId=AMlL9loRZNEFKsqDwdnyjRu_0Jnvw29aDfoZI2nAxWQtM8C95V5Q5LZ76xP13PIRGoCOB0bIRZFJG6p66tF3cO8fknqJ-bLjbzE-XEUGmnCplxMI0RERotWU9BgUDLqF2DiFa1qt8xIuZ4MvO1YjtXGlhK4ESAZYqHY5vdY9GaTFTHAIEX7OiqFO07oaLmQ85dGhIfj0PHI_johOm-RL705XuKo3ErIL45vc_9PdNExbae175azbJCJqBbiczwypHCz5lf0310Nhpo_HTGnN7Cj2qfVx22_o7htu-eMtrzDd5hYE5UbuTLE5EGi8FJIDMQ7_kY6i7WXG5zll_d4tEKI&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-ppc64le.tar.xz?uploadId=AD-QybSCbx5AtyxVFP_zOUXmdCl05J95Oyp1QFB_W6lf1vmlXOZjpdKAYd-4L_4FTLyjo34oX4rDdBnfl-BoPUzzcjQ4wCxmRARrDNErNZXh8WbAJg6-PQzrVYUKf3EqAP8N_8yWckdxrS0DiQ494XqNruNOVwvKVxPboc2onfJbWW_ZgZT9O9PcvdXuSaKsBI13Ai_G2wJYsDb_wX7Cs0QtfNudADtFaZ5Fgzk48SK83SrO_f9GbHWxHab767TjAqM7hXsVzTH0T3Uh8N81Qkcq7AgbeteYEWeGZ2D-iRcWE0SnL3jRAtMFITG_qxCy01tMShYHDFb2oAsMOc_Olng&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.gz?uploadId=AAwOyFGuwNTBenVpMm0focOuZcnDW2g2d8M1FsVPr4FmPOjDRl4-Tv0ecly20EwmTfgYWGMqWwHxnR8472eQmadv-sG_KUXU-HRsndQKaZtihGUc6kbXE_xrR2tVYRQvRZvaKtsOFS3zfovVkB4-GHudNOqVGhNYyAHdkV1CM3Smnl8O2JCR5RO2Qfr0LE02iTaMk3vXDnJf65EoP41QBiI5KJXgiWkgJyMhor9WL31mDwVGZm8kOOdgBDOB_LZQF2qCk9LYWSCAy_8se9K2WtKqwWjkAn7XR2Xy9-W0eiSCg5DwJtHAswn3moQal2MPBVsd0OdZhmXPis7W2Nd0nWQ&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-s390x.tar.xz?uploadId=ANEZzsnpRXUgADvxhKSOUYXW7t7hxbJ8qxUTqTT7C2YHpgAse6Zn_e3BGjDxZUsMPLqfSYdHXXyy7IuWw5OhqFIGpnExWcKufbbx4SYcrYEDiZApoFHWwDUcwnmkJ99IMCh74dnL7W53-EkZuP-yGpDur3L_X5I9UUVjPkdKLgNI9Eno-uhcuv4UKcuHeLmE9QeivLTMO8RRiuvSqP0DCygRWxgnpyePwD7nQrh2JkE2IAt8D9yacsRxVY0oC6KTDDzgj4ttDIkz15qTXiVbhWPF0e6vzoLMgtWgFYZrPjIexCqkbm5RA6qNAMj-qxIUw6fxwulYTFtaN_zqDtOrvvo&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-linux-x64.tar.xz?uploadId=AB0nJ81_nbe8AA148NLE4Upjz0ASRbpbhIQVzEe3E24H6xKq5QiNASuuto5fKmsoeWW6VkQ0HTYll-EkEM9jtNMe-N3uOXjSfMrn2k0Za5hsncZv67kxEDbK1_F6coNcGI_cbQ_m6Un1VmLflRDqSPEM4kxZrQ2pRArEsfLo66t3FPEQdQtngUm1IbxIz5LDivhSEIKnL2FrQw1av0_1qRR9X4MMCueOo5WJF803JmE5df94NPWOle5Sv4NxI3o1dPPj4IFbmYgE6hEcmtC7kANhOSnIlKm5XwToPOwOC60O0YNWw0fh0EczU5GIYufceUKmo6S1KqBqtVgqURkw93g&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.7z?uploadId=AA-QUfH57CHLUnp4zMoteTRVdHCm62UUaGR1ThGNmJ6J5YDGsjxbKE1xGTbT95_u-T2ADPSqOyH69wRDwI4XEDWxm_Sw411XiuzpXfZeDREctWd_aJnLIcSJXrQtUkyn0FSWkbtMra6koyKOrM_d-rf2GDDMukpYfo6XYQheM04frt1NjefGOnP2sfanSP_Uys6ghF2iq3427OqfjJueTrIqRBhpuJY50Vi53ZUKAjgWiTYwtHtza3xFTryYv8jQ45oqOnmfS-UwLVTxgebJuDblOEupILAMfLTEN0eMLBFV0D-sDocHhgPguO2yxv4dUmuujYruXMN6ckAYawkPTuA&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z Connection was closed before we received a valid response from endpoint URL: "https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.7z?uploadId=AOJKmxdqJZpGbcdtMJZR0A4rJwZHQsOsBB7XWGnOh6-CQ0Xh8kLWXfr-gtf8aKszYSdssvfXaCWOE27QY0Y5KGMejyQpQHSvrnZ6_hEl3mO8-MdYAy0PFtfqIBo9HfKlohlXhVv3-QeWDXc_PbicVyydWbDnCiUJERwxFmkLPRqzWWfBMTlid151dqgD_WoXgpD-quF9OD2h5YcgJ_-xRd5orGJ-bAAuD-5t_DD-a_oLJ3UPB1oZ5mLXsYcBTmibIxa4fMj4Uhs0meYAiC3Eaajh7Vx4sDP3lWH-0YlFB6SVr0TkylKWpmbY2GR0VkZgb-H3GseeTAJGuKj4YtJ0FZA&partNumber=3".
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x86.zip?uploadId=AAqI5JU0k7ReQ25MvcfNJaUi394xfgSlaL5ly2aKUaUiKVAhJPLiBULxLhUQ4rYvSw9I4Bf_8klqsz66njzUA2YfioWH9-s6tUzs6fPGnGhqlF2cs_v2usDXBHYDhqisiAkoNEg5lQCbo6ljVC-B0Saw2zY9jWE37ajXtO_fcI6zdDhAmRSnG9C6tTIx9UY3mFN4i_TaqXcu-fn1LEfNFr6ZKnuk7KUn_g5iX72bomuHzPe9ad3jimx5JJwQ9fnScOKdyL8_P2hxPXRpWn00Iej0Gssu-ljMqXBdlG4bKsS_On4ajXOfB-fR3EN0C8AacJt1CQzLoR5KJybGEU_yEwo&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-win-x64.zip?uploadId=AHG6FbmDzD4q4xehQgIRvTjTrsOsEQyFX4YVfVM0m3QPrzNtBts07XqdqRRNeWDA8gMp8lSYD2MzHC5kvpGRONr1JsPuqn8oIFq3ZNRlMU2I0dI57SfzWCPmPRhW7_zPn9_iutokUyHIaDPuj8kHhbM-oEXdtIi-gPo_aG6Xs3cja9iyfhH1VtA-t0TX89Po5XDjLfoc5U5no9TJsB4XXGGnWn1A76PoQuoc37ytb49DqSp2Hekte6rgN2ZLhglQXweap6qPqsMGAaFXUKv-ZMbw8wAe6Lwo22Gwi4gDzq-EtZUFyo6-rbaRf1xtB4I1-kGe5_63hSWdsKOM5uVg97U&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-x64.msi to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x64.msi SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x64.msi?uploadId=AP4c7XUSrPT5bxe2-O_NMN2I45wgE_UFZM94P_lN3Y17FjXmFG_llHaCObiHOmFyTxwyMvCSKOziSIPEa_4puQaaJ2vSyhTdIRo3AlrME9zcOJ0glO4CuQ3KeWz_WfAAEbECldzqIdw3cxuwhpnS3Jj7VPUwtoMFpBgDqi6_yaVE_vNM2Nd6d42jYaQb8f7tQruH6KgqUcXP9w8A3FFaeLkP1-qimKjpifH4yDbjHWgjcnwoCh24e5pEgusRPq7UM65p76pVQcuXqog13GXMErRbMILeEGEe9MLdZ9Cb9eLdiuZlhABrBLXRkD73ebEm-w-1YO2YhKEHyfqEZDlV8no&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1-x86.msi to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x86.msi SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1-x86.msi?uploadId=AOQzX2Yx7ZNW_rN9FPslfmTV7YYQoOclj-ONBSslE5Poy0GVTxMObVa1kAl9RqqabxX6r_uptKjwWGaAGzN_DLAXV9A-JjvFW8i9d2xuBPWC7yO8-l-p6rtwTTjT-yu4YrVxjb54-gz1mBQDtXBj6VOFb0YzglG4MP3BnUSS6wGprnqMxUo5kZgVW7S4qWhqsG8f1fYlxtBKFp8tLaDp33QRoofnOJgxQui7jiAyF8J068U8cq2SmdJqulmR_B_biR6V7wdYo0LetMU0ZW5HV9-zXQYd-b5pIR6wT1pK_6NeoHk1-K0LDzCE264VmLtCFJiUnll6gJtOrPDS7ysZC1g&partNumber=4 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1.pkg to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1.pkg SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1.pkg?uploadId=AO6MuwqBn4EJDVxIKoUum-rAvZaXF8ETUfFLcu4gtm271pga0BuoEnEUC-Sd8ZRL2MFUE-i9XlCB2KMEN2B2Iji5D_hs78Nx1NSmDKKdYZSNfWACxUbl2K8SBbJmBUfRiGSLJv5fk0ui8ne9qOvllysIBy4MODDs4bbf2zgc3WQPNV_U1LUAxNbkW73NmQmAie1mnW5spoP7FQQgh4q0nFdYW27niJihu7OEGMxDS9eLwQptxrcFbSxajpTn4qja_Y8QJUbiDAu_spk7WmCkDAj78l8Ap1-d61_xpwsPMKwUDPQCTr1jXix-HmRFfAbGt5UW_Hph5MlJa2QTxRf9O1c&partNumber=9 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/node-v18.18.1.tar.gz to s3://dist-prod/nodejs/release/v18.18.1/node-v18.18.1.tar.gz SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/node-v18.18.1.tar.gz?uploadId=ADBk6NL9Q2ehez8_McAh1Mf7CR4zyh1QK6kPXYmpj9MDU7Jz0HalsCPX-6mfqTSGcct3FBdnDSbadj3EuB1uIEUo8vdi4bzQvEuGSUsjeTvEjVdVBRZtQ40b_CcIVXZZnQJ7tBj66uejgkMt_zkpkqkgvyQNI6FNeWa0K9c1t8t6BzxBDRIPrq5Y-BkyzDpaUDi0KeELWe0mT9vXeYGTtLReoN3oHR2aSsmr0weiLE7agGf2bhW09EWaPJ8PY8DxrjEVxD5WpnGxipRTaNs4S8XX6O3vK7d9YdlcXxo49CZ0eR32ze8n3LC6o1W7fy6XSyysY27YjWcwkLTrO6dzJmY&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node.exe to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node.exe SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node.exe?uploadId=ALfJRer4UbzGgzZAAC8E447xGQBu2Jva9QUTLsObZH6BoPzSxx9PaSK9efBK-gkyZZUsj05NjUwuqznjS6ilQufuVQ4AG-jbAeVyfPNNuN1i5SabdlV0433dg38nJIIY7-OL9gUp-f18nkPULFmLeKvoMObMzV3BioYLKMJ9JZpMPavvahzupJBI__i1k5XWofooRqb3wVOsMvcXV49uT5ZZxsitP7hW5cIjZSz9mhDrfh4RmVUgAI8tHHyhGDPBIACqby_HB2ChdlMaCpUfgMOS8S22TiJ4DSavjiXqAT4qo7Iw4FITgcs-r6n_x3AvGJg4B8FgNR62NhuUHYvXL60&partNumber=3 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node.lib to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node.lib SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node.lib?uploadId=AON6QbdBHLTVcjEC5SOke9-pPSp9FYvUZED-5SWbFU3Y-npTbO1Arj5_k0FP0Lapgfkg5l5pYx6OAZnTtVla4RMHfk-wdKMPIVtt6fUDkYLcfqIeBoCQVf5Pm8fJra-YY6qJsng1YrpjKe9xi2lSepRbChKOvOB75cuHXoyFLXDFEa0xQCAkR-aMhans_W_Cad8YqH3gwVLTJhyCS1kStiIWvluKOuRx9zt7qpIkQHcP5v5fTJWEPyBLMM3aYWeoUahni1JGaVBgkvY03fWkBMQWmc5Jckl4sc0PtBAU9el5RfFpPxdqhgzIQQyUTDHrNU8DGqI0ruROAf6IQzuDbbs&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node_pdb.7z to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.7z?uploadId=AA5kO1d2ZgdVAlWKzAF3NGddqb8WIGngKT8D4VrJxExlYz2B2A7NaHYOqFUzzMi9aWgA3ofzf7cf2jjT5JhrXytavKVsR0QBv3l72ti2E04xFnjhJQNWYSfFzgzH6dYwQtlxC05KAFjOWrIkU5NPge1CBuTpYtfJQgMaYGYh8yjH9Y5YP4bCmnhlnKB6nE9C-rvQ8cfc79-hvxPMelBJWP8gsQx6D8V3tBGrPboIQKUSZMt2BxjxaRvQxbR44wjzp6x_ohxV4t7XFbJdGpKAJDVpClzHZ9L0dxG8SBVFB7GD9fRXg36rafyl3Wy07oDHNbNREtHDpVLDJcbcyDRTANk&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x64/node_pdb.zip to s3://dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x64/node_pdb.zip?uploadId=AH8ieQkTJSvGmxBlv6PbrBh1NjuAXibvLkqgvCa_13-pCedouLVWSccfKhd2hDc-IeNQzNuB19Z2DY-7zh3K5i5ULhGW11_xPoQF4KbwuCaE3y2sBEk-Ylg5E5CzIY3RhJfyd2zA1KnjrrVE29o9qPtt-Lyss6IwXWDtdCcFu0bNjh-fcen5-iZXq4fjg1Ir75j8K2dZZGlLhsp9e5BvOVcXsPghcBkK5vJBSgho5DktNcaUi8YeoSAsGyxNSvAfjyBMw5SM0tXGTf_0QO-KNlWkExvAPsC-7EpQA4iyTTVpU2SV0wUfwiH5D2P6jOgQxvCkQojycFCAps-962m2-v8&partNumber=5 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node.exe to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node.exe SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node.exe?uploadId=AHnm_19Q4ivEGat7YNepjZOnNDPl6CSp5Npjev_9XgvK-Kbmp_7eGtTyp7X6qd26LA_Bvoh3eEERG4kjrN1U6ouSb6vMBXtjTqdsaSZzr49c98ca9p5eNWQ7NeBnbD99F3y_y555xMmlEZ3vVzUH0bSWAgwuegLNBNXWNDSlWLPyqyf0TV1kO6n0VsmnFa7YsHc6QNQsijh-9RlDmr_sqMLJwrsSjuP87muUS5spJSGrguWYPsc_Q2q0lQBv6__lWWi1UUTsakjeT-6CxjZvHXT69R1GBfSIFjGW3bdW_jyWbzPzxxNCU97zvy5B96ARhuuNN0hPMK3pshpd_Bv4nZc&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node.lib to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node.lib SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node.lib?uploadId=ANnBZCq-6ijn_8ZcDCdA32R3ZA_kVl4hw_gL98dmCXJcrvQMuEOBB0NU39VZZihB6XyvBb9ZN3b0wlSMCwhdlW3rOvEHWJzkZJKmmTLK9RviH-yl4whPjmvkITyBXBAC8wyLRjCuLpqs_y2SuhDLtxMacpe9D3uWqIaSXAuvPnS3WqOLABzJas2BE2VPnK4AVkvon5YlSoud2EYnR1quqe0uzmMdfXNYXlKAyAD2ukQDf_FF1n29SekhxbcLlBLioCI-SNZHarYlCmPKPLxjNyNQgCos_ntwvk8b0LF-Lowu7D-LLQViNPAElzMfF50ZEXrUsBTw1DmTICqkFj54HeI&partNumber=1 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node_pdb.7z to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.7z SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.7z?uploadId=AJ3AQL3aZCrLsy7myySv_L61kLKVTa_7t_U68o-pUOHUYY1lS29itps-etR8ZS8xFrjwqmpXOUiaOxIugul14pijacW7diecbA3fLm_kV8e6ROwI0bU2d0N-oF4Ao68CJNqbrfaQuUg8Xx_XmPcQVcgqTiSNbYXn33VRqJM9ngZTDl5OZvPtFysq6bff6EL7n6NojFy9oxeXbdmBAILyCIk2Y3N8DRSNXFhH_yima17OVJ-cqW12xt43rV_0NGP-Xl8F1H16UETzzSx_S2gn4zpN_pnsl6md4QOLj0z1GTND1UPiWED3P_95nDjeshrFzs7InJgN8AlZ1MdDf5A3VD8&partNumber=2 EOF occurred in violation of protocol (_ssl.c:2427)
upload failed: nodejs/release/v18.18.1/win-x86/node_pdb.zip to s3://dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.zip SSL validation failed for https://07be8d2fbc940503ca1be344714cb0d1.r2.cloudflarestorage.com/dist-prod/nodejs/release/v18.18.1/win-x86/node_pdb.zip?uploadId=ABmd5cHDQxWUZ4F_1XW_YTuBDpYxAU5QZWK7bFdb6xzkbWidqPfoWiWIjMybXNAtG_VH9rHPUT_mybY_mpGqCwOIWJzaydvvXFH44kIx5dRDPwv2HMMhwQbl4A3W8MVGkhVUnRSAqosRt7Mc2wLG-b7tPD5MrmkLxX2HnMrQK9Lbuz_zqoQbaF_WHhnD9EPMmN5_dPMflpmEKtvqaJTdako82Ws2wV7Wsxal0ZoGVrQaARlG71rbzGJTfp2lqMEat7CknovUyWnxketGWognkKyhj4EMX8zfQKN7zndS4Z4gNCiA7J5el4fssVPeYeBKsQ_dlFHOtBI6_EmowyUUkx0&partNumber=5 EOF occurred in violation of protocol (_ssl.c:2427)

The errors are very slow -- presumably the uploads are being attempted and erroring one by one.

This is just running tools/release.sh as per https://github.com/nodejs/node/blob/main/doc/contributing/releases.md#15-promote-and-sign-the-release-builds.

[flakey5]

Something like that. Roll out in smaller steps than a big bang. Do you have a suggestion for a first good step?

First step could be /metrics, that way if anything does go wrong with how we route to the worker we can fix it without disrupting too much

I'm running into errors trying to promote Node.js 18.18.1

@MoLow

[richardlau]

FYI I've had to comment out the calls to upload_to_cloudflare.sh from the promotion scripts so that tools/release.sh can actually sign the releases (the errors from the upload were causing the release script to bail out).

_promote.sh-1267-# https://github.com/nodejs/build/issues/3508
_promote.sh:1313:#    . ${__dirname}/upload_to_cloudflare.sh $site $subdir
_promote.sh-1371-    fi
--
resha_release.sh-505-# https://github.com/nodejs/build/issues/3508
resha_release.sh:551:# . ${__dirname}/upload_to_cloudflare.sh $site $2
resha_release.sh-601-
--

[ovflowd]

Why would there be SSL validation failures? @flakey5 do you want to share this with the Cloudflare team? Afaik we have one Cloudflare Engineer and our Account Manager on the #nodejs-build channel

[jasnell]

Check the certs provided by the server. As a temporary workaround also try setting allow unauthorized to skip the cert validation. It's possibly with either a mismatch in the cname expected by the cert or a CA issue.