✅ Deploy Preview for karpenter-docs-prod canceled.

I just submitted this PR and wanted to reach out for help and guidance. If I missed anything or there are changes you’d like me to make, please let me know—I’m more than happy to address them promptly. Thanks for taking the time to review this!

(I would want to point out that this doesn't quite solve the linked issue, though is a related cost modification that would be helpful for Karpenter)

Hey @jasonwbarnett, thanks for taking the time to put up the PR!

I'm a karpenter user that is currently facing the same needs.
In fact, it seems like the implementation is similar to what I've suggested here: #5163 (comment)

I have just two question/suggestions regarding the code, if I may:

1. I'm curious whether there's no need to also edit the UpdateSpotPricing function (in the same way as you did for UpdateOnDemandPricing)?

   To my knowledge, spot instances incur the same (e.g. Datadog) extra hourly costs as ondemand instances do, and my understanding of the pricing.go code was that the UpdateOnDemandPricing and UpdateSpotPricing functions act separately. So I wonder if there's no need to also edit the UpdateSpotPricing function (as I did here for example). If I've got something wrong and you could point it out for me I'd appreciate it!

2. Also, with some time having passed since my initial take at it I've come to suspect that you might also need to apply the same edits to the ondemand/spot prices in the Reset function (in addition to the UpdateOnDemandPricing and UpdateSpotPricing functions) in order to make it a more complete and reliable solution. I'm curious what you think?

   My reasoning is as follows:

   • There is a write path and a read path for the ondemand/spot prices.
   • The write path uses the pricingProvider. Karpenter first initializes ondemand/spot prices with static defaults in populateInitialSpotPricing called by Reset (in case network requests fail), then attempts to update ondemand/spot prices in UpdateOnDemandPricing/UpdateSpotPricing (which usually succeed).
   • The read path has to do with the instanceTypesProvider? And karpenter makes provisioning and consolidation decisions based on the pricing values read.
   • The problem is there is no fixed order between the read and write paths? So there may be a race condition between the "read path" and the "update" action of the write path. That is, a provisioning/consolidation decision may happen before the first price update(= UpdateOnDemandPricing/UpdateSpotPricing). This will lead to a brief period of time where karpenter makes provisioning/consolidation decisions while disregarding the extra hourly cost info. Therefore the extra hourly cost info should also be reflected on the initial static default values as well.

   I'm hoping for a second opinion on this because I'm not entirely confident in my understanding the code.

Again, thanks to the team and community for getting the PR going. I'd appreciate any second opinions and/or corrections regarding my comment!