Enhancement: Implement retrieval and calculation of prices info using GCP CloudBilling API

[dm3ch]

Empediment from: #31

GCP doesn't have an API that tells you how much the VM type costs.
But they have an https://cloud.google.com/billing/docs/reference/rest/v1/services.skus which is really hard to use, but can provide information how does VM parts costs (CPU / RAM / etc).

So to not to bring too complex pricing logic at the beginning we leveraged already calculated pricing data provided by https://github.com/Cyclenerd/google-cloud-pricing-cost-calculator project.

It seems it would make sense to try to reimplement runtime price retrieval to the calculation of instance prices on Karpenter provider side (this project).
Also rewriting initial prices to use the own calculation logic, if possible.

P.S. Don't forget about GPU, sustainable usage discount, disks in calculation, custom machine types

[jwcesign]

Karpenter doesn't need disk pricing information actually.
Sustainable usage discount, I guess it should be handled in karpenter core.
We will back after MVP demo.

[dm3ch]

Yeah, let's postpone it until there is a working MVP.

Regarding disks, I meant that we need to calculate disk sizes in case node have local disks as part of instance type. But actually I haven't checked if there're instance types with predefined disks.

Regarding the sustainable usage discount, as far as I understand,the following RFC is going to do some for that. I haven't read it fully yet, and didn't understood how it implements default cloud discounts (not user specific). Cause google have sustainable usage discount for specific node types (when you're selecting instance type during creation). But maybe I haven't read the google doc and don't fully understand how it meant to work.

Also just added info that we need to think latter about custom instance types as well....

[sthomson-wyn]

Committed use discounts are something that could also be considered

[nia-potato]

@dm3ch there is now a working MVP, and i think this is the last issue we need to resolve to get a stable release.

if you have any capacity to still work on this, it would be appreciated.

[dm3ch]

@dm3ch there is now a working MVP, and i think this is the last issue we need to resolve to get a stable release.

if you have any capacity to still work on this, it would be appreciated.

Hi folks,

I initially worked on the pricing retrieval part. During the first iteration, I ran into several issues while trying to aggregate and interpret the raw data coming from the CloudBilling API.

To move things forward at that time, I decided to take a shortcut and used an external API (gcloud-compute.com) that already provides up-to-date on-demand and spot pricing in an aggregated form. That implementation became the MVP version.

Since then, I haven’t had the chance to reimplement the retrieval logic using the official Pricing API. Unfortunately, I’m quite busy with my main work at the moment, so I can’t commit to any specific ETA for completing that part.

If this is the only blocker for the stable release, perhaps it might make sense to proceed with the current pricing retrieval approach for now, or to reassign this task to someone who can take it over.

P.S. I’d be happy to help in my spare time, but to be honest, the data parsing and aggregation part seems rather complex. Unlike AWS, GCP requires us to build the machine price from individual component prices, and since the data isn’t normalized, it involves a lot of manual mapping and regex-based processing to make it work correctly.

[dm3ch]

By the way, were there any issues discovered during the alpha or beta testing phases with the current pricing mechanism, or is the goal to replace it mainly to remove the dependency on the external API?

[nia-potato]

By the way, were there any issues discovered during the alpha or beta testing phases with the current pricing mechanism, or is the goal to replace it mainly to remove the dependency on the external API?

it has been working so far for my testing, but im not sure i can speak for everyone else.

If this is the only blocker for the stable release, perhaps it might make sense to proceed with the current pricing retrieval approach for now, or to reassign this task to someone who can take it over.

that would also make sense to me @jwcesign

[jwcesign]

It makese sense to me.
Just one new problem: #143

if this is fixed, we will release a version