Integrate power consumption estimation on public cloud virtual machines as a degraded mode

[bpetit]

Problem

Until the cloud provider does install scaphandre on its hypervisors, we should enable cloud customers to estimate their power usage and thus their climate impact.

Solution

Integrate statistical models like https://github.com/etsy/cloud-jewels for GCP (look for other models) as sensors to enable using scaphandre on cloud providers even if they didn't implement (yet) a scaphandre-like solution from the hypervisor layer.

Alternatives

Implements a ratio based approached like powertop (cpu time consumed / cpu time globally consumed) in this context (VM). Mix it with provider public informations about hypervisor hosts hardware ?

Additional context

This is more thatn a feature request, it is a starting point for a wider study and discussions.

[mrchrisadams]

@bpetit did you imagine this as a kind of synthetic, 'modelled' sensor, that provides numbers based on the detected number of CPUs, RAM and so on?

[mrchrisadams]

There's some work likely to be released by the German Minisitry of the environment in December that will provide some numbers that could inform this, as it's based on actual recorded energy usage figures for a known set of machines:

https://www.umweltbundesamt.de/en/press/pressinformation/video-streaming-data-transmission-technology

However, I haven't seen the underlying data yet.

Although, it maybe worth speaking to the folks at https://datacenterlight.ch - I know they pretty see much see openness and transparency as a one of their differentiators and might be able to share the some numbers for an basic version of this.

[bpetit]

Hi,

First, thanks for involving in this project !

@bpetit did you imagine this as a kind of synthetic, 'modelled' sensor, that provides numbers based on the detected number of CPUs, RAM and so on?

I think so. I imagine a sensor that gathers metrics about resource consumption on the machine, plus data/characteristics about bare metal machines running the VM, that the cloud provider accepts to disclose.

There's some work likely to be released by the German Minisitry of the environment in December that will provide some numbers that could inform this, as it's based on actual recorded energy usage figures for a known set of machines

I'll have a look at that, thanks !

Although, it maybe worth speaking to the folks at https://datacenterlight.ch - I know they pretty see much see openness and transparency as a one of their differentiators and might be able to share the some numbers for an basic version of this.

I'm definitely interested to talk with people from ungleich/datacenterlight to imagine a proof of concept. (I love their work <3)
If you know someone there we could discuss with I'd be very interested !

[bpetit]

I'm investigating on the feasibility to feed a centralized database of processor models per cloud provider that would both benefit from and to scaphandre. If you have access to an instance on any public cloud provider, could you send the content of /proc/cpuinfo in this thread please ? With name of the cloud provider, model of the instance, content of the file ? I'd like to check that this file contains enough data for most of the cloud providers.
@mrchrisadams @florimondmanca @uggla @jdrouet

[florimondmanca]

@bpetit Here's what I've got on my personal VM:

• Cloud provider: DigitalOcean
• Model: Basic (Shared CPU, 1vCPU, 1GB RAM)
• cpuinfo:

$ cat /proc/cpuinfo
processor	: 0
vendor_id	: GenuineIntel
cpu family	: 6
model		: 85
model name	: Intel(R) Xeon(R) Gold 6140 CPU @ 2.30GHz
stepping	: 4
microcode	: 0x1
cpu MHz		: 2294.608
cache size	: 25344 KB
physical id	: 0
siblings	: 1
core id		: 0
cpu cores	: 1
apicid		: 0
initial apicid	: 0
fpu		: yes
fpu_exception	: yes
cpuid level	: 13
wp		: yes
flags		: ... REDACTED
bugs		: ... REDACTED
bogomips	: 4589.21
clflush size	: 64
cache_alignment	: 64
address sizes	: 40 bits physical, 48 bits virtual
power management:

[jdrouet]

@bpetit my cloud provider is Ikoula

cat /proc/cpuinfo 
processor       : 0
vendor_id       : GenuineIntel
cpu family      : 6
model           : 85
model name      : Intel(R) Xeon(R) Gold 6134 CPU @ 3.20GHz
stepping        : 4
microcode       : 0x2000043
cpu MHz         : 3200.078
cache size      : 25344 KB
physical id     : 0
siblings        : 1
core id         : 0
cpu cores       : 1
apicid          : 0
initial apicid  : 0
fpu             : yes
fpu_exception   : yes
cpuid level     : 13
wp              : yes
flags           : fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush acpi mmx fxsr sse sse2 syscall nx pdpe1gb rdtscp lm constant_tsc rep_good nopl cpuid pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand hypervisor lahf_lm abm 3dnowprefetch cpuid_fault invpcid_single pti intel_ppin ibrs ibpb stibp fsgsbase tsc_adjust bmi1 hle avx2 smep bmi2 erms invpcid rtm mpx rdseed adx smap clflushopt clwb xsaveopt xsavec xgetbv1 xsaves pku ospke
bugs            : cpu_meltdown spectre_v1 spectre_v2 spec_store_bypass l1tf mds swapgs taa itlb_multihit
bogomips        : 6400.06
clflush size    : 64
cache_alignment : 64
address sizes   : 46 bits physical, 48 bits virtual
power management:

[florimondmanca]

Ikoula

https://www.ikoula.com/fr/cloud-public

Oh, Cloud offerings seem super nice. VM Micro is basically the same specs and price than my DO droplet. Except Ikoula is French, has servers in the EU, low-carbon, etc… I've been looking for some bog-standard public Cloud like this at a competitive price for a while! Sweet. :-)

[pydubreucq]

Hi,
I'm VP Bare Metal at Scaleway

I'll give you cpuinfo of our Bare Metal offers

First is our Ultimate Performance Range:

Offer: UP-BM2-XL
CPU: 8 x Intel® Xeon® Platinum 8280 - 128 cores / 256 threads / 2,25 GHz

scaleway-cpuinfo-up-bm2-xl.txt

Offer: UP-BM2-M
CPU: 2 x AMD EPYC™ 7742 - Zen 2 - 224 cores / 448 threads / 2,7 GHz

scaleway-cpuinfo-up-bm2-m.txt

[pydubreucq]

The second one is Bare Metal General Purpose Range

Offer: GP-BM1-S & GP-BM1-M (Same CPU)
CPU: 1 × Intel® Xeon E3 1240v6 - 4 cores / 8 threads / 3,7 GHz

scaleway-cpuinfo-gp-bm1-s_and_m.txt

Offer: GP-BM1-L
CPU: 1 × AMD EPYC 7281 - 16 cores / 32 threads / 2,1 GHz

scaleway-cpuinfo-gp-bm1-l.txt

[bpetit]

Thanks a lot @jdrouet and @florimondmanca for the data. This seems to confirms that, most of the time, on most cloud providers, instances cpuinfo contains enough data to guess the baremetal hypervisor cpu model. This is great to move forward on that feature.

Hi, and thanks a lot @pydubreucq for the data about scaleway's bare metal machines. Even if this thread is more about guessing the consumption in a virtual machine without having access to the bare metal, this is valuable data. Would you have by any chance the same data for the machines that are running scalway cloud's instances ?

[pydubreucq]

Integrate power consumption estimation on public cloud virtual machines as a degraded mode

I missed something :) Sorry for the noice :)

I'll try to get these info for Instances.

[pydubreucq]

Scaleway DEV Range

DEV1-S
scaleway-instance-dev1-s.txt

DEV1-M
scaleway-instance-dev1-m.txt

DEV1-L
scaleway-instance-dev1-l.txt

DEV1-XL
scaleway-instance-dev1-xl.txt