9.investigation.md

ntroduction

---

This is a more in depth exploration of how to design an investigation.

We’ll take the example of the Green Software Foundation website: greensoftware.foundation

This is a simple, static site served on Netlify, but as you will see as we progress through this example, there are a lot of steps and a lot of nuance to capturing its SCI score.

We’ll examine the whole process in 8 steps:

• Decide your functional unit? (IRL, this will probably be first)
• Decide the application boundary? (IRL, after more domain specific SCI for X, this and the functional unit will be prescribed in the standard for different domains)
• Which components are you going to include and how will you group them?
• For each component, what observations can you gather?
• Which models will you need to use to induce those observations into impacts?
• Write the IMP file
• Execute the IMP file
• Visualize the IMP file
• Share the IMP file.

Choosing your components

---

The application boundary is all the individual pieces that you want to observe in order to capture a representative SCI score for your application. What carbon-emitting processes have to happen to enable someone to use your app?

There is some judgement required to define the boundary. This should certainly include the energy used to run your app and the embodied carbon of the hardware used to run it. The SCI specification suggests the following infrastructure should be in scope:

The calculation of SCI shall include all supporting infrastructure and systems that significantly contribute to the software’s operation:

• compute resources
• storage
• networking equipment
• memory
• monitoring
• idle machines
• logging
• scanning
• build and deploy pipelines
• testing
• training ML models
• operations
• backup
• resources to support redundancy
• resources to support failover
• end user devices
• IoT devices
• Edge devices

This isn’t necessarily comprehensive, but it gives a good sense of what’s expected, but other factors might be appropriate in some circumstances and several of these suggestions might be irrelevant for your application!

Here’s an example inventory for the GSF website:

• Github storage (storing website source code on Github servers)
• Netlify builds (creating build artefacts from source code using Netlify)
• Netlify static site storage (storing the static site data on a Netlify server)
• Cache storage across content delivery network (caching static site data at several nodes across a CDN)
• Data transferred over network (transferring site data from server to user)
• End users viewing site in browser (energy required to display site in the user’s browser)
• Embodied carbon of Github server (for storing and serving source code)
• Embodied carbon of static site servers, incl CDN (for storing and serving static site)
• Embodied carbon of end user devices (for viewing content)

The next step in an investigation is to work out how these components should be grouped in your IMP file.

Grouping

---

The individual components identified to be within the application boundary can be grouped so that several similar components can be bundled together under a common parent. The reason this is important is because when you aggregate impacts up your tree, you get an aggregated value for each parent node, so designing your grouping well can make it easier to gain insights into which parts of your stack are emitting the most carbon. It can also be helpful for navigating your IMP.

Let’s explore the grouping for the components above:

• Development
  • Github storage (storing website source code on Github servers)
• Servers
  • Netlify builds
  • Storing static site data at origin server
  • Embodied carbon for web server
  • Embodied carbon for CDN
  • Embodied carbon for Github server
• User-devices
  • End user operational carbon
  • embodied carbon for end user devices
• Networking
  • Networking energy to serve static site over the wire

The specific hierarchy of components should reflect the classes of activity you want to break your total carbon emissions into, because this gives you the best insight into where to focus mitigation strategies. For different applications, the right hierarchy might be different, maybe you don’t have much development activity, but you do want to divide 10,000 servers by region to see which geographic location you should focus efforts on. There’s no fixed rule, it’s about deciding what surfaces information you can action, and what makes sense when you aggregate information up the tree from child to parent.

Observations

---

Observations are data you can collect about a particular component. Ideally, you would always directly measure power consumption for each individual component, but that is rarely possible, especially if you are running applications in the cloud or on shared hardware.

Instead, you have to audit what data is available that can be translated into the data we want using some model. Typically, the smaller the pipeline of operations required to link the observation to the energy value you actually want, the better (although this isn’t always true, as simple coefficients are simple to implement but may be inaccurate).

Sometimes, the observation might be quite abstracted away from the target value, meaning that heuristics, models and coefficients may need to be chained together. The right place to start is a straightforward audit of the data that is available for each component under investigation.

For example, for a virtual machine running on Azure, you can check the Azure Portal and find a dashboard that shows metrics such as CPU utilization, memory utilization, network traffic etc. This data is available via a monitor API, meaning you can programmatically scrape the data rather than manually extract it from the dashboard.

We can categorize observations into several types, such as:

• Direct measurements
  • Power / energy
  • Proxies:
    • Cpu-util
    • Mem-util
    • Data transfer
    • Cost
• Indirect measurements
  • Manufacturer data sheets
  • Analog systems
  • Digital twins
• Heuristics and generalizations
  • Coefficients gathered from literature
  • Regional/global averages
  • Qualitative estimates
  • Educated guesses

For each component in your tree, you can investigate which kinds of data are available. Typically, you might have access to direct measurements if you manage an on-premise system, but you’ll typically have indirect measurements for cloud VMs and services, and maybe only heuristic type data for components like operational carbon for internet hardware like switches and routers, networking energy, etc. There may be multiple data elements required for each component.

Here’s an example component from the GSF website:

• Servers
  • Netlify builds
    • Direct measurements:
      • Number of builds per day from Netlify dashboard
    • Indirect measurements:
      • CPU util and memory util during build, inferred from running build process on local machine and gathering metrics using top
    • Heuristics and generalizations
      • Coefficient for mem -> carbon from CCF
      • Power curve relating CPU util to TDP factor from Teads article
      • Processor TDP from GSF data based on assumed server specs used for Netlify build

Functional Unit

---

Determining the appropriate functional unit involves answering the following questions:

• What is a sensible functional unit to use to express the SCI for this application?
• Is data for this functional unit available?

For the GSF website, it made sense to express the carbon emissions as a mass per page visit. This data was available from Google analytics at a daily time resolution.

A sensible functional unit is one that allows you to demonstrate variations in the carbon efficiency of your application even as it scales. For a website, visits makes a lot of sense because the page exists for the purpose of enabling visitors, and total energy usage scales with the number of visits. Therefore, it won’t appear as though there was backwards progress if the site’s overall carbon emissions increase as a result of more visitors. Likewise, normalizing to visits protectss against misinterpreting total carbon emissions decreasing due to fewer visitors ad improvements in carbon efficiency.

The other factor to consider when choosing a functional unit is comparability with other similar systems. This might not be strictly relevant if you are only using IF for internal, introspective purposes, but if you want to benchmark against other systems then a common functional unit enables that. Two websites might have very different technology stacks, scales, deployment details and purposes, but their carbon intensity can both be measured in mass of carbon per visit, making it possible to compare the efficiency of one against another.

Models/Pipelines

---

At this stage of project planning you should have established the components in your tree and their organization, the observations available for each component, and the functional unit in which you wish to express your final SCI value.

The final piece of preparatory work is to connect, for each component, the available observations to SCI. This means writing out, step-wise, the operations needed to convert an observation into SCI, and working out which IF plugins can do the job.

In some cases, new plugins might be needed, but many studies can be performed using our standard library of builtins alone.

Let’s take a look at a simple example:

First, networking energy.

We have observations of the data transferred over the network when a user loads our website. We then have to rely on a coefficient published elsewhere to convert that value into energy. The CCF provided a coefficient of 0.000392 kWh/GB.

There is therefore only a single operation required to convert our observation into energy, and one subsequent one to convert energy to carbon (multiply energy in kWh by grid intensity in gCO2/kWh), and then one more to convert carbon into SCI (a division by the functional unit).

The plugin required to do the initial operation is Coefficient, passing the observation of data transferred in GB and the coefficient value. Then, we can use Multiply to calculate the product of energy and carbon intensity, and Divide to divide by the functional unit. So the pipeline for this component is:

Observation -> Coefficient -> Multiply -> Divide

We can also sketch out the necessary inputs and outputs:

Coefficient:
  Inputs:
    Data-transferred: GB
    Coefficient or networking energy: kWh/GB
  Outputs:
    Energy: kWh

Multiply:
  Inputs:
    Energy: kWh
    Carbon intensity: gCO2e/kWh
  Outputs:
    Carbon: gCO2e

Divide:
  Inputs:
    Carbon, gCO2e
    site-visits: visits
  Outputs:
    SCI: gCO2e/visit

This will now be straightforward to configure in an IMP file.

Writing the IMP

---

Writing the IMP is now just a case of collating all the information you have collected and expressing it in the IMP yaml format.

The metadata is simple - you give your IMP a descriptive name and a short description of its contents. Then you can optionally add tags that help make your IMP discoverable.

From section 5.5 you should have a good idea of the plugins required for each component’s pipeline, so you can start adding them to the IMP in the plugins: initialize block.

Each time you use a plugin, for example Sum, it needs to be instantiated with a unique name and appropriate config that allows it to be included correctly in a certain component’s pipeline. For example, you might use Sum in both the networking component and the servers component in your tree. This means you need two instances of Sum in your plugins:initialize block, each having the config that will make them work correctly in their component’s pipeline and a name that will allow you easily invoke the right instance in the right component. For example, you might have something like:

plugins:
  initialize:
    instance-of-sum-for-networking-component:
      path: "builtin"
      method: Sum
      config:
        input-parameters:
          - carbon-from-routers
          - carbon-from-switches
        output-parameter: carbon
    instance-of-sum-for-server-component:
      path: "builtin"
      method: Sum
      config:
        input-parameters:
          - carbon-from-cpu-util
          - carbon-from-mem-util
        output-parameter: carbon

With the metadata and plugins configured, you can start building your tree. The tree is where you organize your individual components according to the hierarchy you sketched out earlier. The root of the tree is where your total metrics aggregated across all the components will be calculated. The root has children, each of which is a component that can have children of its own. In this way, you can build up complicated architectures, with each component being a simple block of yaml data.

Remember, each component has this general structure:

component:
  pipeline:
    compute:
  defaults:
  inputs:
  

You simply add values to each of these fields to configure your component. The pipeline is where you can list out, in order, the plugins that you want to execute for this specific component. It is an array where each value must match an instance name from the plugins:initialize block you created earlier. For now, they can all go under compute - separating out plugins into phases is a more advanced topic we will cover in future modules.

The inputs field will contain the observations you have made for that component. You already scoped out what observations were available, here you can actually get those data and copy them into yaml format with a timestamp and duration. Every element in inputs must have a timestamp and duration.

A complete component might look as follows:

component-1:
  pipeline:
    compute:
      - plugin-instance-1
      - plugin-instance-2
      - plugin-instance-3
  inputs:
    - timestamp: "2023-12-12T00:00:01.000Z"
      duration: 5
      cpu/utilization: 20

One final piece of config to add to the IMP context is the aggregation config. This is where you define how exactly metrics are aggregated across your tree (as explained in the previous module).

aggregation:
  metrics:
    - carbon
  type: "both"

This means the carbon metric will be aggregated, and it will be aggregated across time and components.

You now have an executable IMP!

Executing the manifest

---

Executing your IMP is now simply a case of passing its path to if-run.

The command to run is: if-run -m imp.yml

This will run your IMP and display the outputs to the console.

You can also add -o output-file.yml  to redirect the outputs to a file instead of the console.

Visualizing and interpreting outputs files

---

You can now display your results using the Impact Framework visualizer. You’ll need to pass a URL to your file. There are several options for how to do this. The simplest is to host your file on a service like Github, and then pass the URL of the raw file to the visualizer. Alternatively, you could serve the file locally and pass the local URL to the visualizer.

The visualizer will display a donut chart with sections for each top level component in the tree. You can click into any segment. If the clicked segment has child components, the donut chart will now display the proportions of the clicked segment accounted for by each child. If a clicked segment has no children, then it will display a flyout with the granular information about that component instead.

If you prefer not to serve your own IMP you can use this prepared example:

https://viz.if.greensoftware.foundation/?url=https%3A%2F%2Fraw.githubusercontent.com%2FGreen-Software-Foundation%2Fif-db%2Frefs%2Fheads%2Fmain%2Fmanifests%2Fgsf-website%2Fgsf-website-output.yml

Sharing IMPs

---

An Impact Framework output file can be thought of as an executable audit. It includes your emissions and SCI estimates, but it also includes all the input data, configuration and model pipelines that were executed so anyone can inspect your work.

Not only that, but your output file can be run through Impact Framework to check that the results were calculated correctly. Let’s say you receive an output file from a peer and you want to check their working and suggest improvements. Step 1 is to check that their reported results truly resulted from executing the stated models over the given input data.

The tool for this is if-check .

You pass the file to if-check and it re-executes the file, verifying that the given results match the re-executed ones.

if-check -m outputs.yml

Next, you can inspect the model pipelines - do you agree with the models that were executed? Do you agree with the coefficient values used? If not, swap them out, rerun the IMP and send it back to the authors. In this way, environmental impact reporting becomes a discussion, rather than a one-way transfer of information. If the IMP is made public, this process can be used to crowdsource consensus for your environmental impact measurements.

---

Key Takeaways

---

• You learned how to design measurement investigation
• You learned about observations, proxies and models
• You learned how to view a file in the Impact Framework visualizer
• You gained some ideas for sharing and critiquing IMPs