Encoding version information in the KGO

[alejandrobodas]

In the PMC on 29 May 2020, there was some discussion about encoding the KGO with some version information. It was agreed that it was a nice idea only if it can be done automatically by the CI test. This issue will explore the possibility of automatically updating that information to the KGO when the CI test fails.

[mvdebolskiy]

Would it be beneficial to use git lfs for the creation and versioning of the KGO files within the workflows? The files are not that big and can be stored in a separate repo that will be cloned only by a script so that bandwidth and data quotas are not used up that much.

[alejandrobodas]

When I looked into this I concluded that we would hit the bandwidth limit easily, but I may not be understanding how it works.
Please can you give more details of your proposed solution?

[mvdebolskiy]

I just assumed that testing is only needed for contributors and action runs and given the size of all datasets is around 100 Mb, unless you get >10 answer-changing PRs a month - the bandwidth quota won't be met. Especially, if the testing description is moved to the wiki for developers, so the general users won't use it much.

[alejandrobodas]

Currently, the test datasets are downloaded every time the CI tests are triggered (push, pull request), so my understanding is that the bandwidth limit will be hit after 10 tests. If my understanding is correct, then we would have hit the bandwidth limit during the last couple of weeks.

[mvdebolskiy]

Oh. Ok. Then what can be done is: in the workflows and in' driver/download_test_data.sh only a json file containing a list of *.nc files in a google folder (a google apps script can manage that). Your google folder will contain all the KGO files with different versions (I can show you how to set up the google apps scripts). In this case, there will be no need to update the workflows with new google links every time there is an answer-changing PR.

With regards to the versioning of the files, instead of vXXX, the filenames can contain creation date and tag/commit they were created with and in download_test_data.sh that part would be an optional argument if for some reason the older versions of KGO are needed.

[alejandrobodas]

Sorry for my slow reply. That sounds like a nice approach. I'm not familiar with google apps scripts, would you be able to provide guidance on how to to do that? For the moment, I'd like to keep the vXXX versioning, and perhaps move to date/tag in the future.

[mvdebolskiy]

You can add a script at https://script.google.com/home/start (documentation).
Before you write a script:

1. Make a folder with *.nc files.
2. Make an empty .txt file which would contain the list of *.nc files in the folder. It can also be .json file
3. Get the .txt file Id and folder Id.
4. Example java_script function:

function write_nc_list() {
  var listfile = DriveApp.getFileById('.<txt file id>')
  var folder = DriveApp.getFolderById('<folder id>'); 
  var list = [];
  list.push(['Name','ID','Size']); // comment this out if you do not need the header
  var files = folder.getFiles(); //
  while (files.hasNext()){
    file = files.next();
    var row = []
    var fname = file.getName()
    if (fname.includes('.nc')) {
      row.push(file.getName(),file.getId(),file.getSize())
      list.push(row);
    }
  }
  listfile.setContent(JSON.stringify(list))
}

5. This function will fill your .txt with JSON that has filenames, id's size (you can add other entries f.e. date modified) which can later be read in the github workflow or download_test_data.sh to construct download links for curl. There are also options to make triggers for the script (f.e. run it every day) or just execute the script manually once you add new kgo to the folder.
   This way you can have the only the .txt file link hardcoded in the workflows. If you have any trouble, we can arrange a zoom.