To extend the mapping capabilities of metadata.json to have very specific features for each templates

[guptaaryan16]

Clear and concise description of the problem

Right now, the metadata.json works like this

{
	training”:{
		…Options
	}

}

And we just import all these options and assume that all templates are going to have them.

This causes two specific issues, as given below

• We are not able to choose specific metadata and training options(maybe sub options) for each template
• We can't configure special options for some templates like having specific options for text classification template for question-answering, text summarisation etc.

Suggested solution

We propose two approaches to solve this problem, namely

• We can add a templates sub option for each training option which contains all templates for that specific training option and check it during rendering of the templates. This works as follows:

[Changes in metadata.json]
"deterministic": {
     "name": "deterministic",
     "type": "checkbox",
     "description": "Should the training be deterministic?",
 +   "templates": ['vision-classification', 'text-classification', ..]
   }

[Changes in TabTraining.vue(and other Vue components)]
<div v-if="deterministic.templates.includes(store.config.template)">

Then we can selectively choose which option will be available for each template. This approach makes it easier to track each template and option. The only problem of this approach seems to be hard for us to specify an option like a specific argparser or backend.

• We can configure data_options.json file in the templates. This file will contain all options related to each template that can may contain sub options check as well. This approach makes more sense while contributing new templates as you need to add all options you want to configure directly.
  This will require the following changes

[New file templateOptions.json]
{
   "templates": 
   {
       "template-vision-classification": 
       {
           "training": [
               "argparser", 
               "deterministic", 
               "torchrun",
               "spawn",
               "nproc_per_node",
               "nnodes",
               "master_addr",
               "master_port",
               "backend"
           ]
}

[Changes in TabTraining.vue(and other Vue components)]
<FormCheckbox
     :label="deterministic.description"
     :saveKey="deterministic.name"
     v-show="trainingOptions.includes('deterministic')"
   />

const trainingOptions = ref(templates[store.config.template]["training"])

This approach seems a bit complex but can provide more control over the options and templates. Also it can help introduce template specific features like having specific evaluation functions and other metadata options.

Additional context

This issue was discuss in our weekly meeting this week.

cc @vfdev-5 @theory-in-progress

[vfdev-5]

@guptaaryan16 thanks a lot for expanded "Suggested solution" section. I would prefer the 2nd option