Log regression task metrics in multitask model

[ntravis22]

Recently per-task metrics were added to multitask_model, however we did not include any for regression tasks and we did not check that the metric keys are present which can throw an error, so this addresses both of those concerns.

[MattGPT-ai]

Are there metrics for regression type models that we could put into scores, or do those perhaps go into the "classification report" - e.g. Pearson, spearman. In the regression models, results is written as

            eval_metrics = {
                "loss": eval_loss.item(),
                "mse": metric.mean_squared_error(),
                "mae": metric.mean_absolute_error(),
                "pearson": metric.pearsonr(),
                "spearman": metric.spearmanr(),
            }

So maybe we could either check for the base model class that defines evaluate, or just check for the keys. Then maybe we could write e.g. scores[(task_id, 'mse')]. What do you think?

[ntravis22]

Are there metrics for regression type models that we could put into scores, or do those perhaps go into the "classification report" - e.g. Pearson, spearman. In the regression models, results is written as

            eval_metrics = {
                "loss": eval_loss.item(),
                "mse": metric.mean_squared_error(),
                "mae": metric.mean_absolute_error(),
                "pearson": metric.pearsonr(),
                "spearman": metric.spearmanr(),
            }

So maybe we could either check for the base model class that defines evaluate, or just check for the keys. Then maybe we could write e.g. scores[(task_id, 'mse')]. What do you think?

Ok I added mse.

[MattGPT-ai]

Oh actually, can we just add all four of the metrics?

[ntravis22]

Oh actually, can we just add all four of the metrics?

Done

[alanakbik]

@ntravis22 @MattGPT-ai Could you paste a script to test this PR?

[ntravis22]

@alanakbik Here is a script to test:

from flair.data import Sentence, Corpus
from flair.embeddings import TransformerEmbeddings
from flair.models import TextRegressor
from flair.trainers import ModelTrainer
from flair.trainers.plugins.base import TrainerPlugin
from flair.nn.multitask import make_multitask_model_and_corpus


class MetricPlugin(TrainerPlugin):
    """Test plugin. In practice this could do something like logging metrics to WandB."""

    @TrainerPlugin.hook
    def metric_recorded(self, record):
        print(f"Metric: {record}")

sentences = [Sentence("This is a sentence") for _ in range(100)]
for sentence in sentences:
    sentence.add_label("regression_label", 1.0)
corpus = Corpus(sentences)
model = TextRegressor(TransformerEmbeddings('bert-base-uncased'), label_name="regression_label")

multitask_model, multicorpus = make_multitask_model_and_corpus([(model, corpus)])

trainer = ModelTrainer(multitask_model, multicorpus)
trainer.train('regression_model/', plugins=[MetricPlugin()])

Running this with the changes in this PR you can see lines printed like:

Metric: MetricRecord(dev/Task_0/mse at step 6, 1743547102.8555)
Metric: MetricRecord(dev/Task_0/mae at step 6, 1743547102.8556)
Metric: MetricRecord(dev/Task_0/pearson at step 6, 1743547102.8556)
Metric: MetricRecord(dev/Task_0/spearman at step 6, 1743547102.8556)