Simple API to run load and evaluate datasets based on tasks and input data

[yoavkatz]

We need a simple way for a user to load a Unitxt dataset based on existing files or data structures.
This is required for ilab (@Roni-Friedman ) and also for other use cases.

I thought that maybe a dataframe is a common interface that everyone knows.

load_dataset(task="tasks.classification.multi_class", test_set = pd.from_csv("test.csv"))
load_dataset(task="tasks.classification.multi_class", test_set = pd.from_csv("test.csv"), train_set=pd.from_csv("train.csv" )

And

evaluate(predications, task="tasks.classification.multi_class", test_set = pd.from_csv("test.csv"))

The assumption is that the fields of input dataframe are the same as the task, and a clear error message will be presented if not.

[elronbandel]

Is unitxt.api.produce not enough? you could use it with produce(pd.from_csv("test.csv").to_list(), task="your_task")

unitxt/src/unitxt/api.py

Lines 132 to 139 in 582d96f

	def produce(instance_or_instances, dataset_query: Optional[str] = None, **kwargs):
	is_list = isinstance(instance_or_instances, list)
	if not is_list:
	instance_or_instances = [instance_or_instances]
	result = _get_produce_with_cache(dataset_query, **kwargs)(instance_or_instances)
	if not is_list:
	result = result[0]
	return result

[yoavkatz]

I did not even think about produce. I'll check. It has to be more prominently documented (Currently it's under "Production and Inference").

[yoavkatz]

I tried:

Set up question answer pairs in a dictionary

data = [
{"question": "What is the capital of Texas?", "answers": ["Austin"]},
{"question": "What is the color of the sky?", "answers": ["Blue"]},
]

test_dataset = produce(data,task="tasks.qa.open", template="templates.qa.open.title")

But the output dataset does not contain references field, so it can not be evaluated.

[elronbandel]

You are right. I forgot it was meant for inference. We should document it properly.