Functioning of the Macro object

[RobinPicard]

We want to replace the Outlines function by a new object called Macro that will encapsulate a model, a prompt template and an output type (see #1346 and #1454). It would look like the following:

from pydantic import BaseModel
from outlines import from_transformers
from outlines.templates import Template
from outlines.types import JsonType
from transformers import AutoModelForCausalLM, AutoTokenizer

class OutputModel(BaseModel):
    result: int

model = from_transformers(
    AutoModelForCausalLM.from_pretrained("WizardLM/WizardMath-7B-V1.1"),
    AutoTokenizer.from_pretrained("WizardLM/WizardMath-7B-V1.1")
)
template = Template.from_str("What is 2 times {{ num }}?")
output_type = JsonType(OutputModel)

macro = Macro(model, template, output_type)

result = macro(3)

There are 3 issues with that implementation:

• a macro cannot be modified after having been instantiated. Ideally we would like to be able to do macro.with_model(...) to be able to change the model used by the macro while keeping the same output type and prompt template
• it requires already having a model instance to instantiate a macro, meaning that we cannot rely on lazy loading and need to have the model loading before the macro is actually called
• it does not allow to load a pre-defined model instead of having to redefine one each time. Ideally we would like to be able to associate a model to a name and then to use that name as an alias (the model would be saved locally for instance). We could imagine the following cli: outlines <macro_name> --with-model <name-of-predefined-model>

A solution considered is to ask the user to provide a function that returns a model instead of a model instance. That way we could rely on lazy loading and we could save the code of the function to be able to create a predefined model.

3 questions:

• what do you think about the solution above?
• do you have ideas of better alternatives?
• can you see other constraints/features that should be supported for macros?