Model FastFitTrainable not supported for text-classification when using other model

[SanneW7]

I tried changing the RoBERTa model to RobBERT (Dutch) and I got an error saying: "The model 'FastFitTrainable' is not supported for text-classification." This is really weird because text-classification is done in the example on your GitHub. I hope you can help me with the error.

Here is the code I use:

[nishkalavallabhi]

I get a warning message "The model 'FastFit' is not supported for text-classification" when I try to use a trained fastfit model for inference, but the model actually runs and shows me results.

[baptistecumin]

The difference between your two examples is if you use the model = trainer.train() vs model = FastFit.from_pretrained() method. I believe right now you must use the latter for it to work.

In the first case, the output is a

model.save_pretrained('fast-fit')
model = FastFit.from_pretrained('fast-fit') 
model

prints:

FastFit(
  (encoder): MPNetModel(
    (embeddings): MPNetEmbeddings(
      (word_embeddings): Embedding(30527, 768, padding_idx=1)
      (position_embeddings): Embedding(514, 768, padding_idx=1)
      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )

In the second case case, the output is a FastFitTrainable, which is not useable for inference.

model = trainer.train()
model

prints:

FastFitTrainable(
  (encoder): MPNetModel(
    (embeddings): MPNetEmbeddings(
      (word_embeddings): Embedding(30527, 768, padding_idx=1)
      (position_embeddings): Embedding(514, 768, padding_idx=1)
      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )

The code has a nice hint: fastfit just subclasses fastfittrainable and overrides the forward() function. Just do inf_model = FastFit(model.config), and you should be able to set up your model for inference as well.

[elronbandel]

Thanks @baptistecumin your answer is correct. I also added to the trainer an export_model function that produce the FastFitModel that is compatible with the classification pipeline.

If there is any Huggingface models expert in the crowed who can suggest a more elegant way to do it will be much apprecited.

[SanneW7]

Thanks a lot for the help and suggestions. It leaded me to the following solution:

def main():
    # Load datasets
    data_files = {"train": "clean_train24.csv", "test": "dev24.csv"}
    dataset = load_dataset('csv', data_files=data_files)
    dataset["validation"] = dataset["test"]
    dataset["train"] = sample_dataset(dataset["train"], label_column="class_label", num_samples_per_label=3, seed=44)
    print(dataset['train'])

    # iterable_dataset = dataset['train'].to_iterable_dataset()
    for row in dataset['train']:
      print(row)

    print("Trainen")
    trainer = FastFitTrainer(
        model_name_or_path="pdelobelle/robbert-v2-dutch-base",
        label_column_name="class_label",
        text_column_name="tweet_text",
        num_train_epochs=40,
        per_device_train_batch_size=32,
        per_device_eval_batch_size=64,
        max_text_length=128,
        dataloader_drop_last=False,
        num_repeats=4,
        optim="adafactor",
        clf_loss_factor=0.1,
        dataset=dataset,
    )

    model = trainer.train()
    trainer.export_model()
    model.save_pretrained("fast-fit")
    
    tokenizer = RobertaTokenizer.from_pretrained("pdelobelle/robbert-v2-dutch-base")
    model = FastFit.from_pretrained("fast-fit")
    classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)
    data = pd.read_csv('dev24.csv')
    predictions = []
    y_true = []

    for indx, row in data.iterrows():
        classification = classifier(row['tweet_text'])
        correct_label = row['class_label']
        predictions.append(classification[0]['label'])
        y_true.append(correct_label)
    print(predictions)

main()

[daboe01]

i have the same issue despite using FastFit and not FastFitTrainer.

from fastfit import FastFit
from transformers import AutoTokenizer, pipeline

model = FastFit.from_pretrained("fast-fit")
model

FastFit(
  (encoder): MPNetModel(
    (embeddings): MPNetEmbeddings(
      (word_embeddings): Embedding(30527, 768, padding_idx=1)
      (position_embeddings): Embedding(514, 768, padding_idx=1)
      (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
      (dropout): Dropout(p=0.1, inplace=False)
    )
    (encoder): MPNetEncoder(
      (layer): ModuleList(
        (0-11): 12 x MPNetLayer(
          (attention): MPNetAttention(
            (attn): MPNetSelfAttention(
              (q): Linear(in_features=768, out_features=768, bias=True)
              (k): Linear(in_features=768, out_features=768, bias=True)
              (v): Linear(in_features=768, out_features=768, bias=True)
              (o): Linear(in_features=768, out_features=768, bias=True)
              (dropout): Dropout(p=0.1, inplace=False)
            )
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
          (intermediate): MPNetIntermediate(
            (dense): Linear(in_features=768, out_features=3072, bias=True)
            (intermediate_act_fn): GELUActivation()
          )
          (output): MPNetOutput(
            (dense): Linear(in_features=3072, out_features=768, bias=True)
            (LayerNorm): LayerNorm((768,), eps=1e-05, elementwise_affine=True)
            (dropout): Dropout(p=0.1, inplace=False)
          )
        )
      )
      (relative_attention_bias): Embedding(32, 12)
    )
    (pooler): MPNetPooler(
      (dense): Linear(in_features=768, out_features=768, bias=True)
      (activation): Tanh()
    )
  )
  (projection): Linear(in_features=768, out_features=128, bias=False)
  (clf): Linear(in_features=768, out_features=17999, bias=True)
  (dropout): Dropout(p=0.1, inplace=False)
  (batch_norm): BatchNorm1d(768, eps=1e-05, momentum=0.1, affine=True, track_running_stats=True)
  (clf_criterion): CrossEntropyLoss()
  (sim_criterion): SupConLoss()
  (all_docs): ParameterList(
      (0): Parameter containing: [torch.int64 of size 17999x10]
      (1): Parameter containing: [torch.int64 of size 17999x10]
  )
)

tokenizer = AutoTokenizer.from_pretrained("sentence-transformers/paraphrase-mpnet-base-v2")
classifier = pipeline("text-classification", model=model, tokenizer=tokenizer)
print(classifier("Cataract"))

The model 'FastFit' is not supported for text-classification. Supported models are ['AlbertForSequenceClassification', 'BartForSequenceClassification', 'BertForSequenceClassification', 'BigBirdForSequenceClassification', 'BigBirdPegasusForSequenceClassification', 'BioGptForSequenceClassification', 'BloomForSequenceClassification', 'CamembertForSequenceClassification', 'CanineForSequenceClassification', 'LlamaForSequenceClassification', 'ConvBertForSequenceClassification', 'CTRLForSequenceClassification', 'Data2VecTextForSequenceClassification', 'DebertaForSequenceClassification', 'DebertaV2ForSequenceClassification', 'DistilBertForSequenceClassification', 'ElectraForSequenceClassification', 'ErnieForSequenceClassification', 'ErnieMForSequenceClassification', 'EsmForSequenceClassification', 'FalconForSequenceClassification', 'FlaubertForSequenceClassification', 'FNetForSequenceClassification', 'FunnelForSequenceClassification', 'GemmaForSequenceClassification', 'GPT2ForSequenceClassification', 'GPT2ForSequenceClassification', 'GPTBigCodeForSequenceClassification', 'GPTNeoForSequenceClassification', 'GPTNeoXForSequenceClassification', 'GPTJForSequenceClassification', 'IBertForSequenceClassification', 'JambaForSequenceClassification', 'JetMoeForSequenceClassification', 'LayoutLMForSequenceClassification', 'LayoutLMv2ForSequenceClassification', 'LayoutLMv3ForSequenceClassification', 'LEDForSequenceClassification', 'LiltForSequenceClassification', 'LlamaForSequenceClassification', 'LongformerForSequenceClassification', 'LukeForSequenceClassification', 'MarkupLMForSequenceClassification', 'MBartForSequenceClassification', 'MegaForSequenceClassification', 'MegatronBertForSequenceClassification', 'MistralForSequenceClassification', 'MixtralForSequenceClassification', 'MobileBertForSequenceClassification', 'MPNetForSequenceClassification', 'MptForSequenceClassification', 'MraForSequenceClassification', 'MT5ForSequenceClassification', 'MvpForSequenceClassification', 'NezhaForSequenceClassification', 'NystromformerForSequenceClassification', 'OpenLlamaForSequenceClassification', 'OpenAIGPTForSequenceClassification', 'OPTForSequenceClassification', 'PerceiverForSequenceClassification', 'PersimmonForSequenceClassification', 'PhiForSequenceClassification', 'Phi3ForSequenceClassification', 'PLBartForSequenceClassification', 'QDQBertForSequenceClassification', 'Qwen2ForSequenceClassification', 'Qwen2MoeForSequenceClassification', 'ReformerForSequenceClassification', 'RemBertForSequenceClassification', 'RobertaForSequenceClassification', 'RobertaPreLayerNormForSequenceClassification', 'RoCBertForSequenceClassification', 'RoFormerForSequenceClassification', 'SqueezeBertForSequenceClassification', 'StableLmForSequenceClassification', 'Starcoder2ForSequenceClassification', 'T5ForSequenceClassification', 'TapasForSequenceClassification', 'TransfoXLForSequenceClassification', 'UMT5ForSequenceClassification', 'XLMForSequenceClassification', 'XLMRobertaForSequenceClassification', 'XLMRobertaXLForSequenceClassification', 'XLNetForSequenceClassification', 'XmodForSequenceClassification', 'YosoForSequenceClassification'].