Fix for issue #19933: langchain_huggingface

[Soumil32]

This partially fixes issue #19933 in which Langchain was not taking the taking steps to apply the correct formatting for different chat models.

The change involves passing the prompt in the ChatML format of the pipeline. This way the pipeline can automatically format accordingly.

In addition, the argument return_full_text=False is being passed into the pipeline in huggingace_pipeline.py. This causes the pipeline to only return the newly generated text whereas the default behaviour is to return the entire chat history which we do not need. The output can then be parsed back into an AIMessage.

This still only works when ChatHuggingFace is used. If either model_id or tokenizer are not set when initialising ChatHuggingFace , it defaults to gpt2 tokenizer which is what caused the issue in the first place. The changes I have made correctly handle parsing the model response back into AIMessage. The initial issue still might need to be investigated but doing this should work as the tokenizer should be picked up from the pipeline passed into HuggingFacePipeline.

from langchain_huggingface import ChatHuggingFace, HuggingFacePipeline
from langchain_core.messages import HumanMessage, SystemMessage
from transformers import pipeline, AutoTokenizer, AutoModelForCausalLM

model_id = "hugging-quants/Meta-Llama-3.1-8B-Instruct-BNB-NF4" #replace with your model
tokenizer = AutoTokenizer.from_pretrained(model_id)
terminators = [
    tokenizer.eos_token_id,
    tokenizer.convert_tokens_to_ids("<|eot_id|>")
]
model = AutoModelForCausalLM.from_pretrained(
  model_id,
  torch_dtype=torch.bfloat16,
  low_cpu_mem_usage=True,
  device_map="auto",
)

pipe = pipeline(
    "text-generation", model=model, tokenizer=tokenizer, max_new_tokens=40, eos_token_id=terminators, pad_token_id=tokenizer.eos_token_id
)

messages = [
    SystemMessage(content="You are a translator which converts the users input from English to French. Never refer to yourself. Just provide the translated text."),
    HumanMessage(content="Hi, how are you?")
]


hf = HuggingFacePipeline(pipeline=pipe)
chat_model = ChatHuggingFace(llm=hf, model_id=model_id)

response = chat_model.invoke(messages)

print(response)

Sorry if my language isn't clear. It is my first time contributing to any OSS project. If you have any questions or clarification is need, please say so! 😊

Edit: Did some more testing and my changes also fix the iisue the issue all together instead of partially! model_id or tokenizer do not need to be set anymore.

[vercel]

The latest updates on your projects. Learn more about Vercel for Git ↗︎

1 Skipped Deployment

Name	Status	Preview	Comments	Updated (UTC)
langchain	⬜️ Ignored (Inspect)	Visit Preview		Aug 12, 2024 7:02pm

[Soumil32]

@efriis whats the status on this?

[Soumil32]

This also seems to fix issue #24437, #22487

[Soumil32]

@ccurme Is there anything i still need to do?

[ccurme]

Could we add some tests to demonstrate the updated behavior?

[ccurme]

Can we do this in a separate issue / PR?