Function Calling Playground

Description

This repository is a little playground experimenting with function calling and IBM granite models.

Function calling is a technique that allows Large Language Models (LLMs), such as the IBM Granite-3.0-8B-Instruct model, to integrate with external tools and access information beyond their training data. This capability addresses the inherent limitations of LLMs in terms of knowledge and reasoning, as they generate responses based on the data they were trained on.

Some key aspects of function calling are:

• Overcoming Knowledge Limitations: Traditional LLMs are restricted by the data they were trained on. Function calling enables them to retrieve missing information by invoking external tools.
• Adapting to Personalized Queries: By using external tools, LLMs can better respond to personalized user queries that require up-to-date or specific data not available in their training sets.
• Avoiding Extensive Fine-Tuning: Function calling offers a way to adapt LLMs to specific use cases without the need for extensive fine-tuning of the foundation model for each individual scenario. Thus, CPU-based inferencing becomes a good option.

Usage

Structure and configuration

The repository consists of two parts:

• setup contains all Ansible-related files, which help you setting up this little demo.
• src contains the actual application. I you like to add further tools, simply add a well defined and documented function to the tools.py module. The main application will automatically integrate and provide it to the LLM.

If you want to use the demo as it is, please get a weather API key here as well as a stock price API key from here. Both need to be passed as environment variables to the application. You can insert them into the provided inventory.yml (two placeholders already exist).

If you extend the tools.py module and want to pass values as environment variables, simply add them as key-value pairs to the application_environment map alongside the other two API keys. Everything under application_environment gets passed as environment variable to the application.

Setting up the demo using Ansible

To deploy the demo on a Power10 LPAR, adjust the inventory.yml accordingly (hostname/IP, password, user, etc.). On your local machine, install Ansible in a Python environment:

$ python -m pip install ansible

Next, download existing playbooks from our main repository (we'd like to avoid duplicates):

$ ansible-playbook setup/local-setup,yml

You'll notice that three new files exist in the setup directory. Lastly, run the main playbook to set everything up:

$ ansible-playbook -i setup/inventory.yml setup/entrypoint-playbook.yml

The demo should be available under http://<HOSTNAME>:<PORT> as specified in the inventory.yml.