test-llama-v2-with-math.ipynb

{"metadata":{"kernelspec":{"language":"python","display_name":"Python 3","name":"python3"},"language_info":{"pygments_lexer":"ipython3","nbconvert_exporter":"python","version":"3.6.4","file_extension":".py","codemirror_mode":{"name":"ipython","version":3},"name":"python","mimetype":"text/x-python"}},"nbformat_minor":4,"nbformat":4,"cells":[{"source":"<a href=\"https://www.kaggle.com/code/gpreda/test-llama-v2-with-math?scriptVersionId=144057186\" target=\"_blank\"><img align=\"left\" alt=\"Kaggle\" title=\"Open in Kaggle\" src=\"https://kaggle.com/static/images/open-in-kaggle.svg\"></a>","metadata":{},"cell_type":"markdown"},{"cell_type":"markdown","source":"# Introduction  \n\n\nWe will use LlaMa v2 model to test if it can be used to perform simple math operations.\n\nThe model used is:\n\n* **Model**: Llama 2  \n* **Variation**: 7b-chat-hf  \n* **Version**: V1  \n* **Framework**: PyTorch  \n","metadata":{}},{"cell_type":"markdown","source":"# Imports and utils","metadata":{}},{"cell_type":"code","source":"from transformers import AutoTokenizer\nimport transformers\nimport torch\nimport warnings\nfrom time import time\nwarnings.filterwarnings('ignore')\n!pip install xformers","metadata":{"_uuid":"8f2839f25d086af736a60e9eeb907d3b93b6e0e5","_cell_guid":"b1076dfc-b9ad-4769-8c92-a6c4dae69d19","_kg_hide-input":true,"_kg_hide-output":true,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def load_model_tokenize_create_pipeline():\n    \"\"\"\n    Load the model\n    Create a \n    Args\n    Returns:\n        tokenizer\n        pipeline\n    \"\"\"\n    # adapted from https://huggingface.co/blog/llama2#using-transformers\n    time_1 = time()\n    model = \"/kaggle/input/llama-2/pytorch/7b-chat-hf/1\"\n    tokenizer = AutoTokenizer.from_pretrained(model)\n    time_2 = time()\n    print(f\"Load model and init tokenizer: {round(time_2-time_1, 3)}\")\n    pipeline = transformers.pipeline(\n        \"text-generation\",\n        model=model,\n        torch_dtype=torch.float16,\n        device_map=\"auto\",)\n    time_3 = time()\n    print(f\"Prepare pipeline: {round(time_3-time_2, 3)}\")\n    return tokenizer, pipeline","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"code","source":"def test_model(tokenizer, pipeline, prompt_to_test):\n    \"\"\"\n    Perform a query\n    print the result\n    Args:\n        tokenizer: the tokenizer\n        pipeline: the pipeline\n        prompt_to_test: the prompt\n    Returns\n        None\n    \"\"\"\n    # adapted from https://huggingface.co/blog/llama2#using-transformers\n    time_1 = time()\n    sequences = pipeline(\n        prompt_to_test,\n        do_sample=True,\n        top_k=10,\n        num_return_sequences=1,\n        eos_token_id=tokenizer.eos_token_id,\n        max_length=200,)\n    time_2 = time()\n    print(f\"Test inference: {round(time_2-time_1, 3)}\")\n    for seq in sequences:\n        print(f\"Result: {seq['generated_text']}\")","metadata":{"_kg_hide-output":false,"_kg_hide-input":false,"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Initialize the model, tokenizer and create a pipeline","metadata":{}},{"cell_type":"code","source":"tokenizer, pipeline = load_model_tokenize_create_pipeline()","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"After we initialized the model and the tokenizer, we created a pipeline. Creatin of the pipeline takes the longest time.\nNow let's test the model with few mathematical prompts.","metadata":{}},{"cell_type":"markdown","source":"# Can LlaMa v2 do simple math?","metadata":{}},{"cell_type":"markdown","source":"## Prompt #1: Perform a simple sum","metadata":{}},{"cell_type":"code","source":"prompt_to_test = 'Prompt: Adrian has three apples. His sister Anne has ten apples more than him. How many apples has Anne?'\ntest_model(tokenizer, pipeline, prompt_to_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prompt #2: Ask for the area of a circle, giving the radius","metadata":{}},{"cell_type":"code","source":"prompt_to_test = 'Prompt: A circle has the radius 5. What is the area of the circle?'\ntest_model(tokenizer, pipeline, prompt_to_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"## Prompt #3: Calculate an equation with two unknowns","metadata":{}},{"cell_type":"code","source":"prompt_to_test = 'Prompt: Anne and Adrian have a total of 10 apples. Anne has 2 apples more than Adrian.\\\nHow many apples has each of the children Anne and Adrian?'\ntest_model(tokenizer, pipeline, prompt_to_test)","metadata":{"trusted":true},"execution_count":null,"outputs":[]},{"cell_type":"markdown","source":"# Conclusions\n\n\nAfter we initialized the model and the tokenizer, which took on GPU T4 x2 at the first run under 200 sec. (this time might be variable), then each prompt only took less than 10 sec. \n\nThe simple math questions are answered sometime correct, sometime incorrectly. \n\nIt might be related to the temperature factor (that is not set here explicitly).","metadata":{}}]}