Skip to content

Latest commit

 

History

History
78 lines (63 loc) · 3.95 KB

README.md

File metadata and controls

78 lines (63 loc) · 3.95 KB

Llama-v2-7B-Chat: State-of-the-art large language model useful on a variety of language understanding and generation tasks

Llama 2 is a family of LLMs. The "Chat" at the end indicates that the model is optimized for chatbot-like dialogue. The model is quantized to w4a16(4-bit weights and 16-bit activations) and part of the model is quantized to w8a16(8-bit weights and 16-bit activations) making it suitable for on-device deployment. For Prompt and output length specified below, the time to first token is Llama-PromptProcessor-Quantized's latency and average time per addition token is Llama-TokenGenerator-KVCache-Quantized's latency.

This is based on the implementation of Llama-v2-7B-Chat found here. This repository contains scripts for optimized on-device export suitable to run on Qualcomm® devices. More details on model performance across various devices, can be found here.

Qualcomm AI Hub Models uses Qualcomm AI Hub Workbench to compile, profile, and evaluate this model. Sign up to run these models on a hosted Qualcomm® device.

Deploying Llama 2 on-device

Please follow the LLM on-device deployment tutorial.

Sample output prompts generated on-device

  1. --prompt "what is gravity?" --max-output-tokens 30
-------- Response Summary --------
Prompt: what is gravity?
Response: Hello! I'm here to help you answer your question. Gravity is a fundamental force of nature that affects the behavior of objects with mass
  1. --prompt "what is 2+3?" --max-output-tokens 30
-------- Response Summary --------
Prompt: what is 2+3?
Response: Of course! I'm happy to help! The answer to 2+3 is 5.
  1. --prompt "could you please write code for fibonacci series in python?" --max-output-tokens 100
-------- Response Summary --------
Prompt: could you please write code for fibonacci series in python?
Response: Of course! Here is an example of how you could implement the Fibonacci sequence in Python:
```
def fibonacci(n):
    if n <= 1:
        return n
    else:
        return fibonacci(n-1) + fibonacci(n-2)
```
You can test the function by calling it with different values of `n`, like this:
```
print(fibonacci(5))

License

  • The license for the original implementation of Llama-v2-7B-Chat can be found here.

References

Community

Usage and Limitations

This model may not be used for or in connection with any of the following applications:

  • Accessing essential private and public services and benefits;
  • Administration of justice and democratic processes;
  • Assessing or recognizing the emotional state of a person;
  • Biometric and biometrics-based systems, including categorization of persons based on sensitive characteristics;
  • Education and vocational training;
  • Employment and workers management;
  • Exploitation of the vulnerabilities of persons resulting in harmful behavior;
  • General purpose social scoring;
  • Law enforcement;
  • Management and operation of critical infrastructure;
  • Migration, asylum and border control management;
  • Predictive policing;
  • Real-time remote biometric identification in public spaces;
  • Recommender systems of social media platforms;
  • Scraping of facial images (from the internet or otherwise); and/or
  • Subliminal manipulation