Docs: clarify deprecated status and Llama 2-only scope in README

.github/workflows/tests.yml

@@ -0,0 +1,23 @@
+name: tests
+
+on:
+  push:
+  pull_request:
+
+jobs:
+  pytest:
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout
+        uses: actions/checkout@v4
+      - name: Set up Python
+        uses: actions/setup-python@v5
+        with:
+          python-version: '3.10'
+      - name: Install dependencies
+        run: |
+          python -m pip install --upgrade pip
+          pip install -r requirements.txt
+          pip install pytest
+      - name: Run tests
+        run: pytest -q

README.md

@@ -1,3 +1,15 @@
+## Fork additions (evaluation + CI)
+
+This repository is a fork of Meta’s Llama inference code. I added a lightweight NLP evaluation layer on top of text generation to support quick, reproducible regression checks:
+- Reference-based metrics: BLEU, ROUGE-L, and simple diversity measures
+- A minimal unit test suite to validate evaluation behavior
+- A minimal GitHub Actions workflow for automated checks (fork workflows may require maintainer approval upstream)
+
+Where to look:
+- `example_text_completion.py` (adds `--reference` + metric reporting)
+- README diff summary near the bottom (documents the evaluation layer)
+- Upstream PR: #1408 (documentation + eval utilities/tests/CI notes)
+
 ## **Note of deprecation**
 
 Thank you for developing with Llama models. As part of the Llama 3.1 release, we’ve consolidated GitHub repos and added some additional repos as we’ve expanded Llama’s functionality into being an e2e Llama Stack. Please use the following repos going forward:
@@ -95,6 +107,18 @@ torchrun --nproc_per_node 1 example_text_completion.py \
     --max_seq_len 128 --max_batch_size 4
 ```
 
+#### Optional evaluation metrics
+
+`example_text_completion.py` accepts an optional `--reference` string to evaluate generations with simple BLEU, ROUGE-L, and diversity metrics. For example:
+
+```
+torchrun --nproc_per_node 1 example_text_completion.py \
+    --ckpt_dir llama-2-7b/ \
+    --tokenizer_path tokenizer.model \
+    --max_seq_len 128 --max_batch_size 4 \
+    --reference="The meaning of life is to learn and grow."
+```
+
 ### Fine-tuned Chat Models
 
 The fine-tuned models were trained for dialogue applications. To get the expected features and performance for them, a specific formatting defined in [`chat_completion`](https://github.com/facebookresearch/llama/blob/main/llama/generation.py#L212)
@@ -140,3 +164,30 @@ For common questions, the FAQ can be found [here](https://ai.meta.com/llama/faq/
 
 ## Original Llama
 The repo for the original llama release is in the [`llama_v1`](https://github.com/facebookresearch/llama/tree/llama_v1) branch.
+
+
+diff --git a/README.md b/README.md
+index 0000000..1111111 100644
+--- a/README.md
++++ b/README.md
+@@ -1,6 +1,26 @@
+-Note of deprecation
++## Note of deprecation
+ Thank you for developing with Llama models. As part of the Llama 3.1 release, we’ve consolidated GitHub repos and added some additional repos as we’ve expanded Llama’s functionality into being an e2e Llama Stack. Please use the following repos going forward:
+
+ llama-models - Central repo for the foundation models including basic utilities, model cards, license and use policies
+ PurpleLlama - Key component of Llama Stack focusing on safety risks and inference time mitigations
+ llama-toolchain - Model development (inference/fine-tuning/safety shields/synthetic data generation) interfaces and canonical implementations
+ llama-agentic-system - E2E standalone Llama Stack system, along with opinionated underlying interface, that enables creation of agentic applications
+ llama-cookbook - Community driven scripts and integrations
+ If you have any questions, please feel free to file an issue on any of the above repos and we will do our best to respond in a timely manner.
+
+ Thank you!
+
+**Important:** This repository is **deprecated** and is maintained primarily for historical reference and minimal Llama 2 example code.  
+For current Llama releases and the end-to-end Llama Stack, please use the repositories listed above.
+
+**Scope note:** The instructions and examples below apply to **Llama 2** only and may not reflect the latest best practices for newer releases.
+
+ (Deprecated) Llama 2
+ We are unlocking the power of large language models. Llama 2 is now accessible to individuals, creators, researchers, and businesses of all sizes so that they can experiment, innovate, and scale their ideas responsibly.

example_text_completion.py

@@ -2,9 +2,11 @@
 # This software may be used and distributed according to the terms of the Llama 2 Community License Agreement.
 
 import fire
+from typing import List, Optional
 
 from llama import Llama
-from typing import List
+from llama.nlp_metrics import evaluate
+
 
 def main(
     ckpt_dir: str,
@@ -14,6 +16,7 @@ def main(
     max_seq_len: int = 128,
     max_gen_len: int = 64,
     max_batch_size: int = 4,
+    reference: Optional[str] = None,
 ):
     """
     Entry point of the program for generating text using a pretrained model.
@@ -22,13 +25,12 @@ def main(
         ckpt_dir (str): The directory containing checkpoint files for the pretrained model.
         tokenizer_path (str): The path to the tokenizer model used for text encoding/decoding.
         temperature (float, optional): The temperature value for controlling randomness in generation.
-            Defaults to 0.6.
         top_p (float, optional): The top-p sampling parameter for controlling diversity in generation.
-            Defaults to 0.9.
         max_seq_len (int, optional): The maximum sequence length for input prompts. Defaults to 128.
         max_gen_len (int, optional): The maximum length of generated sequences. Defaults to 64.
         max_batch_size (int, optional): The maximum batch size for generating sequences. Defaults to 4.
-    """ 
+        reference (str, optional): Optional reference text for NLP evaluation (BLEU, ROUGE, diversity).
+    """
     generator = Llama.build(
         ckpt_dir=ckpt_dir,
         tokenizer_path=tokenizer_path,
@@ -37,33 +39,42 @@ def main(
     )
 
     prompts: List[str] = [
-        # For these prompts, the expected answer is the natural continuation of the prompt
         "I believe the meaning of life is",
         "Simply put, the theory of relativity states that ",
         """A brief message congratulating the team on the launch:
 
         Hi everyone,
-        
+
         I just """,
-        # Few shot prompt (providing a few examples before asking model to complete more);
         """Translate English to French:
-        
+
         sea otter => loutre de mer
         peppermint => menthe poivrée
         plush girafe => girafe peluche
         cheese =>""",
     ]
+
     results = generator.text_completion(
         prompts,
         max_gen_len=max_gen_len,
         temperature=temperature,
         top_p=top_p,
     )
+
     for prompt, result in zip(prompts, results):
+        generation = result["generation"]
+
         print(prompt)
-        print(f"> {result['generation']}")
+        print(f"> {generation}")
+
+        if reference:
+            metrics = evaluate(generation, reference)
+            print("\n=== NLP Evaluation ===")
+            print(metrics.pretty())
+
         print("\n==================================\n")
 
 
 if __name__ == "__main__":
     fire.Fire(main)
+

llama/nlp_metrics.py

@@ -0,0 +1,120 @@
+# llama/nlp_metrics.py
+from __future__ import annotations
+
+import math
+import re
+from collections import Counter
+from dataclasses import dataclass
+from typing import Iterable, List, Tuple
+
+
+_WORD_RE = re.compile(r"\b\w+\b", re.UNICODE)
+
+
+def tokenize(text: str) -> List[str]:
+    # simple, robust tokenization without extra deps
+    return _WORD_RE.findall(text.lower())
+
+
+def ngrams(tokens: List[str], n: int) -> List[Tuple[str, ...]]:
+    if n <= 0:
+        return []
+    return [tuple(tokens[i : i + n]) for i in range(0, max(0, len(tokens) - n + 1))]
+
+
+def distinct_n(text: str, n: int = 2) -> float:
+    toks = tokenize(text)
+    ng = ngrams(toks, n)
+    if not ng:
+        return 0.0
+    return len(set(ng)) / len(ng)
+
+
+def repetition_rate(text: str, n: int = 3) -> float:
+    toks = tokenize(text)
+    ng = ngrams(toks, n)
+    if not ng:
+        return 0.0
+    c = Counter(ng)
+    repeated = sum(v for v in c.values() if v > 1)
+    return repeated / len(ng)
+
+
+def _lcs_len(a: List[str], b: List[str]) -> int:
+    # classic DP LCS length, O(len(a)*len(b)) – fine for short eval strings
+    dp = [0] * (len(b) + 1)
+    for i in range(1, len(a) + 1):
+        prev = 0
+        for j in range(1, len(b) + 1):
+            cur = dp[j]
+            if a[i - 1] == b[j - 1]:
+                dp[j] = prev + 1
+            else:
+                dp[j] = max(dp[j], dp[j - 1])
+            prev = cur
+    return dp[-1]
+
+
+def rouge_l_f1(hypothesis: str, reference: str) -> float:
+    h = tokenize(hypothesis)
+    r = tokenize(reference)
+    if not h or not r:
+        return 0.0
+    lcs = _lcs_len(h, r)
+    prec = lcs / len(h)
+    rec = lcs / len(r)
+    if prec + rec == 0:
+        return 0.0
+    return (2 * prec * rec) / (prec + rec)
+
+
+def _modified_precision(h: List[str], r: List[str], n: int) -> float:
+    h_ngrams = Counter(ngrams(h, n))
+    r_ngrams = Counter(ngrams(r, n))
+    if not h_ngrams:
+        return 0.0
+    clipped = {k: min(v, r_ngrams.get(k, 0)) for k, v in h_ngrams.items()}
+    return sum(clipped.values()) / sum(h_ngrams.values())
+
+
+def bleu(hypothesis: str, reference: str, max_n: int = 4, smooth: float = 1e-9) -> float:
+    h = tokenize(hypothesis)
+    r = tokenize(reference)
+    if not h or not r:
+        return 0.0
+
+    # brevity penalty
+    bp = 1.0 if len(h) > len(r) else math.exp(1 - (len(r) / max(1, len(h))))
+
+    # geometric mean of modified n-gram precisions (with smoothing)
+    log_p_sum = 0.0
+    for n in range(1, max_n + 1):
+        p = _modified_precision(h, r, n)
+        log_p_sum += math.log(max(p, smooth))
+
+    return bp * math.exp(log_p_sum / max_n)
+
+
+@dataclass
+class NLPMetrics:
+    bleu4: float
+    rouge_l_f1: float
+    distinct2: float
+    repetition3: float
+
+    def pretty(self) -> str:
+        return (
+            f"BLEU-4: {self.bleu4:.3f} | "
+            f"ROUGE-L(F1): {self.rouge_l_f1:.3f} | "
+            f"Distinct-2: {self.distinct2:.3f} | "
+            f"Repetition-3: {self.repetition3:.3f}"
+        )
+
+
+def evaluate(generated: str, reference: str) -> NLPMetrics:
+    return NLPMetrics(
+        bleu4=bleu(generated, reference, max_n=4),
+        rouge_l_f1=rouge_l_f1(generated, reference),
+        distinct2=distinct_n(generated, n=2),
+        repetition3=repetition_rate(generated, n=3),
+    )

tests/conftest.py

@@ -0,0 +1,4 @@
+from pathlib import Path
+import sys
+
+sys.path.insert(0, str(Path(__file__).resolve().parents[1]))

tests/test_nlp_metrics.py

@@ -0,0 +1,41 @@
+import math
+
+from llama.nlp_metrics import (
+    bleu,
+    distinct_n,
+    evaluate,
+    repetition_rate,
+    rouge_l_f1,
+)
+
+
+def test_bleu_smoke():
+    score = bleu("the cat sat on the mat", "the cat sat on the mat")
+    assert 0.0 <= score <= 1.0
+    assert not math.isnan(score)
+
+
+def test_rouge_l_f1_smoke():
+    score = rouge_l_f1("the cat sat on the mat", "the cat sat on the mat")
+    assert 0.0 <= score <= 1.0
+    assert not math.isnan(score)
+
+
+def test_distinct_n_smoke():
+    score = distinct_n("one two three four five", n=2)
+    assert 0.0 <= score <= 1.0
+    assert not math.isnan(score)
+
+
+def test_repetition_rate_smoke():
+    score = repetition_rate("repeat repeat repeat repeat", n=2)
+    assert 0.0 <= score <= 1.0
+    assert not math.isnan(score)
+
+
+def test_evaluate_smoke():
+    metrics = evaluate("the cat sat on the mat", "the cat sat on the mat")
+    assert 0.0 <= metrics.bleu4 <= 1.0
+    assert 0.0 <= metrics.rouge_l_f1 <= 1.0
+    assert 0.0 <= metrics.distinct2 <= 1.0
+    assert 0.0 <= metrics.repetition3 <= 1.0