Skip to content

MODEL_COMPARISON.md

Latest commit

 

History

History
304 lines (240 loc) · 7.84 KB

MODEL_COMPARISON.md

File metadata and controls

304 lines (240 loc) · 7.84 KB
Raw

HuggingFace Model Comparison for SMS Spam Detection

Quick Model Selection Guide

Choose Based on Priority:

Priority Model Code
🎯 Balanced Phi-3-mini HF_MODEL = 'microsoft/Phi-3-mini-4k-instruct'
⚡ Speed Llama 3.2 3B HF_MODEL = 'meta-llama/Llama-3.2-3B-Instruct'
🎓 Accuracy Mistral 7B HF_MODEL = 'mistralai/Mistral-7B-Instruct-v0.3'
📋 Instructions Zephyr 7B HF_MODEL = 'HuggingFaceH4/zephyr-7b-beta'

Detailed Comparison

1. microsoft/Phi-3-mini-4k-instruct ⭐ DEFAULT

Specifications:

  • Parameters: 3.8B
  • Context Length: 4,096 tokens
  • Average Response Time: 1-2 seconds
  • Memory Usage: ~8GB

Strengths:

  • ✅ Fast inference
  • ✅ Good balance of speed and accuracy
  • ✅ Works well with short prompts
  • ✅ Optimized for instruction following
  • ✅ Lower memory footprint

Weaknesses:

  • ⚠️ Smaller context window (4k vs 8k)
  • ⚠️ May struggle with very nuanced messages

Best Use Cases:

  • Quick spam detection
  • Real-time classification
  • Limited compute resources
  • Straightforward spam/ham decisions

Expected Performance on SMS Spam:

  • Accuracy: 93-95%
  • Precision: 90-93%
  • Recall: 88-92%
  • F1-Score: 89-92%

2. meta-llama/Llama-3.2-3B-Instruct

Specifications:

  • Parameters: 3B
  • Context Length: 8,192 tokens
  • Average Response Time: 0.8-1.5 seconds
  • Memory Usage: ~6GB

Strengths:

  • ✅ Fastest inference time
  • ✅ Larger context window (8k)
  • ✅ Latest Meta model
  • ✅ Efficient architecture
  • ✅ Low latency

Weaknesses:

  • ⚠️ Slightly lower accuracy than larger models
  • ⚠️ May require more prompt engineering

Best Use Cases:

  • High-volume processing
  • When speed is critical
  • Real-time applications
  • Batch processing large datasets

Expected Performance on SMS Spam:

  • Accuracy: 91-94%
  • Precision: 88-91%
  • Recall: 86-90%
  • F1-Score: 87-90%

3. mistralai/Mistral-7B-Instruct-v0.3

Specifications:

  • Parameters: 7B
  • Context Length: 8,192 tokens
  • Average Response Time: 2-3 seconds
  • Memory Usage: ~14GB

Strengths:

  • ✅ Higher accuracy
  • ✅ Better understanding of nuanced language
  • ✅ Handles complex scenarios well
  • ✅ Larger context window
  • ✅ Strong reasoning capabilities

Weaknesses:

  • ⚠️ Slower inference (2-3s vs 1-2s)
  • ⚠️ Higher memory requirements
  • ⚠️ May use more API quota

Best Use Cases:

  • Maximum accuracy required
  • Complex or ambiguous messages
  • Multilingual spam detection
  • When speed is not critical

Expected Performance on SMS Spam:

  • Accuracy: 94-97%
  • Precision: 92-95%
  • Recall: 90-94%
  • F1-Score: 91-94%

4. HuggingFaceH4/zephyr-7b-beta

Specifications:

  • Parameters: 7B
  • Context Length: 8,192 tokens
  • Average Response Time: 2-3 seconds
  • Memory Usage: ~14GB

Strengths:

  • ✅ Excellent instruction following
  • ✅ Good with structured prompts
  • ✅ Consistent output format
  • ✅ Well-tuned for chat/assistant tasks
  • ✅ Good reasoning

Weaknesses:

  • ⚠️ Slower inference
  • ⚠️ May be verbose (need token limits)
  • ⚠️ Higher memory usage

Best Use Cases:

  • When you need consistent output format
  • Complex multi-step reasoning
  • When using detailed prompts
  • Hybrid classification systems

Expected Performance on SMS Spam:

  • Accuracy: 93-96%
  • Precision: 91-94%
  • Recall: 89-93%
  • F1-Score: 90-93%

Performance vs Cost Trade-offs

Speed Ranking (Fastest to Slowest)

  1. 🥇 Llama-3.2-3B (0.8-1.5s)
  2. 🥈 Phi-3-mini (1-2s)
  3. 🥉 Mistral-7B (2-3s)
  4. 4️⃣ Zephyr-7b (2-3s)

Accuracy Ranking (Highest to Lowest)

  1. 🥇 Mistral-7B (94-97%)
  2. 🥈 Zephyr-7b (93-96%)
  3. 🥉 Phi-3-mini (93-95%)
  4. 4️⃣ Llama-3.2-3B (91-94%)

Cost Efficiency (API Calls per Day)

All models have the same free tier limit (~1,000 requests/day), but:

  • Faster models = More messages processed per day
  • Slower models = Fewer messages but potentially higher accuracy

Recommended Sample Sizes

Model Recommended LLM_SAMPLE_SIZE Processing Time Daily Capacity
Llama-3.2-3B 150-200 ~1.5 minutes Can run 5-6 times/day
Phi-3-mini 100-150 ~1 minute Can run 7-10 times/day
Mistral-7B 75-100 ~1.5 minutes Can run 10-13 times/day
Zephyr-7b 75-100 ~1.5 minutes Can run 10-13 times/day

Switching Models in the Notebook

Simply change one line in the configuration cell:

# Before running the LLM evaluation, change this line:
HF_MODEL = 'your-preferred-model'

Then re-run the LLM configuration and evaluation cells.

Testing Multiple Models

To compare all models, you can run the LLM evaluation cell multiple times:

# Model comparison loop (optional - will use significant API quota)
models_to_test = [
    'microsoft/Phi-3-mini-4k-instruct',
    'meta-llama/Llama-3.2-3B-Instruct',
    'mistralai/Mistral-7B-Instruct-v0.3',
    'HuggingFaceH4/zephyr-7b-beta'
]

LLM_SAMPLE_SIZE = 50  # Reduced for testing multiple models

for model_name in models_to_test:
    HF_MODEL = model_name
    client = InferenceClient(token=HF_API_KEY)
    print(f"\n{'='*60}")
    print(f"Testing: {model_name}")
    print(f"{'='*60}\n")
    
    # Run evaluation (paste LLM evaluation code here)
    # ... evaluation code ...
    
    time.sleep(10)  # Pause between models

⚠️ Warning: Testing all models will use 200 API calls (4 models × 50 samples).

Model Selection Decision Tree

Start Here
    │
    ├─ Need maximum speed? ──────────────────────► Llama-3.2-3B
    │
    ├─ Need balanced performance? ────────────────► Phi-3-mini ⭐
    │
    ├─ Need maximum accuracy? ────────────────────► Mistral-7B
    │
    └─ Need structured outputs? ──────────────────► Zephyr-7b

Prompt Optimization by Model

For Phi-3-mini (Default)

system_prompt = (
    "You are a strict SMS spam filter. Respond with ONLY one word: "
    "either 'Spam' or 'Ham'. No explanations."
)

✅ Works well - keep it simple

For Llama-3.2-3B

system_prompt = (
    "Classify this SMS message as 'Spam' or 'Ham'. "
    "Output only the classification word."
)

✅ Direct and concise prompts work best

For Mistral-7B

system_prompt = (
    "You are an expert SMS spam detection system. Analyze the message "
    "and respond with 'Spam' for unsolicited/fraudulent messages or "
    "'Ham' for legitimate messages. Output only one word."
)

✅ Can handle more detailed instructions

For Zephyr-7b

system_prompt = (
    "Task: SMS spam classification\n"
    "Input: An SMS message\n"
    "Output: Exactly one word - 'Spam' or 'Ham'\n"
    "Instructions: Classify as 'Spam' if unsolicited or fraudulent, "
    "otherwise 'Ham'."
)

✅ Structured prompts work best

Summary Recommendation

For most users: Start with Phi-3-mini (default)

  • Good balance of speed and accuracy
  • Proven performance on spam detection
  • Fast enough for interactive use
  • Low resource requirements

Upgrade to Mistral-7B if:

  • You need the highest possible accuracy
  • Speed is less important
  • You have complex/nuanced spam patterns

Switch to Llama-3.2-3B if:

  • Speed is your top priority
  • You're processing large volumes
  • You want to maximize daily throughput

Try Zephyr-7b if:

  • You need very consistent output formatting
  • You're building a hybrid system
  • You want strong reasoning capabilities

Additional Notes

  • All models support the same API interface
  • Switching is as simple as changing one variable
  • Performance may vary based on prompt engineering
  • Test with your specific dataset for best results
  • Monitor API usage across all models at https://huggingface.co/settings/tokens