Skip to content

summary.en.md

Latest commit

 

History

History
95 lines (79 loc) · 7.84 KB

summary.en.md

File metadata and controls

95 lines (79 loc) · 7.84 KB

OWASP Top 10 for LLMs

Disclaimer: This is a personal summary and interpretation based on a YouTube video. It is not official material and not endorsed by the original creator. All rights remain with the respective creators.

This document summarizes the key takeaways from the video. I highly recommend watching the full video for visual context and coding demonstrations.

Before You Get Started

  • I summarize key points to help you learn and review quickly.
  • Simply click on Ask AI links to dive into any topic you want.

AI-Powered buttons

Teach Me: 5 Years Old | Beginner | Intermediate | Advanced | (reset auto redirect)

Learn Differently: Analogy | Storytelling | Cheatsheet | Mindmap | Flashcards | Practical Projects | Code Examples | Common Mistakes

Check Understanding: Generate Quiz | Interview Me | Refactor Challenge | Assessment Rubric | Next Steps

Introduction to OWASP Top 10 for LLMs

The speaker kicks off by outlining the OWASP Top 10 vulnerabilities for Large Language Models (LLMs), noting the rapid evolution of AI and the need for secure architectures. They share their background in app security and highlight real-world headlines, hype, and actual attacks like data leaks in tools such as ChatGPT and Copilot. OWASP's resources, including the Gen AI Security Project, are recommended for governance, red teaming, and secure implementation.

  • Key Takeaway: The Top 10 includes familiar issues like prompt injection (similar to SQL injection), excessive agency, and misinformation, emphasizing that AI security builds on traditional concepts but adapts to LLM specifics.
  • Link for More Details: Ask AI: OWASP Top 10 for LLMs

Architecture of LLM Applications

Early LLM apps typically featured a front-end, application code interacting with an LLM, and embeddings stored in vector databases for proprietary data. Now, agentic AI (essentially LLM-powered automation) dominates, using web fronts, control planes, message buses, and agents that handle tasks autonomously via prompts. Model Context Protocol (MCP) enables LLMs to interact with external systems, but it raises concerns about unverified servers and rapid adoption.

  • Key Takeaway: Shift from simple embeddings to event-driven architectures with agents requires careful evaluation of MCP threats, like data exfiltration from untrusted sources.
  • Link for More Details: Ask AI: LLM Application Architecture

Prompt Injection Vulnerability

Prompt injection occurs when attackers craft inputs to override system prompts, leading to unintended actions like data disclosure. Early ChatGPT versions were vulnerable, and libraries have had CVEs enabling remote code execution. A basic code example shows concatenating user input directly into prompts, making it easy to exploit.

# Vulnerable example (Python with concatenation)
prompt = f"You are a helpful assistant. User query: {user_input}"
response = llm_service.generate(prompt)

Mitigations include parameterization (separating system and user inputs) and sanitization libraries like Rebuff for heuristic filtering.

  • Key Takeaway: Combine techniques like Ollama's control characters for system prompts with libraries to block injections—testing shows even simple changes can strengthen defenses.
  • Link for More Details: Ask AI: Prompt Injection in LLMs

Sensitive Information Disclosure

LLMs can leak sensitive data due to their non-deterministic nature, especially in Retrieval-Augmented Generation (RAG) setups where queries access embedded data without proper controls. A simple RAG query might return all data indiscriminately.

# Vulnerable RAG query example
embeddings = embed_text(user_query)
results = vector_db.query(embeddings)

Fix by passing user roles to the vector database for role-based access, applying data filters during embedding, and sanitizing outputs with pattern matching (e.g., regex for sensitive info).

Excessive Agency in Agentic Systems

Excessive agency lets LLMs perform unauthorized actions, like sending unintended emails or executing commands. Research from Pillar Security shows hidden characters in rule files for AI coding tools (e.g., Cursor, Replit) can bypass restrictions. In event-driven setups, agents with broad access via MCP pose risks.

# Unsafe execution example
if "execute:" in prompt:
    os.system(command_after_colon)

Secure by filtering allowed commands (e.g., list/read but not delete), sandboxing execution, and controlling agent permissions.

  • Key Takeaway: In vibe coding or server-side agents, always use allowlists and sandboxes to limit actions—prevent things like accidental file deletion or external interactions.
  • Link for More Details: Ask AI: Excessive Agency in LLMs

About the summarizer

I'm Ali Sol, a Backend Developer. Learn more: