21.10.2024
6 min
1927
The OWASP GenAI Security Project is a flagship initiative that shapes global security standards for generative AI and LLM systems. The article reviews the OWASP LLM Top-10, which collects the main vulnerabilities of AI applications, as well as accompanying projects: Agentic Initiative, Red Teaming Guide, Security Landscape, Governance Checklist, and Center of Excellence. It describes in detail the latest approaches to testing, threat modeling, and protecting multi-component agent systems. If you are implementing AI or working in the security field, this material will become your main reference point in 2025.
*On March 27, the project was renamed The OWASP Gen AI Security Project. See the official announcement here. This project has become the flagship project for OWASP, let’s understand what it is and why it happened.
What is it:
A list of the ten most critical vulnerabilities in applications that use Large Language Models (LLM).
Started in May 2023 by Steve Wilson* to document the main risks of LLM systems.
Why is it needed:
Closes the gap between classic web application security and specific LLM threats.
Contains a description of typical vulnerabilities in AI systems and advice on countermeasures.
Constantly updated: the third version of the Top 10 LLM 2025 was released at the end of 2024 (vulnerabilities in Retrieval-Augmented Generation (RAG) systems were added, DOS attacks were edited, etc.).
The OWASP LLM Top-10 is most frequently cited in the application environment, used by country regulators and other organizations such as NIST and MITRE. If you google LLM vulnerabilities, you will find it. Within the OWASP organization, the project has grown and now has the status of a flagship. The main landing page https://genai.owasp.org unites other projects within it, which will be discussed below.
The project uses an open voting mechanism, which allows taking into account the experience of the community, but also introduces subjective assessment criteria for specific vulnerabilities and does not always reflect real risks. There are no normal incident statistics for AI applications yet, so this is a forced approach.
Later, Steve Wilson published a book with a detailed review of version 1.1 of the LLM Top 10 (summer 2024).
In the “Top 10” list of security threats, prompt injections are in first place, but there is a debate in the expert community about whether it should be considered a vulnerability. Let’s try to dot the i’s. Technically, prompt injections are a method of attack on programs that use LLM. It allows, for example, to steal system instructions or poison the RAG system.
When the information security service expresses concern about the vulnerability of an AI program to injections, this is often associated with the risk of generating toxic or malicious content. Most often, such attacks are specified and called jailbreaks. For example, a bank chatbot begins to recommend customers to buy cryptocurrency instead of regular deposits, or, even worse, violates the law. And although this does not lead to classic cybersecurity problems, legal consequences may arise for the company, which may lead to the need to remove the genAI system from the release. You can read more about this discussion on Simon Willison’s blog, who with his light hand gave the attack the name “prompt injection”: https://simonwillison.net/2024/Mar/5/prompt-injection-jailbreaking/
As noted above, many niche initiatives have begun to emerge around the Top-10 for LLM Applications. Let’s analyze their varieties in more detail:
AI Security Solutions Landscape – a landscape of security products in the GenAI area.
Agentic Initiative – documents on the security of agent AI systems.
GenAI Red Teaming Guide – a large report on GenAI testing practices.
Governance Checklist – a checklist of types of AI system threats for managers.
Center of Excellence Guide – building best practices in terms of KPIs for IT departments and communication with business departments.
Посилання: https://genai.owasp.org/resource/llm-applications-cybersecurity-and-governance-checklist-english/
What is it:
AI Security Solutions and Vendors Report.
The document also includes a description of the LLM SecOps solution development and operation process, listing protection methods for each phase of the cycle. The report is planned to be updated several times a year.
I use this map to monitor new features of top AI Security vendors, for example, the French Giskard added tools for auto-testing agent systems, and ZenGuardAI was able to make an AI Firewall against prompt attacks with a latency of 50ms (though the firewall itself does not work properly even in English, but that’s okay).
This is what the solution map looks like at the end of 2024, we are looking forward to updates in 2025.
Additionally, in the document you will find a description of the LLM SecOps process and its stages.
Link: https://genai.owasp.org/initiatives/#agenticinitiative
What is it:
Autonomous AI Agent Security Research Working Group (Agentic AI). Includes about 30 active participants from IBM, Palo Alto Networks, Twillio, Salesforce, META (banned in the Russian Federation), etc. Historically, it arose during the work on LLM Top-10 2025, when the team could not reach an agreement on the Excessive Agency item.
The group’s work started in December 2024. The project is not limited to one document, and its goal is to study how the emergence of multi-agent AI-systems will affect the threat landscape and what consequences it has for cybersecurity.
What is an agent system:
Agents accept natural language (text queries, files, images, audio or video). The implementation is built on agent frameworks such as LangChain, LangFlow, AutoGen, Crew.AI, etc.
One or more LLM models, local or remote, are used for logic and decision-making.
Agents interact with services and tools through: Calling functions and tools at the framework/application level. Calling functions directly by the LLM model, which returns the call code to the agent.
Additional agent infrastructure services include: External storage for long-term memory. Data sources, including vector databases and RAG.
Below is a schematic example of a single-agent system architecture:
The solution architecture becomes more complex when there are many agents.
Multi-agent system. A clear illustration that a modern AI application is not just a call to the OpenAI API and a database with documents that are cut into chunks.
The main areas of work of the Agentic Initiative group:
Threats and Mitigations Guide: first version published in February, version 1.1 is currently in development.
Repository with helloworld examples of agent systems and attack cases: planned for release in April.
Guide to securing agent applications: release in spring.
MAESTRO framework for modeling threats to multi-agent systems by Ken Huang (author of Gen AI Security)
Agentic Security Landscape report (scheduled for release in the summer), which will reflect key incidents and an overview of solutions.
Work on streams is currently actively underway on LinkedIn – the group’s press releases are often flashing, so when diving into the material, be prepared to read a lot of text, sometimes of draft quality.
Link https://genai.owasp.org/initiatives/#ai-redteaming
What is it:
A guide to comprehensive testing (Red Teaming) of generative AI systems.
Published in late 2024, it contains about 50 pages. Since such a volume is difficult to read, the working group plans to shorten the text and release a more user-friendly version in 2025.
A detailed description of attacks, relevant datasets, tools. There is even our Open Source Llamator.
The project is led by Krishna Sankar, the working group is currently active, if you like cracking AI programs and want to share your techniques and improve the current guide, join the working group, you can write to me, I will tell you where to start. I remember how a week before the release, Red Team Synack contributed, sharing their experience of pentesting real projects.
Link: https://genai.owasp.org/resource/llm-applications-cybersecurity-and-governance-checklist-english/
What is it:
A security risk checklist for CIOs, C-suite executives, and legal professionals. It will help you understand what AI is, what common threats are, and how generative systems differ from classic ML models.
Structured and practical recommendations on security, governance, and privacy for AI projects.
One of the first OWASP AI projects (appeared in 2023). Contains a mix of technical and organizational measures, but does not provide clear prioritization and does not always indicate the justification for recommendations.
The most interesting thing in the guide is the indication that one of the main risks for a company now is not implementing AI into business processes. Today, adaptations of technologies based on LLM provide an important competitive advantage.
Link: https://genai.owasp.org/resource/llm-and-generative-ai-security-center-of-excellence-guide/
What is it:
A high-level document dedicated to the practices of implementing generative AI, aimed primarily at IT managers and directors of large companies. The purpose of the document: to provide tools for creating a Security Center of Excellence (the name sounds pompous in English, so I didn’t translate it), which includes:
Developing policies to implement GenAI security practices.
Risk management during the operation of AI systems.
Employee training.
Interaction between technical teams and business leaders.
In the document, the reader will see examples of acceptable KPIs/OKRs for IT departments when implementing generative AI. Below is a sample timeline for implementing best practices and creating an AI expertise center:
Unlike the LLM AI Cybersecurity & Governance Checklist: the previous document focuses on a list of risks in simpler language for non-technical readers, while the Center of Excellence Guide is dedicated to building security practices around GenAI in the company.
Link https://github.com/OWASP/www-project-llm-verification-standard
What is it:
A standard checklist to help design and test LLM-based applications.
Focuses on architecture, model lifecycle, integration, monitoring, and anomaly detection. Shares recommendations by company type: startups, midsize companies, and large corporations.
Never released, version 0.0.1 was published in February 2024.
As far as I understand, the project was attempted by contributors with support from Snyk and Lakera, but never completed. Use the LLM Sec Ops recommendations from AI Security Solutions Landscape.
We still live in a reality where new attacks are released every month and it is unlikely that the work of the various OWASP working groups will be consolidated into a single document anytime soon. Nevertheless, the guides are popular among application developers and many technical papers and products from AI Security vendors continue to rely on the OWASP threat classification on a par with NIST, MITRE.
