- OWASP Top 10 LLM Apps Newsletter
- Posts
- OWASP Top 10 for LLM Applications Newsletter - July '24 Edition
OWASP Top 10 for LLM Applications Newsletter - July '24 Edition
OWASPers have spoken ! V2 voting results !!
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/0103275c-91fd-4e60-b1a4-6bea8caed811/image.png?t=1720759549)
Greetings Gen AI Security Enthusiasts and OWASP Community Members!
Musings from the editorial desk.
The OWASP T10 for LLM Applications v2 deliberations are underway! We have successfully moved past the candidate stage and the voting phase. We have lots of interesting candidates and the voting results are very insightful.
That is the focus of this news letter - please read on …
On a side note, if you're curious about the Alfa Romeo in the picture above, here's the scoop:
I'm a huge fan of Lego (and FLL!) and have been collecting larger sets mostly space ships from Star Wars, cars like SP3, Chiron and Sian for build animation cinematography - a long term project.
I also collect Formula 1 race car models - various scales (1:43 to 1:8, yet to own a 1:1 😇), from Alfa Romeo, McLaren, Porsche 963 et al.
The Alfa Romeo 2022 (C42) and 2023 (C43) cars have particularly stunning aesthetics ! I love the Brandywine-white color combination on C42 and the black on C43 (see a collage from my collection, below). Years ago, we owned a Brandywine Lexus RX 350, w/ sparkles! A beautiful SUV …
In the picture above (created by ChatGPT 4o), the LLM missed the color—turned it into purple. This minor oversight sheds light on the complex behavior of large language models (LLMs) and the gaps …
Let me explain …
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/aafd7b25-13cf-4fd9-9d9b-b433c8614f3f/Alfa_Romeo_01.png?t=1720905234)
Alfa Romeo F1 car models to build !!
I asked ChatGPT 4o to generate an image of a wasp riding an Alfa Romeo C43. The resulting image (at the top of this newsletter) closely resembles the model (including the looong nose/front wing, the driver race number 03 and the whole F1 track with other cars), but the color is off—purple instead of Brandywine.
It’s not that GPT 4o doesn’t have enough data to get it right.
In the top blog picture, it almost nailed the background cars, like the teal colored Aston Martin Aramco AMR23 2023 !
In the picture below, it accurately depicts the C43 Alfa Romeo race car (stickers are off, driver race number is missing but the logo looks right) and even includes the the Matrix Digital Rain !
The point is, GPT hasn’t experienced Brandywine in California sunlight - this nuance isn’t documented anywhere either. As a result, it has not developed any specific aesthetic affinity, and so unceremoniously picks a red-ish shade - purple … Many subtleties of our world are left unsaid...
Neither a hallucination nor an emergent behavior could bring out the exact color tone - which of course is a topic we will touch upon !!
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/14ff4589-7331-45ad-b873-59de2905211e/image.png?t=1720914504)
📋 Let us dive into the v2 discussions …📋
We will look at the V2 from multiple POV in four sections.
1. OWASP T10 for LLM Applications v2 : Status
I am sure y’all remember the organized v2 candidate list from our June Newsletter [Here].
For those enquiring minds who want to know, the raw list is also available in our GitHub [Here]
Voting on existing Entries, New entries proposal, Voting on new entries | Completed ✔️ |
Merging & Down selection ⛘ | August 1, 2024 |
Data Analysis & Voting for Ranking 📋 | September 1, 2024 |
Entry Cleanup 🧹 | September 15, 2024 |
Publish 🏁 | October 1, 2024 |
Like we talked about earlier, the general focus for v2 is three fold:
Span Security (of the operator), Safety (of the user) and Trust (by the user) of LLM Applications
Be the one-stop-shop to answer the question "What are the top risks that I should worry about when I deploy my application that has LLM components?"
Make the artifacts approachable, accessible and consumable by a wide variety of audience - folks who have high information overload and a low attention span (current company included !)
In addition to the Top 10 for LLM list, we are pursuing multiple tracks to facilitate these goals.
Many interesting initiatives are being deliberated - Solutions doc, AI Red Teaming Best Practices, Data Collective and so forth. Lots of opportunities for all of us to participate.
Your turn to join the lists - voice your opinions; pick the ones you are passionate about and contribute !!
P.S: I am partial towards the AI Red Teaming efforts ! Here’s a shameless plug for your participation (Editor’s prerogative 😇)!!
Check out the initial proposal - OWASP Top 10 for LLMs - AI Red Teaming & Evaluation Guidelines Initiative - join the discussions at the #team-llm-redteam slack channel, shape the body of work and more importantly, contribute.
Of course I can’t hog the space 🙃 Scott has done an excellent job on the Solutions doc work. Outline and initial content here
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/637e89e2-2c01-49ac-ab69-07dcba9faa84/image.png?t=1721602546)
2. v2 Voting Results
Back to the main feature - trends first, analytics next, and finally the gory details
Trends
Agents are rising in importance. We have a whole channel #team-llm_v2_agents_discussion led by our favorites John Sotiropoulos and Fabrizio Cilli. There are 2 candidates viz., Vulnerable Autonomous Agents and Agent Autonomy Escalation. Autonomous Agents bring together different aspects of of LLMs
Agents embody autonomy and inherently relate to the existing entry on excessive agency
Agents are a different way to use LLMs and change the application paradigm from the traditional almost client server model of LLM apps (and also the usual inference endpoint of traditional ML) to a distributed one. They bring new attack vectors and considerations including environment, adaptability, and new patterns such as reflection, and scenarios including emerging on-device agents and models, and challenge the notion of human-on-the-loop and how we scale it
The Gen AI security is maturing and separating itself from traditional ML and web applications. Agents, RAG all are newer paradigms (the ideas are not new, but the applications are) that figure prominently in the voting
As expected, Prompt Injection has very high priority
An interesting one is the system prompt exploits, like bypassing system instructions have risen in importance. There are 2 candidates that deal with this aspect - the C06:Bypassing System Instructions Using System Prompt Leakage and the C27:System Prompt Leakage
System Instructions is the superset that can contain system prompts, roles, response characteristics (like be succinct or be more detailed et al) and so forth) Not sure if bias or malicious code creation is blocked by system instructions.
The discussion chain in the slack channel, mentions two separate “vulnerabilities” - viz., the disclosure of sensitive information and second, use this information to jailbreak the LLM (may be straightforward, may be use the exposed information along with other information)
So probably a solution is to add to three items :
Add System Instruction as another internal information and methods to protect it in “LLM06: Sensitive Information Disclosure”
Add system instruction leakage as another source for prompt injection (basically modify system instructions to do harm) to “LLM01: Prompt Injection & other insecure input handling”
The System Instructions leakage would be another prompt injection vulnerability, adding to “LLM02: Insecure Output Handling” also is interesting. We filter any system instructions out ! I hope that itself is not yet another system instruction
Analytics
Steve has done a good job in coalescing the results:
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/1fab3b9e-7e5a-45a6-ac78-e7719ab88eeb/image.png?t=1721599130)
Gory Details
3. Notes from the Core Team v2 deliberations
The slack channel #team-llm-v2-brainstorm has lots of interesting discussions. I have a proposal mapping [here], based on the discussions from the v2 brainstorm discussions. We had a v2 core team conversation, which you can watch - video [here] and the slides [here].
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/06fc3e75-471a-441c-97b3-b029abdeb634/image.png?t=1721600425)
Ads has captured the key points from the discussion:
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/ab318f99-226e-4efd-b7db-e7a5cd360267/image.png?t=1721600956)
4. OWASP T10 for LLM Applications v2 : Next Steps
The v2 is in development, evolving and still formative. It is up to all of us to shape the entries, their order and the write ups !
The Data Collective has a call out for data and their work is here. It is important to contribute as this will form the basis for v2.
![](https://media.beehiiv.com/cdn-cgi/image/fit=scale-down,format=auto,onerror=redirect,quality=80/uploads/asset/file/c5739df6-13fb-4f2b-8232-e5ffd18d11f3/image.png?t=1721600874)
🔗 OWASP LLM Documents & Links 🔗
📱Stay Connected📱
Aubrey King has expanded our social media reach across various platforms including LinkedIn, Youtube, Twitter (Agree with Will, not going to say X), and soon even more! Follow and don’t be shy to re-share any content we post! BTW, if you're presenting anything related to our project externally, do let us know at [email protected] so we can review and blast out to our social media followers!
💭 Closing Thoughts 💭
As you can see, we are making good progress towards OWASP Top 10 LLM V2. And, we have more initiatives like the Solutions doc, AI Red Teaming Best Practices, Data Collective and so forth. An inflection point probably. So a good time to track, participate and contribute …
We look forward to seeing you in our Slack channels and at our upcoming meetings!
Till next time … Stay Secure & Stay Sharp …