A student group project on LLM security testing by
By Joeri, Khalid, Thomas, Gijs & Jeroen — Fontys ICT Students 23rd June 2026
Introduction
Large language models are showing up everywhere, including in education. But how safe are they, really? For our school group project at Fontys ICT, a university of applied sciences in the Netherlands, we were commissioned by Satu Korhonen, Founder of Helheim Labs, to security test an AI-powered capture-the-flag game she built. Our job was to play through their CTF game and, on top of that, look at the LLM behind it with a security mindset and report real vulnerabilities.
This blog post is for anyone who is new to Helheim Labs or curious about what it looks like when students get to test a real LLM-powered product. We will walk you through how we approached the work, what we found, and what we would do differently next time.
What Is a CTF and How Does an LLM Fit In?
A capture-the-flag game is a challenge-based format common in cybersecurity. Players work through puzzles and scenarios, and finding a flag, a hidden string like flag[FinlandisCool] (not a real one, find the actual flags yourself in their game!), proves you completed a challenge. Helheim Labs built their CTF platform around an LLM, meaning the challenges involve interacting with an AI rather than traditional software.
Playing the CTF was the fun part. But our actual assignment from Satu went a step further: beyond finding flags as intended by the game, we were asked to look at the LLM itself and identify security vulnerabilities worth reporting.
Working Across Borders
As a group, we were able to meet up physically to work together. However, our client Satu was based in Finland, which meant all communication with them happened entirely online. We coordinated across time zones, handled feedback asynchronously, and managed the whole client relationship remotely.
One early decision shaped how we structured our work: we used OWASP’s frameworks to structure how we tested and to classify what we found. This gave our findings a recognized structure and made them easier to communicate. If you are testing an LLM for the first time, their resources are a solid starting point.
What We Found
Once we started looking at the LLM with a security lens, we found a range of vulnerabilities. Due to the sensitivity of some findings, we are only sharing one example here. For the full picture, you’ll have to play the game, as sharing all vulnerabilities would reveal how to win.
One finding we can share is a hallucination. As part of our OWASP-based testing methodology, we used a prompt specifically designed to test whether the model would fabricate information about non-existent subjects. We asked the model to detail the literary works of a fictional author named Elara, someone who does not exist.
Figure 1: Prompt used to test for hallucinations.
As you can see in Figure 1, rather than acknowledging that the author does not exist, the model confidently produced an entire bibliography complete with book titles, publication years, and detailed plot summaries. This maps to hallucination behavior as defined in OWASP’s LLM framework, and in a learning context it could easily mislead a someone treating the output as a reliable source.
What We Took Away as a Team
Beyond the findings themselves, this project pushed our knowledge forward in ways a classroom assignment rarely does. Working with OWASP’s frameworks in a semi-real setting gave us a much better feel for how AI-specific vulnerabilities differ from traditional software issues, like those covered in the OWASP Top 10 for Web Applications. They are less predictable, harder to reproduce, and require a different mindset to test properly.
By the end of the project we had not only sharpened our technical skills but also developed a broader understanding of secure AI application design, responsible reporting, and why structured frameworks exist in the first place. That kind of hands-on experience is difficult to replicate in a purely theoretical setting.
Where We Went Out of Scope
There is something about finding a vulnerability that makes you want to keep pulling the thread. At some point, our enthusiasm got ahead of our scope. We went further in certain areas than the brief intended, and some of our findings, while interesting to us, were not necessarily what Satu was most focused on.
In hindsight, a better approach would have been to continuously check our direction against what Satu actually needed. The most valuable findings for a client are not always the most technically interesting ones. For example, we spent time investigating content bias in the model’s responses, which, while a valid concern in a real-world application, was simply less relevant for a CTF game environment. Next time, we would prioritize failures that matter from the client’s perspective and confirm scope alignment early and often rather than assuming our instincts matched theirs.
Conclusion
Testing an LLM-powered application is different from traditional software testing. The attack surface is less predictable, the outputs are probabilistic, and the failures can be subtle.
Playing the CTF was genuinely fun. The game challenged us to think creatively, find hidden flags in our own way, and explore how an LLM behaves when you push it in unexpected directions. At the same time, working with OWASP’s frameworks gave us the structure we needed to go beyond just playing the game and actually identify real vulnerabilities in a systematic way. The combination of creative exploration and structured methodology made this project more valuable than either approach would have been on its own.
We came away with real findings, real lessons about professional scoping, and a clearer picture of what responsible LLM security testing looks like in practice. Helheim Labs built something genuinely worth testing, and getting to work on a real platform made this one of the more memorable projects we have done at Fontys ICT.
Joeri, Khalid, Thomas, Gijs & Jeroen — Fontys ICT Students