New publication in EMSE!

December 05, 2025

2025

Our paper, “Evaluating and Improving the Robustness of Security Attack Detectors Generated by LLMs,” has been published in Empirical Software Engineering (EMSE)!

This achievement was possible thanks to my co-authors: Jinhan Kim, Tommaso Aiello, Rocío Cabrera Lozoya, Antonino Sabetta, and Professor Paolo Tonella. A special thank you to Software Institute (USI), USI Università della Svizzera italiana, and Sec4AI4Sec, which made this research possible!

Key Takeaways:

  • LLMs as code generators: Large Language Models are not just text generators; they can produce functional security code. Yet, they may lack domain-specific knowledge needed for robust attack detection.
  • Knowledge-Augmented Code Generation: We enhance LLMs with Retrieval Augmented Generation (RAG), injecting external security knowledge to guide generation. This shows the potential of hybrid LLM systems that combine learned patterns with curated information to solve knowledge-intensive coding tasks.
  • Self-Evaluating Models: Inspired by Self-Consistency in LLM reasoning, our Self-Ranking mechanism generates multiple candidate solutions and selects the most reliable detector, demonstrating how LLMs can become self-aware of robustness.
  • LLM in action for cybersecurity: Applied to XSS and SQLi attack detection, our approach demonstrates that LLMs can produce high-quality, interpretable security solutions that generalize across tasks, reducing reliance on manual labeling or expert-crafted rules.