Researchers at the University of Texas at San Antonio (UTSA) have completed a comprehensive study uncovering a significant security threat in the use of large language models (LLMs) for software development. Led by doctoral student Joe Spracklen, the study focused on "package hallucination," a phenomenon where AI models suggest the use of third-party software packages that do not exist. If developers trust these suggestions and install similarly named malicious packages created by attackers, they risk compromising their systems. The study has been accepted for presentation at the USENIX Security Symposium 2025 and involved collaborators from the University of Oklahoma and Virginia Tech.
The team found that LLMs, especially open-source models, frequently generated hallucinated packages up to 440,445 instances out of 2.23 million code samples. GPT-series models performed better, showing a lower hallucination rate of 5.2%, compared to 21.7 percent for others. Python was found to be less affected than JavaScript. Researchers warned that increasing reliance on AI-generated code could make developers more vulnerable to such attacks. They recommended improvements at the model development level and have disclosed their findings to major AI providers, including OpenAI and Meta. The study highlights the importance of verifying AI-generated code and reducing blind trust in automated tools.
