AI Tools Have Major Security Flaw That Hackers Can Exploit

AI tools and chatbots are trusted by millions of developers and everyday users. New research reveals these popular AI systems have a serious weakness that cybercriminals are exploiting.

What Is the LegalPwn Attack?

Security researchers at Pangea Labs have found a dangerous new cyberattack called LegalPwn. This attack tricks popular AI tools into thinking malicious code is safe by hiding it inside fake legal text.

The attack works because AI models are trained to respect legal-sounding language. Hackers take advantage of this by wrapping their dangerous code in phony legal disclaimers, compliance notices, and copyright warnings.

How the Attack Works

LegalPwn is a type of prompt injection attack. The researchers tested twelve major AI models and found most were vulnerable to this social engineering trick.

The attack uses six different legal contexts to fool AI systems:

• Legal disclaimers
• Compliance mandates
• Confidentiality notices
• Terms of service violations
• Copyright violation notices
• License agreement restrictions

In their experiments, researchers embedded malicious reverse shell code (which gives hackers remote access to computers) inside fake copyright notices and disclaimers. The AI models often failed to detect the dangerous code and classified it as harmless.

Real Tools Are at Risk

This isn’t just a lab experiment. The attack affects tools millions of people use every day. Google’s Gemini CLI was tricked into recommending users run dangerous code on their systems. The tool even went as far as suggesting users execute the malicious commands.

GitHub Copilot also fell for the trick. It classified dangerous reverse shell code as a simple calculator when the malware was hidden inside a fake copyright notice. This shows how the attack can fool AI tools into misidentifying serious security threats.

Which AI Models Are Vulnerable?

Most major AI companies have vulnerable models according to the research:

• ChatGPT 4.1 and 4o
• Google Gemini (both Flash and Pro versions)
• DeepSeek R1 model
• Meta Llama 3.3
• xAI Grok

However, some models showed better resistance to these attacks. Anthropic’s Claude 3.5 Sonnet and Claude 4 Sonnet, Microsoft’s Phi 4, and Meta’s Llama Guard consistently blocked the LegalPwn attempts across all test scenarios.

Why Human Review Still Matters

The research shows a clear gap between AI and human security analysis. While AI models failed to spot malicious code wrapped in legal text, human security analysts correctly identified threats every time during the study.

Even when researchers gave AI models specific security instructions, the LegalPwn technique still worked in many cases. This shows that current AI safety measures aren’t enough to stop determined attackers.

The researchers found that stronger system prompts (instructions that tell AI models to prioritize security) helped reduce successful attacks. However, they weren’t foolproof, and more sophisticated versions of the attack could still bypass these protections.

Final Thoughts

This discovery shows we can’t rely completely on AI for security decisions. Companies need human oversight for AI-assisted security work.

The LegalPwn attack proves that cybercriminals will find creative ways to exploit any weakness in our security systems.

As AI tools become more common, maintaining human expertise and oversight remains critical for keeping our systems safe.

For more information on this story, refer to the official LegalPwn report (PDF) and the report from Hackread.

We want to know your thoughts. What do you think about this story? Let us know in the comment section below!

Be sure to stay up-to-date with the latest streaming news, reviews, tips, and more by following the TROYPOINT Advisor with updates weekly.