OpenAI’s Guardrails Can Be Bypassed by Simple Prompt Injection Attack

Just weeks after its release, OpenAI’s Guardrails system was quickly bypassed by researchers. Read how simple prompt injection attacks fooled the system’s AI judges and exposed an ongoing security concern for OpenAI.

A new report from the research firm HiddenLayer reveals an alarming flaw in the safety measures for Large Language Models (LLMs). OpenAI recently rolled out its Guardrails safety framework on October 6th as part of its new AgentKit toolset to help developers build and secure AI agents.

It is described by OpenAI as an open-source, modular safety layer to protect against unintended or malicious behaviour, including concealing Personal Identifiable Information (PII). This system was designed to use special AI programs called LLM-based judges to detect and block harmful actions like jailbreaks and prompt injections.

For your information, a jailbreak is a prompt that tries to get the AI to bypass its rules, and a prompt injection ...