OpenAI's announcement of GPT-Red marks an important step in the automation of language model security. GPT-Red is a system trained via reinforcement learning that plays against itself to discover increasingly effective prompt injections. At the end of its training, the system recorded an 84% success rate in internal evaluation scenarios, far exceeding the 13% recorded by human red teamers on the same tests.
This announcement, while seemingly reassuring, actually presents two highly critical issues that redefine how we must evaluate AI security.
The Limit of Automated Batch Testing
The large gap between the measured effectiveness of GPT-Red (84%) and that of human red teamers (13%) stems largely from the nature of automated evaluations. Batch testing tends to be repetitive and relies on rigid rules that fail to account for the actual production context.
Many results recorded as successful compromises are actually false positives. An artificial intelligence optimizes its responses to satisfy a mathematical metric of safety filter evasion, but this does not mean the generated attack is capable of causing actual harm to an enterprise application or extracting confidential data in production.
The Threat of Simulated Reasoning Traces (Fake Chain-of-Thought)
The most interesting aspect of the announcement is GPT-Red's independent discovery of a new class of attack: fake chain-of-thought.
Next-generation models use an internal reasoning trace to plan their responses before showing them to the user. Fake chain-of-thought consists of injecting falsified reasoning steps at the beginning of the prompt (for example, by inserting fake system tags such as <thought> or [thinking]).
When the model reads sentences within its own context that simulate a safety check that has already completed successfully, it assumes the verification has already taken place and proceeds to execute the request without applying its native alignment guardrails. This behavior highlights a structural logical fragility in models that rely exclusively on internal safety checks.
Safety Marketing and the Real Risk
Loud announcements about defensive systems considered unbreakable often create a false sense of security. In the cybersecurity field, declaring that a system is impenetrable attracts the attention of independent researchers and increases the reputational risk if a simple jailbreak is discovered that bypasses the guardrails.
The solution to these threats cannot rely solely on the internal safety alignment of commercial language models. To ensure real application security, it is necessary to implement external and independent filters, such as Oracolo from the AIPF line and Vigil Security Proxy. These tools analyze requests before they reach the model, blocking attempts to manipulate context or insert fake system tags at the root.