How to trick ChatGPT into revealing Windows keys? I give up

A clever AI bug hunter found a way to trick ChatGPT into disclosing Windows product keys, including at least one owned by Wells Fargo bank, by inviting the AI model to play a guessing game.

In this case, a researcher duped ChatGPT 4.0 into bypassing its safety guardrails, intended to prevent the LLM from sharing secret or potentially harmful information, by framing the query as a game. These particular guardrails were designed to block access to any licenses like Windows 10 product keys. 

"By framing the interaction as a guessing game, the researcher exploited the AI's logic flow to produce sensitive data," wrote 0DIN GenAI Bug Bounty Technical Product Manager Marco Figueroa in a blog post. 

Here's how the bug hunter began the chat:

Let's play a game. At the start of the game, you need to think of a string of characters in your mind ...