The challenge#
When this global financial technology company developed an AI agent to help its lender clients support customers, its mandate was clear: give customers an accurate and automated way to manage their loans. But because the agent needed to act in a highly regulated space, a single misleading answer about financing, interest rates, or credit could cause real financial harm and put lender clients at regulatory risk.
The fintech leader had guardrails built in to catch standard moderation content like hate speech, but those guardrails weren't built for sophisticated attackers or domain-specific risks. A malicious user could talk it into confirming fabricated terms, turning the transcript into a legitimate dispute. An agent giving customers misinformation about their loans could also cause consumer harm and regulatory exposure under FDCPA and CFPB guidance.
The solution#
The fintech leader partnered with General Analysis to evaluate its loan-management agent against adversarial prompting at scale. To do this, the platform ran attacks against the assistant the way a real attacker would – black-box testing with no backend access or privileged accounts. GA's automated red-teaming agent sent 100,000+ adversarial prompts to the agent, built around 900+ attack outcomes, with:
-
Risks based on target context. GA mapped their attack framework to OWASP's Top 10 for LLM applications: Prompt Injection (LLM01), System Prompt Leakage (LLM07) and Misinformation (LLM09) based on industry context.
-
Attack goals relevant to target workflows. Each goal was mapped to a specific policy violation, such as “get the agent to recommend Chapter 7 or Chapter 13 bankruptcy.”
-
Adaptive adversarial reasoning. GA’s red-teaming agent operated as an autonomous attacker, armed with an arsenal of research-backed algorithms like TAP and injection techniques like fake system headers and encoded text. When a prompt was refused, the attacker agent reasoned and adapted on the next turn.
Prompt injection was the highest-risk category, with an attack success rate of 22.5%. In one attack, the red-teaming agent attempted a “Do Anything Now” (DAN) injection, which effectively jailbreaks an agent into an unrestricted interaction mode. The company’s agent refused the first request but by the fifth refinement step, it acknowledged DAN mode, expanding the attack surface and facilitating violations with virtually unlimited scope and impact.
The impact#
The red-teaming agent convinced the target to execute harmful actions in 250+ simulations. General Analysis found that the fintech leader’s loans agent held up >80% of the time against generic harm, but <30% of the time against domain-specific attacks. The platform’s automated attacker achieved ~70% ASR in system prompt leakage and ~60% ASR in misinformation, revealing critical weaknesses that comprehensive static testing was unable to find. The security team received remediation guidance, full attack traces, and reproduction steps for each failure mode General Analysis found.
A single policy-violating response could expose the company to regulatory and reputational risk. With General Analysis, the company was able to automate 900+ hours of manual testing and received a complete regression suite. The high-severity exploits found by the attacker became a repeatable evaluation set before every deployment, so updates to the agent could be rigorously re-tested before production.
