Code at AI Speed, Risk at AI Scale

Executive Summary

The first clue was not a zero-day. It was the tempo. 

During a recent cloud incident response, attacker activity did not look like one operator moving through a linear checklist. Credential discovery, cloud enumeration, source-code review, CI/CD probing, database access, and operational disruption were unfolding in parallel. From the defender’s seat, it looked less like a single intruder walking a path and more like multiple agents pursuing different attack paths simultaneously. 

This is part of a pattern across public reporting. Sygnia found that in the AI-assisted cloud intrusion, the attacker moved across application services, AWS resources, source control, CI/CD, runtime infrastructure, and data stores at unusual speed. The important detail was not a novel exploit, it was concurrency. Familiar techniques were chained together faster than traditional response models expect.

This investigation changed the way we think about attackers. We realized we were witnessing another major shift; this time the rapid emergence of security risks driven by AI-generated applications. The same force that lets one attacker move like a small team can let one builder ship code like a small engineering group. 

We saw that second use of AI during a penetration test for a financial services organization managing billions of dollars in client assets, during which we assessed a customer onboarding application that processed highly sensitive personal and financial information, including government-issued identification, identity verification data, and payment details. 

Portions of the application had been built or heavily accelerated with Claude. It was built to solve a real product problem: the need to enable applicants resume an application before they had full user accounts. The code was functional, and it followed familiar framework patterns.  

This code solves a legitimate business need. A prospective customer may start an application having credentials, or they may leave and return. The application needs some way to restore the current state of completion, associate activity with the right applicant, and protect the applicant journey from abuse. 

But the implementation introduced a critical flaw. 

The application treated possession of an applicant globally unique identifier (GUID) as enough context to obtain an access token for that applicant. In practical terms, if someone had another applicant’s GUID, the flow could issue or restore access for that applicant without proving that the requester was the proper owner of the applicant’s session, identity, device, or communication channel. 

The result was not merely “Broken Access Control.” 

It was an authentication and authorization failure in the applicant onboarding flow that could expose client PII. 

What Actually Broke

The vulnerable design tried to solve a legitimate onboarding problem: applicants needed a way to resume an application before they had full account credentials. To support that, the application used a temporary applicant-access model. The flaw was that the token issuance and restoration flow treated possession of an applicant GUID as sufficient proof to issue an access token. A GUID can identify a record, but it does not prove identity, session ownership, device control, or control of a verified communication channel. In effect, the GUID became a bearer secret. 

The system confirmed that an applicant record existed but did not reliably confirm that the requester was entitled to act as that applicant. That distinction created a path to access applicant data outside the requester’s authorized scope, including names, contact information, application status, financial details, identity verification data, payment information, SSNs, and co-applicant details. This was a confidentiality failure involving regulated client data. The root cause was tied to the AI-assisted implementation strategy: the plan included tokens, expiration, rate limiting, and logging, but missed the critical pre-issuance question of what proves a requester should receive or restore an applicant token. Figure 1 shows a sanitized mock of that Claude-generated implementation plan. 

The implementation plan had the language of security. It described temporary tokens, expiration, rate limiting, audit logging, session restoration, and suspicious-activity detection. Those are all useful controls. 

But none of them answer the most important question: 

What must be proven before the backend issues or restores an applicant access token?

That was the failure point. The implementation focused on how the token would behave after issuance. The vulnerability existed before issuance, when the system decided whether the requester was entitled to receive a token at all.

A short-lived token is still dangerous if it is issued to the wrong person. A signed token is still dangerous if the signing service grants it based on an identifier alone. Audit logging may help an investigation later, but it does not prevent the initial exposure. Rate limiting may slow abuse, but it does not prove authorization.

The issue was not that the system lacked security controls; the issue was that the controls were wrapped around the wrong trust decision.

How to Secure Enterprise AI: From Adoption to Incident Readiness

Download the eBook

Why the Finding Was Easy to Miss

The application did not look obviously broken. In fact, it looked like someone had tried to add structure and security around a difficult workflow.

That is what made the finding important.

Applicant onboarding is messy by design. A user may not have a password yet. They may not have completed identity verification. They may return to an application from the same browser, a different device, or a link sent through email. Product teams need to deliver a smooth experience, especially in financial services where abandoned applications have direct business impact.

AI is very good at helping teams build that kind of workflow quickly. It can produce routes, middleware, token services, storage patterns, expiration logic, and test scaffolding. But it does not automatically understand the institution’s risk model. It does not know which identifiers are public, which are sensitive, which are guessable, which are exposed in URLs, and which must never become proof of identity.

That is the distinction between a working but insecure implementation and a safe implementation.

The vulnerable flow worked because it restored applicant state. It failed because it did not require strong proof that the person asking to restore that state was the applicant.

What a Safer Design Would Require

A GUID can locate an applicant record, but it should not authorize access to it. Temporary access should require independent proof of applicant control, such as a one-time password (OTP), one-time link, signed continuation token, or trusted session. Any token issued should be narrowly scoped, short-lived, revocable, and monitored, but the primary control is proving entitlement before granting access.

How We Used LLM for Code Review

The same democratization that changed software development is now changing vulnerability analysis.

A few years ago, reviewing this kind of business-logic flaw required either an experienced application security engineer, a detailed threat model, or expensive tooling with highly customized rules. Those still matter. But AI changes the economics of first-pass review. Tasks that previously required hours or days of manual review can now be completed in minutes, allowing security engineers to focus on validating findings, understanding business impact, and exploring complex attack paths at machine speed.

As part of the penetration test, an LLM was run against portions of the client’s codebase to identify security-relevant patterns, trust-boundary issues, and business-logic flaws. The review surfaced multiple significant vulnerabilities, including the applicant-token issue described above. In that case, the model identified that the application’s temporary-access flow treated possession of an applicant GUID as sufficient context to issue or restore an access token, without requiring stronger proof that the requester controlled the applicant’s session, identity, device, or communication channel. The screenshots below show an example of an LLM-assisted security analysis identifying the core trust-boundary flaw in the applicant-token flow. What once required extensive manual reasoning can now be surfaced in minutes, compressing vulnerability analysis and enabling security teams to review complex codebases at scale.

This finding captured the larger theme of the assessment: a vibe-coded security architecture flaw was identified through vibe-coded code review.

This is the uncomfortable loop now facing AppSec teams. AI not only helps teams write code faster. It also helps reviewers, testers, and potentially attackers reason about that code faster. When given enough context, an LLM can quickly identify the questions that matter most for business-logic vulnerabilities.

For defenders, this creates an opportunity. AI-assisted review can help AppSec teams scale first-pass analysis across large or unfamiliar codebases, especially for flaws that traditional SAST may not reliably catch, due to a focus on insecure coding patterns and unsafe data flows rather than business logic.  But it also raises the stakes. If a security team can use an LLM to identify missing trust boundaries in a codebase, an attacker with access to exposed source code, leaked repositories, or stolen implementation artifacts can ask an LLM those same kinds of questions.

The lesson is not that AI-assisted review replaces expert testing; it does not. The lesson is that vulnerability analysis is being democratized in the same way software development has been democratized. The cost of finding certain classes of flaws is lower, just as the cost of creating the code decreased first.

The Democratization Loop

The cyber security industry has spent the last year talking about the democratization of software development. That part is real. A person with access to Claude, ChatGPT, Cursor, Copilot, or another coding assistant can generate application code, integrations, scripts, and internal tools at a speed that previously required a larger engineering effort.

But the same curve applies to vulnerability analysis.

A person with access to an LLM can ask:

• Can this token be issued without proving ownership?
• Does this session restoration flow trust an identifier as a secret?
• Could this expose applicant PII?
• What negative tests should exist for this endpoint?

These questions are defensive in the hands of a security team. They are offensive in the hands of a threat actor reviewing exposed source code, leaked implementation notes, or stolen repositories.

The AI-assisted cloud incident showed how familiar attack techniques can be executed at unfamiliar speed. The financial institution penetration test showed how AI-assisted development can create working application code with unsafe trust assumptions. Together, they point to the same conclusion: security teams are no longer defending only against faster attackers. They are defending against faster software creation, faster vulnerability discovery, and faster attack-path execution.

How Organizations Should Defend

Organizations should not respond by banning AI coding assistants outright. That usually pushes usage into unmanaged tools and personal accounts. The better response is visibility, review, and containment.

First, make AI-assisted code visible. Pull requests should indicate whether AI generated or substantially modified the code, which tool was used, and whether the workflow touches sensitive areas such as authentication, authorization, session restoration, payments, document access, administrative actions, data exports, or regulated client information.

Second, make security requirements explicit in prompts. A prompt that asks for “temporary access tokens for applicants” is incomplete. A safer prompt says: applicant identifiers must not be treated as secrets; token issuance must require independent proof of applicant control; tokens must be scoped to the verified applicant; session restoration must not work based only on a GUID; and negative tests must prove that one applicant cannot obtain access for another applicant.

Third, review AI-generated code against business intent, not just syntax. Reviewers should ask: What does the requester have to prove before receiving a token? What is the token allowed to access? Can an identifier from the request influence authorization? What PII becomes available after token issuance? What happens if a valid identifier belongs to someone else?

Finally, use AI defensively, but constrain it. Redact sensitive code. Integrate AI-assisted security analysis and code review into the secure development lifecycle to identify insecure patterns, authorization flaws, business logic weaknesses, and deviations from security requirements. Use AI to augment – not replace – expert review. Redact sensitive code. Do not send secrets, credentials, raw PII, client identifiers, or proprietary implementation details to unmanaged tools. Use enterprise-approved environments where possible and require human validation of every finding and every proposed fix.

Conclusion

AI-assisted development is now part of the enterprise software lifecycle. The financial institution penetration test showed the risk clearly: AI helped produce a working applicant access flow, but the implementation treated possession of an applicant GUID as enough context to obtain an access token. That missing validation step created a path to client PII exposure.

The investigation into the AI-enabled cloud intrusion showed the other side of the equation. AI can help attackers chain known techniques faster, pursue multiple workstreams in parallel, and adapt to complex environments with less friction.

Together, the cases point to the next AppSec reality: AI is not just changing how code is written. It is changing how code is reviewed, how vulnerabilities are found, and how quickly weaknesses become attack paths.

The organizations that adapt will not be the ones that ban the AI accelerant. They will be the ones that govern it, inspect it, and use it defensively with the same speed and context that attackers are beginning to use offensively.

Contributing Reviewer: Ofir Swisa, Adversarial Tactics Team Leader