AI Threat Modeling Needs a Faster Starting Point

On a busy day recently, threat-modeling expert Adam Shostack opened an email from a client. Someone at that organization had vibe-coded an app and put it to work with customer data. The client wanted to know what risks the tool posed — and what to do about them. They needed answers quickly, so Shostack gave himself 15 minutes to analyze the system.

Within that quarter-hour, he produced what he called “a list of meaningful threats,” including hallucination and bias — problems that STRIDE, the long-standing application-security threat model framework, would not have surfaced. “I found some interesting stuff that wasn’t obvious to me when I started,” Shostack says. The tool he used was PHANTOM-B, a framework he developed to focus threat modeling on the LLM components of a system.

PHANTOM-B is designed to complement, not replace, STRIDE. While STRIDE applies across an entire application, PHANTOM-B zeroes in on the parts that interact with large language models. The name is an acronym for a set of AI-specific threats: Prompt injection, Hallucination, Anthropomorphization, Non-explainability, Training issues, Overreliance (including data quality or “poison”), Missing security engineering, and Bias.

“I’m a big fan of: Let’s make it inexpensive to do this work,” says Shostack, who presented PHANTOM-B at Black Hat USA 2026. He argues that CISOs and their organizations are squeezed between two pressures: the demand to secure AI quickly and the difficulty of applying existing security tools to it. “Businesses need an entry point, which is something that you can fit into a one-hour meeting or a 10-minute conversation with an executive,” he says.

Why traditional threat modeling falls short

Threat modeling has always been a struggle. Organizations of all sizes struggle to do it consistently because it takes time and often can’t keep pace with the number of applications being built and updated. As a result, many companies reserve it for their most critical systems, leaving much of their software without the same scrutiny.

“AI didn’t break threat modeling. It exposed weaknesses that were already there,” says Jeff Williams, founder of OWASP and founder and CTO of Contrast Security. He argues that threat modeling has never been standardized or automated enough to scale. The hardest part is not naming threats but figuring out which applications work. “Traditionally, we’ve attempted that with surveys, questionnaires, interviews, stale Visio diagrams, and spreadsheets,” Williams says. “The resulting model is often incomplete before the exercise even begins, and it becomes obsolete as soon as the application changes.”

Generative and agentic AI amplify those weaknesses because their behavior is not easy to predict. Conventional software follows rules that engineers can trace from input to output. AI systems interpret natural-language instructions and generate probabilistic responses, so the same request may produce different results. And when agents can access data or call tools, those results can trigger actions elsewhere.

“Traditional threat modeling was applied to systems that were generally deterministic. The modern generative and agentic AI systems are non-deterministic, so modification of how we view their risk profiles is needed,” says Brian Glas, vice president of consulting services at CODIFIC and one of the project leads for the OWASP Top 10. This distinction also appears in Microsoft’s AI threat modeling guidance, which notes that the boundary between data and commands can blur, creating new attack surfaces and enabling failures to spread across tools, memory, and agents.

Faster cycles, more complexity

At the same time, faster development cycles are leaving teams with less time to think through how systems should be designed and secured. And complexity also makes it harder to examine threats in isolation. A weakness in a document, for instance, could alter a model’s response and cause an agent or tool to take an unsafe action in another system. OWASP warns that these risks can spread across the entire retrieval-augmented generation pipeline.

“A chaining of threats or weaknesses results in the exploitation of the application,” Glas says. The answer, he and others suggest, is to make threat modeling an ongoing practice — a recommendation that is made in NIST’s Artificial Intelligence Risk Management Framework.

But despite a growing body of guidance — including Shostack’s PHANTOM-B, Microsoft’s AI threat modeling recommendations, NIST’s Generative AI Profile, and OWASP’s Top 10 for LLM Applications — there is still no definitive playbook for threat modeling AI systems.

“Threat modeling is not a mature practice,” Glas says. “I believe we are still trying to figure this out.”

What 15 minutes can do

A 15- or 30-minute threat-modeling exercise sounds deceptively manageable. Shostack says a short session is not supposed to be exhaustive. It should, however, identify enough meaningful risks to guide the next decision: Is the risk acceptable, does the system need to change, or is a deeper review required?

“One of the things that we’ve learned from the agile world is that when you make the period of work small, you iterate more and faster,” he says. If the first session misses the mark, the team can erase the whiteboard and try again without losing days or weeks of work. “You make the experiments cheap, and when the experiment is cheap, you can run it repeatedly.”

The result should be a set of concrete stories about how the system could fail. Those scenarios can help CISOs understand the risks they are accepting and help technical teams choose appropriate controls, which can mean limiting data access, narrowing tool permissions, adding human approval points, or even reconsidering whether an LLM is needed at all.

Gaps that PHANTOM-B tries to fill

PHANTOM-B aims to fill in gaps left by other frameworks. It touches on relevant but sometimes exotic concepts such as anthropomorphization, asking whether users or developers are treating the model as if it understands intent, reasons like a person, or will reliably follow an instruction such as “Do not do X.” That misplaced trust can shape how a system is designed and how much authority it receives.

Non-explainability raises another problem. If an LLM screens résumés, evaluates medical images, or makes business decisions, an organization may need to justify its output. But asking the model to explain itself may produce a plausible-sounding answer rather than an accurate record of how it reached the original result. Because LLMs are probabilistic, rerunning the same request may not reproduce the same output, making investigation and debugging more difficult.

The “missing security engineering” prompt is a reminder that adding an LLM does not eliminate conventional software risks. According to Shostack’s white paper, it can magnify them, particularly when organizations rush to deploy vibe-coded software that no one fully understands.

This tool, together with the work from OWASP, NIST, CISA, and others can help teams understand the emerging risk landscape. “Threat modeling is a creative process,” Williams says.

Mistakes to avoid when threat-modeling AI

LLM-powered tools may be relatively new, but they are still apps. They use third-party libraries, process untrusted input, handle sensitive data, expose APIs, and depend on identity and authorization controls.

Focusing heavily on new AI threats while ignoring the basics can leave organizations exposed to preventable vulnerabilities. “The biggest mistake is forgetting that all the existing risks still apply,” Williams says. He advises CISOs not to jump straight to prompt injection or model manipulation while skipping the fundamentals. “That’s backwards,” Williams says. “Traditional application security has to be the foundation. The novel AI risks are an additional layer on top of it, not a replacement for it.”

Glas agrees. “Details have changed; fundamentals of risk have not,” he says. “Don’t throw away the security lessons learned in the last 25 years to chase velocity.” Glas also points to another common problem: missing context. Teams may not fully understand how an LLM fits into the broader application, including its access controls, trust boundaries, inputs, and data flows. Without that view, important risks can easily slip through the cracks.

Trying to catalog every conceivable attack is not the answer, either. “You can’t possibly track them all,” Glas says. “The number of possible attacks is almost limitless.” Instead, he recommends starting with the assets that need protection and the preventive and detective controls already in place. From there, teams can model different attack types to uncover weak or missing controls and determine what is needed to reduce risk to an acceptable level.

Why this matters for CISOs

The takeaway for security leaders is that threat modeling for AI doesn’t have to be a massive, months-long exercise to be useful. A short, structured session like the one Shostack ran can surface non-obvious risks and drive faster decisions about whether to accept, fix, or review deeper. But the experts interviewed here all caution that speed shouldn’t come at the cost of abandoning the fundamentals of application security. For CISOs, the real challenge may be finding the balance: adopting new frameworks like PHANTOM-B while ensuring that the traditional security controls that protect any application — AI-driven or not — remain firmly in place.

As AI becomes more embedded in business processes, the ability to quickly and consistently assess its risks will likely become a competitive differentiator. Teams that can integrate lightweight threat modeling into their development cycles may be better positioned to catch problems before they become incidents. That said, as Glas notes, the practice is still maturing, and there is no one-size-fits-all playbook yet. The best approach may be to start small, iterate, and learn from each session — while keeping the fundamentals in view.