As the capabilities of generative artificial intelligence (AI) continue to advance, the risks associated with large language models (LLMs) have become increasingly prominent. While the focus on “prompt injection” has gained traction as a potential threat vector, this narrative simplifies a more complex and alarming reality. Recent discussions highlight the emergence of “promptware,” a sophisticated class of malware execution mechanisms targeting LLMs. A structured seven-step “promptware kill chain” has been proposed to help policymakers and security practitioners address the evolving AI threat landscape.
The first step in the promptware kill chain is known as Initial Access, where malicious payloads infiltrate AI systems. This can occur directly, through the input of harmful prompts, or indirectly via “indirect prompt injection.” In the latter case, attackers embed malicious instructions within content that LLMs retrieve, such as web pages or emails. As LLMs evolve into multimodal systems capable of processing diverse input types, including images or audio, the possibilities for embedding these threats expand significantly.
The architecture of LLMs presents a fundamental challenge. Unlike traditional computing systems that maintain a clear separation between executable code and user data, LLMs treat all input as a continuous stream of tokens. This lack of an architectural boundary means that malicious instructions can be processed with the same authority as legitimate commands, allowing for significant exploitation opportunities.
Understanding the Phases of the Promptware Kill Chain
Prompt injection is just the beginning of a complex, multi-stage operation that mirrors traditional malware campaigns such as Stuxnet or NotPetya. After Initial Access, the attack progresses to Privilege Escalation, often referred to as “jailbreaking.” In this phase, attackers bypass the safety protocols and policy guardrails established by AI developers. Techniques akin to social engineering are employed, convincing the LLM to assume a persona that disregards its built-in restrictions. This escalation enables the attacker to fully leverage the model’s capabilities for malicious purposes.
Following privilege escalation comes the Reconnaissance phase, where the compromised LLM is manipulated to disclose information about its connected services and operational capabilities. This strategic manipulation allows the attacker to advance through the kill chain without raising alarms, effectively turning the LLM’s reasoning capabilities against itself.
The next phase is Persistence. A transient attack that fades after a single interaction is manageable, but when an attack can embed itself into the long-term memory of an AI agent, it poses serious risks. For example, a worm could infect a user’s email archive, ensuring that malicious code is executed every time the AI summarizes past emails.
Command and Control, Lateral Movement and Actions on Objectives
Command-and-Control (C2) is the stage where persistent threats evolve. This phase allows attackers to dynamically modify their strategies and objectives, transforming a static threat into a more adaptable and dangerous entity. While not strictly necessary for the kill chain to progress, C2 enables the promptware to adapt its behavior based on the attacker’s commands.
The sixth stage, Lateral Movement, is critical as it allows the attack to extend from the initial victim to other users or systems. With AI agents increasingly integrated into our emails, calendars, and enterprise platforms, the potential for malware propagation grows exponentially. In a self-replicating attack, an infected email assistant could unwittingly forward malicious payloads to all contacts, akin to a computer virus spreading through a network.
Finally, the kill chain culminates in Actions on Objective. The ultimate goal of promptware is often not merely to manipulate a chatbot but to achieve tangible malicious outcomes. This can include data exfiltration, financial fraud, or even physical-world implications. There are documented instances of AI agents being coerced into actions like selling cars for a dollar or transferring cryptocurrency to an attacker’s wallet. Alarmingly, agents with coding capabilities can be tricked into executing arbitrary code, giving attackers complete control over the AI’s underlying systems.
Real-World Examples of the Promptware Kill Chain
Two notable research studies have illustrated the effectiveness of the promptware kill chain. The first, titled “Invitation Is All You Need,” demonstrated how attackers gained initial access by embedding a malicious prompt in the title of a Google Calendar invitation. This prompt utilized a delayed tool invocation technique, causing the LLM to execute the embedded instructions. Because the prompt was part of a Google Calendar artifact, it persisted in the user’s long-term memory, leading to lateral movement when the Google Assistant was instructed to launch other applications.
The second study, “Here Comes the AI Worm,” showcased a similar end-to-end realization of the kill chain. In this scenario, initial access was gained through a prompt embedded in an email sent to the victim. The injected prompt coerced the LLM to replicate itself and exfiltrate sensitive user data, resulting in off-device lateral movement when the email assistant was asked to draft new messages. This action inadvertently spread the infection to new recipients, demonstrating the sublinear propagation of the attack.
Implications and Defensive Strategies
Understanding the promptware kill chain provides a framework for identifying and mitigating these emerging threats. Given that prompt injection is not something that can be easily fixed within current LLM technology, a comprehensive defensive strategy is essential. This strategy should assume initial access is inevitable and focus on breaking the chain at subsequent steps. This includes:
- Limiting privilege escalation
- Constraining reconnaissance efforts
- Preventing persistence of malicious payloads
- Disrupting command-and-control capabilities
- Restricting the actions an AI agent is permitted to take
By shifting from reactive measures to systematic risk management, organizations can better secure critical systems that are increasingly reliant on AI technology.
As the landscape of AI continues to evolve, ongoing research and proactive measures will be crucial in addressing the challenges posed by promptware and similar threats. Staying informed and prepared will be vital for organizations seeking to navigate the complexities of AI integration without compromising security.
For readers interested in the implications of AI security, sense free to share your thoughts and engage in the discussion.
