Rising AI Prompt Injection Attacks Pose New Security Risks, Warns NCSC

The National Cyber Security Centre (NCSC) has warned about AI prompt injection attacks, exposing organizations to a new class of stealthy manipulation. Unlike conventional vulnerabilities, these attacks exploit the way LLMs interpret text, which forces security teams to rethink their defenses to protect critical systems.

How does prompt injection differ from traditional vulnerabilities?

A prompt injection occurs when untrusted user input is combined with developer-provided instructions in a large language model (LLM) prompt. It allows attackers to embed hidden commands within normal content and manipulate the model’s behavior. Unlike traditional vulnerabilities, where code and data are clearly separated, LLMs process all text as part of the same sequence, which makes every input potentially influential and creates unique security challenges.

On the other hand, SQL injection exploits databases by inserting malicious commands into queries that take advantage of the clear boundary between executable instructions and stored data. LLMs lack this distinction and treat all text uniformly, which enables attackers to hide harmful instructions within seemingly harmless content and makes prompt injection harder to prevent than traditional injection attacks.

“Under the hood of an LLM, there’s no distinction made between ‘data’ or ‘instructions’; there is only ever ‘next token’. When you provide an LLM prompt, it doesn’t understand the text it in the way a person does. It is simply predicting the most likely next token from the text so far. As there is no inherent distinction between ‘data’ and ‘instruction’, it’s very possible that prompt injection attacks may never be totally mitigated in the way that SQL injection attacks can be,” NCSC explained.

Prompt injection attacks have increasingly drawn attention over the past few years as experts warn that attackers can exploit AI models by embedding hidden instructions within user input. These manipulations can trick models into generating harmful or misleading outputs, which poses risks to data integrity, system security, and user trust. As AI adoption grows, understanding and mitigating this threat has become a top priority for organizations.

Strategies to mitigate prompt injection attacks

To protect systems using LLMs, organizations should start by designing secure architectures that separate trusted instructions from untrusted inputs. This includes filtering or restricting user-generated content before it reaches the LLM and applying deliberate constraints around how the model communicates with internal systems. They must also embed user input within clearly tagged or bounded segments of a prompt to further reduce the risk that hidden commands will be interpreted as legitimate instructions.

Additionally, ongoing monitoring and proactive testing are equally important in enterprise environments. Administrators should incorporate logging of LLM inputs and outputs, external API calls, and model-triggered actions to detect anomalies early. They must also adopt a red-teaming mindset and conduct formal security reviews at each stage of development.