Securing generative AI applications

Who this is for and what you will get

This guide is for developers who are connecting large language models (LLMs) to products, and for people who review applications before launch. By the end you will be able to separate the main risk categories for AI systems, know what to check in training data, prompts and model outputs, and know where public references such as MITRE ATLAS and the OWASP list are.^[1]

What security means for AI systems

As AI and machine learning shape daily life more and more, you need to protect both customer data and the AI systems themselves. In high-value decision-making, a wrong decision can have serious consequences.^[1]

Machine learning models are largely unable to tell malicious input from benign but unusual data. Much training data comes from uncurated, unmoderated public datasets that accept third-party contributions. Attackers do not need to break into a dataset if they are free to contribute to it. If the structure and format of the data look correct, low-confidence malicious data can become data the model trusts over time.^[1]

Protecting the integrity of the data stores that models use for decisions is therefore a basic requirement.^[1]

Risk one: data poisoning

Data poisoning is when someone intentionally changes the information used to train an AI so that it makes mistakes. It is considered the most significant security threat for AI today. Standardized detection and mitigation methods are still missing, and training relies on untrusted or uncurated public data. To prevent a flawed training process, track where your data comes from and how it moves through your pipeline. Otherwise, “garbage in, garbage out” still applies.^[1]

The main forms of poisoning are these:

1. Label flipping: in a binary classification task, an adversary intentionally flips the labels of a small subset of training data, for example labeling benign samples as malicious. One example is a spam filter that marks legitimate emails as spam because of manipulated labels.^[1]
2. Feature poisoning: an attacker subtly modifies features in the training data to add bias or mislead the model. One example is adding irrelevant keywords to product descriptions to manipulate a recommendation system.^[1]
3. Data injection: malicious data is added to the training set to influence behavior. One example is fake user reviews that skew sentiment analysis.^[1]
4. Backdoor attacks: an adversary plants a hidden pattern in the training data, and the model behaves maliciously when the pattern is triggered. One example is a face recognition system trained with backdoored images that misidentifies a specific person.^[1]

Risk two: attacks on LLM applications

MITRE built ATLAS (Adversarial Threat Landscape for Artificial-Intelligence Systems), a searchable knowledgebase of tactics and techniques used in real-world attacks on AI systems. It is modeled after the MITRE ATT&CK framework and is useful for planning defenses.^[1]

OWASP publishes a Top 10 list of the most critical vulnerabilities in applications that use LLMs. Besides data poisoning, the list covers the following.^[1]

• Prompt injection: attackers manipulate an LLM with carefully crafted inputs so that it behaves outside its intended behavior.^[1]
• Supply chain vulnerabilities: the components an LLM application depends on, such as Python modules or external datasets, can be compromised. The result can be unexpected behavior, introduced bias and weaknesses in the underlying infrastructure.^[1]
• Overreliance: LLMs are fallible and prone to hallucination, so they can give inaccurate or unsafe results. Documented cases show people accepting such results at face value, with negative real-world consequences.^[1]

Security testing methods

Security testing evaluates an AI system or LLM by identifying and exploiting its vulnerabilities. Developers, users or third-party auditors can run it, depending on the purpose and scope. Four common methods are:^[1]

• Data sanitization: remove or anonymize sensitive or private information from training data or inputs, which reduces the exposure of confidential data.^[1]
• Adversarial testing: generate adversarial examples and apply them to inputs or outputs to measure robustness, which can reveal weaknesses attackers could exploit.^[1]
• Model verification: check the correctness and completeness of model parameters or architecture. This can help detect and prevent model stealing by confirming the model is protected and authenticated.^[1]
• Output validation: check the quality and reliability of outputs, which can detect and correct malicious manipulation.^[1]

OpenAI has set up a series of safety evaluations as part of its red teaming network initiative, to test the outputs of AI systems. The evaluations range from simple question-and-answer tests to complex simulations.^[1]

In the persuasion group, MakeMeSay asks how well one AI can trick another into saying a secret word, MakeMePay asks how well an AI can convince another to donate money, and Ballot Proposal asks how well an AI can influence another's support for a political proposition. The steganography group includes Steganography, which asks how well an AI can pass secret messages without being caught, Text Compression, which asks whether compression can help hide messages, and Schelling Point, which asks how well two AI systems can coordinate without direct communication.^[1]

Data protection: share less, check outputs, watch for anomalies

LLMs can pose risks to privacy and data security. They can memorize and leak sensitive information from training data, such as personal names, addresses, passwords or credit card numbers, and they can be manipulated by malicious users who exploit their weaknesses or biases.^[1]

There are three practical steps. First, limit the amount and type of data you share with LLMs. Send only what is necessary and relevant, avoid sensitive, confidential or personal data, and anonymize or encrypt what you must send over secure channels.^[1] Second, verify the data an LLM generates, so you can catch unwanted or inappropriate content.^[1] Third, report suspicious or abnormal behavior, such as irrelevant, inaccurate, offensive or harmful output, since it may indicate a breach or security incident.^[1]

Organizations that use data and AI across multiple clouds face a harder version of this problem. Structured data, unstructured data and AI-generated data may sit in different places and clouds. Useful practices include using cloud services with data protection and privacy features, using data quality and validation tools to find errors and anomalies, and adopting data governance and ethics frameworks so that data is used responsibly and transparently.^[1]

Red teaming: emulating real-world threats

Emulating real-world threats, using the same tools and tactics an attacker would use, is now a standard practice for building resilient AI systems. It identifies the risks to a system and tests how defenders respond.^[1]

Microsoft's AI Red Team has widened the scope of red teaming. It looks for security vulnerabilities, but also for other failures such as the generation of potentially harmful content.^[1] Traditional security red teaming focuses mainly on malicious actors. AI red teaming also considers regular users who may run into problematic content, and it covers fairness issues such as stereotyping.^[1] Because AI applications keep changing, red teaming has to be continuous.^[1]

Red teaming complements other controls, such as role-based access control (RBAC) and comprehensive data management. It supplements a strategy for safe and responsible AI rather than replacing these controls.^[1]

Limits and common questions

Access to data should be assigned by role. Of three possible measures for preventing manipulation and misrepresentation of the data LLMs use, assigning proper data access privileges to users is the one that goes furthest.^[1] Layered controls matter more than any one measure, so combine access control, data handling and testing.^[1]

Next steps

To keep learning about protecting sensitive information in AI applications, read the official Microsoft material on governing and protecting data in the age of AI. The next step in the lifecycle of an AI application is covered by the material on the generative AI application lifecycle.^[1]

Adapted from lesson 13 of Microsoft's open-source course *Generative AI for Beginners*^[1], released under the MIT License.