Model inversion

Model inversion is an attack in which someone asks an AI model many questions and studies its answers to piece together sensitive information about the data it learned from. The attacker usually needs no special access, because repeated questions through the model’s public interface can be enough. A well-known example involves a face recognition model. By probing it over and over, an attacker can rebuild a rough likeness of a person whose photos were used to train it.

Why it matters

Suppose your organization trains or fine-tunes a model on personal data, such as client records, donor details or staff photos. In that case the model itself can leak some of that information, even if no one ever sees the original files. This is one reason to limit who can query a model trained on personal data. When a vendor offers to build a model from your records, ask who will be able to send it questions, whether there is a limit on how many questions they can send, and what protections the vendor uses against this kind of attack. Related risks include membership inference and training data extraction. Vendors may also bring up differential privacy when you ask these questions.