Membership inference

Membership inference is an attack that tries to tell whether a particular person’s record was part of the data an AI model was trained on. The attacker gives the model that record, then compares how the model responds to it with how the model responds to records it has never seen. Picture a model trained on the client list of a local health service. If someone can confirm that your neighbor’s record was in that list, they have learned that your neighbor used the service, even though the model never revealed a single detail about them.

Why it matters

The danger here is that even a plain yes or no can expose something private. MITRE ATLAS, a public catalog of attacks on AI systems, lists membership inference among its known techniques. If your organization trains or fine-tunes a model on records about clients, donors or volunteers, ask the vendor how they protect against this. Ask whether they use methods such as differential privacy and whether they have tested the model for this risk. Related attacks to know by name are training data extraction and model inversion.