Training data extraction is an attack that gets an AI model to reveal pieces of the data it was trained on. Models can memorize rare or often-repeated passages. The right kind of prompt can make them repeat those passages back. For example, imagine a chatbot that was trained on old support emails. Someone could coax it into typing out a customer’s name, phone number and the problem that customer wrote in about.
Why it matters
Your organization might train or fine-tune a model on sensitive records, such as donor files, client case notes or staff emails. If you do, plan for the chance that some of that text could come back out to anyone who uses the model. Think carefully before you feed those records into training, and remove personal details first where you can. If you use an outside AI tool, ask the vendor a direct question: is our content used to train your models, and can we turn that off? Get the answer in writing.
Related risks include membership inference, model inversion and sensitive information disclosure. For how to keep your content out of a vendor’s training, see training opt-out.
Published Updated