Handling an AI incident

This lesson covers what counts as an AI incident at a small organization and what to do in the first hours and days after one. When something goes wrong, the first afternoon matters most. A short plan written ahead of time turns a scary moment into a list of steps.

What counts as an AI incident

An AI incident is any event where an AI tool causes, or nearly causes, harm to your people, your data, your money or your reputation. You don’t need a hacker for it to count. Most incidents at small organizations start with an ordinary person trying to get work done faster.

Here are some typical examples. All of them are illustrations, not reports of real events:

• Restricted data in the wrong tool. A staff member pastes client names, health details or donor records into an AI tool your organization has not approved.
• Wrong facts sent out. An AI-written grant report or donor newsletter goes out with made-up numbers, a wrong name or a promise you can’t keep.
• A fake voice or video. Someone receives a convincing call or video message that sounds like your executive director asking for an urgent payment. This is a deepfake.
• An exposed account or key. The password to your organization’s AI tool account, or an API key (a secret code that lets software use an AI service on your behalf), ends up in a shared document, an email or a public website.
• Attacked software. Your organization runs AI software on its own server, and an attacker uses a known weakness in it to get in.

Near misses count too. If someone almost wired money to a deepfake caller, or caught a wrong figure in a funder report an hour before sending it, treat it as an incident. Near misses are free lessons.

A worked example: one afternoon at a small food bank

This example is made up to show how the steps fit together.

A small food bank uses volunteers to help with intake. On a Tuesday afternoon, a volunteer wants to sort the client list by neighborhood. She copies the list (names, phone numbers and addresses of about 200 households) and pastes it into the AI chat tool on her personal account. She mentions it to the office manager at 2 p.m., pleased with the result.

Contain (2:00 to 2:30). The office manager thanks the volunteer for speaking up and asks her to stop using the tool for work. Together they open the chat history on the volunteer’s account and delete the conversation, using the delete option the tool provides. The office manager also checks the account’s settings for any option that lets the tool use chats to improve its service, and turns it off. She writes down each step and the time.

Assess (2:30 to 3:15). The office manager answers four questions in the incident log:

• What data was involved? Names, phone numbers and home addresses of clients.
• Whose data? Clients of the food bank, some of them in vulnerable situations.
• How many people? About 200 households.
• Did it leave our control? Yes. It went to an outside company under a personal account with no agreement with the food bank.

Notify (3:15 to 4:30). She calls the executive director, who agrees this is serious. They read the AI company’s published guidance on deleting data and send a request through its official channel asking that the data be removed. The director calls the board member who handles legal questions to ask whether the food bank must tell clients, its main funder or a regulator. They agree to get a clear answer before the end of the week, because some laws set deadlines for reporting breaches of personal data.

Fix and learn (4:30 to 5:00). The office manager adds a line to the volunteer handbook: client data never goes into any AI tool except the one on the approved list. She plans a 10-minute briefing at the next volunteer shift. Nobody is blamed. The volunteer is thanked again for reporting quickly, because that is exactly what the food bank wants people to do.

By 5 p.m., the food bank has stopped the problem, knows how big it is, has the right people involved and has changed one rule. That is a good afternoon’s work.

The steps to take

Use these four steps in order. Each can take minutes or days depending on the incident.

1. Contain

Stop the harm from growing. Containing comes before investigating.

• Stop the activity. Ask the person to stop using the tool, pause the automated process or take the software offline.
• Remove or revoke access. Change passwords, cancel exposed API keys and disconnect any apps linked to the account.
• Delete the content where the vendor’s tools let you, such as chat history or uploaded files.
• Write down what you did and when. You will need this record later, and memory fades fast.

2. Assess

Work out how serious it is. Answer these questions in writing:

• What data or systems were involved?
• Whose data was it: staff, clients, donors, volunteers?
• How many people could be affected?
• Did the data leave your control, and where did it go?

The answers shape everything that follows. A wrong date in an internal memo is minor. Client health details in an unapproved tool are not.

3. Notify

Tell the right people, in the right order.

• Tell leadership right away, even if you’re not sure how bad it is.
• Tell the vendor through its official support or security channel, especially if an account or key was exposed.
• Decide, with advice, whether affected people, funders or a regulator must be told. Some laws set deadlines for reporting personal data breaches, so don’t let this decision drift. If you don’t have in-house legal help, ask a board member, a pro bono lawyer or your insurer.

4. Fix and learn

Close the gap so it doesn’t happen again.

• Correct anything published. Send a correction to donors or funders if wrong facts went out.
• Update your AI policy, your approved tools list or the settings on the tools you use.
• Brief staff and volunteers on what changed and why. Keep it short and practical.

If you run AI software on your own server

Some organizations host AI software themselves, often with help from a volunteer or contractor. If a security advisory for that software says the weakness is “known exploited,” treat it as urgent. That label means the weakness is listed in the CISA Known Exploited Vulnerabilities catalog, a list kept by a US government agency of weaknesses that attackers have actually been seen using. You can read more about it in our glossary entry on CISA KEV. Ask whoever looks after the server to apply the fix or take the software offline the same day. We track these in our advisories.

Keep an incident log

A simple log is one of the most useful things a small organization can have. A shared spreadsheet or a single document is enough. For each event, record:

• the date
• what happened
• the data involved
• the actions taken
• who was told

Log near misses as well. Over a year, the log shows you patterns, such as one tool that keeps causing trouble or one team that needs more training. For a look at how a large AI data incident unfolded, see our guide on the ChatGPT data breach.

Common mistakes

• Blaming the person who reported it. If people get in trouble for speaking up, the next incident stays hidden. Thank reporters, then fix the process.
• Investigating before containing. Every hour spent working out who did what is an hour the key stays active or the content stays online.
• Assuming deletion means it’s gone. Deleting a chat is a good step, but it doesn’t prove the data is erased everywhere. Record what you deleted and still do the assessment.
• Deciding alone about notifications. Whether to tell clients, funders or a regulator is a leadership decision made with advice, not a call for the office manager to make alone.
• Paying an urgent request without checking. For any unexpected payment request by voice or video, hang up and call back on a number you already have.
• Skipping the log for small things. Near misses are where you learn cheaply.

Try it this week

Set a timer for 15 minutes and write a one-page incident card. Keep it to these four parts:

1. Who to call. Name the person to tell first, with a phone number, and a backup if they’re away.
2. The first three containment steps. For example: stop using the tool, change the password or cancel the key, delete the content where possible.
3. Where the log lives. Give the exact location of the shared document or spreadsheet.
4. Who decides on notifications. Name the person who decides whether to tell affected people, funders or a regulator, and who they ask for advice.

Print it, pin it near the office computer and share it with staff and regular volunteers.

Key takeaways

• An AI incident is any event, including a near miss, where an AI tool puts your people, data, money or reputation at risk.
• Contain first, then assess, notify and fix, and write down every action with the time.
• Get advice quickly on whether to tell affected people, funders or a regulator, because some laws set reporting deadlines.
• Treat any “known exploited” advisory for AI software you run yourself as urgent.
• A one-page incident card and a simple log make the first afternoon calm instead of chaotic.