An inference server is software that loads an AI model and answers requests sent to it by other programs or people over a network. Organizations use one when they want to host a model on their own computers instead of using a vendor’s cloud service. For example, a small charity might set one up on an office machine so staff can ask an open-weight model to summarize grant reports without sending the text to an outside company. For the work the model does when it answers, see inference.
Why it matters
Running your own inference server gives you more control, but it also makes you responsible for keeping it safe. If the server can be reached from the internet without strong access controls, or if nobody installs updates, an attacker may be able to get into the machine it runs on. Before you or a volunteer set one up, ask who will apply updates, who is allowed to connect, and whether it can be reached from outside your office network. If a vendor or consultant runs it for you, ask them the same questions and get the answers in writing. You can also check the vulnerabilities list on this site for known problems with inference software.
Published Updated