Most operational software should not use AI, and most of it does not need to. Recording a sale, calculating a settlement, enforcing a permission, posting to a ledger: these must be exactly right every time, and the tools for that are relational databases, tested code and explicit rules.
But every operation has a layer of work that is not like that. Someone reads a form and types what it says. Someone looks at a photograph and writes a description. Someone reads a long thread and summarises it for a colleague. Someone decides which category a request belongs in. This work is interpretive, it is repetitive, and it is where language models are genuinely useful.
The design question is how to use them there without letting them anywhere near the parts that must not be wrong.
A working example
In the retail platform Opsenium built for Bishops Emporium, staff photograph an item at the counter and the system produces a draft catalogue entry: a title, a description, a category and attributes. The model is given the photographs and a strict output schema. Its response is validated against that schema, scrubbed for the kinds of phrasing that models produce and antique dealers do not, and presented as a draft.
A person reviews it, corrects it and saves it. The audit trail records both the draft and the saved version. The model never sets a price, never touches stock and never publishes anything.
The same platform includes a help assistant on the till and in the admin. It answers questions from a handbook that was written by hand and deploys with the code, so its knowledge is exactly what the shop has documented, no more. It has one action available, raising a support case, and it asks for consent before taking it.
The controls that matter
The details differ by system, but the shape of the controls does not.
Constrain the output. Free text is a liability. Ask the model for a structure, validate it, and reject what does not fit. A draft that fails validation is a failed draft, not a corrupted record.
Separate proposing from committing. The model proposes; a named person commits. This is the single most important control, because it keeps responsibility with a human and keeps the audit trail honest.
Ground the model in your own material. For assistants, answer from a maintained corpus rather than general knowledge, and treat the corpus as part of the codebase: versioned, reviewed and deployed together.
Test the failure modes. Models make characteristic mistakes in each domain. Find them, write checks for them, and test the checks. A scrubbing function with a test suite is worth more than a prompt with a warning in it.
Keep the keys and the costs where you can see them. Call models through resources in your own cloud subscription, in your own region, with keys in a vault and a usage cap. AI features should be as operable as everything else.
Say no where it does not fit. If a task needs to be exactly right, build the deterministic version. The credibility of AI features depends on the ones you chose not to build.
What to look for
If you are evaluating where AI might help in an operation, look for tasks where a person currently reads or writes unstructured material as a routine step, where a wrong draft is cheap because a person will review it, and where the volume is enough that the saved minutes add up.
Then ask of any proposed feature: what is the schema, who commits, what is the model grounded in, how are its mistakes caught, and where do the keys live. If the answers are clear, the feature is probably worth building. If they are not, it is probably a demonstration rather than a capability.
AI in operational software is unglamorous when it works. It removes a few minutes of typing from a task a hundred times a week, and the person doing that task barely notices it is there. That is the standard to aim for.