AI agents in business software: what they may do alone, and what a person approves

An AI agent does more than answer a question. It reads, decides on its next step and acts through the tools it has been given, such as your customer records, your mailbox or your accounting. That makes it useful in some parts of a process and risky in others. This article is about drawing that line.

What is an AI agent in business software?

A program in which a language model chooses its own next step and carries it out through tools, such as looking up a record, filling in a form or sending a message, until the task is done or a limit is reached. Anthropic, which makes the Claude models, defines agents as "systems where LLMs dynamically direct their own processes and tool usage". An LLM is a large language model, the kind of AI that reads and writes text.

The alternative is software in which the steps are fixed in advance and an AI model does one of them, such as reading an invoice. In Anthropic's words, there the models and tools "are orchestrated through predefined code paths". Its advice is to find "the simplest solution possible", which "might mean not building agentic systems at all", and it notes that for many applications a single, well-prepared call to the model "is usually enough".

So before choosing an agent, ask whether the process really has steps that cannot be fixed in advance. If it does not, fixed steps are the better fit: Anthropic says they "offer predictability and consistency for well-defined tasks". Where AI helps inside such steps, and where a fixed rule should decide, is the subject of AI in business software.

Which tasks can it do alone?

Tasks where it only reads and prepares, and where a wrong step is easy to notice and cheap to undo: collecting information from several places, sorting incoming requests, filling in drafts, checking documents against a list, or preparing a case for a person to decide. The less it can change, the more can be left to it.

An invented example: a building services firm receives service requests by email. An agent reads each request, looks up the customer and their installation, sorts the request by urgency and prepares a draft job for the dispatcher. For that it needs to read the mailbox and the customer records and to create drafts. It does not need to send emails, change prices or delete anything, so it should not be able to.

The OWASP Gen AI Security Project, which publishes a list of the top risks in software built on language models, calls the opposite "Excessive Agency" and traces it to "excessive functionality; excessive permissions; excessive autonomy". Its first advice is to "limit the extensions that LLM agents are allowed to call to only the minimum necessary". Its own example is an assistant that summarises emails: it "may only require the ability to read emails", not to delete or send them.

Which decisions stay with a person?

Anything that spends money, sends something to a customer or the public, changes what someone owes, or affects a person's rights. The agent can prepare such a decision and set out its reasons, but a person approves it before it takes effect. OWASP advises the same: use human-in-the-loop control "to require a human to approve high-impact actions before they are taken".

Two details make this work. First, the approval is enforced by the software, not requested in the agent's instructions. OWASP's advice is to "implement authorization in downstream systems rather than relying on an LLM to decide if an action is allowed or not". Instructions can be overridden by what the agent reads; a button that only a person can press cannot. Second, the person sees what they approve: the draft itself, the data behind it and what will happen when they press the button.

Decisions about people carry a legal duty as well. Under the Swiss Data Protection Act, a person affected by a decision "based exclusively on automated processing" that has "a legal consequence for or a considerable adverse effect" on them must be informed, and may request that it "be reviewed by a natural person" (Art. 21, with exceptions). More on this in AI in business software.

How do you check what it did?

Through a record of every step: what the agent read, which tool it used with which input, what came back and what it did next. With that record, a person can review a sample each week, follow up a complaint or undo a step. A limit on how many actions it may take in a given time, with an alert when the limit is reached, helps catch problems before they grow.

OWASP lists both as ways to limit the damage: "log and monitor the activity" of the agent's tools and the systems they reach, and "implement rate-limiting" so that problems can be discovered "before significant damage can occur".

Checking starts before the agent touches real data. Anthropic recommends "extensive testing in sandboxed environments, along with the appropriate guardrails", and notes that it is common to include "stopping conditions (such as a maximum number of iterations) to maintain control". In practice: let it work through sample cases in a test version first, compare what it did with what a person would have done, and give it a fixed maximum number of steps per task.

Where are the limits?

An agent can be wrong, can be misled by what it reads, and can cost more than planned. None of this can be ruled out completely, so the limits are built around the agent rather than left to it: narrow access, approval for what matters, a cap on steps and spending, and a person who owns the result.

• It can be wrong, convincingly: OWASP describes language models producing content "that seems accurate but is fabricated". In an agent, one wrong step can feed the next: Anthropic warns of "the potential for compounding errors".
• It can be misled: an email or a document can contain instructions aimed at the AI. OWASP notes that such inputs "do not need to be human-visible/readable", and that two techniques for improving answers "do not fully mitigate" the risk.
• It costs more than a single answer: agent-based systems "typically make repeated calls to an LLM" (OWASP), and Anthropic writes that "the autonomous nature of agents means higher costs".
• It is not accountable: someone in your company owns each process the agent works in, and that person decides when it may do more.

A sensible first agent prepares and a person approves. Widen what it may do alone only once its record shows that it handles those cases well.