Design risk-based human approval for AI agent actions

Start with observable consequences

Treat the first pass as an architecture exercise. List the facts that change the consequence of an operation, then mark where each fact comes from and whether the system can trust it. That list can be broader than ActionProxy's current YAML: tool, operation, destination, amount, data class, scope, reversibility, requester, environment, and recent frequency may all matter to a design.

Turn the combinations you can actually establish into explicit outcomes. A bounded call may fit an allow lane, another may need human context, and a prohibited action belongs in deny. Any signal that lacks a trusted mapping should remain a documented gap rather than an optimistic condition.

The current YAML recognizes a smaller condition set

ActionProxy's evaluator recognizes actionId, appId, workflowId, operationKind, risk, customerVisible, recipientDomain, amount, currency, and approverGroup. Amount can be an exact number or use gt, gte, lt, lte, and eq thresholds. Every condition listed on a rule must match; an unknown condition key does not match.

Trust mappings narrow that list further on canonical HTTP. workflowId and approverGroup are absent until a trusted resolver supplies them, and caller risk metadata is not authoritative. Recipient-domain derivation is coarse, while customer visibility is available only for explicit known-tool mappings. Putting a desired signal in ordinary metadata does not make it trusted policy context.

An illustrative first matrix

For a support assistant, searching public documentation might be allowed. Sending a reply to a known customer could require approval when the message includes an attachment or a new recipient. Bulk outbound email could be denied until a separate campaign process exists. These are examples, not universal defaults; the right boundary depends on the business process and the facts your gateway can trust.

For refunds, a team might allow a narrow amount to the original payment method, review a higher band, and deny requests beyond an explicit ceiling. Amount alone is not enough: order identity, currency, requester, prior refund state, and destination can matter just as much.

Show enough context to make a decision

A review card should name the action, meaningful arguments, requester, reason for review, applicable bounds, and the effect expected after approval. Raw JSON alone can hide the important difference between two calls. A friendly summary alone can omit the field that changes the risk. Present both a concise explanation and the structured values that authorization will bind.

Measure what each rule sends to review

Track why calls enter review, how often they are rejected or edited, which rules dominate the queue, and where outcomes are unknown. A high approval rate may mean the rule is correctly catching rare-but-valid work, or it may mean the lane is too broad. Read samples before changing policy.

Move a pattern into the allow lane only after its inputs are reliably bounded and the team accepts the residual consequence. Move recurring prohibited patterns into explicit deny rules. Keep a change record so a queue improvement does not become an invisible expansion of authority.