AI agents in healthcare: what they are and where clinicians stay in control

An AI agent is not a chatbot with a grander job title.

A useful healthcare agent can observe a defined part of a patient’s state, work towards a specific goal, choose from a limited set of permitted actions, use approved tools, record what it did and stop when the situation exceeds its authority. A clinician remains responsible for clinical judgement and consequential care decisions.

That definition matters because “agent” is becoming a convenient label for almost any software that produces text. A note summariser may be useful. A search box may be useful. Neither becomes an agent merely because a vendor adds a friendly name and a chat window.

The practical test is not whether the software sounds intelligent. It is whether you can answer six plain questions about how it behaves.

There is also a distinction that most agent discussions blur: deterministic and probabilistic agents solve different problems.

Two kinds of agent, and why healthcare needs both

Kalibra’s design distinguishes two patterns: deterministic Kali workflows and probabilistic Kay workflows. Availability and review controls must be confirmed for your deployment.

Kali is deterministic. It follows programmed rules and structured branches inside an approved workflow. Given the same state, rule version and inputs, it should make the same transition. The intended design lets the clinic inspect which rule fired, which data it used and what happened next.

Kay is probabilistic. It can interpret open-ended language, explain a complex record and handle questions that nobody could sensibly pre-script. That flexibility brings variation. The same prompt can produce different wording or emphasis. A capable model may still omit a fact, overstate an inference or answer outside the intended boundary.

A harness can reduce those risks through structured retrieval, released content, permissions, task-specific tests, source checks, abstention rules and restricted tools. It cannot turn a probabilistic model into a deterministic rules engine.

The useful architecture blends them:

1. deterministic policy establishes eligibility, risk and authority;
2. Kay interprets, asks or explains within that boundary;
3. deterministic checks validate any proposed state change;
4. a named person approves the consequential action;
5. Kali carries the approved structured flow and records progress.

This is a design direction, not a claim that every hand-off is already implemented in every Kalibra deployment. Kalibra’s current public boundary is clear: no autonomous execution today.

The six-part agent test

1. What state can it see?

An agent acts on a representation of the world. In a clinic, that might include selected records, laboratory results, assessments, wearable trends, the active care plan and the status of previously approved actions.

The word selected is important. More data is not automatically better data. An agent that ingests everything but cannot distinguish a current medication list from a five-year-old one has more context and less understanding.

Ask for a source map:

• Which systems and documents can the agent read?
• How does it distinguish current, historical and patient-reported data?
• What happens when two sources disagree?
• Does it show what is missing?
• Can a clinician trace a statement back to the original source?

If the answer is “our model handles that”, the answer is incomplete.

2. What goal is it pursuing?

“Improve health” is not an operational goal. Neither is “support the clinician”.

A bounded goal sounds more like this:

Prepare the patient record for review by identifying material changes since the previous consultation, unresolved follow-up and missing information. Do not propose a treatment decision.

The narrower wording is a feature. It tells the agent what success means and what work remains outside its remit.

Bad objectives create bad behaviour. Researchers have shown how a seemingly reasonable proxy can encode the wrong goal: a widely used population-health algorithm used healthcare cost as a proxy for health need, producing racial bias in who received extra help. The lesson is bigger than that one system. Before asking whether an agent is accurate, ask whether it is optimising the right thing.

3. What is it allowed to do?

Permissions turn an impressive demonstration into a governed workflow.

There is a material difference between permission to:

• read a record;
• draft a summary;
• prepare a patient message;
• place that message in a clinician’s approval queue;
• send the message;
• change a care plan.

Those actions should not inherit the same authority merely because they sit in one interface. A clinic needs a capability-by-capability permission model.

In Kalibra’s proposed operating model, AI may perform bounded mechanical work such as gathering, extracting, normalising, comparing, drafting, reminding, routing and tracking. Diagnosis, ambiguity resolution, treatment decisions and approval remain human responsibilities.

4. Which tools can it use?

An agent becomes consequential when it can do more than generate language. It might query a structured record, calculate a deterministic score, prepare a form, create a task or route an item for review.

Every tool raises three practical questions:

1. Can the agent only read, or can it write?
2. Is the output reversible?
3. Who is told when the tool fails or returns an ambiguous result?

Do not accept “integrates with your stack” as an answer. Ask to see the exact action, the exact system changed, the audit entry and the rollback path.

5. When must it stop?

The safest action is often no action.

An agent needs explicit stopping rules for contradictory evidence, missing critical information, an out-of-scope request, a high-risk signal, a failed tool, low confidence or a situation requiring clinical interpretation.

This is where many demonstrations become misleading. The happy path is choreographed. The real test is what happens when the patient record contains:

• two medication lists with different doses;
• a laboratory result without a collection date;
• a wearable trend that conflicts with the patient’s report;
• a request that falls outside the approved workflow.

A trustworthy system should surface the conflict and route it to the right person. It should not smooth the problem into fluent prose.

6. Who owns the result?

“Human in the loop” does not answer this question.

Which human? Reviewing what? At what point? With enough time and evidence to change the outcome? What happens if nobody responds?

Ownership should be named:

• the agent prepares;
• a particular role reviews;
• that person may modify or reject;
• an approved action goes to a named recipient;
• exceptions have an escalation path;
• the system records the decision and its source.

If everyone is nominally involved, nobody is accountable.

Chatbot, scribe, copilot or agent?

These categories overlap, but the distinctions are useful.

None is inherently superior. A well-designed scribe is better than a badly governed agent. Use the least authority required for the job.

A clinic example: preparing a complex follow-up

Imagine a patient returning to a longevity clinic after three months. The record includes two new laboratory reports, wearable data, a symptom questionnaire, a message about travel and three incomplete actions from the previous plan.

A useful agent might:

1. identify what changed since the last review;
2. separate measured data from patient-reported information;
3. flag the incomplete actions without assuming non-adherence;
4. identify that recent travel may affect the interpretation of sleep data;
5. show one conflicting date across two documents;
6. prepare a review brief with links to every source;
7. ask the practitioner to resolve the conflict and decide what happens next.

It should not diagnose the cause of the symptoms, silently decide which source is correct or alter the care plan.

The value is not artificial bedside manner. It is less time reconstructing the patient and more time deciding what to do.

What the agent must not do

A healthcare agent should not:

• manufacture certainty when evidence conflicts;
• hide the source or reasoning behind a consequential suggestion;
• turn a risk score into a diagnosis;
• use a patient’s silence as proof of consent or adherence;
• send clinical instructions merely because a draft exists;
• optimise engagement by producing more alerts;
• expand its own permissions;
• continue when a stopping rule has been triggered.
• use probabilistic text generation to decide its own eligibility or expand its authority;
• hide a generative judgement behind a deterministic-looking score.

Regulation will depend on intended use, market and deployment. Whatever the classification, a clinic still needs to understand the system it is placing inside care.

Seven questions to ask in the next vendor demo

1. Show us a case where the agent decides to do nothing.
2. Show us contradictory data and how the conflict reaches a clinician.
3. Show the evidence behind one recommendation.
4. Show every system the agent can write to.
5. Show who can approve, reject, reverse and audit an action.
6. Show what happens when a tool fails.
7. Tell us the exact clinical decisions the agent is prohibited from making.

The answers will tell you more than a perfect scripted consultation.

The standard to hold

Healthcare needs both repeatability and judgement under uncertainty. Deterministic agents provide a controlled path for known work. Probabilistic agents can handle language and ambiguity, but only inside a harness that limits what their fluency can cause.

Ask the vendor to demonstrate the point where the agent stops and returns the decision to a practitioner.

Explore the rest of Kalibra’s Clinical AI guides, read our approach to healthcare AI governance or see how fragmented health data enters a clinical workflow.

Sources

• NIST. Artificial Intelligence Risk Management Framework 1.0. 2023.
• US FDA. Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff. 2026.
• World Health Organization. Ethics and governance of artificial intelligence for health. 2021.
• Topol EJ. High-performance medicine: the convergence of human and artificial intelligence. Nature Medicine. 2019.
• Obermeyer Z et al. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019.

This guide is educational and does not provide medical or legal advice. Regulatory obligations depend on the system’s intended use, market and deployment.