AI agents in healthcare: what they are and where clinicians stay in control
An AI agent is not a chatbot with a grander job title.
A useful healthcare agent can observe a defined part of a patient's state,
work towards a specific goal, choose from a limited set of permitted actions,
use approved tools, record what it did and stop when the situation exceeds its
authority. A clinician remains responsible for clinical judgement and
consequential care decisions.
That definition matters because “agent” is becoming a convenient label for
almost any software that produces text. A note summariser may be useful. A
search box may be useful. Neither becomes an agent merely because a vendor
adds a friendly name and a chat window.
The practical test is not whether the software sounds intelligent. It is
whether you can answer six plain questions about how it behaves.
There is also a distinction that most agent discussions blur: deterministic
and probabilistic agents solve different problems.
Two kinds of agent, and why healthcare needs both
Kalibra uses two deliberately different patterns.
Kali is deterministic. It follows programmed rules and structured branches
inside an approved workflow. Given the same state, rule version and inputs, it
should make the same transition. The clinic can inspect which rule fired,
which data it used and what happened next.
Kay is probabilistic. It can interpret open-ended language, explain a
complex record and handle questions that nobody could sensibly pre-script.
That flexibility brings variation. The same prompt can produce different
wording or emphasis. A capable model may still omit a fact, overstate an
inference or answer outside the intended boundary.
A harness can reduce those risks through structured retrieval, released
content, permissions, task-specific tests, source checks, abstention rules and
restricted tools. It cannot turn a probabilistic model into a deterministic
rules engine.
The useful architecture blends them:
- deterministic policy establishes eligibility, risk and authority;
- Kay interprets, asks or explains within that boundary;
- deterministic checks validate any proposed state change;
- a named person approves the consequential action;
- Kali carries the approved structured flow and records progress.
This is a design direction, not a claim that every hand-off is already
implemented in every Kalibra deployment. Kalibra's current public boundary is
clear: no autonomous execution today.
The six-part agent test
1. What state can it see?
An agent acts on a representation of the world. In a clinic, that might include
selected records, laboratory results, assessments, wearable trends, the active
care plan and the status of previously approved actions.
The word selected is important. More data is not automatically better data.
An agent that ingests everything but cannot distinguish a current medication
list from a five-year-old one has more context and less understanding.
Ask for a source map:
- Which systems and documents can the agent read?
- How does it distinguish current, historical and patient-reported data?
- What happens when two sources disagree?
- Does it show what is missing?
- Can a clinician trace a statement back to the original source?
If the answer is “our model handles that”, the answer is incomplete.
2. What goal is it pursuing?
“Improve health” is not an operational goal. Neither is “support the
clinician”.
A bounded goal sounds more like this:
Prepare the patient record for review by identifying material changes since
the previous consultation, unresolved follow-up and missing information. Do
not propose a treatment decision.
The narrower wording is a feature. It tells the agent what success means and
what work remains outside its remit.
Bad objectives create bad behaviour. Researchers have shown how a seemingly
reasonable proxy can encode the wrong goal: a widely used population-health
algorithm used healthcare cost as a proxy for health need, producing racial
bias in who received extra help. The lesson is bigger than that one system.
Before asking whether an agent is accurate, ask whether it is optimising the
right thing.
3. What is it allowed to do?
Permissions turn an impressive demonstration into a governed workflow.
There is a material difference between permission to:
- read a record;
- draft a summary;
- prepare a patient message;
- place that message in a clinician's approval queue;
- send the message;
- change a care plan.
Those actions should not inherit the same authority merely because they sit in
one interface. A clinic needs a capability-by-capability permission model.
In Kalibra's operating view, AI may perform bounded mechanical work such as
gathering, extracting, normalising, comparing, drafting, reminding, routing
and tracking. Diagnosis, ambiguity resolution, treatment decisions and
approval remain human responsibilities.
4. Which tools can it use?
An agent becomes consequential when it can do more than generate language. It
might query a structured record, calculate a deterministic score, prepare a
form, create a task or route an item for review.
Every tool raises three practical questions:
- Can the agent only read, or can it write?
- Is the output reversible?
- Who is told when the tool fails or returns an ambiguous result?
Do not accept “integrates with your stack” as an answer. Ask to see the exact
action, the exact system changed, the audit entry and the rollback path.
5. When must it stop?
The safest action is often no action.
An agent needs explicit stopping rules for contradictory evidence, missing
critical information, an out-of-scope request, a high-risk signal, a failed
tool, low confidence or a situation requiring clinical interpretation.
This is where many demonstrations become misleading. The happy path is
choreographed. The real test is what happens when the patient record contains:
- two medication lists with different doses;
- a laboratory result without a collection date;
- a wearable trend that conflicts with the patient's report;
- a request that falls outside the approved workflow.
A trustworthy system should surface the conflict and route it to the right
person. It should not smooth the problem into fluent prose.
6. Who owns the result?
“Human in the loop” does not answer this question.
Which human? Reviewing what? At what point? With enough time and evidence to
change the outcome? What happens if nobody responds?
Ownership should be named:
- the agent prepares;
- a particular role reviews;
- that person may modify or reject;
- an approved action goes to a named recipient;
- exceptions have an escalation path;
- the system records the decision and its source.
If everyone is nominally involved, nobody is accountable.
Chatbot, scribe, copilot or agent?
These categories overlap, but the distinctions are useful.
| System | Typical job | What makes it different |
|---|---|---|
| Chatbot | Answers a user's question | Usually responds turn by turn and may not own a continuing task |
| Scribe | Captures or drafts documentation | Works around the clinical encounter and should not decide care |
| Copilot | Assists a user inside a workflow | Offers options while the user remains the active operator |
| Agent | Pursues a bounded goal across steps | Can choose permitted next steps, use tools and update state |
None is inherently superior. A well-designed scribe is better than a badly
governed agent. Use the least authority required for the job.
A clinic example: preparing a complex follow-up
Imagine a patient returning to a longevity clinic after three months. The
record includes two new laboratory reports, wearable data, a symptom
questionnaire, a message about travel and three incomplete actions from the
previous plan.
A useful agent might:
- identify what changed since the last review;
- separate measured data from patient-reported information;
- flag the incomplete actions without assuming non-adherence;
- identify that recent travel may affect the interpretation of sleep data;
- show one conflicting date across two documents;
- prepare a review brief with links to every source;
- ask the practitioner to resolve the conflict and decide what happens next.
It should not diagnose the cause of the symptoms, silently decide which source
is correct or alter the care plan.
The value is not artificial bedside manner. It is less time reconstructing the
patient and more time deciding what to do.
What the agent must not do
A healthcare agent should not:
- manufacture certainty when evidence conflicts;
- hide the source or reasoning behind a consequential suggestion;
- turn a risk score into a diagnosis;
- use a patient's silence as proof of consent or adherence;
- send clinical instructions merely because a draft exists;
- optimise engagement by producing more alerts;
- expand its own permissions;
- continue when a stopping rule has been triggered.
- use probabilistic text generation to decide its own eligibility or expand
its authority; - hide a generative judgement behind a deterministic-looking score.
Regulation will depend on intended use, market and deployment. Whatever the
classification, a clinic still needs to understand the system it is placing
inside care.
Seven questions to ask in the next vendor demo
- Show us a case where the agent decides to do nothing.
- Show us contradictory data and how the conflict reaches a clinician.
- Show the evidence behind one recommendation.
- Show every system the agent can write to.
- Show who can approve, reject, reverse and audit an action.
- Show what happens when a tool fails.
- Tell us the exact clinical decisions the agent is prohibited from making.
The answers will tell you more than a perfect scripted consultation.
The standard to hold
Healthcare needs both repeatability and judgement under uncertainty.
Deterministic agents provide a controlled path for known work. Probabilistic
agents can handle language and ambiguity, but only inside a harness that
limits what their fluency can cause.
The strongest agent may be the one that knows when to prepare, when to ask,
when to route and when to stop.
Explore the rest of Kalibra's Clinical AI guides,
read our approach to healthcare AI governance or see how
fragmented health data enters a clinical workflow.
Sources
- NIST. Artificial Intelligence Risk Management Framework 1.0. 2023.
- US FDA. Clinical Decision Support Software: Guidance for Industry and Food
and Drug Administration Staff. 2026. - World Health Organization. Ethics and governance of artificial intelligence
for health. 2021. - Topol EJ. High-performance medicine: the convergence of human and artificial
intelligence. Nature Medicine. 2019. - Obermeyer Z et al. Dissecting racial bias in an algorithm used to manage the
health of populations. Science. 2019.
This guide is educational and does not provide medical or legal advice.
Regulatory obligations depend on the system's intended use, market and
deployment.
