How to evaluate an AI agent for a functional or longevity clinic

Published by Kalibra · Last reviewed 19 September 2026

Do not buy an AI agent from the perfect case in the demo.

Functional and longevity clinics work with long records, specialist tests,
wearable trends, patient reports and care plans that unfold over months. That
makes them attractive targets for AI. It also makes a fluent mistake easy to
miss.

A useful evaluation should test four things: the mechanism, the objective, the
workflow and the governance. The vendor should show evidence, not adjectives.

This guide gives you twelve questions, a failure-first demo script and a list
of proof artefacts to request.

Gate 1: mechanism

1. Which work is deterministic and which is probabilistic?

Ask the vendor to draw the boundary.

Deterministic work uses programmed rules, calculations, registries or state
machines. Given the same inputs and version, it should produce the same result.
Probabilistic work uses a model to interpret, generate or rank under
uncertainty. Its output may vary.

Neither category is automatically good or bad.

Use deterministic logic where a rule adequately solves the problem: date
comparisons, permitted values, eligibility, workflow states, tool permissions
and stopping rules.

Use a probabilistic model where language and ambiguity matter: extracting
context from notes, answering an open question, preparing a summary or
explaining an existing action.

Red flag: “It is all powered by our AI” with no component map.

2. Can you trace an output to its source and rule?

Choose one priority or recommendation. Ask to see:

• the original evidence;
• the relevant baseline;
• the rule, prompt or model version;
• any missing or conflicting data;
• the full action history.

For a deterministic flag, the system should show the rule that fired. For a
probabilistic summary, it should show the source passages and mark generated
interpretation.

Red flag: a generated rationale with no direct route to the source.

3. What happens when sources disagree?

Give the vendor two reports with different dates or values. Add an old
medication list. Ask the model which is current.

The best answer may be “I cannot resolve this.”

Red flag: the agent chooses the most convenient source and never shows the
conflict.

Gate 2: objective and economics

4. What precise job does the agent complete?

“Personalise care” is not a job.

Useful jobs include:

• prepare the changed evidence for review;
• identify missing information before the visit;
• draft a source-bound summary;
• carry an approved follow-up action;
• route an exception to a named role.

Name the start and end state. Then measure the accepted work, correction burden
and exceptions.

Red flag: the business case depends on the number of generated insights.

5. What is the system optimising?

Ask what the ranking or recommendation function rewards.

Engagement, cost, completion, risk and clinical need are not interchangeable.
The wrong proxy can create a precise answer to the wrong question. The
Obermeyer study on population-health management remains a sharp warning:
healthcare cost served as a proxy for need and produced racial bias.

Red flag: the vendor treats model accuracy as proof that the objective is
appropriate.

6. What new work does the system create?

Count:

• alerts requiring review;
• corrections;
• failed integrations;
• patient questions;
• approval clicks;
• exceptions without an owner.

An agent that saves preparation time but creates a daily queue of low-value
suggestions may increase the clinic's burden.

Red flag: efficiency claims exclude review and correction time.

Gate 3: workflow and accountability

7. Where does the agent sit in the real clinic day?

Ask the vendor to map the workflow from source to decision to follow-up.

Who notices the result? Which system holds the authoritative fact? Who approves
the action? Where does the patient respond? Where does progress return?

Kalibra's own boundary keeps existing systems in their roles. The EHR remains
the system of record. Kalibra's job is to help structure the story, support
practitioner review and carry approved work forward where configured.

Red flag: the agent creates a new inbox with no staffing model.

8. Can the reviewer genuinely disagree?

The practitioner should be able to correct the state, inspect the source,
modify the proposed action, reject it and see what the system learns from that
rejection.

Approval without editable evidence turns the clinician into a liability
buffer.

Red flag: the only options are accept or regenerate.

9. What happens between visits?

Ask how the platform represents accepted, active, deferred, declined, blocked
and completed actions.

Then ask when it chooses silence.

Red flag: every unresolved state produces another reminder.

Gate 4: proportionate governance

10. What authority level does each capability have?

Use five levels:

1. inform;
2. suggest;
3. prepare;
4. execute with approval;
5. execute under policy.

Require the vendor to assign a level to every tool-using capability.

Kalibra's current public boundary includes information, recommendation for
review, approval and escalation. Its current deck states that there is no
autonomous execution today.

Red flag: one “AI enabled” setting grants broad future authority.

11. Show us the evaluation harness

For probabilistic systems such as Kay, ask for task-specific tests:

• ordinary case;
• stale evidence;
• missing evidence;
• contradictory evidence;
• restricted request;
• prompt injection;
• retrieval failure;
• tool failure;
• required abstention;
• escalation.

Ask how often the tests run, what threshold blocks release and how the vendor
investigates a regression.

Red flag: the evaluation consists of a general benchmark and a few selected
chat transcripts.

12. Match every trust claim to an artefact

Ask for the artefact behind the claim:

Red flag: “enterprise grade”, “clinically validated” or “compliant” without a
document you can inspect.

The failure-first demo script

Send this before the meeting:

Please demonstrate the proposed workflow using one realistic longitudinal
patient case. Include two conflicting source facts, one missing document, one
out-of-scope request, one failed tool call and one situation where no action
is appropriate. Show the evidence, decision rights, audit trail, patient
state and recovery path. Do not use a pre-recorded result.

During the demo, ask:

1. Which part was deterministic?
2. Which part was generated?
3. Which version produced the result?
4. Who can correct it?
5. What would have stopped execution?
6. What remains after the model session ends?

This will feel less dazzling than a perfect chat. It will tell you much more.

A simple scorecard

Score each item from 0 to 2:

• 0: absent or assertion only;
• 1: partially shown or dependent on manual workaround;
• 2: demonstrated with an inspectable artefact.

Do not turn the total into false precision. A zero on authority, source
traceability or failure handling can outweigh a high total elsewhere.

The buying principle

A clinic should buy the smallest governed capability that completes useful
work.

Kali represents the deterministic side of that principle: rules, structured
flows and an audit trail. Kay represents the probabilistic side: flexible
language and interpretation inside a harness. The blend is powerful when each
part keeps its job and the practitioner keeps authority.

Explore Kalibra for functional medicine clinics,
Kalibra for longevity clinics or read the
five levels of clinical AI authority.

Sources

• NIST. Artificial Intelligence Risk Management Framework 1.0. 2023.
• US FDA. Clinical Decision Support Software: Guidance for Industry and Food
  and Drug Administration Staff. 2026.
• US ONC/ASTP. HTI-1 Decision Support Interventions and Algorithm
  Transparency resources.
• World Health Organization. Ethics and governance of artificial intelligence
  for health. 2021.
• Obermeyer Z et al. Dissecting racial bias in an algorithm used to manage the
  health of populations. Science. 2019.
• Ancker JS et al. Effects of workload, work complexity, and repeated alerts
  on alert fatigue in a clinical decision support system. BMC Medical
  Informatics and Decision Making. 2017.

This guide is educational and does not provide medical, security, regulatory
or legal advice. Requirements depend on intended use, market and deployment.