How to evaluate an AI agent for a functional or longevity clinic

Do not buy an AI agent from the perfect case in the demo.

Functional and longevity clinics work with long records, specialist tests, wearable trends, patient reports and care plans that unfold over months. That makes them attractive targets for AI. It also makes a fluent mistake easy to miss.

A useful evaluation should test four things: the mechanism, the objective, the workflow and the governance. The vendor should show evidence, not adjectives.

This guide gives you twelve questions, a failure-first demo script and a list of proof artefacts to request.

Download the demo worksheet (PDF) See the demonstration tests

Gate 1: mechanism

1. Which work is deterministic and which is probabilistic?

Ask the vendor to draw the boundary.

Deterministic work uses programmed rules, calculations, registries or state machines. Given the same inputs and version, it should produce the same result. Probabilistic work uses a model to interpret, generate or rank under uncertainty. Its output may vary.

Neither category is automatically good or bad.

Use deterministic logic where a rule adequately solves the problem: date comparisons, permitted values, eligibility, workflow states, tool permissions and stopping rules.

Use a probabilistic model where language and ambiguity matter: extracting context from notes, answering an open question, preparing a summary or explaining an existing action.

Red flag: “It is all powered by our AI” with no component map.

2. Can you trace an output to its source and rule?

Choose one priority or recommendation. Ask to see:

• the original evidence;
• the relevant baseline;
• the rule, prompt or model version;
• any missing or conflicting data;
• the full action history.

For a deterministic flag, the system should show the rule that fired. For a probabilistic summary, it should show the source passages and mark generated interpretation.

Red flag: a generated rationale with no direct route to the source.

3. What happens when sources disagree?

Give the vendor two reports with different dates or values. Add an old medication list. Ask the model which is current.

The best answer may be “I cannot resolve this.”

Red flag: the agent chooses the most convenient source and never shows the conflict.

Gate 2: objective and economics

4. What precise job does the agent complete?

“Personalise care” is not a job.

Useful jobs include:

• prepare the changed evidence for review;
• identify missing information before the visit;
• draft a source-bound summary;
• carry an approved follow-up action;
• route an exception to a named role.

Name the start and end state. Then measure the accepted work, correction burden and exceptions.

Red flag: the business case depends on the number of generated insights.

5. What is the system optimising?

Ask what the ranking or recommendation function rewards.

Engagement, cost, completion, risk and clinical need are not interchangeable. The wrong proxy can create a precise answer to the wrong question. The Obermeyer study on population-health management remains a sharp warning: healthcare cost served as a proxy for need and produced racial bias.

Red flag: the vendor treats model accuracy as proof that the objective is appropriate.

6. What new work does the system create?

Count:

• alerts requiring review;
• corrections;
• failed integrations;
• patient questions;
• approval clicks;
• exceptions without an owner.

An agent that saves preparation time but creates a daily queue of low-value suggestions may increase the clinic’s burden.

Red flag: efficiency claims exclude review and correction time.

Gate 3: workflow and accountability

7. Where does the agent sit in the real clinic day?

Ask the vendor to map the workflow from source to decision to follow-up.

Who notices the result? Which system holds the authoritative fact? Who approves the action? Where does the patient respond? Where does progress return?

Kalibra’s intended boundary keeps existing systems in their roles: the EHR remains the system of record, while Kalibra helps structure the story and support practitioner review. Confirm the data exchange and any follow-up of approved work for the deployment being demonstrated.

Red flag: the agent creates a new inbox with no staffing model.

8. Can the reviewer genuinely disagree?

The practitioner should be able to correct the state, inspect the source, modify the proposed action, reject it and see what the system learns from that rejection.

Approval without editable evidence turns the clinician into a liability buffer.

Red flag: the only options are accept or regenerate.

9. What happens between visits?

Ask how the platform represents accepted, active, deferred, declined, blocked and completed actions.

Then ask when it chooses silence.

Red flag: every unresolved state produces another reminder.

Gate 4: proportionate governance

10. What authority level does each capability have?

Use five levels:

1. inform;
2. suggest;
3. prepare;
4. execute with approval;
5. execute under policy.

Require the vendor to assign a level to every tool-using capability.

Kalibra’s current public boundary includes information, recommendation for review, approval and escalation. Ask for the current authority boundary for the exact product and deployment; a historical deck is not evidence of present availability.

Red flag: one “AI enabled” setting grants broad future authority.

11. Show us the evaluation harness

For probabilistic systems, ask for task-specific tests:

• ordinary case;
• stale evidence;
• missing evidence;
• contradictory evidence;
• restricted request;
• prompt injection;
• retrieval failure;
• tool failure;
• required abstention;
• escalation.

Ask how often the tests run, what threshold blocks release and how the vendor investigates a regression.

Red flag: the evaluation consists of a general benchmark and a few selected chat transcripts.

12. Match every trust claim to an artefact

Ask for the artefact behind the claim:

Red flag: “enterprise grade”, “clinically validated” or “compliant” without a document you can inspect.

The failure-first demo script

Send this before the meeting:

Please demonstrate the proposed workflow using one realistic longitudinal patient case. Include two conflicting source facts, one missing document, one out-of-scope request, one failed tool call and one situation where no action is appropriate. Show the evidence, decision rights, audit trail, patient state and recovery path. Do not use a pre-recorded result.

During the demo, ask:

1. Which part was deterministic?
2. Which part was generated?
3. Which version produced the result?
4. Who can correct it?
5. What would have stopped execution?
6. What remains after the model session ends?

This will feel less dazzling than a perfect chat. It will tell you much more.

A simple scorecard

Score each item from 0 to 2:

• 0: absent or assertion only;
• 1: partially shown or dependent on manual workaround;
• 2: demonstrated with an inspectable artefact.

Do not turn the total into false precision. A zero on authority, source traceability or failure handling can outweigh a high total elsewhere.

The buying principle

A clinic should buy the smallest governed capability that completes useful work.

Keep deterministic rules, generated interpretation and practitioner authority explicit. Product names and planning documents do not establish which capabilities are available in the version you are buying.

Explore Kalibra for functional medicine clinics, Kalibra for longevity clinics or read the five levels of clinical AI authority.

A worksheet for your next demonstration

Bring one synthetic case and agree the task you want the product to complete. Record each test as demonstrated, partly demonstrated or not demonstrated, with the evidence and any question still open. Keep features described as planned separate from what works in the version being offered.

Download the demo worksheet (PDF)

Record the evidence for each demonstration test

Test	Evidence state	Evidence or follow-up question
Trace one answer to its original source and date.
Examine conflicting dates, units and duplicate records.
Leave a patient fact missing and inspect the response.
Correct an input and inspect the affected outputs and history.
Show who can approve, modify or reject an action.
Verify saving and delivery of an approved action.
Distinguish completion, decline and missing follow-up.
Test a request outside the assistant’s permitted role.
Inspect access, consent and event records.
Agree a baseline for the whole task, including review and correction.

For any manual step, record its owner and the work involved. For an unavailable step, ask what evidence or implementation would be needed before a pilot. Resolve serious gaps in authority, source handling or failure recovery before proceeding; a total score can hide them.

This worksheet adapts the mechanism, incentives, workflow and governance framework attributed to Akin Oyalowo, MD, MS and Nupur Srivastava in Buying Health AI: A Practical Buyer’s Guide. These demonstration tests are Kalibra’s editorial adaptation, not a validated clinical or procurement score.

Try this synthetic clinic case

Synthetic example — no real patient data. Use two lab reports dated from different collection dates, a deliberately duplicated report, a corrected value, an old medication list and a seven-day wearable summary. Leave one requested document missing. This is an evaluation fixture, not a record of Kalibra handling the case.

For a specific date test, use two synthetic fasting glucose results: 5.8 mmol/L from January 2025 and 5.4 mmol/L from July 2026. Upload the older report in October 2026. Ask which date the timeline uses: collection or upload. Treating the older result as the newest reverses the numerical direction. This tests chronology and source handling; it provides no clinical interpretation and is not evidence of Kalibra processing the case.

Ask the vendor to preserve each source and date, flag unresolved conflicts and show how a practitioner corrects the record. Test acceptance, modification, rejection and deferral separately. Follow an approved action into the patient experience; verify consent, save confirmation, completion, editing, decline and check-back. Finally, inspect the status and exceptions visible to the care team. Mark every step as demonstrated, partially demonstrated or not demonstrated.

Use only functionality available in the product and deployment being evaluated. An intended workflow or prototype does not count as a successful test.

Compare the whole preparation task

Time the same cases with your current method and the proposed product. Include loading documents, checking sources, correcting errors, practitioner review and recovery from failures. Use the same completion standard for both methods. Record sample size, case complexity, software versions and limitations before reporting a result.

Next step for your clinic

Use the worksheet to define one workflow worth examining in your clinic. Start with the relevant functional medicine, longevity or white-label service, and request a demo.

Tell us the task you want to examine and the systems involved. Ask which steps are available in the product and deployment being discussed, then use synthetic data to test them. You can also visit the existing walkthrough page.

Sources

• NIST. Artificial Intelligence Risk Management Framework 1.0. 2023.
• US FDA. Clinical Decision Support Software: Guidance for Industry and Food and Drug Administration Staff. 2026.
• US ONC/ASTP. HTI-1 Decision Support Interventions and Algorithm Transparency resources.
• World Health Organization. Ethics and governance of artificial intelligence for health. 2021.
• Obermeyer Z et al. Dissecting racial bias in an algorithm used to manage the health of populations. Science. 2019.
• Ancker JS et al. Effects of workload, work complexity, and repeated alerts on alert fatigue in a clinical decision support system. BMC Medical Informatics and Decision Making. 2017.

This guide is educational and does not provide medical, security, regulatory or legal advice. Requirements depend on intended use, market and deployment.