How to Evaluate an AI Tool for Your Clinic

Start with the job, not the demo

Most AI tools for medicine look impressive in a two-minute demo and collapse when they meet a clinic schedule. Before you evaluate features, write one sentence that names the job: what decision or task should be faster, safer, or more consistent after you adopt the tool?

If you cannot name the job, you are shopping for novelty. Digital Curbside evaluates tools against a single weekly workflow for exactly this reason.

• Name the clinical or research job in one sentence.
• Define what “done” looks like for that job today.
• Decide what failure would look like if the tool is wrong.

Test with your own messy inputs

Vendors will show clean examples. You should bring your own notes, PDFs, OR lists, or inbox threads — the ones with typos, abbreviations, and incomplete context.

Run three real cases. Score each on time saved, edits required, and whether you would trust the output without re-checking primary sources.

Privacy, provenance, and exit cost

Ask where data is stored, whether it is used for training, and how you export or delete it. Prefer tools that keep PHI out of the loop unless your institution has already cleared them.

Also price the exit: if the vendor disappears next year, can you keep your notes, prompts, and outputs in a format you control?

Adopt only if it survives a normal week

A tool that works once on a Saturday afternoon is not a workflow. Try it across five clinic or research days. If it still saves time after the novelty fades — and if you can explain how it works to a colleague in under a minute — it earns a permanent slot.