October 1, 2026 |

ICYMI at #AIS2026: using AI in outcomes reporting is a lot like onboarding a research assistant. There’s real efficiency in having the help, and a significant amount of background teaching, modeling, and guardrail-setting needed before you can trust what comes back.
That comparison landed the way it did because of where the rest of the Summit went. Several sessions dealt with how we communicate the value of IME initiatives to our internal stakeholders, and another focused on aggregating data across grants to provide just-in-time insights. All of that asks for more evidence, faster, for people who weren’t part of the original analysis.
So our session, “From Slide Decks to Systems: Using AI to Generate and Consume CME Outcomes Responsibly,” turned out to be a very relevant place to land.
Some of our key takeaways:
Curiosity and potential are outpacing trust and confidence in the technology. Researchers spend years learning to do this work rigorously, and right now it’s about changing the script from seeing AI as cheating to seeing it as a form of rigorous methodology when it’s used appropriately. Appropriate use is the key phrase.
There are two different things people mean when they say “we used AI,” and they need different guardrails. Substitution is work someone already did by hand, like coding open comments, cleaning data, merging files, or building the deck. There’s a known answer to check against. New capability is work that wasn’t possible before, like testing whether a finding survives every reasonable analytic path, or pooling results across a whole portfolio. There’s nothing to check it against, so expert review has to be built into the process rather than added at the end.
Here’s the case we walked through. A real program, self-reported outcomes came back modest, renewal was close, and there was pressure to show something valuable. So the dataset went to a model with one instruction: find what’s significant. What came back was 168 comparisons. Seven looked significant, which is about what you’d expect from chance alone when you run that many. None of them held up once we corrected for the number of tests. And the finding we could have led with came down to two respondents labeled cardiologists who were actually nurses.
None of it was fabricated. Someone just would have had to choose which result to put on the slide.
Here’s the part I keep coming back to. Supporters want to use AI to help communicate value to their internal stakeholders across an entire portfolio of grant funding. MECs and consultants want to use it to increase value and personalize what we deliver. Both of those are reasonable and both are already happening.
The hard part is the handoff. The supporter’s AI is reading the report, not the dataset behind it. Whatever context didn’t make it onto the page is gone by the time that number gets pooled with fourteen other programs.
So the practical question isn’t whether we use AI. It’s what has to arrive with the report so the number still means something when it hits the supporter’s desk: how many people it’s based on, what was measured and when, which numbers can and can’t be combined, and which parts were AI-assisted. Now, more than ever before, does a supporter have the ability to take that into consideration in their summation of the insights and value from what they funded with AI tools.
All of this points back to needing the human in the loop, with the expertise required to guide the work and evaluate what’s coming back.
Thank you to our panel and moderator, and to everyone who worked through what could be on the checklist with us.
Stay tuned for an Almanac article with the checklist you helped us build: companion guidelines for what providers should consistently do in outcomes reporting, and a list of things supporters can confidently ask for.
This content was provided by Medscape Education