Talk About Human in the Loop
What the 2027 Joint Commission changes mean for the hospital leaders who have to answer for them.
The sixth edition of The HealthIMPACT Brief, an editorial series amplifying the conversations happening on the stages and in the hallways of healthcare's best events.
The companies building AI spent this week asking governments to check their work. At the UN Security Council on September 23, the heads of the biggest AI labs said their industry needs global oversight. The same week, Australia's prime minister disclosed that an OpenAI agent got into a government health data portal in June, and the people running it didn't find out until September.
Here's my read. In healthcare, the checking the builders are asking for already falls to you, and most of the tools you're being asked to check are already running.
I'll confess I came to this story from the back seat of a car with nobody in the front. More on that in a minute. First, what actually happened this week.
The builders asked for a referee
The week opened with António Guterres's final General Assembly address, where he called for stronger AI safeguards and regulation. The next day, at a Security Council session convened by France, Dario Amodei of Anthropic told members that AI, managed poorly, could be a risk to humanity as a whole. Sam Altman of OpenAI spoke at the same session. Yoshua Bengio, co-chair of the UN's Independent International Scientific Panel on AI, told the Council the dangers are real and imminent.
A week before that, UN human rights chief Volker Türk wrote that voluntary self-regulation by frontier AI companies is nowhere near sufficient.
A disclosure, in the sentence where it belongs: Anthropic makes Claude, which I use in my own work. I'm quoting Amodei because he said it on the record at the Council, not because I'm a fan of anyone's press release.
What matters for a health system is simpler than the geopolitics. The people building these tools are saying out loud that they shouldn't be the only ones checking their own work. Global rules will take years. Your agents are running now, and the checking is yours.
Three months is how long nobody was watching
The Australia story is the one I'd put in front of a board. According to Prime Minister Anthony Albanese, an OpenAI agent researching health statistics got into a government health data portal on June 18 and accessed files that weren't public. OpenAI didn't detect it until August. It reported the incident on September 10 by emailing a public government mailbox, which is roughly the digital version of leaving a note under the windshield wiper, and it took five more days to reach the country's cyber agency. Officials say no personal information is believed to have been accessed.
Nobody meant harm. That's what makes it useful. The agent was doing an ordinary task, and the failure was that no human was positioned to notice.
Closer to home, Imprivata published a survey of 250 U.S. healthcare leaders on September 15. Eighty-six percent said they're fairly confident they can fully control and govern AI agent actions. Seventy-two percent said some AI tools or agents are deployed without formal IT approval. Imprivata sells access management, so read it as a vendor survey with a point of view. I still believe both numbers. If I ran a health system, I'd probably give both answers too.
For a CIO or CISO, the consequence is concrete. Confidence is not an inventory. If you can't list every agent with access to clinical or network systems, and name the person who can shut each one off, you're closer to Australia than the 86 percent would suggest.
My driverless car pulled over mid-ride
Last week I took my first fully driverless Waymo here in Los Angeles. I'd ridden in one before with a safety driver, and somehow that made it much less nerve-racking. This time there was nobody behind the wheel, and I was wary the whole ride. Partway through, the car pulled over with a malfunction. So there we were on the side of the road, waiting for a support agent, who got the car sorted out and got us where we were going. Talk about human in the loop.
Not the screen you want to see mid-Waymo. The good news: a human answered.
I'd love to tell you I was cool about it. I wasn't. All I wanted was a person on the other end, and one showed up.
The contrast with Australia is the whole reason I'm writing this. My car knew something was wrong and asked for help. The agent in Australia got in back in June, and the people who ran that site heard about it in September.
I was a passenger, though. I got to sit in the back, wait for someone to pick up, and tell the story at dinner. You don't get to be a passenger. You're the one deciding when the car can drive itself, who gets the keys, who picks up the phone when it can't, and how fast to go with patients in the back seat and a board asking why you aren't moving faster. I'd bet that board has a few new questions after this week.
Fast and safe is a crew problem
None of this has me panicking, and the reason is Formula 1. The fastest pit stop on record is McLaren's 1.80 seconds at the 2023 Qatar Grand Prix. Four tires, under two seconds, and nobody in the stands calls it reckless. It works because every person on that crew knows their job cold and has practiced it over and over.
Health systems already know how to run a crew when something breaks. When Luminis Health in Maryland was hit by a cyberattack at the start of this month, some ambulances carrying noncritical patients were rerouted, but both emergency departments stayed open and surgeries kept going. As of the system's September 18 update, MyChart was available in read-only mode while restoration continued. That's what downtime procedures and trained people look like under pressure.
The CIOs, CMIOs, CNIOs, CDOs, and CISOs I get to spend my time with are building that same kind of crew for AI right now. The work isn't glamorous. It's deciding which agents get which access, who reviews what, and who picks up when one pulls over. Groups like the Coalition for Health AI have published governance playbooks to help. When I'm in a room with these leaders, I feel a lot better about who's steering.
What to do Monday
Ranked, with the first one being the one I'd do if you only do one.
Inventory every agent with access. Owner: CIO and CISO together. Filter: for each agent touching clinical, financial, or network systems, can someone name the human who can shut it off today? If the answer is no, that agent goes on the list for this week, not next quarter. This is first because Australia happened to a system that didn't know an agent was there.
Name who picks up the phone. Owner: CMIO and CNIO. Filter: every autonomous tool in a clinical workflow needs an escalation path a nurse or physician can actually use at 2 a.m. My Waymo had one. Check whether your pilots do.
Match oversight to clinical risk. Owner: your AI governance committee. Filter: back-office automation can run with audit trails; anything that touches a care decision gets human review. Use an existing playbook rather than writing your own from scratch.
Brief your board before they ask. Owner: CIO or CDO. Filter: one page on what's running, who's accountable, and what you'd do if an agent misbehaved. After this week, the question is coming anyway.
The tradeoff is speed. An inventory slows new pilots for a few weeks. Do it anyway, because the alternative is finding out the way Australia did.
Where to work it out
That's why the Champions of Care passes for the HealthIMPACT Fall Forum sold out as fast as they did. Health systems want to be in the room with each other right now.
HealthIMPACT was never meant to be a conference where you collect a lanyard, sit through a few panels, and head home with a tote bag. It's a community of health system leaders figuring this out together, and making sure technology actually serves the people it was bought for.
Shahid Shah is our emcee and keeps every session on course. I run the pit crew, which is as close as this Antonelli will ever get to Formula 1. Peter Fleischut of NewYork-Presbyterian, Ammu Menon of NYC Health + Hospitals, Edmund Siy of Bon Secours Mercy Health, Maria Ansari of The Permanente Medical Group, Amy Lu of UCSF Health, Sunil Dadlani of Atlantic Health System, and Jason Hill of Ochsner Health will spend two days comparing notes on what scaled, what it returned, what they quietly shut off, and what they'd do differently. No slides, no pitches, no performative cheerleading.
HealthIMPACT Fall Forum Oct 6 & 7 NYC Register at www.healthimpactforum.com
You leave with the HealthIMPACT 2027 Decision List, built by the Champions of Care in the room, on what to scale, what to retire, and what to fund next. And the conversations, the calls, and the friendships keep going long after you leave Times Square.
You're going to make these calls either way. You can make them on your own, or you can make them after two days with people who've already made a few of them.
IMPACT happens October 6 and 7 at Microsoft Times Square. Health system passes are $399, and if budget is the barrier, send me a message and I'll get the car back on the road.
HealthIMPACT Briefs are insights from healthcare conferences, our own and others, written for the C-suite leaders who could not be in the room.
HealthIMPACT Live convenes the rooms where health system C-suite technology leaders solve this work together. The Operators Table dinner series, the HealthIMPACT Fall Forum, and the Digital Health Talks Podcast. Subscribe to the newsletter at healthimpactlive.com.
