What we ask before a model goes near a patient

What decision is this actually changing?

The first question is the one most often skipped: what will somebody do differently because of this output. If there is no clear answer, the feature is a demonstration rather than a tool, and it will add noise to a working environment that already has plenty.

A surprising number of proposed models fail here. They predict something real and interesting that nobody can act on, or they predict something a clinician already knows by the time the model is confident enough to say it.

Who is the output for, and what do they already know?

A suggestion aimed at a consultant, a junior doctor, a nurse in charge and a bed manager are four different products, even if the underlying model is identical. They have different context, different authority and different amounts of time.

Getting this wrong produces the classic failure where an advisory is technically correct and lands on somebody who cannot act on it, who then forwards it to somebody who can at which point the software has invented a new manual step.

What happens when it is wrong in each direction?

Both directions, separately, because they are rarely symmetrical. A missed flag and a false flag have different costs, different visibility and different failure paths.

False positives are the ones that quietly kill a feature. Each one is individually cheap and collectively they teach people to dismiss the advisory without reading it, at which point the true positives stop working too.

So the question is not only how often it is wrong, but what the wrong answer costs the person receiving it and how quickly it erodes their willingness to look.

Can we show the working?

If the reasoning cannot be surfaced in a form a clinician can evaluate in seconds, we do not ship it. That is a hard constraint rather than a preference, and it has ruled out approaches that scored better on paper.

It is worth being clear that this is a design decision with a real cost. Sometimes the more opaque model is more accurate. We take the trade knowingly, because an unverifiable output in this setting is not usable regardless of its accuracy.

Who is accountable for the output?

Not 'the algorithm'. A named clinical owner responsible for the advisory being appropriate, and a named engineering owner responsible for it behaving as described.

This sounds bureaucratic and is the opposite: it is what makes it possible to change or withdraw something quickly, because there is no question about whose call it is.

How will we turn it off?

Asked before launch, not after. If disabling an advisory requires a release, it will stay on through a problem because the alternative is worse and that is a decision made by accident rather than by anyone.

So every advisory ships with a way to narrow its scope or switch it off without a deployment, and the people who would need to do that know it exists before they need it.

What we do not ask

We do not ask whether it is impressive. There is a persistent pull towards features that demonstrate well and help little, and in this domain that pull has to be actively resisted rather than merely noticed.

We also do not ask whether the technique is current. Whether something uses the newest available approach is close to irrelevant to whether it should be near a patient, and letting it into the conversation early tends to reorder the other questions around it.

And we do not ask whether a competitor has shipped something similar. That is a real commercial question and a terrible design input, because it answers what exists rather than what would help.