Accuracy is the easy part to talk about
Every model can be described with a number, and every number sounds reassuring in a slide. The number tells you how the model behaved on data somebody curated, at a moment that has already passed.
It tells you very little about the thing that actually determines whether the software is safe: what a tired person does with the output at three in the morning, when the model is confident and wrong.
That is a design problem, not a modelling problem, and it is where most of our work goes.
Suggestion, not instruction
There is a meaningful difference between software that says 'this patient is deteriorating' and software that says 'these four observations, in this order, are the pattern that flagged this patient'. The first asks for belief. The second asks for a look.
We build for the second. The output is framed as something to check, positioned where checking it is cheap, and worded so that ignoring it is a legitimate clinical act rather than a rebellion against the computer.
This is partly language and partly layout, and both matter more than people expect. A suggestion presented in the visual language of an alarm will be read as an alarm no matter what the text says.
Showing the working
A model that cannot explain itself has no business near a bedside. In practice, 'explain' does not mean a research-grade interpretability report. It means answering the three questions a clinician actually asks in the ten seconds they have.
What did you look at. Which inputs, over what window, from which source. If the model is reading a value that was entered by hand four hours ago and never confirmed, the clinician needs to know that before anything else.
What tipped it. Not a full derivation, but the handful of contributors that moved the result. If the honest answer is that no single factor dominated, say that too.
How sure is this, in language that survives contact with a busy ward. Confidence expressed as a bare percentage invites false precision. Confidence expressed as a band, with the conditions that would narrow it, invites judgement.
The override is the feature
Systems that make overriding difficult produce two outcomes, both bad. Either people comply against their judgement, or they learn to route around the software entirely.
So the override is not an escape hatch bolted on at the end. It is a first-class action: one step, no justification screen that punishes the clinician for having an opinion, and no silent penalty for using it.
We do ask for a reason, optionally and briefly, because overrides are the most useful signal the system produces. A model that is being overridden in a particular ward, for a particular cohort, at a particular time of day is telling you something that no validation set will.
What gets recorded
Every advisory the system produces, every clinician who saw it, and every decision taken with or against it goes into the record. Not as a compliance artefact, but because a suggestion that vanishes when dismissed cannot be reviewed, and a suggestion that cannot be reviewed cannot be improved.
The record is also the answer to the question a hospital will eventually be asked: what did the software say, when, and what did the clinician do about it. That question deserves a precise answer rather than a reconstruction.
Where we will not put a model
Anywhere the output would act without a person in the loop. There is a category of automation that is entirely appropriate in a warehouse and entirely inappropriate in a ward, and the boundary is not subtle.
Anywhere the reasoning cannot be surfaced. If we cannot answer the three questions above, the feature does not ship, however good the number looks.
And anywhere the failure mode is silent. A model that degrades quietly as the underlying population shifts is more dangerous than one that fails loudly on day one, because nobody is looking.
If any of the above matches a problem you are dealing with, we are happy to go deeper than a blog post reasonably can.