An open letter to Health AI Startups
I am going to tell you in this manner because I want to try and maintain the friendship with you in spite of what I am about to say
I want to thank you for the introduction and opportunity with [your Health AI Startup], and the faith you showed in me when you made that introduction.
However, and for several reasons, I have decided not to continue to explore this opportunity.
For the last several years I have intentionally avoided becoming entangled with Health AI startups because, for a number of reasons, I have seen that almost all have done things that ultimately and antithetically make true health AI adoption an even more unlikely prospect.
Most, in my honest observation, do untenable things with how they promote themselves (thanks IBM Watson for Health), how they handle patient data (thanks Google/DeepMind) or how they address the failures of their solution (thanks EPIC Sepsis Model), such that my fairly black-and-white ethical and legal mind simply doesn’t want to be involved in whatever it is they are selling.
First, one of the key reasons I have declined to work with most Health AI startups is that, in my honest and experienced opinion, many are started by people who are little more than tech evangelists who simply set out to exploit a relationship that they have with a clinical organisation or funder.
They sell what sounds like a world-changing idea wrapped up in buzzwords and tech-speak, but for which they actually have limited or very poor understanding and/or capability of ever producing into a tangible solution.
Like Theranos, they oversell, overpromise and are eventually caught out, possibly for being what some might consider to be snake-oil salesmen and women. In effect, like a virtual beehive they are all buzz and no honey1. A hype-factory.
They create a job and income for themselves and a few others and, if they are very lucky, they may even produce a virtual product or idea that they can later exploit… but more about that in point (iv) of my third point in a moment.
Second, I am yet to see one that is actually doing something truly different to the pack. They all set out claiming to have, but rarely possess, a point of difference that sets them apart or suggests they will be successful where many others before them have failed. The claims they make always include one or more of the following:
- better data – we have access to a better data source than anyone else and that data source means we will succeed where others failed. This claim is used too often to count, and unfortunately data is never a substitute for knowledge or wisdom.
- more data – at some point in the future we are going to integrate additional or larger datasets such as genome or -omics data and this will fix any failures in our models or modelling approach or bring unlimited and untold accuracy. As with better data, this claim is made by almost every Health AI startup and again, data is not a substitute for knowledge or wisdom. More data being given to flawed models, at best, results in even more skew and random, uncontrolled and complicated errors. At worst, it breaks the model entirely or results in untold potential harm.
- special person – that we have this particular person means we will succeed where all others failed. Almost always, the special person is the tech evangelist him/herself. The special quantity they are said to bring is the originating idea he/she believes they created and seeded as the core intellectual property of the health AI startup they wish us to hitch our wagon to (and that investors in their vapourware are ‘investing in’).
- a better approach or model – in order to keep it comprehensible to the masses we are or are going to use the same tools as everyone else (cool math or ML models), but our special person has discovered some previously undiscovered and top secret esoteric knowledge that will make them work better or smarter than anyone else’s. The sad fact is that every time this claim has been made, the forensic after-play analysis has shown they were doing the same things as tens or hundreds of others before them who failed, but were simply hoping for a different outcome. As one person put it to me this week, they hope that with slightly different potted plants in the office they will be the ones to crack it… this time.
Third, of the rare few Health AI startups that manage to get something to market, there are at least four consistent observations we can make, including finding their solutions:
- are never true AI – in almost all observable cases they are either sub-ML mathematical models based on linear regression or similar, or ML classifiers with the term AI used as a buzzword to upsell or hype what they are doing.
- have poor accuracy and have all been observed to be universally flawed – those that do ‘make something’ and get it in front of clinicians or health consumers are:
- linear regression over-smoothed math models with poor accuracy and low sensitivity and specificity that presents with between 21-74% false positive or false negative rates – e.g. pacmed.nl and EPIC Sepsis model (see my previous post here)
- math models that are heavily overfitted and in some cases are later proven to have been trained on the same patients that the developers go on to use as the patients in validation and accuracy tests – e.g. the IBM Watson for Cancer 1000 patient accuracy test in 2013 (again, see my previous post here)
- machine learning-based image classifiers that because of high fallibility have to be supervised even once they are in use, and whose outputs are often regularly ‘confused’, ‘misrepresented’ or ‘completely misunderstood’ – e.g. solutions to identify tumours in x-ray and ultrasound images that while academic research claims are as good or better than radiologists have proven in practice to make mistakes in around 1/3rd of cases, the retinal image detectors used in the DeepMind solution at Moorfields Hospital in London that eye doctors regularly mis-understand or mis-describe, or the cataract image detectors the NHS began rolling out in 2022 that have incredibly increased cataract diagnoses, many of which, at a rate of around 1400% and like mine, have proven to be complete misdiagnoses.
- have incredibly poor safety resulting from many missed or incorrect diagnoses and under- or over -alarming that mean serious health conditions either go undetected/undiagnosed or, at the other extreme, staff get alarm fatigue and begin to ignore the solution entirely – e.g. see IBM Watson for Health, EPIC Sepsis Model, pacmed.nl and Babylon Health’s GP at Hand (again, see my previous post here)
- as the flaws in their model start to become apparent, they very quickly use the fact that they actually got some
suckersusers to buy in to sell up and bug out at breakneck speed. If they can’t sell up quick enough they extract all the value they can and bankrupt/shutter the startup – e.g. IBM sold elements of Watson for Health to Fransisco Partners who pretty much only used the hardware platform as ‘tin’ to run their own solutions. The founder of Babylon AI in the UK tried unsuccessfully to broker a deals with several large tech companies before eventually only managing to sell off the South London GP clinic and ‘GP at Hand’ app to eMed. eMed appear to have gutted most of the GP at Hand app to turn it into little more than a four or five question triage solution that directs you to see a doctor today, this week, or ‘soon’ – much of the clever ‘AI’ seems to have gone. Babylon AI announced they were in financial dire straights or almost bankrupt in three countries (US, UK and Rwanda) prior to the sale and several articles suggest the founder has moved overseas with any remaining funds and the proceeds of the sale to ‘try again in another market’. While not strictly in healthcare (but having healthcare customers) the Engine B solution started by former staff of Microsoft UK and KPMG’s London office that I did actually do some of the work on was offered for sale to a US accountancy firm at the point where it was possibly going to have financial difficulties (i.e. needed cash). That accountancy firm initially purchased 10 percent of the inflated value of the company, keeping the lights on for about another year. They have since purchased the entire company but some commentators believe this was solely in order to get a foothold in the UK audit market. That while they acquired the common data model and automated workflows Engine B had been developing, the purchase was more about the customers Engine B was onboarding or had access to than the SAP-like ML models and knowledge graphs.
Patients, at best, see no change for all the money that is ultimately taken from them and distributed to the Health AI startup. At worst, they may be negatively impacted or, in some cases, even harmed by the introduction of what have universally proven to be ‘bad’ Health AI solutions.
Whether we talk in terms of the potential or actual benefits, the results these Health AI startups bring never seem to come close to cancelling out the financial costs, ethical risks and other negatives visited on health service consumers.
Fourth, almost all Health AI startups focus on data, data, data and ignore clinical workflow and expert clinical knowledge. They believe they can address the complexity of the human model, diseases and comorbid interaction by creating complicated and often incomprehensible math models based around a small and ‘simpler’ subset of data fields.
In this way we see linear regression used to filter something as complicated as cardiac diseases (which can have wildly variable aetiologies related to congenital, degenerative, pharmacological, infective or acute disease) down to a collection of six or seven fields that are used to generate what are laughably claimed to be propensity scores and predictions that can not only never take into account anything not contained within that subset of data, but which use a method they claim addresses confounders that they often go on to admit they never even attempted to identify or quantify (i.e. the unknown unknowns).
Data should never be the starting point.
Health AI should be thought of much like the human patients it may be used to make predictions about – the skeleton should be based entirely on expert clinical knowledge; the internal organs should be a well-selected model suitable to answering the question being asked or producing the result you require; the skin represents the protection that comes when we define and detail the proper testing we will undertake to verify and validate, and the application of common sense in how we construct and use the results of the model; and the data comes last – represented by the good quality clean and safe food we should all eat that keeps the entire structure running.
At the start it was my hope that [your Health AI Startup] was going to be different to all the other run-of-the-mill Health AI startups. Unfortunately, the reality has shown me this may actually not be the case. [Your Health AI Startup] manages to trip, slip or land squarely on aspects of every one of the four points I raise above.
This isn’t about you or any particular thing you may have said, done or not done. In fact, I have honestly and fairly tried to surface whether there is a point of difference that would necessitate re-evaluation of my general position.
All I have uncovered is the presence of misplaced belief in ‘big data’ and the existing like so many that came before cookie-cutter approaches that others have tried and failed, and all of the usual cliches of AI startups everywhere.
The client-facing tech work I have done during most of the last 15 years or so has been designing solutions to solve what my clients often thought were intractable problems. When I am not doing that I am being brought in as a troubleshooter to get something that isn’t working or has become ‘stuck’, moving again.
Your special person has decided and knows what they want to do and doesn’t therefore have a need for someone like me. There is no problem in this situation for me to solve, nothing new for me to design, and nothing even for me to troubleshoot.
Overall, you would both be better served to find warm bodies that can sit in chairs and do little more than take instructions without question, and undertake the data cleansing and SKLearn modelling tasks that will bring the picture in your special person’s mind into fruition.
See more here substack.com
Please Donate Below To Support Our Ongoing Work To Defend The Scientific Method
PRINCIPIA SCIENTIFIC INTERNATIONAL, legally registered in the UK as a company 
incorporated for charitable purposes. Head Office: 27 Old Gloucester Street, London WC1N 3AX.
Trackback from your site.