OODA loop revisited – medical errors, heuristics, and AI.

OODA loop revisited – medical errors, heuristics, and AI.

My OODA loop post is actually one of the most popular on this site.   I  blame Venkatesh Rao of Ribbonfarm and his Tempo book and John Robb’s Brave New War for introducing me to Boyd’s methodology.   Venkatesh focuses on philosophy and management consulting, and Robb focuses on COIN and human social networks. Both are removed from healthcare, but applying Boyd’s principles to medicine: our enemy is disease, perhaps even ourselves.

Consider aerial dogfighting.  The human OODA loop is – Observe, Orient, Decide, Act.   You want to “get inside your opponent’s OODA loop” and out-think them, knowing their actions before they do, assuring victory.  If you know your opponent’s next move, you can anticipate where to shoot and end the conflict decisively.  Quoting Sun Tzu in The Art of War:

Sun Tzu Art of War OODA loops and AI

If you know the enemy and know yourself, you need not fear the result of a hundred battles. If you know yourself but not the enemy, for every victory gained you will also suffer a defeat. If you know neither the enemy nor yourself, you will succumb in every battle.

Focused, directed, lengthy and perhaps exhausting training for a fighter pilot enables them to “know their enemy” and anticipate action in a high-pressure, high-stakes aerial battle.  The penalty for failure is severe – loss of the pilot’s life.   Physicians prepare similarly – a lengthy and arduous training process in often adverse circumstances.  The penalty for failure is also severe – a patient’s death.  Given adequate intelligence and innate skill, successful pilots and physicians internalize their decision trees – transforming the OODA loop to a simpler OA loop – Observe and Act.  Focused practice allows the Orient and Decide portions of the loop to become automatic and intuitive, almost Zen-like.  This is what some people refer to as ‘Flow’ – an effortlessly hyperproductive state where total focus and immersion in a task suspends the perception of the passage of time.

For a radiologist, ‘flow’ is when you sit down at your PACS at 8am, continuously reading cases, making one great diagnosis after another, smiling as the words appear on Powerscribe. You’re killing the cases and you know it.  Then your stomach rumbles – probably time for lunch – you look up at the clock and it is 4pm.  That’s flow.

Flow is one of the reasons why experienced professionals are highly productive – and a smart manager will try to keep a star employee ‘in the zone’ as much as possible, removing extraneous interruptions, unnecessary low-value tasks, and distractions.

Kahneman defines this as fast type 1 thinking, intuitive and heuristic : quick, easy, and with sufficient experience/training, usually accurate.  But type 1 thinking can fail : a complex process masquerades as a simple one, additional important data is undiscovered or ignored, or a novel agent is introduced.  In these circumstances type 2 critical thinking is needed : slow, methodological, deductive and logical.  But humans err, substituting heuristic thinking for analytical thinking, and we get it wrong.

For the enemy fighter pilot, its the scene in Top Gun where Tom Cruise hits the air brakes to drop behind an attacking Mig to deliver a kill shot with his last missile. For a physician, it is an uncommon or rare disease presenting like a common one, resulting in a missed diagnosis and lawsuit.

To those experimenting in deep learning and Artificial intelligence, the time to train or teach the network far exceeds the time needed to process an unknown through the trained network.  Training can take hours to days, evaluation takes seconds.

Narrow AI’s like Convolutional Neural Networks take advantage of their speed to go through the OODA loop quickly, in a process called inference.  I suggest a deep learning algorithm functions as an OA loop on the specific type of data it has been trained on.  Inference is quick.

I believe that OODA loops are Kahneman’s Type 2 slow thinking.  OA loops are Kahneman’s Type 1 fast thinking.  Narrow AI inference is a type 1 OA loop.   An AI version of type 2 slow thinking doesn’t yet exist.*

And like humans, Narrow AI can be fooled.

Can your classifier tell the difference between a chihuahau and blueberry muffin?

If you haven’t seen the Chihuahua vs. blueberry muffin clickbait picture, consider yourself sheltered. Claims that narrow AI can’t tell the difference are largely, but not entirely, bogus.  While Narrow AI is generally faster than people, and potentially more accurate, it can still make errors. But so can people. In general, classification errors can be reduced by creating a more powerful, or ‘deeper’ network. I think collectively we have yet to decide how much error to tolerate in our AI’s. If we are willing to tolerate an error of 5% in humans, are we willing to tolerate the same in our AI’s, or do we expect 97.5%?  Or 99%? Or 99.9%?

The single pixel attack is a bit more interesting.  While similar images such as the ones above probably won’t pass careful human scrutiny, and frankly adversarial images unrecognizable to humans can be misinterpreted by a classifier:

Convolutional Neural Networks can be fooled by adversarial images

Selecting and perturbing a single pixel is much more subtle, and probably could escape human scrutiny.  Jaiwei Su et al address this in their “One Pixel Attack” paper, where the modification of one pixel in an image had between a 66% to 73% chance of changing the classification of that image.  By changing more than one pixel, success rates respectively rose.  The paper used older, less deep Narrow AI’s like VGG-16 and Network-in-network.  Newer models such as DenseNets and ResNets might be harder to fool.  This type of “attack” represents a real-world situation where the OA loop fails to account for unexpected new (or perturbed) information, and is incorrect.

Contemporaneous update: Google has developed images that use an adversarial attack to uniformly defeat classification attempts by standard CNN models.  By making “stickers” out of these processed images, the presence of such an image, even at less than 20% of the image size, is sufficient to change the classification to what the ensemble dictates, rather than the primary object in an image.  They look like this:

adversarial images capable of overriding CNN classifier
https://arxiv.org/pdf/1712.09665.pdf

 

I am not aware of defined solutions to these problems – the obvious images that fool the classifier can probably be dealt with by ensembling other, more traditional forms of computer vision image analysis such as HOG or SVM’s.  For a one-pixel attack, perhaps widening the network and increasing the number of training samples by either data augmentation or adversarially generated features might make the network more robust.  This probably falls into the “too soon to tell” category.

There has been a great deal of interest and emphasis placed lately on understanding black-box models.  I’ve written about some of these techniques in other posts.  Some investigators feel this is less relevant.  However, by understanding how the models fail, they can be strengthened.  I’ve also written about this, but from a management standpoint.  There is a trade off between accuracy at speed, robustness, and serendipity.  I think the same principle applies to our AI’s as well.  By understanding the frailty of speedy accuracy vs. redundancies that come at the expense of cost, speed, and sometimes accuracy, we can build systems and processes that not only work but are less likely to fail in unexpected & spectacular ways.

Let’s acknowledge the likelihood of failure of narrow AI where it is most likely to fail, and design our healthcare systems and processes around that, as we begin to incorporate AI into our practice and management.  If we do that, we will truly get inside the OODA loop of our opponent – disease – and eradicate it before it even had a chance.  What a world to live in where the only thing disease can say is, “I never saw it coming.”

 

*I believe OODA loops have mathematical analogues. The OODA loop is inherently Bayesian – next actions iteratively decided by prior probabilities. Iterative deep learning constructs include LSTM and RNN’s (Recurrent Neural Networks) and of course, General Adversarial Networks (GANs). There have been attempts to not only use Bayesian learning for hyperparameter optimization but also combining it with RL(Reinforcement Learning) & GANs.  Time will only tell if this brings us closer to the vaunted AGI (Artificial General Intelligence)**.

**While I don’t think we will soon solve the AGI question, I wouldn’t be surprised if complex combinations of these methods, along with ones not yet invented, bring us close to top human expert performance in a Narrow AI. But I also suspect that once we start coding creativity and resilience into these algorithms, we will take a hit in accuracy as we approach less narrow forms of AI.  We will ultimately solve for the best performance of these systems, and while it may even eventually exceed human ability, there will likely always be an error present.  And in that area of error is where future medicine will advance.

© 2018

Do we need more medical imaging?

 

starshipdream
Fanpic of the starship enterprise with deep dream

The original captain of the starship Enterprise, James T. Kirk addressed his ship with the invocation of, “Computer, …” .  For an audience in the late 1960’s it was a imagined miracle hundreds of years in the future.  In the early 1990’s, MIT’s SAIL Laboratory was dreaming of Project Oxygen – an ever-present, voice activated computer that could be spoken to and give appropriate responses.

 

“Hi, Siri” circa 2011
alexa
“Hello Alexa” circa 2016

 

 

 

 

 

 

 

 

 

Cloud computing, plentiful memory, on-demand massive storage and GPU-powered deep learning brought this future into our present.  Most of us already have the appliance (a smartphone) capable of connecting us to scalable cloud computing resources. Comparing current reality to the 1960’s expectations, this advancing world of ubiquitous computing is small, cheap, and readily available.

But imaging is not.  The current paradigm holds imaging as a rare, special, and expensive medical procedure.  In the days of silver-film radiology, with tomographic imaging and cut-film feeders for interventional procedures, it was a scarce resource.  In the first days of CT and MRI, requests for anything more complicated than an x-ray needed to pass through a radiologist.  These machines, and the skills necessary to operate them, were expensive and in short supply.

But is it still?  In a 2017 ER visit – the point of access to health care for > 50% of patients –  if your symptoms are severe enough, it is almost a certainty you will receive imaging early in your ER visit.  Belly pain? – CT that.  Worst headache of your life? – CT again.   Numbness on one side of your body?  Diffusion Weighted MRI.  And it is ordered on a protocol circumventing Radiology approval – why waste time in the era of 24/7 imaging with final interpretations available in under an hour.

I’ve written briefly about how a change to value-based care will upend traditional fee for service (FFS) delivery patterns.  But with that change from FFS, and volume to value, should we think about Radiology and other diagnostic services differently?  Perhaps medical imaging should be not rationed, but readily and immediately available – an equal to the history and physical.

I call this concept Ubiquitous Imaging ©, or Ubiquitous Radiology.   Ubiquitous Imaging is the idea that imaging is so necessary for the diagnosis and management of disease that it should be an integral part of every diagnostic workup, and guide every treatment plan where it is of benefit.  “A scan for every patient, when it would benefit to the patient.”

This is an aggressive statement.  We’re not ready for it just yet.  But let me explain why Ubiquitous Imaging is not so far off.

  1.  Imaging is no longer a limited good in the developed world
  2.  Artificial intelligence will increase imaging productivity, similar to PACS
  3.  Concerns about radiation dose will be salved by improvements in technology
  4.  Radiomics will greatly increase the value of imaging
  5.  Contrast use may be markedly decreased by an algorithm
  6.  Imaging will change from a cost center to an accepted part of preventative care in a value-based world.
  7. Physicians may shift from the current subspecialty paradigm to a Diagnosis-Acute Treatment-Chronic Care Management paradigm to better align with value based care.

Each of these points may sound like science fiction.  But the groundwork for each of these is being laid now:

In the US in 2017, there are 5,564 hospitals registered with the AHA.  Each of these will have some inpatient radiology services.  As of 2007, there were at least 10,335 CT Scanners operating in the US, and 7810 MRI scanners.  Using OECD library data from 2015, with 41 CT’s & 39 MRI’s per million inhabitants of the US, and a total US census of 320,000,000 we can calculate the number of US CT and MRI scanners in 2015 to be 13,120 and 12,480 respectively.

If proper procedures are followed with appropriate staffing and a lean/six sigma approach to scanning, it is conceivable that a modern multislice CT could scan one patient every ten minutes (possibly better), and be run almost 24/7 (downtime for maintenance & QA).  Thus, one CT scanner could image 144 patients daily. 144 scans/day x 365 days/year x 13120 CT scanners = 689,587,200 potential scans yearly – two scans a year for every US resident!

MRI imaging is more problematic because physics dictates the length of scans.  The T1 and T2 relaxation times are set by the length of the sequence in milliseconds, and making scans faster runs up against the laws of physics.  While there are some ‘shortcuts’, we pay for those with T2* effects and decreased resolution.  Stronger magnets & gradients help, but at higher cost and a risk of energy transfer to the patient.  So at optimal efficiency and staffing, the best you could probably get is 22 studies daily (a very aggressive number).  22 MRI studies/day x 365 days/year x 12480 MRI’s = 100,214,400 studies yearly.  Or enough to scan 1/3 of the US population yearly.  (Recent discussions at RSNA 2017 suggest MRI scans might be able to be shortened to the length of CT)

Think about this.  We can CT scan every US citizen twice in a one year period, and we continue to think about imaging as a scarce resource.  One in three US citizens can be scanned with MRI annually.  Imaging is not scarce in the developed world.

X-ray is the most commonly performed imaging procedure, including mammography & fluoroscopy, accounting for up to 50% of radiology studies.  The CT/MR/US and nuclear medicine studies occupy the other 50%.  A bit of backing out on the number above will suggest capacity on the order of 2.256 billion possible studies a year.

We’ve done the studies – how will we interpret them?  A physician (MD) examines every study and interprets them, delivering a report.  There are about 30,656 radiologists in the USA (2012 AMA physician masterfile).  Nieman HPI suggests that estimate may be low, and gives an upper range of 37,399 radiologists.

A busy radiologist on a PACS system could interpret 30,000 studies a year.  30,656 x 30,000 = 919,680,000 potentially interpretable studies from our workforce.  Use the high estimate and the capacity number rises to 1.12 billion.  That’s a large variance from the 2.256 billion studies performed.  However, it is suggested that about 50% of studies, usually X-ray and Ultrasound, are performed and interpreted by non-radiologists.  So, that gets us back to 1.12 billion studies.

Recall that Radiologists did not always interpret studies on computer monitors (PACS).  Prior to PACS, a busy radiologist would read 18,000 studies a year.  Radiologists experienced a jump in productivity when we went from interpreting studies based on film to interpreting studies on PACS systems.

Artificial Intelligence algorithms are beginning to appear in Radiology at a rapid pace.  While it is early in the development of these products, there is no question in the minds of most informed Radiologists that computer algorithms will be a part of radiology.  And because AI solutions in radiology will not be reimbursed additionally, cost justification needs to come from productivity.  An AI algorithm in Radiology needs to justify its price by making the radiologist more efficient, so that cost is borne by economies of scale.

Now imagine that the AI algorithms develop accuracy similar to a radiologist.  Able to ‘trust’ the algorithms and thereby streamline their daily work processes, Radiologists no longer are limited to interpreting 30,000 studies a year.  Perhaps that number rises to 45,000.  Or 60,000.  I can’t in good conscience consider a higher number.  The speed of AI introduction, if rapid and widespread, may cause some capacity issues, but the aging population, retiring radiologists, well-informed medical students responding to the “invisible hand” and perpetual trends toward increasing demand for imaging services will form a new equilibirum.  Ryan Avent of the Economist (who’s book Wealth of Humans is wonderful reading) has a more resigned opinion, however.

One of the additional functions of Radiologists is to manage the potentially harmful effects of the dose of ionizing radiation used in X-rays.  We know that high levels of ionizing radiation cause cancerWhether lower levels of radiation cause cancer is controversialHowever, it is likely that some (low) percentage of cancer is actually CAUSED by medical imaging.  To combat this, we have used the ALARA paradigm in medical imaging, and in recent years to combat concerns associated with higher doses received in advanced imaging, the image gently campaign.

Recently, James Brink MD of the American College of Radiology (ACR) testified to the US congress about the need for contemporary research on the effects of the radiation doses encountered in medical imaging.  Without getting too much into the physics of imaging, more dose usually yields crisper, “prettier” images at higher resolution.

But what if there was another way to do this?  Traditionally, Radiologists have relied upon equipment makers to improve hardware and extract better signal/noise ratios which would allow for a lower radiation dose.  But in a cost-concious era, it is difficult to argue for more expensive new technologies if there is no reimbursement advantage.

However, an interesting pilot study used an AI technique on CT scans to ‘de-noise’ the images, improving their appearance.   The noise was added after artificially after the scan, rather than present at the time of imaging.  A number of papers at NIPS 2017 dealt with super-resolution.  Could similar technologies exist for imaging?  Paras Lahkani seems to think so.

Put hardware & software improvement together and we might be able to substantially decrease dose in ionizing radiation.  If this dose is low enough, and research bears out that there is a dose threshold below which radiation doesn’t cause any real effects, we could “image gently” with impunity.

Are we using the information in diagnostic imaging effectively?  Probably not.  There is just too much information on a scan for a single radiologist to report entirely.  But with AI algorithms also looking at diagnostic images, there is much more information that we can extract from the scan than we currently are.  The obvious use case is volumetrics.

The burgeoning science of Radiomics includes not only volumetrics, but also relationships between the data present on the scan we may not be able to perceive directly as humans.  Dr. Luke Oakden-Rayner caused a brief internet stir with his preliminary precision radiology article in 2017, using an AI classifier (a CNN) to predict patient survival from CT images.  While small, it showed the possibility of advanced informational discovery on existing datasets and application of those findings in a practical manner.  Radiomics feature selection has similar problems to that of genomics feature selection, in that the large number of data variables may predispose to more chance correlations than in traditionally designed, more focused experiments.

At the RSNA 2017, a number of machine learning companies were making their debut.  One of the more interesting offerings was Subtle Medical, a machine learning application designed to reduce contrast dose in imaged patients.  Not only would this be disruptive to the contrast industry by reducing the amount of administered contrast by a factor of 5 or higher (!), but it would remove one of the traditional concerns about contrast – its potential toxicity.  CT uses iodinated contrast, and MRI uses Gadolinium-based contrast.  Using less implies less toxicity and less cost, so this is a win all-around.

The economics of imaging could fill a book, let alone a blog post.  In a fee-for service world, imaging was a profit center, and increasing capacity and maximizing the number of imaging services was sensible to encourage a profitable service line.  With declining reimbursement, it has become less so (but still profitable).  However, as we transition to value-based care, how will radiology be seen?  Will it be seen as a cost-center, with radiologists fighting over a piece of the bundled payment pie, or something else?  Will it drive reduced or increased imaging utilization?  Target metrics and ease of attainment in the ACO drive this decision, with easier targets correlated with greater imaging. Particularly if imaging is seen as providing greater value, utilization should continue to rise.

Specialty training as it exists currently may not be sufficient to prepare for the way medicine is practiced in the future.  A specialty (and sup-specialty) approach was reasonable when information was not freely available, and the amount of information to know was overwhelming without specialization.  But as we increase efficiencies in medical care, care access goes along a definable path: Patient complaint -> Investigation -> Diagnosis -> Acute Treatment ->Chronic Treatment.  Perhaps it would make more sense to organize medicine along those lines as well?  Particularly in the field of diagnosis, I am not the only physician recognizing the shift occurring.  A well-thought out opinion piece written by Saurabh Jha MD and Eric Topol MD, Radiologists and Pathologists as Information Specialists, broaches that there is more similarity between the two specialties than differences, particularly in an age of artificial intelligence.  Should we call for a new Flexner report, ending the era of physician-basic scientists and beginning the dominance of physician-informaticists and physician-empaths?

Perhaps it is time to consider imaging not as a limited commodity, but instead to recognize it as a widely available resource, to be used as much as is reasonable.  By embracing AI, radiomics, new payment models, the radiologist as an informatician, and basic research on radiation safety, we can get there.

©2017 – All rights reserved

CheXNet – a brief evaluation

CheXNet – a brief evaluation

Chest X-Ray deep dreamed - our AI & deep learning future
Chest Radiograph from ChestX-ray14 dataset processed with the deep dream algorithm trained on ImageNet

NOTE: Controversy over the report and dataset continues.  I have updated the post since first written as new information has become available.  I recommend you read through the post and its addendum.

 

Andrew Ng released CheXNet yesterday on ArXiv (citation) and promoted it with a tweet which caused a bit of a stir on the internet and related radiology social media sites like Aunt Minnie.  Before Radiologists throw away their board certifications and look for jobs as Uber drivers, a few comments on what this does and does not do.

First off, from the Machine Learning perspective, methodologies check out.  It uses a 121 layer DenseNet, which is a powerful convolutional neural network.  While code has not yet been provided, the DenseNet seems similar to code repositories online where 121 layers are a pre-made format.  80/20 split for Training/Validation seems pretty reasonable (from my friend, Kirk Borne), Random initialization, minibatches of 16 w/oversampling positive classes, and a progressively decaying validation loss are utilized, all of which are pretty standard.  Class activation mappings are used to visualize areas in the image most indicative of the activated class (in this case, pneumonia).  This is an interesting technique that can be used to provide some human-interpretable insights into the potentially opaque DenseNet.

The last Fully Connected (FC) layer is replaced by a single output (only one class is being tested for – pneumonia) coupled to a sigmoid function (an activation function – see here) to give a probability between 0 and 1.   Again, pretty standard for a binary classification.  The multiclass portion of the study was performed seperately/later.

The test portion of the study was 420 Chest X-rays read by four radiologists, one of whom was a thoracic specialist.  They could choose between the 14 pathologies in the ChestX-ray14 dataset, read blind without any clinical data.

So, a ROC curve was created, showing three radiologists similar to each other, and one outlier.The radiologists lie slightly under the ROC curve of the CheXNet classifier.  But, a miss is as good as a mile, so the claims of at or above radiologist performance are accurate, because math.  As Luke Oakden Rayner points out, this would probably not pass statistical muster.

So that’s the study.  Now, I will pick some bones with the study.

First, only including one thoracic radiologist is relevant, if you are going to make ground truth agreement of 3 out of four radiologists.  General radiologists will be less specific than specialist radiologists, and that is one of the reasons why we have moved to specialty-specific reads over the last 20 years.  If the three general rads disagreed with the thoracic rad, the thoracic rad’s ground truth would be discarded.  Think about this – you would take the word of the generalist over the specialist, despite greater training.  Even Google didn’t do this in their retinal machine learning paper.  Instead, Google used their three retinal specialists as ground truth and then looked at how the non-specialty opthalmologists were able to evaluate that data and what it meant to the training dataset.  (Thanks, Melody!)  Nevertheless, all rads lie reasonably along the same ROC curve, so methodologically it checks out.

Second, the Wang ChestXray14 dataset is a dataset that was data-mined from NIH radiology reports.  This means that for the dataset, ground truth was whatever the radiologists said it was.  I’m not casting aspersions on the NIH radiologists, as I am sure they are pretty good.  I’m simply saying that the dataset’s ground truth is what it says it is, not necessarily what the patient’s clinical condition was.  As proof of that, here are a few cells from the findings field on this dataset.

Findings field from the ChestX-ray14 dataset (representative)

In any case, the NIH radiologists more than a few times perhaps couldn’t tell either, or identified one finding as the cause of the other (Infiltrate & Pneumonia mentioned side by side) and at the top you have the three fields “atelectasis” “consolidation” & “Pneumonia” – is this concurrent pneumonia with consolidation with some atelectasis elsewhere, or is it “atelectasis vs consolidation cannot r/o pneumonia” (as radiologists we say these things). While the text miner purports to use several advanced NLP tools to avoid these kinds of problems, in practice it does not seem to do so. (See addendum below)  Dr. Ng, if you read this, I have the utmost respect for you and your team, and I have learned from you.  But I would love to know your rebuttal, and I would urge you to publish those results.  Or perhaps someone should do it for reproducibility purposes.

Finally, I’m bringing up these points not to be a killjoy, but to be balanced.  I think it is important to see this and prevent someone from making a really boneheaded decision of firing their radiologists to put in a computer diagnostic system (not in the US, but elsewhere) and realizing it doesn’t work after spending a vast sum of money on it.  Startups competing in the field who do not have deep healthcare experience need to be aware of potential pitfalls in their product.  I’m saying this because real people could be really hurt and impacted if we don’t manage this transition into AI well.  Maybe all parties involved in medical image analysis should join us in taking the Hippocratic Oath, CEO’s and developers included.

Thanks for reading, and feel free to comment here or on twitter or connect on linkedin to me: @drsxr

Addendum: ChestX-ray14 is based on the ChestX-ray8 database which is described in a paper released on ArXiv by Xiaosong Wang et al. The text mining is based upon a hand-crafted rule-based parser using weak labeling designed to account for “negation & uncertainty”, not merely application of regular expressions. Relationships between multiple labels are expressed, and while labels can stand alone, for the label ‘pneumonia’, the most common associated label is ‘infiltrate’.  A graph showing relationships between the different labels in the dataset is here (from Wang Et Al.)

Label map from the ChestX-ray14 dataset by Wang et. al.

Pneumonia is purple with 2062 cases, and one can see the largest association is with infiltration, then edema and effusion.  A few associations with atelectasis also exist (thinner line).

The dataset methodology claims to account for these issues at up to 90% precision reported in ChestX-ray8, with similar precision inferred in ChestX-ray14.

No Findings (!) from NIH CXR14 dataset
“No Findings”
No Findings (!) from NIH CXR14 Dataset
“No Findings”

However, expert review of the dataset (ChestX-ray14) does not support this.  In fact, there are significant concerns that the labeling of the dataset is a good deal weaker.  I’ll just pick out two examples above that show a patient likely post R lobectomy with attendant findings classified as “No Findings” and the lateral chest X-ray which doesn’t even belong in the study database of all PA and AP films.  These sorts of findings aren’t isolated – Dr. Luke Oakden-Rayner addresses this extensively in this post, from which his own observations are garnered below:

Sampled PPV for ChestX-Ray14 dataset vs reported
Dr. Luke Oakden Rayner’s own Positive Predictive Value on visual inspection of 130 images vs reported

His final judgment is that the ChestX-ray14 dataset is not fit for training medical AI systems to do diagnostic work.  He makes a compelling argument, but I think it is primarily a labelling problem, where the proposed 90% acccuracy on the NLP data mining techniques of Wang et al does not hold up.  ChestX-ray14 is a useful dataset for the images alone, but the labels are suspect.  I would call upon the NIH group to address this and learn from this experience.  In that light, I am surprised that the system did not do a great deal better than the human radiologists involved in Dr. Ng’s group’s study, and I don’t really have a good explanation for it.

Copyright © 2017

Machine Intelligence in Medical Imaging Conference – Report

blueI heard about the Society of Imaging Informatics in Medicine’s (SIIM) Scientific Conference on Machine Intelligence in Medical Imaging (C-MIMI) on Twitter.  Priced attractively, easy to get to, I’m interested in Machine Learning and it was the first radiology conference I’ve seen on this subject, so I went.  Organized on short notice so I was expecting a smaller conference.

cmimipacked

I almost didn’t get a seat.  It was packed.

The conference had real nuts and bolts presentations & discussions on healthcare imaging machine learning (ML).  Typically, these were Convolutional Neural Networks (CNN‘s/Convnets) but a few Random Forests (RF) and Support Vector Machines (SVM) sneaked in, particularly in hybrid models along with a CNN (c.f.  Microsoft).  Following comments assume some facility in understanding/working with Convnets.

Some consistent threads throughout the conference:

  • Most CNN’s were trained on Imagenet with the final fully connected (FC) layer removed; then re-trained on radiology data with a new classifer FC layer placed at the end.
  • Most CNN’s were using Imagenet standard three layer RGB input despite being greyscale.  This is of uncertain significance and importance.
  • The limiting of input matrices to grids less than image size is inherited from the Imagenet competitions (and legacy computational power).  Decreased resolution is a limiting factor in medical imaging applications, potentially worked-around by multi-scale CNN’s.
  • There is no central data repository for a good “Ground Truth” to develop improved machine imaging models.
  • Data augmentation methods are commonly used due to lower numbers of obtained cases.

Keith Dryer DO PhD gave an excellent lecture about the trajectory of machine imaging and how it will be an incremental process with AI growth more narrow in scope than projected, chiefly limited by applications.  At this time, CNN creation and investigation is principally an artisanal product with limited scalability.  There was a theme – “What is ground truth?” which in different instances is different things (path proven, followed through time, pathognomonic imaging appearance).

There was an excellent educational session from the FDA’s Berkman Sahiner.  The difference between certifying a type II or type III device may keep radiologists working longer than expected!  A type II device, like CAD, identifies a potential abnormality but does not make a treatment recommendation and therefore only requires a 510(k) application.  A type III device, as in an automated interpretation program creating diagnosis and treatment recommendations will require a more extensive application including clinical trials, and a new validation for any material changes.  One important insight (there were many) was that the FDA requires training and test data to be kept separate.   I believe this means that simple cross-validation is not acceptable nor sufficient for FDA approval or certification.  Adaptive systems may be a particularly challenging area for regulation, as similar to the ONC, significant changes to the software of the algorithm will require a new certification/approval process.

Industry papers were presented from HK Lau of Arterys, Xiang Zhou of Siemens, Xia Li of GE, and Eldad Elnekave of Zebra medical.  The Zebra medical presentation was impressive, citing their use of the Google Inception V3 model and a false-color contrast limited adaptive histogram equalization algorithm, which not only provides high image contrast with low noise, but also gets around the 3-channel RGB issue.  Given statistics for their CAD program were impressive at 94% accuracy compared to a radiologist at 89% accuracy.

Scientific Papers were presented by Matthew Chen, Stanford; Synho Do, Harvard; Curtis Langlotz, Stanford; David Golan, Stanford; Paras Lakhani, Thomas Jefferson; Panagiotis Korfiatis, Mayo Clinic; Zeynettin Akkus, Mayo Clinic; Etka Bullar, U Saskatchewan; Mahmudur Rahman, Morgan State U; Kent Ogden SUNY upstate.

Ronald Summers, MD PhD from the NIH gave a presentation on the work from his lab in conjunction with Holger Roth, detailing the specific CNN approaches to Lymph Node detection, Anatomic level detection, Vertebral body segmentation, Pancreas Segmentation, and colon polyp screening with CT-colonography, which had high False Positives.  In his experience, deeper models performed better.  His lab also changes unstructured radiology reporting into structured reporting through ML techniques.

Abdul Halabi of NVIDIA gave an impressive presentation on the supercomputer-like DGX-1 GPU cluster (5 deliveries to date, the fifth of which was to Mass. General, a steal at over $100K), and the new Pascal architecture in the P4 & P40 GPU’s.  60X performance on AlexNet vs the original version/GPU configuration in 2012.  Very impressive.

Sayan Pathak of Microsoft Research and the Inner Eye team gave a good presentation where he demonstrated that a RF was really just a 2 layer DNN, i.e. a sparse 2 layer perceptron.   Combining this with a CNN (dNDE.NET), it beat googLENet’s latest version in the Imagenet arms race.  However, as one needs to solve for both structures simultaneously, it is an expensive (long, intense) computation.

Closing points were the following:

  • Most devs currently using Python – Tensorflow +/- Keras with fewer using CAFFE off of  Modelzoo
  • DICOM -> NIFTI -> DICOM
  • De-identification of data is a problem, even moreso when considering longitudinal followup.
  • Matching accuracy to the radiologist’s report may not be as important as actual outcomes report.
  • There was a lot of interest in organizing a competition to advance medical imaging, c.f. Kaggle.
  • Radiologists aren’t obsolete just yet.

It was a great conference.  An unexpected delight.  Food for your head!

 

 

 

Value and Risk: the Radiologist’s perspective (Value as risk series #4)

Public DomainMuch can be written about Value-based care. I’ll focus on imaging risk management from a radiologist’s perspective. What it looks like from the Hospital’s perspective , the Insurer’s perspective, and in general have been discussed previously.

When technology was in shorter supply, radiologists were gatekeepers of limited Ultrasound, CT and MRI resources. Need-based radiologist approval was necessary for ‘advanced imaging’. The exams were expensive and needed to be protocoled correctly to maximize utility. This encouraged clinician-radiologist interaction – thus our reputation as “The Doctor’s doctor.”

In the 1990’s-2000’s , there was an explosion in imaging utilization and installed equipment. Imaging was used to maximize throughput, minimize patient wait times and decrease length of hospital stays. A more laissez-faire attitude prevailed where gatekeeping was frowned upon.

With a transition to value-based care, the gatekeeping role of radiology will return. Instead of assigning access to imaging resources on basis of limited availability, we need to consider ROI (return on investment) in the context of whether the imaging study will be likely to improve outcome vs. cost. (1) Clinical Decision Support (CDS) tools can help automating imaging appropriateness and value. (2)

The bundle’s economics are capitation of a single care episode for a designated ICD-10 encounter. This extends across the inpatient stay and related readmissions up to 30 days after discharge (CMS BPCI Model 4). A review of current Model 4 conditions show mostly joint replacements, spinal fusion, & our example case of CABG (Coronary Artery Bypass Graft).

Post CABG, a daily Chest X-ray (CXR) protocol may be ordered – very reasonable for an intubated & sedated patient. However, an improving non-intubated awake patient may not need a daily CXR. Six Sigma analysis would empirically classify this as waste – and a data analysis of outcomes may confirm it.

Imaging-wise, patients need a CXR preoperatively, & periodically thereafter. A certain percentage of patients will develop complications that require at least one CT scan of the chest. Readmissions will also require re-imaging, usually CT. There will also be additional imaging due to complications or even incidental findings if not contractually excluded (CT/CTA/MRI Brain, CT/CTA neck, CT/CTA/US/MRI abdomen, Thoracic/Lumbar Spine CT/MRI, fluoroscopy for diaphragmatic paralysis or feeding tube placement, etc…). All these need to be accounted for.

www.n2value.com

 

In the fee-for-service world, the ordered study is performed and billed.  In bundled care, payments for the episode of care are distributed to stakeholders according to a pre-defined allocation.

Practically, one needs to retrospectively evaluate over a multi-year period how many and what type of imaging studies were performed in patients with the bundled procedure code. (3) It is helpful to get sufficient statistical power for the analysis and note trends in both number of studies and reimbursement. Breaking down the total spend into professional and technical components is also useful to understand all stakeholder’s viewpoints. Evaluate both the number of studies performed and the charges, which translates into dollars by multiplying by your practice’s reimbursement percentage. Forward-thinking members of the Radiology community at Nieman HPI  are providing DRG-related tools such as ICE-T to help estimate these costs (used in above image). Ultimately one ends up with a formula similar to this:

CABG imaging spend = CXR’s+CT Chest+ CTA chest+ other imaging studies.

Where money will be lost is at the margins – patients who need multiple imaging studies, either due to complications or incidental findings. With between a 2% to 3% death rate for CABG and recognizing 30% of all Medicare expenditures are caused by the 5% of beneficiaries that die, with 1/3 of that cost in the last month of life (Barnato et al), this must be accounted for. An overly simplistic evaluation of the imaging needs of CABG will result in underallocation of funds for the radiologist, resulting in per-study payment dropping  – the old trap of running faster to stay in place.

Payment to the radiologist could either be one of two models:

First, fixed payment per RVU. Advantageous to the radiologist, it insulates from risk-sharing. Ordered studies are read for a negotiated rate. The hospital bears the cost of excess imaging. For a radiologist in an independent private practice providing services through an exclusive contract, allowing the hospital to assume the risk on the bundle may be best.

Second, a fixed (capitated) payment per bundled patient for imaging services may be made to the radiologist. This can either be in the form of a fixed dollar amount or a fixed percentage of the bundle.  (Frameworks for Radiology Practice Participation, Nieman HPI)  This puts the radiologist at-risk, in a potentially harmful way. The disconnect is that the supervising physicians (cardio-thoracic surgeon, intensivist, hospitalist) will be focusing on improving outcome, decreasing length of stay, or reducing readmission rates, not imaging volume. Ordering imaging studies (particularly advanced imaging) may help with diagnostic certitude and fulfill their goals. This has the unpleasant consequence of the radiologist’s per study income decreasing when they have no control over the ordering of the studies and, in fact, it may benefit other parties to overuse imaging to meet other quality metrics. The radiology practice manager should proceed with caution if his radiologists are in an employed model but the CT surgeon & intensivists are not. Building in periodic reviews of expected vs. actual imaging use with potential re-allocations of the bundle’s payment might help to curb over-ordering. Interestingly, in this model the radiologist profits by doing less!

Where the radiologist can add value is in analysis, deferring imaging unlikely to impact care. Reviewing data and creating predictive analytics designed to predict outcomes adds value while, if correctly designed, avoiding more than the standard baseline of risk. (see John’s Hopkins Sepsis prediction model). In patients unlikely to have poor outcomes, additional imaging requests can be gently denied and clinicians reassured. I.e. “This patient has a 98% chance of being discharged without readmission. Why a lumbar spine MRI?” (c.f. AK Moriarty et al) Or, “In this model patients with these parameters only need a CXR every third day. Let’s implement this protocol.” The radiologist returns to a gatekeeping role, creating value by managing risk, intelligently.

Let’s return to our risk/reward matrix:

www.n2value.com

 

For the radiologist in the bundled example receiving fixed payments:

 

Low Risk/Low Reward: Daily CXR’s for the bundled patients.

 

High Risk/Low Reward: Excess advanced imaging (more work for no change in pay)

 

High Risk/High Reward: Arbitrarily denying advanced imaging without a data-driven model (bad outcomes = loss of job, lawsuit risk)

 

Low Risk/High Reward: Analysis & Predictive modeling to protocol what studies can be omitted in which patients without compromising care.

 

I, and others, believe that bundled payments have been put in place not only to decrease healthcare costs, but to facilitate transitioning from the old FFS system to the value-based ‘at risk’ payment system, and ultimately capitated care. (Rand Corp, Technical Report TR-562/20) By developing analytics capabilities, radiology providers will be able to adapt to these new ‘at-risk’ payment models and drive adjustments to care delivery to improve or maintain the community standard of care at the same or lower cost.

  1. B Ingraham, K Miller et al. Am Coll Radiol 2016 in press
  2. AK Moriarty, C Klochko et al J Am Coll Radiol 2015;12:358-363
  3. D Seidenwurm FJ Lexa J Am Coll Radiol 2016 in press

Defining value in healthcare through risk

High-low-norisk

For a new definition of value, it’s helpful to go back to the conceptual basis of payment for medical professional services under the RBRVS. Payment for physician services is divided into three components: Physician work, practice expense, and a risk component.

Replace physician with provider, and then extrapolate to larger entities.

Currently, payer (insurer, CMS, etc…) and best practice (specialty societies, associations like HFMA, ancillary staff associations) guidelines exist. This has reduced some variation among providers, and there is an active interest to continue this direction. For example, level 1 E&M clearly differs from a level 5 E&M – one might disagree whether a visit is a level 3 or 4, but you shouldn’t see the level 1 upcoded to 5. Physician work is generally quantifiable in either patients seen or procedures done, and for any corporate/employed practice, most physicians will be working towards the level of productivity they have contractually agreed to, or they will be let go/contracts renegotiated. Let’s hope they are fairly compensated for their efforts and not subjected solely to RVU production targets, which are falling out of favor vs. more sophisticated models (c.f. Craig Pedersen, Insight Health Partners).

Unless there is mismanagement in this category, provider work is usually controllable, measurable, and with some variation due to provider skill, age, and practice goals, consistent. For those physicians who have been vertically integrated, their current EHR burdens and compliance directives may place a cap on productivity.

Practice expenses represent those fixed expenses and variable expenses in healthcare – rent, taxes, facility maintenance, and consumables (medical supplies, pharmaceuticals, and medical devices). Most are fairly straightforward from an accounting standpoint. Medical supplies, pharmaceuticals, and devices are expenses that need management, with room for opportunity. ACO and super ACO/CIO organizations and purchasing consortiums such as Novation, Amerinet, and and Premier have been formed to help manage these costs.

Practice expense costs are identifiable, and once identified, controllable. Initially, six sigma management tools work well here. For all but the most peripheral, this has happened/is happening, and there are no magic bullets out there beyond continued monitoring of systems & processes as they evolve over time as drift and ripple effects may impact previously optimized areas.

This leaves the last variable – risk. Risk was thought of as a proxy for malpractice/legal costs. However, in the new world of variable payments, there is not only downside risk in this category, but the pleasant possibility of upside risk.

It reasons that if your provider costs are reasonably fixed, and practice expenses are as fixed as you can get them at the moment, that you should look to the risk category as an opportunity for profit.

As a Wall St. options trader, the only variable that really mattered to me for the price of the derivative product was the volatility of the option – the measure of its inherent risk. We profited by selling options (effectively, insurance) when that implied volatility was higher than the actual market volatility, or buying them when it was too low. Why can’t we do the same in healthcare?

What is value in this context? The profit or loss arising from the assumption and management of risk. Therefore, the management of risk in a value-based care setting allows for the possibility of a disproportionate financial return.

www.n2value.com

The sweet spot is Low Risk/High Return. This is where discovering a fundamental mispricing can return disproportionately vs. exposure to risk.

Apply this risk matrix to:

  • 1 – A medium sized insurer, struggling with hospital mergers and former large employers bypassing the insurer directly and contracting with the hospitals.
  • 2 – A larger integrated hospital system with at-risk payments/ACO model, employed physicians, and local competitors which is struggling to provide good care in the low margin environment.
  • 3 – group radiology practice which contracts with a hospital system and a few outpatient providers.

& things get interesting. On to the next post!

Some reflections on the ongoing shift from volume to value

As an intuitive and inductive thinker, I often use facts to prove or disprove my biases. This may make me a poor researcher, though I believe I would have been popular in circa 1200 academic circles. Serendipity plays a role; yes I’m a big Nassim Taleb fan – sometimes in the seeking, unexpected answers appear. Luckily, I’m correct more often than not. But honestly – in predicting widely you miss more widely.

One of my early mentors from Wall St. addressed this with me in the infancy of my career – take Babe Ruth’s batting average of .342 . This meant that two out of three times at bat, Babe Ruth struck out. However, he was trying to hit home runs. There is a big difference between being a base hit player and a home run hitter. What stakes are you playing for?

With that said, this Blog is for exploring topics I find of interest pertaining mostly to healthcare and technology. The blog has been less active lately, not only due to my own busy personal life (!) but also because I have sought more up-to-date information about advancing trends in both the healthcare payment sector and the IT/Tech sector as it applies to medicine. I’m also diving deeper into Radiology and Imaging. As I’ve gone through my data science growth phase, I’ll probably blog less on that topic except as it pertains to machine learning.

The evolution of the volume to value transition is ongoing as many providers are beginning to be subject to at least a degree of ‘at-risk’ payment. Stages of ‘at-risk’ payment have been well characterized – this slide by Jacque Sokolov MD at SSB solutions is representative:

Sokolove - SSB solutions slide 1

In 2015, approximately 20% of medicare spend was value-based, with CMS’s goal 50% by 2020. Currently providers are ‘testing the waters’ with <20% of providers accepting over 40% risk-based payments (c.f. Kimberly White MBA, Numerof & Associates). Obviously the more successful of these will be larger, more data-rich and data-utilizing providers.

However, all is not well in the value-based-payment world. In fact, this year United Health Care announced it is pulling its insurance products out of most of the ACA exchange marketplaces. While UHC products were a small share of the exchanges, it sends a powerful message when a major insurer declines to participate. Recall most ACO’s (~75%) did not produce cost savings in 2014, although more recent data was more encouraging (c.f. Sokolov).   Notably, out of the 32 Pioneer ACO’s that started, only 9 are left (30%) (ref. CMS). The road to value is not a certain path at all.

So, with these things in mind, how do we negotiate the waters? Specifically, as radiologists, how do we manage the shift from volume to value, and what does it mean for us? How is value defined for Radiology? What is it not? Value is NOT what most people think it is. I define value as: the cost savings arising from the assumption and management of risk. We’ll explore this in my next post.