{"id":13421,"date":"2017-11-17T11:48:08","date_gmt":"2017-11-17T16:48:08","guid":{"rendered":"http:\/\/n2value.com\/blog\/?p=13421"},"modified":"2018-01-25T11:42:35","modified_gmt":"2018-01-25T16:42:35","slug":"chexnet-a-brief-evaluation","status":"publish","type":"post","link":"https:\/\/n2value.com\/blog\/chexnet-a-brief-evaluation\/","title":{"rendered":"CheXNet \u2013 a brief evaluation"},"content":{"rendered":"
\"Chest
Chest Radiograph from ChestX-ray14 dataset processed with the deep dream algorithm trained on ImageNet<\/figcaption><\/figure>\n

1\/25\/18: NOTE:\u00a0 Since the November release of the CheXNet paper on ArXiV, there has been a healthy and extensive online discussion on twitter, reddit, and online blogs.\u00a0 The Stanford paper has undergone at least two revisions with some substantial modifications, most importantly the replacement of ROC curves with F1 scores and a bootstrap calculation of significance.\u00a0 Some details about the methodology which were not released in the original version have come out, particularly the “re-labeling” of ground truth by Stanford radiologists.\u00a0 My comment about the thoracic specialist has completely borne out on further release of information. And the problems with ChestXRay14’s labeling (why the Stanford docs re-labeled) are now well-known.
\n<\/strong><\/em><\/p>\n

The investigation and discussion of this paper has been spearheaded by Luke Oaken Rayner, who has spent months corresponding with the author and discussing the paper.\u00a0 For further Information, see below.<\/strong>
\n<\/em><\/p>\n

The discussion on CheXNet appears to be over, and there has been a great deal of collective learning in it.\u00a0 The Stanford group should be lauded for their willingness to engage in open peer review and modify their paper substantially after it.\u00a0 There is no question that a typical 18-24 month process of review and discussion was fast-tracked in the last two months.\u00a0 Relevant blog links are below after my December addendum. \u00a0 This will be my last update on this post, as it is “not so brief” any longer! <\/strong><\/em><\/p>\n

 <\/p>\n

Andrew Ng released CheXNet <\/a>yesterday on ArXiv (citation)<\/a> and promoted it with a tweet <\/a>which caused a bit of a stir on the internet <\/a>and related radiology social media sites <\/a>like Aunt Minnie.\u00a0 Before Radiologists throw away their board certifications and look for jobs as Uber drivers, a few comments on what this does and does not do.<\/p>\n

First off, from the Machine Learning perspective, methodologies check out.\u00a0 It uses a 121 layer DenseNet<\/a>, which is a powerful convolutional neural network.\u00a0 While code has not yet been provided, the DenseNet seems similar to code repositories online where 121 layers are a pre-made format.\u00a0 80\/20 split for Training\/Validation seems pretty reasonable (from my friend, Kirk Borne<\/a>), Random initialization, minibatches of 16 w\/oversampling positive classes, and a progressively decaying validation loss are utilized, all of which are pretty standard.\u00a0 Class activation mappings are used to visualize areas in the image most indicative of the activated class <\/a>(in this case, pneumonia).\u00a0 This is an interesting technique that can be used to provide some human-interpretable insights into the potentially opaque DenseNet.<\/p>\n

The last Fully Connected (FC) layer is replaced by a single output (only one class is being tested for – pneumonia) coupled to a sigmoid function (an activation function – see here<\/a>) to give a probability between 0 and 1.\u00a0\u00a0 Again, pretty standard for a binary classification.\u00a0 The multiclass portion of the study was performed seperately\/later.<\/p>\n

The test portion of the study was 420 Chest X-rays read by four radiologists, one of whom was a thoracic specialist.\u00a0 They could choose between the 14 pathologies in the ChestX-ray14 dataset, read blind without any clinical data.<\/p>\n

So, a ROC curve was created, showing three radiologists similar to each other, and one outlier.The radiologists lie slightly<\/span> under the ROC curve of the CheXNet classifier.\u00a0 But, a miss is as good as a mile, so the claims of at or above radiologist performance are accurate, because math.\u00a0 As Luke Oakden Rayner points out, this would probably not pass statistical muster.<\/p>\n

So that’s the study.\u00a0 Now, I will pick some bones with the study.<\/p>\n

First, only including one thoracic radiologist is relevant, if you are going to make ground truth agreement of 3 out of four radiologists.\u00a0 (Addendum: And, for statistical and methodological reasons discussed online, the 3 out of 4 implementation was initially flawed as scored)<\/em>\u00a0 General radiologists will be less specific than specialist radiologists, and that is one of the reasons why we have moved to specialty-specific reads over the last 20 years.\u00a0 If the three general rads disagreed with the thoracic rad, the thoracic rad’s ground truth would be discarded.\u00a0 Think about this – you would take the word of the generalist over the specialist, despite greater training.\u00a0 (1\/25 Addendum: proven right on this one.\u00a0 The thoracic radiologist is an outlier with a higher F1 score<\/strong>)\u00a0 Even Google didn’t do this in their retinal machine learning paper<\/a>.\u00a0 Instead, Google used their three retinal specialists as ground truth and then looked at how the non-specialty opthalmologists were able to evaluate that data and what it meant to the training dataset.\u00a0 (Thanks, Melody!)\u00a0 Nevertheless, all rads lie reasonably along the same ROC curve, so methodologically it checks out<\/del>\u00a0the radiologists are likely of equal ability but different sensitivities\/specificities<\/em>. <\/del><\/p>\n

Second, the Wang ChestXray14 dataset is a dataset that was data-mined from NIH radiology reports.\u00a0 This means that for the dataset, ground truth was whatever the radiologists said it was.\u00a0 I’m not casting aspersions on the NIH radiologists, as I am sure they are pretty good.\u00a0 I’m simply saying that the dataset’s ground truth is what it says it is, not necessarily what the patient’s clinical condition was.\u00a0 As proof of that, here are a few cells from the findings field on this dataset.<\/p>\n

\"Findings<\/p>\n

In any case, the NIH radiologists more than a few times perhaps couldn’t tell either, or identified one finding as the cause of the other (Infiltrate & Pneumonia mentioned side by side) and at the top you have the three fields “atelectasis” “consolidation” & “Pneumonia” – is this concurrent pneumonia with consolidation with some atelectasis elsewhere, or is it “atelectasis vs consolidation cannot r\/o pneumonia” (as radiologists we say these things). While the text miner purports to use several advanced NLP tools to avoid these kinds of problems, in practice it does not seem to do so. (See addendum below, further addendum, confirmed by Jeremy Howard<\/strong>)\u00a0 Dr. Ng, if you read this, I have the utmost respect for you and your team, and I have learned from you.\u00a0 But I would love to know your rebuttal, and I would urge you to publish those results.\u00a0 Or perhaps someone should do it for reproducibility purposes.<\/p>\n

Finally, I’m bringing up these points not to be a killjoy, but to be balanced.\u00a0 I think it is important to see this and prevent someone from making a really boneheaded decision of firing their radiologists to put in a computer diagnostic system (not in the US, but elsewhere) and realizing it doesn’t work after spending a vast sum of money on it.\u00a0 Startups competing in the field who do not have deep healthcare experience need to be aware of potential pitfalls in their product.\u00a0 I’m saying this because real people could be really hurt and impacted if we don’t manage this transition into AI well.\u00a0 Maybe all parties involved in medical image analysis should join us in taking the Hippocratic Oath, CEO’s and developers included.<\/p>\n

Thanks for reading, and feel free to comment here or on twitter or connect on linkedin to me: @drsxr<\/a><\/p>\n

December Addendum<\/span>: ChestX-ray14 is based on the ChestX-ray8 database which is described in a paper released on ArXiv<\/a> by Xiaosong Wang et al. The text mining is based upon a hand-crafted rule-based parser using weak labeling<\/a> designed to account for “negation & uncertainty”, not merely application of regular expressions. Relationships between multiple labels are expressed, and while labels can stand alone, for the label ‘pneumonia’, the most common associated label is ‘infiltrate’.\u00a0 A graph showing relationships between the different labels in the dataset is here (from Wang Et Al.)<\/p>\n

\"Label<\/p>\n

Pneumonia is purple with 2062 cases, and one can see the largest association is with infiltration, then edema and effusion.\u00a0 A few associations with atelectasis also exist (thinner line).<\/p>\n

The dataset methodology claims to account for these issues at up to 90% precision reported in ChestX-ray8, with similar precision inferred in ChestX-ray14.<\/p>\n

\"No
“No Findings”<\/figcaption><\/figure>\n
\"No
“No Findings”<\/figcaption><\/figure>\n

However, expert review of the dataset (ChestX-ray14) does not support this.\u00a0 In fact, there are significant concerns that the labeling of the dataset is a good deal weaker.\u00a0 I’ll just pick out two examples above that show a patient likely post R lobectomy with attendant findings classified as “No Findings” and the lateral chest X-ray which doesn’t even belong in the study database of all PA and AP films.\u00a0 These sorts of findings aren’t isolated – Dr. Luke Oakden-Rayner addresses this extensively in this post<\/a>, from which his own observations are garnered below:<\/p>\n

\"Sampled
Dr. Luke Oakden Rayner’s own Positive Predictive Value on visual inspection of 130 images vs reported<\/figcaption><\/figure>\n

His final judgment is that the ChestX-ray14 dataset is not fit for training medical AI systems to do diagnostic work.\u00a0 He makes a compelling argument, but I think it is primarily a labelling problem, where the proposed 90% acccuracy on the NLP data mining techniques of Wang et al does not hold up.\u00a0 ChestX-ray14 is a useful dataset for the images alone, but the labels are suspect.\u00a0 I would call upon the NIH group to address this and learn from this experience.\u00a0 In that light, I am surprised that the system did not do a great deal better than the human radiologists involved in Dr. Ng’s group’s study, and I don’t really have a good explanation for it.<\/p>\n

The evaluation of CheXNet by these individuals should be recognized:<\/em><\/strong><\/p>\n

Luke Oakden-Rayner: CheXNet an in-depth review<\/a><\/p>\n

Paras Lakhani : Dear Mythical Editor: Radiologist-level Pneumonia Detection in CheXNet<\/a><\/p>\n

Bailint Botz: A Few thoughts about ChexNet<\/a><\/p>\n

Copyright \u00a9 2017<\/p>\n","protected":false},"excerpt":{"rendered":"

1\/25\/18: NOTE:\u00a0 Since the November release of the CheXNet paper on ArXiV, there has been a healthy and extensive online discussion on twitter, reddit, and online blogs.\u00a0 The Stanford paper has undergone at least two revisions with some substantial modifications, most importantly the replacement of ROC curves with F1 scores and a bootstrap calculation of […]<\/p>\n","protected":false},"author":1,"featured_media":13422,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"My thoughts on @andrewng 's CheXNet. Special N2value.com post. #radiology #deeplearning","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[29,22,4,24],"tags":[20,27,15],"class_list":["post-13421","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-ai","category-computer-vision","category-data-science","category-radiology","tag-antifragile","tag-machine-learning","tag-patient-care"],"jetpack_publicize_connections":[],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/n2value.com\/blog\/wp-content\/uploads\/2017\/11\/dreamcxr.png","jetpack_shortlink":"https:\/\/wp.me\/p4mtfP-3ut","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/13421","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/comments?post=13421"}],"version-history":[{"count":81,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/13421\/revisions"}],"predecessor-version":[{"id":13630,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/13421\/revisions\/13630"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/media\/13422"}],"wp:attachment":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/media?parent=13421"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/categories?post=13421"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/tags?post=13421"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}