{"id":13649,"date":"2018-02-15T15:51:28","date_gmt":"2018-02-15T20:51:28","guid":{"rendered":"http:\/\/n2value.com\/blog\/?p=13649"},"modified":"2018-02-15T15:58:39","modified_gmt":"2018-02-15T20:58:39","slug":"are-computers-better-than-doctors-will-the-computer-see-you-now-what-we-learnt-from-the-chexnet-paper-for-pneumonia-diagnosis","status":"publish","type":"post","link":"https:\/\/n2value.com\/blog\/are-computers-better-than-doctors-will-the-computer-see-you-now-what-we-learnt-from-the-chexnet-paper-for-pneumonia-diagnosis\/","title":{"rendered":"Are computers better than doctors ? Will the computer see you now ? What we learnt from the ChexNet paper for pneumonia diagnosis \u2026"},"content":{"rendered":"
Author’s Note: This was a fun side-project for the American College of Radiology’s Residents and Fellows Section.\u00a0 Judy Gichoya and I co-wrote the article. \u00a0 The original article was posted by Judy to Medium and appeared on HackerNoon<\/a>.\u00a0 It was really an enlightening gathering of experts in the field.\u00a0 There is a small, but hopefully growing number of radiologists who are also deep learning practitioners.<\/h5>\n

 <\/p>\n

Written by <\/em>Judy Gichoya<\/em><\/a> & <\/em>Stephen Borstelmann MD<\/em><\/a><\/p>\n

 <\/p>\n

In December 2017\u00a0, we (radiologists both in training, staff radiologists and AI practitioners) discussed our role as knowledge experts in world of AI, summarized here https:\/\/becominghuman.ai\/radiologists-as-knowledge-experts-in-a-world-of-artificial-intelligence-summary-of-radiology-ec63a7002329<\/a>. For the month of January, we addressed the performance of deep learning algorithms for disease diagnosis\u00a0, specifically focusing on the paper by the stanford group\u200a\u2014\u200aCheXNet: Radiologist-Level Pneumonia Detection on Chest X-Rays with Deep Learning. <\/strong>We continue to generate a large interest in the journal club\u00a0, with 347 people registered\u00a0, 150 of whom signed on January 24th 2018 to participate in the discussion.<\/p>\n

The paper has had 3 revisions and is available here https:\/\/arxiv.org\/abs\/1711.05225<\/a>\u00a0. Like many deep learning papers that claim super human performance\u00a0, the paper was widely circulated in the news media, several blog posts\u00a0, on reddit<\/a> and twitter.<\/p>\n

\"ngtwitter\"<\/p>\n

Please note that the findings of superhuman performance are increasingly being reported in medical AI papers. For example, this article<\/a> denotes that \u201cMedical AI May Be Better at Spotting Eye Disease Than Real Doctors\u201d<\/p>\n

\"CVDretina\"<\/p>\n

To help critique the ChexNet paper\u00a0, we constituted a panel composed of the author<\/strong> team (most of the authors listed on the paper were kind enough to be in attendance\u200a\u2014\u200athank you!), Dr. Luke(blog<\/a>) and Dr. Paras (blog<\/a>) who had critiqued the data<\/strong> used and Jeremy Howard (past president and chief scientist of Kaggle<\/a>, a data analytics competition site, Ex-CEO of Enlitic,<\/a> a healthcare imaging company, and the Current CEO of Fast.ai<\/a>, a deep learning educational site) to provide insight to deep learning methodology<\/strong>.<\/p>\n

\"chexnet\"<\/p>\n

In this blog we summarise the methodology of reviewing medical AI papers.<\/p>\n

Radiology 101<\/h4>\n

The ChexNet paper reviews performance of AI versus 4 trained radiologists in diagnosing pneumonia. Pneumonia is a clinical diagnosis\u200a\u2014\u200aa patient will present with fever and cough\u00a0, and can get a chest Xray(CXR) to identify complications of pneumonia. Patients will usually get blood cultures to supplement diagnosis. Pneumonia on a CXR is not easily distinguishable from other findings that fill the alevolar spaces\u200a\u2014\u200aspecifically pus\u00a0, blood\u00a0, fluid or collapsed lung called atelectasis. The radiologists interpreting these studies can therefore use terms like infiltrates\u00a0, consolidation and atelectasis interchangeably.<\/p>\n

Show me the\u00a0data<\/h4>\n

The data used for this study is the ChestX-ray14 dataset which is the largest publicly available imaging data set that consists of 112,120 frontal chext xray radiographs of 30,805 unique patients and expands the ChestX-Ray 8, described by Wang, et. al<\/a>. Each radiograph is labeled with one or more of 14 different pathology labels, or a \u2018no finding\u2019 label.<\/p>\n

Labeling of the radiographs was performed using Natural Language Processing (NLP) by mining the text in the radiology reports. Individual case labels were not assigned by humans.<\/p>\n

Critique<\/strong>: Labeling medical data remains a big challenge especially because the radiology report is a tool for communicating to ordering doctors and not a description of the images. For example\u00a0, in an ICU film with a central line, tracheostomy tube and chest tube may be reported as \u201cstable lines and tubes\u201d without detailed description of the every individual finding on the CXR. This can be missclassified by NLP as a study without findings. This image-report disconcordance occurs at a high rate on this dataset.<\/p><\/blockquote>\n

Moreover reportable findings could be ignored by the NLP technique and\/or labeling schema, either through error or pathology outside of one of the 14 labels. The paper\u2019s claims of 90%+ NLP mining accuracy do not appear to be accurate. (SMB,LOR,JH). One of the panelists\u200a\u2014\u200aLuke reviewed several hundred examples and found the NLP labeling about 50% accurate overall compared to the image, with the pneumonia labeling worse\u200a\u2014\u200a30\u201340%.<\/p><\/blockquote>\n

Jeremy Howard notes that the use of an old NLP tool contributes to the inaccuracy due to the preponderance of \u2018No Findings\u2019 cases in the dataset skewing the data\u200a\u2014\u200ahe doesn\u2019t think that the precision of normal findings in this dataset is likely improved over random. Looking at the pneumonia label, it is only 60% accurate. A lot of the discrepancy can be drawn back to the core NLP method, which he characterized as \u201cmassively out of date and known to be inaccurate\u201d. He feels a re-characterization of the labels with a more up-to-date NLP system is appropriate.<\/p><\/blockquote>\n

\"Chest
Chest Xray showing a tracheostomy tube , right internal jugular dialysis line and diffuse infiltrates likely pulmonary edema. The lines and tubes for an ICU patient are easily reported as \u201cStable\u201d<\/figcaption><\/figure>\n

The stanford group tackled the labeling challenge by having 4 radiologists (one specializing in thoracic imaging and 3 non thoracic radiologists) assign labels to a subset of the data for training created through a stratified random sampling, for a minimum of 50 positive cases of each label, with a final N=420.<\/p>\n

Critique<\/strong>: The ChestXRay14 contains many patients with only one radiograph but those who had multiple studies tended to have many. While the text-mined reports may match clinical information, any mismatch between the assigned label and radiographic appearance hurts the predictive power of the dataset.<\/p><\/blockquote>\n

Moreover\u00a0, what do the labels actually mean? Dr. Oakden-Rayner questions what the labels mean\u200a\u2014\u200ado they mean a radiologic pneumonia or a clinical pneumonia? In an immunocompromised patient, radiography of a pneumonia might be negative, largely because the patient cannot mount an immune response to the pathogen. This does not mean that the clinical diagnosis of pneumonia is inaccurate. The imaging appearance and clinical appearance\/diagnosis therefore would not match.<\/p><\/blockquote>\n

The closeness of four of the labels: Pneumonia, Consolidation, Infiltration, and Atelectasis introduces a new level of complexity. Pneumonia is a subset of consolidation and infiltration is a superset of consolidation. While the dataset labels these as 4 separate entities, to the radiologic practitioner they may not be separate at all. It is important to have experts look at images when doing an image classification task.<\/p><\/blockquote>\n

See a great summary of the data problems on this blog<\/a> posting from Luke who was one of the panelists here<\/a>.<\/p>\n

Model<\/h4>\n

The CheXNet algorithm is a 121-layer deep 2D Convolutional Neural Network; a Densenet<\/strong> after Huang & Liu<\/a>. The Densenet\u2019s multiple residual connections reduce parameters and training time, allowing a deeper, more powerful model. The model accepts a vectorized two-dimensional image of size 224 pixels by 224 pixels.<\/p>\n

\"DenseNet,
Densenet<\/figcaption><\/figure>\n

To improve trust in CheXNet\u2019s output, a Class Activation Mapping (GRAD-CAM) heatmap was utilized after Zhou et al<\/a>. This allows the human user to \u201csee\u201d what areas of the radiograph provide the strongest activation of the Densenet for the highest probability label.<\/p>\n

Critique<\/strong>: Jeremy notes that image preprocessing of resizing to 224×224 pixel size images and adding random horizontal flips is fairly standard, but leaves room for potential improvement, as effective data augmentation is one of the best ways to improve a model. Image downsizing to 224×224 is a known issue\u200a\u2014\u200aboth from research and practical experience at Enlitic, larger images perform better in medical imaging (SMB: Multiple top 5 winners of the 2017 RSNA Bone age challenge had image sizes near 512×512). Mr. Howard feels there is no reason to leave Imagenet<\/a> trained models this size any longer. Regarding the model choice, the Densenet model is adequate, but NasNets<\/a> in the last 12 months have shown significant improvement (50%) over older models.<\/p><\/blockquote>\n

Pre-trained Imagenet weights were used, which is fine & a standard approach; but Jeremy felt it would be nice if we had a medical imagenet for some semi-supervised training of an AutoML <\/a>encoder or a siamese network<\/a> to cross validate<\/a> patients\u200a\u2014\u200aleaving room for improvement. Consider that Imagenet consists of color images of dogs, cats, planes and trains\u200a\u2014\u200aand we are getting great results on X-rays? While better than nothing, ANY<\/strong> pretrained network trained on medical images in any modality would probably perform superiorly.<\/p><\/blockquote>\n

The Stanford team\u2019s best idea<\/strong> was to train on multiple labels at the same time\u200a\u2014\u200ait is best to build a single model that predicts multiple classes\u200a\u2014\u200acounterintuitive, but bears out in deep learning models, and likely responsible for their model yielding better results than prior studies. The more classes you train the model on properly, the better results you can expect<\/strong>.<\/p>\n

Results<\/h4>\n

F1 scores<\/a> were used to evaluate both CheXNet model and the Stanford Radiologists.<\/p>\n

\"Precision,
Calculating F1 score<\/figcaption><\/figure>\n

Each Radiologists\u2019 F1 score was calculated by considering the other three radiologists as \u201cground truth.\u201d ChexNet\u2019s F1 score, was calculated vs. all 4 radiologists. A bootstrap calculation<\/a> was added to yield 95% confidence intervals.<\/p>\n

CheXnet\u2019s results are as follows:\"Evaluation-results\"<\/p>\n

From the results, ChexNet outperforms human radiologists. The varying F1 scores can be interpreted to imply that for each study\u00a0, 4 radiologists do not seem to agree with each other on findings. However there is an outlier (rad 4\u200a\u2014\u200awith an F score of 0.442) who is the thoracic trained radiologists who performs better than the ChexNet.<\/p>\n

Moreover CheXNet has State of the Art (SOTA) performance on all 14 pathologies compared to prior publications.\"eval<\/p>\n

In my (JG) search\u00a0, the Machine Intelligence Lab, Institute of Computer Science & Technology, Peking University, directed by Prof. Yadong Mu reports superior performance than the Stanford group. The code is open source and available here\u200a\u2014\u200ahttps:\/\/github.com\/arnoweng\/CheXNet\u00a0<\/a><\/p>\n

\"CheXNet,
Results from various implementations of ChexNet<\/figcaption><\/figure>\n
Results from various implementations of\u00a0ChexNet<\/figcaption><\/figure>\n

Critique<\/strong>\u200a\u2014\u200aVarious studies that assess cognitive fit show that human performance can be affected by lack of clinical information or prior comparisons that may affect their performance. Moreover, before the most recent version of the paper, human performance was unfairly scored against the machine.<\/p><\/blockquote>\n

Clinical significance<\/h4>\n

With the majority of labelled CXRs with pneumothorax having chest tubes present, the question must be raised: \u201care we training the Densenet to recognize pneumothoraces or chest tubes?\u201d<\/p>\n

Peer review<\/strong><\/h4>\n

Luke Oakden-Rayner MD, a radiologist in Australia with expertise in AI & deep learning who was on our panel independently evaluated the ChestXRay-14 dataset<\/a>, and CheXNet<\/a>. He praises the Stanford team for their openness and patience in discussing the paper\u2019s methodology<\/strong>, and their willingness to modify the paper to correct a methodologic flaw which biased against evaluating radiologists.<\/p>\n

Summary<\/h4>\n

For the second AI journal club we analysed the pipeline of AI papers in medicine. You must make sure you are asking the right clinical question to be answered and not doing algorithms for the sake of doing something. Thereafter understand whether your data will help you answer the question you have, looking into details on how the data was collected and labeled.<\/p>\n

To determine human level or super human performance, ensure the baseline metrics are adequate and not biased against one group.<\/p>\n

\"Flowchart,
Pipeline for AI in medicine<\/figcaption><\/figure>\n

The model appears to give at-human performance for experts, or better than human performance for less-trained practitioners. This is in line with research findings and Enlitic\u2019s experience. We should not be surprised by that; the research in Convolutional Neural Networks has consistently reported near-human or super-human performance consistently.<\/p>\n

Take Aways<\/h4>\n
    \n
  1. There is exists a critical gap in the labeling of medical data.<\/li>\n
  2. Do not forget the clinical significance of your results.<\/li>\n
  3. Embrace peer review especially in medicine and AI<\/li>\n<\/ol>\n

    These were the best tweets regarding the problem of labeling medical data\u200a\u2014\u200aaka do not get discouraged to attempt deep learning for medicine.<\/p>\n

    \"twitterJHKL\"<\/p>\n

    The journal club was a success, so if you are a doctor or an AI scientist\u00a0, join us at https:\/\/tribe.radai.club<\/a> to continue with the conversations on AI and medicine. You can listen to the recording of this journal club here\u00a0: https:\/\/youtu.be\/xoUpKjxbeC0<\/a>\u00a0. Our next guest is Timnit Gebru<\/a> who worked on US demographic household prediction using Google Street view images on 22nd February 2018. She will be talking on Using deep learning and Google Street View to estimate the demographic makeup of neighborhoods across the United States (<\/strong>http:\/\/www.pnas.org\/content\/114\/50\/13108<\/a>).<\/p>\n

    Coming soon<\/h4>\n

    For the journal club we developed a human versus AI competition for interepreting the CXRs in the dataset hosted at https:\/\/radai.club<\/a>. We will be publishing the outcome of our crowdsourced labels soon, with a detailed analysis to check whether the model performance improves.<\/p>\n

    Say thanks<\/strong><\/h4>\n

    This I would like to thank the panelists including Jeremy Howard<\/a>, Paras Lakhani<\/a>, Luke Oakden-Rayner<\/a>\u00a0, and the Stanford ML team<\/a>. Thanks to the ACR RFS AI advisory council members including Kevin Seals<\/a>.<\/p>\n

    Article corrections made<\/h4>\n
      \n
    1. This article referred to Jeremy Howard (Ex-CEO of Kaggle<\/a>)\u200a\u2014\u200aupdated to \u201cpresident and chief scientist of Kaggle\u201d<\/li>\n
    2. Article stated NLP performance on that dataset is not likely improved over random.<\/em>Jeremy clarified that the precision of the normal finding<\/em> was what was not likely improved over random<\/li>\n<\/ol>\n
      <\/div>\n

       <\/p>\n

       <\/p>\n

       <\/p>\n","protected":false},"excerpt":{"rendered":"

      Author’s Note: This was a fun side-project for the American College of Radiology’s Residents and Fellows Section.\u00a0 Judy Gichoya and I co-wrote the article. \u00a0 The original article was posted by Judy to Medium and appeared on HackerNoon.\u00a0 It was really an enlightening gathering of experts in the field.\u00a0 There is a small, but hopefully […]<\/p>\n","protected":false},"author":1,"featured_media":13659,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_monsterinsights_skip_tracking":false,"_monsterinsights_sitenote_active":false,"_monsterinsights_sitenote_note":"","_monsterinsights_sitenote_category":0,"jetpack_post_was_ever_published":false,"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_publicize_message":"Very pleased to have been able to co-author with @judywawira the blogpost for the #radaijc with panelists @pranavrajpurkar @DrLukeOR @jeremyphoward @ParasLakhaniMD & others. Was just a fantastic discussion","jetpack_publicize_feature_enabled":true,"jetpack_social_post_already_shared":true,"jetpack_social_options":{"image_generator_settings":{"template":"highway","enabled":false},"version":2}},"categories":[22,24],"tags":[28],"class_list":["post-13649","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-computer-vision","category-radiology","tag-ai"],"jetpack_publicize_connections":[],"aioseo_notices":[],"jetpack_featured_media_url":"https:\/\/n2value.com\/blog\/wp-content\/uploads\/2018\/02\/flowchart.png","jetpack_shortlink":"https:\/\/wp.me\/p4mtfP-3y9","jetpack_sharing_enabled":true,"jetpack_likes_enabled":true,"_links":{"self":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/13649","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/comments?post=13649"}],"version-history":[{"count":3,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/13649\/revisions"}],"predecessor-version":[{"id":13663,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/posts\/13649\/revisions\/13663"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/media\/13659"}],"wp:attachment":[{"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/media?parent=13649"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/categories?post=13649"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/n2value.com\/blog\/wp-json\/wp\/v2\/tags?post=13649"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}