The Alliance held a meeting on November 4th in Arlington, VA
Hosted by the MDIC at the Key Bridge Marriott
“But the main lesson to draw from the birth of computer is that innovation is usually a group effort, involving collaboration between visionaries and engineers, and that creativity comes from drawing on many sources. Only in storybooks do inventions come like a thunderbolt, or a light bulb popping out of the head of a lone individual in a basement or garret or garage.”
Walter Isaacson, American historian.
The Alliance for Digital Pathology Meeting, hosted by the MDIC at the Key Bridge Marriott, in Arlington, VA, on November 4, 2019
The aim of the meeting was to:
Share and discuss progress in the Alliance
Enable networking among the members/participants
Have a full day to share and work on deliverables
Take the content of the prior breakout sessions, continue development and derive practically relevant deliverables
Share the content in the public domain
Please find the presentations and the results of the breakout sessions below. The project overviews can be found in the full program.
Presentations
MDIC Introduction and Welcome
Pamela Goldberg
President & CEO, MDIC
With a brief overview of the scope, aims, and projects of MDIC, Pamela welcomed the Alliance. Pamela outlined the roles and functions including an introduction to the NESTcc ecosystem (https://nestcc.org/) and trial involvement of MDIC.
Alliance Progress Update
Joe Lennerz, MD PhD
On behalf of the Alliance Steering Committee
Joe outlined the overall scope of the Alliance, some of the current accomplishments as well as challenges. He emphasized that the key aim is to select and then prioritize concrete deliverables/workstreams.
Digital Pathology Association
Esther Abels, MS
Digital Pathology Association & PathAI
Esther provided a great pictorial example of an AI/ML problem in pixel-based data recognition and used this example as a starting point to outline the importance of regulatory task forces, the creation of guidelines, and the central role that DPA plays in accomplishing these goals.
Food and Drug Administration
Sara Brenner, MD, MPH
Associate Director for Medical Affairs & Chief Medical Officer, In Vitro Diagnostics, CDRH, FDA
Sarah outlined 4 key elements related to digital pathology and artificial intelligence: 1) the complexity of the regulatory landscape, 2) keeping pace with the technological advances, 3) the issue of data handling, and 4) the complex ecosystem of players.
Patient Centered Outcomes Research Institute (PCORI)
Bill Lawrence, MD, MS
Senior Clinical Advisor, Office of the Chief Engagement and Dissemination Office, PCORI
Bill outlined the mission and strategic goals of PCORI and emphasized the importance of comparative outcomes research and seeking answers to real world questions is key. He also introduced PCORI’s engagement rubric.
Friends of Cancer Research (FOCR)
Laura Lasiter, PhD
Science Policy Analyst, FOCR
Laura outlined the role and functions of FOCR – and in particular their science policy, advocacy and collaborative role across healthcare. She outlined how Friends is helping to develop innovative approaches to speed up patient access to new technologies and improve patient care.
American College of Radiology (ACR)
Bibb Allen, MD, FACR
Chief Medical Officer, ACR Data Science Institute
Bibb outlined how AI has and will be a transformational technology. He emphasized the importance of medical societies as adoption facilitators. Several slides clearly outline how radiology has already adopted their ecosystem for AI/ML. Bibb also described the recently started ACR Institute for Clinical Sciences and emphasized the importance of interoperability of systems in clinical practice.
Healthcare Infrastructure
Joe Lennerz, MD, PhD
Medical Director, Center for Integrated Diagnostics; Associate Chief, Department of Pathology, Massachusetts General Hospital; Associate Professor, Harvard Medical School
Joe outlined that new technologies live in a larger framework that entails the concrete patient care, an IT-layer, as well as revenue cycle management. Introducing new technologies into clinical practice require numerous non-technical components to work in unison. A bottom-up and top-down strategy for coverage was outlined.
Breakout Sessions
Breakout Session 1: Pre-Analytics – Amanda Lowe
Pre-analytical components have widespread impact and require a lot of education. Need to understand the challenges. Pre-scan variability, tissue artifacts at the scanning level, need for criteria for ground truthing. Deliverables are surveys among 1) laboratory personnel about pre-analytical processing, and 2) pharmaceutical companies to understand their challenges.
Breakout Session 2: Pre-Analytics – Amanda Lowe
Continued working on the concrete outline of the survey. The group emphasized the importance of developing a comprehensive questionnaire that captures the various domains of pre-analytical variability.
Breakout Session 1: Slide Scanning – Scott Blakely
Standardization of characterization of digital slide scanning including all stakeholders. The group discussed a set of common definitions that also tackle tissue-independent scanning characteristics. Regulatory implications are decreased time to review and common set of standards/definitions. Standardization may stifle innovation if characterization scheme is made mandatory.
Breakout Session 2: Slide Scanning – Scott Blakely
The group explored the idea of eliminating the need for clinical evaluation of slide scanning. By developing rigorous quality standards and the use of phantoms, possibilities exist where pathologists would no longer be necessary in the validation and regulatory approval of slide scanners. There is precedence for this in the radiology space.
Breakout Session 1: Truthing – Sarah Dudgeon, MPH & Brandon Gallas, PhD
The group emphasized the importance of data sharing and proposed a standard research collaboration agreement template for the Alliance. The group outlined a three-tiered approach (public domain, partially restricted for commercial endeavors, locked-door/completely restricted). Managing the need to use data vs. the ability to share data; this may include glass slides (e.g. slides from proficiency testing; slides from clinical trials). The group also discussed truth sources (e.g. pathologist, outcome, other tissue-derived features).
Breakout Session 2: Truthing – Sarah Dudgeon, MPH & Brandon Gallas, PhD
The group discussed the importance of the generalizability of algorithms developed on datasets. They emphasized using datasets from multiple sites, including expanding the dataset as data source and algorithm needs grow. The group agreed that a FDA qualified MDDT validation dataset would be a good source of ground truth for algorithm evaluation.
Breakout Session 1: ML Models/Use Cases – Matthew Hanna, MD
The group explored a collaboration between different reviewed proposals. In particular, the group proposed an end-to-end solution for data hosting and sharing for a digital pathology challenge. In addition, the groups proposed the use of real-world images and integrated databases (including outcomes and clinical data). Regarding the proposed fellowship program, the group felt it would benefit multiple entities by additional available resources for review and creating high-quality resources for planned (future) submissions. The group suggested a mock submission for locked-down vs. continuous learning model to outline the study requirements for such a study.
Breakout Session 2: ML Models/Use Cases – Matthew Hanna, MD
The group discussed the importance of having a standard annotation format in order to have consistency in interoperability of model testing and performance evaluation. The group also explored conducting human machine interaction studies to see how AI/ML algorithm generated suggestions would affect clinician decision making, which would correlate with the amount of risk mitigation that the algorithm would need to have. The group also touched on the idea of having positive and negative control slides for machine learning algorithms.
Breakout Session 1: Standards – Markus Herrmann, MD, PhD & Mike Isaacs
B
ased on the three existing proposals (remote sign-out, de-identify of WSI, and naming conventions including standard dictionaries), the group proposed a standard dataset that contains pixel, metadata, and annotations. Semantic interoperability (e.g. synoptic), privacy concerns, establish set of requirements, was emphasized as critical quality components of the proposed dataset. The group discussed several open questions regarding establishing, consenting, and hosting such a dataset.
Breakout Session 2: Standards – Markus Herrmann, MD, PhD & Mike Isaacs
The group discussed the importance of having standardized metadata in whole slide images, since the metadata often provide the clinical relevance and are significant to the generalizability of developed algorithms. The group also recommended that taking an agile approach to producing concrete test data, and creating a workspace where developers can investigate what elements would or would not work through product iteration.
Breakout Session 1: Payor Strategy – Joe Lennerz, MD, PhD & Esther Abels, MS
The group discussed the lack of clear return on investment and the urgent need to identify a clinical-value added approach. The group discussed a mock breakthrough device submission aiming to demonstrate equivalence – but with the promise to deliver/add additional value once fully integrated. The group agreed that a mixed cost-model with initial internal funding requirements as opposed to external cost-containment.
Breakout Session 2: Payor Strategy – Joe Lennerz, MD, PhD & Esther Abels, MS
The group had a very productive and informative session that generated two clear deliverable ideas. Firstly, given FDA’s specific priority of using Patient Preference Information (PPI) in medical device evaluation, such as the design and conduct of premarket clinical studies, benefit-risk assessments, and post-market evaluation, the group agreed that a PPI study for digital pathology should be conducted in order to bring the patient perspective into the design and evaluation of digital pathology devices. Secondly, the idea of a universal biomaterials release form was proposed. The form would streamline and generalize the use of patient tissue for clinical practice (such as second opinion consults), and research and development across institutions, while maintaining full patient consent in the process, thereby removing access barriers to important sources of clinically relevant data.
Overall Interests in Individual Projects
The individual projects below were collected from the blueprints submitted via the Projects page, as well as those solicited during the Breakout Sessions.
