QI-Bench Specify ASD Rev 0.3 QI-Bench Specify Scope Description December 2011 Rev 0.3 Required Approvals: Author of this Revision: Andrew J. Buckler Project Manager: Andrew J. Buckler Print Name Signature Date Document Revisions: Revision Revised By Reason for Update Date 0.1 AJ Buckler Initial version June 2011 0.2 AJ Buckler Expanded intent and purpose November 2011 0.3 AJ Buckler Updated purpose and scope December 2011 BBMSC 1 of 14 QI-Bench Specify ASD Rev 0.3 Table of Contents 1. EXECUTIVE SUMMARY ................................................................................................ 3 1.1. APPLICATION PURPOSE .......................................................................................................... 4 1.2. APPLICATION SCOPE .............................................................................................................. 6 1.3. THE REASON WHY THE APPLICATION IS NECESSARY............................................................... 6 1.4. TERMS USED IN THIS DOCUMENT .......................................................................................... 7 2. PROFILES........................................................................................................................... 9 2.1. INFORMATION PROFILES ...................................................................................................... 10 2.2. FUNCTIONAL PROFILES ........................................................................................................ 11 2.3. BEHAVIORAL PROFILES ....................................................................................................... 11 3. CONFORMANCE ASSERTIONS .................................................................................. 11 4. REFERENCES .................................................................................................................. 12 BBMSC 2 of 14 QI-Bench Specify ASD Rev 0.3 1. Executive Summary There is a large and growing body of knowledge at the cellular level increasing utilization of computational methods in drug discovery and development. Likewise, there is a large and growing body of knowledge at the organism level enabling applications as computer aided detection, diagnosis, and targeted therapies [1-5]. However, there is comparatively little available technology to bridge these two bodies of knowledge, compromising the effectiveness of both. Technology linking these scales through quantitative analytic processing of acquired imaging and non-imaging data in translational research [6], coupled with the interpretation of the data through multi-scale models offers means to comprehend disease processes pre-symptomatically and after clinical manifestations at multiple levels of abstraction [7-10]. For example, changes in activities of the receptor tyrosine kinase EGFR can be visualized and quantified through optical imaging of reconstituted luciferase [8]. Positron Emission Tomography enables detection and quantification of molecular processes such as glucose metabolism, angiogenesis, apoptosis and necrosis. Radiolabelled Annexin V molecule uptake by apoptotic and necrotic cells is developed to measure apoptosis, necrosis and other disease processes using PET [11, 12]. Chelated gadolinium attached to small peptides recognizes cell receptors and quantify receptor activities using magnetic imaging techniques. Similarly, microbubbles and nanobubbbles attached to antibodies such as anti-P-selectin may be used to image targeted molecules associated with inflammation, angiogenesis, intravascular thrombus, and tumors [5]. Currently the application of quantitative imaging techniques suffers from the lack of a standardized representation of image features and content [13-16]. The concept of “image biobanking” as an analogue to tissue biobanking has great promise [17, 18]. Tools have begun to be available for handling the complexity of genotype [19-23], and similar advancements are needed in appreciate phenotype, especially as derived from imaging [15, 24-31]. Publicly accessible resources that support large image archives provide little more than file sharing and have so far not yet merged into a framework that supports the collaborative work needed to meet the potential of quantitative imaging analysis. With the availability of tools for automatic ontology-based annotation of datasets with terms from biomedical ontologies, coupled with image archives and means for batch selection and processing of image and clinical data, we believe that imaging will go through a similar increase in capable analogous to what advanced sequencing techniques have brought to molecular biology. Imaging biomarkers are developed for use in the clinical care of patients and in the conduct of clinical trials of therapy. In clinical practice, imaging biomarkers are intended to (a) detect and characterize disease, before, during or after a course of therapy, and (b) predict the course of disease, with or without therapy. In clinical research, imaging biomarkers are intended to be used in defining endpoints of clinical trials. A precondition for the adoption of the biomarker for use in either setting is the demonstration of the ability to standardize the biomarker across imaging devices and clinical centers and the assessment of the biomarker’s safety and efficacy. Currently qualitative imaging biomarkers are extensively used by the medical community. Enabled by the major improvements in clinical imaging, the possibility of developing quantitative biomarkers is BBMSC 3 of 14 QI-Bench Specify ASD Rev 0.3 emerging. For this document “Biomarker” will be used to refer to the measurement derived from an imaging method, and “device” or “test” refers to the hardware/software used to generate the image and extract the measurement. Regulatory approval for clinical use1 and regulatory qualification for research use depend on demonstrating proof of performance relative to the intended application of the biomarker: In a defined patient population, For a specific biological phenomenon associated with a known disease state, With evidence in large patient populations, and Externally validated. The use of imaging biomarkers occurs at a time of great pressure on the cost of medical services. To allow for maximum speed and economy for the validation process, this strategy is proposed as a methodological framework by which stakeholders may work together. 1.1. Application Purpose Driving biological questions addressed by this proposed work include questions associated with formal imaging biomarker qualification such as “is CT volumetry better than unidimensional measures for RECIST assessment of cancer response?”, or “is FDG-PET a surrogate marker for survival?” Biological research questions include “can we find a method to compare preclinical and clinical imaging?” “Can we find quantitative links between imaging mouse and man?” “Can we optimize the frequency of serial imaging?” “What is the influence of location on tumor development?” “Can we build a preclinical in vivo imaging database with translational value, such as supporting the Mouse Models of Human Cancer Center (MMHCC)?” “How well can we compare across imaging modalities?” Each of these problem areas ultimately depends on precise identification of the concepts in each question and an ability to bring data resources to bear that can help answer the question. QI-Bench is created to aggregate large-scale datasets of evidence relevant to characterizing and optimizing imaging biomarkers. We are developing resources that enable many parties to better utilize available data and a neutral broker resource that will provide developers and regulators with unbiased and objective data demonstrating imaging biomarker efficacy. QI-Bench is composed of five linked applications, (1) Specify (detailed description of the context for use and assay specifics), (2) Formulate (gather relevant datasets), (3) Execute (perform batch image analyses), (4) Analyze (statistical analysis of image analyses), and (5) Package (compile evidence for regulatory filing) downstream (Fig. 1). The vision is to deploy the software both as webaccessible resources for collaborative community effort or as instances that may be used within individual organizations for their own purposes. BBMSC 4 of 14 QI-Bench Specify ASD Rev 0.3 Figure 1. Left: Access to QI-Bench, which is composed of five linked applications, is at www.qi-bench.org. Downstream applications include Execute, Analyze, and Package. Right: The current prototype of Specify includes a question-answer paradigm driven capability (right hand side of the screen) using BioPortal to create a triple store (showing in left window on screen). The prototype also incorporates the AIM Template Builder. Specify creates RDF triples (subject-predicate-object units of knowledge) stored in a database based on using the QIBO to guide a question-answer paradigm for naturally interacting with domain experts to create an early version of what this proposal would turn into W3C-compliant SPARQL endpoints (query interface) [32]. Specifically, Specify is developed to allow users to: Specify context for use and assay methods. Use consensus terms in doing so. There are four innovative aspects of the proposed work: 1. We make precise semantic specification uncomplicated for diverse groups of experts that are not skilled proficient in knowledge engineering tools. The key is to bring a level of rigor to the problem space in such a way as to facilitate cross-disciplinary teams to function without requiring individuals to be experts in the representation of knowledge, inferencing mechanisms, or computer engineering associated with grid computing or database query design. 2. We map both medical as well as technical domain expertise into representations well suited to emerging capabilities of the semantic web. The proposed project captures functionality improved from the currently available infrastructures such as caGrid and data integration approaches supported by these infrastructures such as caB2B/caIntegrator of caBIG. It explores how a linked data interface can be created from an object-oriented data interface based on a UML model, annotated with CDEs according to the ISO-11179 metadata registry metamodel standard. The experience could illuminate best practices for combining a semantic web approach on the data interface layer with a modeldriven approach for software development, especially since Common Data Elements are widely used to annotate Case Report Form (CRF) templates for clinical research. BBMSC 5 of 14 QI-Bench Specify ASD Rev 0.3 3. We address the problem of efficient use of resources to assess limits of generalizability. Determining the biological relevance of a quantitative imaging read-out is a difficult problem. For example, having direct tumor volumetry data in the lung and the pancreas, do these results extend to the liver? First, it is important to establish to what extent an intermediate marker is in the causal pathway to a true clinical endpoint. Second, given the combinatorial complexity that arises with multiple contexts for use, multiple imaging protocols, etc., a logical and mathematical framework is needed to establish how extant study data may be used to establish performance in clinical contexts that were not explicitly part of the original studies. Existing tools rarely if ever relate the logical world of ontology with the biostatistical analyses that characterize diagnostic or prognostic performance. Existing tools do not permit the extrapolation of statistical validation results to semantically related situations. Despite decades of using statistical validation approaches, there is no methodology to formally represent the generalizability of a validation study. 4. We provide this capability in a manner that is accessible to varying levels of collaborative models, from individual companies or institutions to larger consortia or public-private partnerships to fully open public access. 1.2. Application Scope Specify refers to the part of the project most closely associated with Protégé, BioPortal, and the application code that builds the triple store to specify quantitative imaging biomarkers and assay methods. This representation is used downstream. Specify is packaged in two forms: 1) as a web-service linking to the databases on the project server dev.bbmsc.com; and 2) as a local installation/instance of the functionality for more sophisticated users. 1.3. The reason why the application is necessary Application of imaging biomarkers often suffers from the lack of a standardized interpretation. This is exacerbated by large measurement variability for different contexts of use. Quantitative imaging capability should allow precise quantification of clinically relevant features when applied in a precise and optimized fashion with a defined standard for interpretation of results. Aggregating results from clinical trials or systematic clinical experience may allow for faster refinement of processes that minimize measurement variance. This is a complex task since there are many distinct imaging techniques, platforms, and interpretation schemes. Broad collaboration across the many relevant stakeholder communities is essential to advance the field, supported by resources for batch analysis over large image archives. In 2009, the Toward Quantitative Imaging (TQI) task force of the Radiological Society of North America (RSNA) developed a working definition of quantitative imaging:2 “Quantitative imaging is the extraction of quantifiable features from medical images for the assessment of normal or the severity, degree of change, or status BBMSC 6 of 14 QI-Bench Specify ASD Rev 0.3 of a disease, injury, or chronic condition relative to normal. Quantitative imaging includes the development, standardization, and optimization of anatomical, functional, and molecular imaging acquisition protocols, data analyses, display methods, and reporting structures. These features permit the validation of accurately and precisely obtained image-derived metrics with anatomically and physiologically relevant parameters, including treatment response and outcome, and the use of such metrics in research and patient care.” Such image-derived metrics may involve the extraction of lesions from normal anatomical background and the subsequent analysis of this extracted region over time, in order to yield a quantitative measure of some anatomic, physiologic or pharmacokinetic characteristic. Computational methods that inform these analyses are being developed by researchers in the field of quantitative imaging, computer-aided detection (CADe) and computer-aided diagnosis (CADx).3,4 They may also be obtained using quantitative outputs, such as those derived from molecular imaging. QIBA and others have already mapped certain sources of imaging variance and have proposed that standardizing image acquisition and review quality process could greatly reduce certain types of variance. If this process of standardization can be implemented throughout the clinical community for the range of imaging techniques, the prospects for robust imaging biomarkers would be significantly improved. A range of strategies will be required to address the various sources of uncertainty but the commitment is to approach this challenge in a systematic fashion in an effort to bring the rigor of measurement science to imaging biomarker application. However, these efforts generally do not relate the logical world of ontology development with the biostatistical analyses that characterize performance. Moreover, existing tools do not permit the extrapolation of statistical validation results along arbitrary ontology hierarchies. Despite decades of using statistical validation approaches, there is no methodology to formally represent the generalizability of a validation activity. Building upon existing tools and content in NCBO‘s BioPortal, we create a system that addresses some of these drawbacks. Specify uses BioPortal as its repository of ontologies; BioPortal encapsulates disparate ontologies and related annotated data in one common interface available via REST Web service. Specify builds on top of these services. Specify is able to work with over any ontology in BioPortal – including QIBO and approximately 200 others. The ontology library is separately curated and updated by the administrators of BioPortal, decoupling us from the underlying representation and versioning of the ontologies. 1.4. Terms Used in This Document The following are terms commonly used that may of assistance to the reader. AAS Application Architecture Specification ASD Application Scope Description BAM Business Architecture Model BBMSC 7 of 14 QI-Bench Specify ASD BRIDG Biomedical Research Integrated Domain Group caBIG Cancer Biomedical Informatics Grid caDSR Cancer Data Standards Registry and Repository CAT Composite Architecture Team CBIIT Center for Biomedical Informatics and Information Technology CFSS Conceptual Functional Service Specification CIM Computational Independent Model DAM Domain Analysis Model EAS Enterprise Architecture Specification ECCF Enterprise Conformance and Compliance Framework EOS End of Support ERB Enterprise Review Board EUC Enterprise Use-case IMS Issue Management System (Jira) KC Knowledge Center NCI National Cancer Institute NIH National Institutes of Health PIM Platform Independent Model PSM Platform Specific Model PMO Project Management Office PMP Project Management Plan QA Quality Assurance QSR FDA’s Quality System Regulation SAIF Service Aware Interoperability Framework SDD Software Design Document SIG Service Implementation Guide SUC System Level Use-case SME Subject Matter Expert SOA Service Oriented Architecture SOW Statement of Work UML Unified Modeling Language UMLS Unified Medical Language System BBMSC Rev 0.3 8 of 14 QI-Bench Specify ASD VCDE Rev 0.3 Vocabularies & Common Data Elements When using the template, extend with specific terms related to the particular EUC being documented. 2. Profiles A profile is a named set of cohesive capabilities. A profile enables an application to be used at different levels and allows implementers to provide different levels of capabilities in differing contexts. Whereas interoperability is the metric with services, applications focus on usability (from a user’s perspective) and reusability (from an implementer’s). Include the following three components in each profile: Information Profile: identification of a named set of information descriptions (e.g. semantic signifiers) that are supported by one or more operations. Functional Profile: a named list of a subset of the operations defined as dependencies within this specification which must be supported in order to claim conformance to the profile. Behavioral Profile: the business workflow context (choreography) that fulfills one or more business purposes for this application. This may optionally include additional constraints where relevant. Fully define the profiles being defined by this version of the application. When appropriate, a minimum profile should be defined. For example, if an application provides access to several business workflows, then one or more should be deemed essential to the purpose of the application. Each functional profile must identify which interfaces are required, and when relevant, where specific data groupings, etc… are covered etc. When profiling, consider the use of your application in: Differing business contexts Different localizations Different information models Partner-to-Partner Interoperability contexts Product packaging and offerings Profiles themselves are optional components of application specifications, not necessarily defining dependencies as they define usage with services. Nevertheless, profiles may be an effective means of creating groupings of components that make sense within the larger application concept. BBMSC 9 of 14 QI-Bench Specify ASD Rev 0.3 2.1. Information Profiles Informatics Services for Quantitative Imaging links several relevant concepts that are distributed across the conceptual hierarchy. As such, a spanning ontology that draws together these concepts is possible using according to the following principles: • Metadata is data • Annotation is data • Data should be structured • Data models should be defined • Annotation may often follow a model from another domain • Figure 1: The semantic infrastructure needed to support quantitative imaging performance assessment encompasses multiple related but distinct concepts and vocabularies to represent them which include characterization of the target population and clinical context for use. Data of all these forms is valuable Specifically, the domain includes linked models and controlled vocabularies for the categories identified in Figure 14. In order to support these capabilities, the following strategy will be employed in the development and/or use of information models and ontologies (Tables 2 and 3): Ontologies: Ontology Available through Extend or just use? Dynamic connection? Example of use Systematized Nomenclature of Medicine--Clinical Terms (SNOMED-CT) UMLS Metathesaurus, NCBO BioPortal Use Dynamically read at runtime Grammar for specifying clinical context and indications for use RadLex (including Playbook) RSNA through www.radlex.org,or NCBO Bioportal Use Dynamically read at runtime Grammar for representing imaging activities Gene Ontology (GO) GO Consortium, www.geneontology.org Use Dynamically read at runtime Nouns for representing genes and gene products associated with mechanisms of action International vocabulary of metrology --- Basic and general concepts and associated terms (VIM) International Bureau of Weights and Measures (BIPM) Use Updated on release schedule How to represent measurements and measurement uncertainty Exploratory imaging biomarkers Paik Lab at Stanford Extend Dynamically read at runtime Grammar for representing imaging biomarkers Table 1: Ontologies utilized in meeting the functionality BBMSC 10 of 14 QI-Bench Specify ASD Rev 0.3 As a practical matter, many (but not all) of these ontologies have been collected within the NCI Thesaurus (NCIT). It may be that there is utility in utilizing this to subsume included ontologies as a design consideration. Information models: Information Model Available through Extend or just use? Dynamic connection? Example of use Biomedical Research Integrated Domain Group (BRIDG) (drawing in HL7RIM and SDTM) caBIG Use Updated on release schedule Data structures for clinical trial steps and regulatory submissions of heterogeneous data across imaging and non-imaging observations Life Sciences Domain Analysis Model (LS-DAM) caBIG Use Updated on release schedule Data structures for representing multiscale assays and associating them with mechanisms of action that link phenotype to genotype Annotation and Image Markup (AIM) caBIG Extend Updated on release schedule Data structures for imaging phenotypes Table 2: Information Models utilized in meeting the functionality 2.2. Functional Profiles A named list of a subset of the operations, defined as dependencies within this specification, which must be supported in order to claim conformance to the profile. 2.3. Behavioral Profiles The business workflow context (choreography) that fulfills one or more business purposes for this application. This may optionally include additional constraints where relevant. 3. Conformance Assertions Conformance Assertions are testable, verifiable statements made in the context of a single RM-ODP Viewpoint (ISO Standard Reference Model for Open Distributed Processing, ISO/IEC IS 10746|ITU-T X.900). They may be made in four of the five RMODP Viewpoints, i.e. Enterprise, Information, Computational, and/or Engineering. The Technology Viewpoint specifies a particular implementation /technology binding that is run within a ‘test harness’ to establish the degree to which the implementation is conformant with a given set of Conformance Assertions made in the other RM-ODP Viewpoints. Conformance Assertions are conceptually non-hierarchical. However, Conformance Assertions may have hierarchical relationships to other Conformance Assertions within the same Viewpoint (i.e. be increasingly specific). They are not, however, extensible in and of themselves. BBMSC 11 of 14 QI-Bench Specify ASD Rev 0.3 4. References 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. BBMSC Jaffer, F.A. and R. Weissleder, Molecular imaging in the clinical arena. JAMA : the journal of the American Medical Association, 2005. 293(7): p. 855-62. Quon, A. and S.S. Gambhir, FDG-PET and beyond: molecular breast cancer imaging. Journal of clinical oncology : official journal of the American Society of Clinical Oncology, 2005. 23(8): p. 1664-73. Smith, J.J., A.G. Sorensen, and J.H. Thrall, Biomarkers in imaging: realizing radiology's future. Radiology, 2003. 227(3): p. 633-8. Li, W., et al., Noninvasive imaging and quantification of epidermal growth factor receptor kinase activation in vivo. Cancer research, 2008. 68(13): p. 4990-7. Klibanov, A.L., Ligand-carrying gas-filled microbubbles: ultrasound contrast agents for targeted molecular imaging. Bioconjugate chemistry, 2005. 16(1): p. 917. Hehenberger, M., Information Based Medicine: From Biobanks to Biomarkers, 2007, IBM Healthcare & Life Sciences: High Tech Connections (HTC) Forum. Sadot, A., et al., Toward verified biological models. IEEE/ACM Trans Comput Biol Bioinform, 2008. 5(2): p. 223-34. Cavusoglu, E.Z.E.a.M.C., A Software Framework for Multiscale and Multilevel Physiological Model Integration and Simulation, in 30th Annual International IEEE EMBS Conference 2008: Vancouver, British Columbia, Canada. Feng, D., Molecular Imaging and Biomedical Process Modeling, in 2nd AsiaPacific Bioinformatics Conference (APBC2004)2004. Chen, J., et al., How Will Bioinformatics Impact Signal Processing Research? IEEE Signal Processing Magazine, 2003: p. 16-26. Toretsky, J., et al., Preparation of F-18 labeled annexin V: a potential PET radiopharmaceutical for imaging cell death. Nuclear medicine and biology, 2004. 31(6): p. 747-52. Zijlstra, S., J. Gunawan, and W. Burchert, Synthesis and evaluation of a 18Flabelled recombinant annexin-V derivative, for identification and quantification of apoptotic cells with PET. Applied radiation and isotopes : including data, instrumentation and methods for use in agriculture, industry and medicine, 2003. 58(2): p. 201-7. Group, B.D.W., Biomarkers and surrogate endpoints: preferred definitions and conceptual framework. Clinical pharmacology and therapeutics, 2001. 69(3): p. 89-95. Zhao, B., et al., Evaluating variability in tumor measurements from same-day repeat CT scans of patients with non-small cell lung cancer. Radiology, 2009. 252(1): p. 263-72. Sheikh, H.R., M.F. Sabir, and A.C. Bovik, A statistical evaluation of recent full reference image quality assessment algorithms. IEEE transactions on image processing : a publication of the IEEE Signal Processing Society, 2006. 15(11): p. 3440-51. 12 of 14 QI-Bench Specify ASD 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. 31. BBMSC Rev 0.3 Buckler, A.J. and R. Boellaard, Standardization of quantitative imaging: the time is right, and 18F-FDG PET/CT is a good place to start. Journal of nuclear medicine : official publication, Society of Nuclear Medicine, 2011. 52(2): p. 171-2. Wong, D., Liaison Committee Discusses Possible Radiotracer Sharing Clearinghouse, in American College of Neuropsychopharmacology2006. Wong, D.F., Imaging in drug discovery, preclinical, and early clinical development. Journal of nuclear medicine : official publication, Society of Nuclear Medicine, 2008. 49(6): p. 26N-28N. Creating the gene ontology resource: design and implementation. Genome Res, 2001. 11(8): p. 1425-33. Ashburner, M., et al., Gene ontology: tool for the unification of biology. The Gene Ontology Consortium. Nature genetics, 2000. 25(1): p. 25-9. Romero-Zaliz, R.C., et al., A Multiobjective Evolutionary Conceptual Clustering Methodology for Gene Annotation Within Structural Databases: A Case of Study on the Gene Ontology Database. IEEE Transactions on Evolutionary Computation, 2008. 12(6): p. 679-701. Brazma, A., et al., Minimum information about a microarray experiment (MIAME)toward standards for microarray data. Nature genetics, 2001. 29(4): p. 365-71. Sirota, M., et al., Discovery and preclinical validation of drug indications using compendia of public gene expression data. Science translational medicine, 2011. 3(96): p. 96ra77. Brown, M.S., et al., Database design and implementation for quantitative image analysis research. IEEE Trans Inf Technol Biomed, 2005. 9(1): p. 99-108. Maier, D., et al., Knowledge management for systems biology a general and visually driven framework applied to translational medicine. BMC Syst Biol, 2011. 5: p. 38. Toyohara, J., et al., Evaluation of 4'-[methyl-14C]thiothymidine for in vivo DNA synthesis imaging. Journal of nuclear medicine : official publication, Society of Nuclear Medicine, 2006. 47(10): p. 1717-22. Yuk, S.H., et al., Glycol chitosan/heparin immobilized iron oxide nanoparticles with a tumor-targeting characteristic for magnetic resonance imaging. Biomacromolecules, 2011. 12(6): p. 2335-43. Veenendaal, L.M., et al., In vitro and in vivo studies of a VEGF121/rGelonin chimeric fusion toxin targeting the neovasculature of solid tumors. Proceedings of the National Academy of Sciences of the United States of America, 2002. 99(12): p. 7866-71. Wen, X., et al., Biodistribution, pharmacokinetics, and nuclear imaging studies of 111In-labeled rGel/BLyS fusion toxin in SCID mice bearing B cell lymphoma. Molecular imaging and biology : MIB : the official publication of the Academy of Molecular Imaging, 2011. 13(4): p. 721-9. Wang, H.H., et al., Durable mesenchymal stem cell labelling by using polyhedral superparamagnetic iron oxide nanoparticles. Chemistry, 2009. 15(45): p. 1241725. http://www.ncbi.nlm.nih.gov/books/NBK5330/, Molecular Imaging and Contrast Agent Database (MICAD). 2011. 13 of 14 QI-Bench Specify ASD 32. Rev 0.3 SPARQL, a query language and protocol for RDF acccess released by the W3C RDF Data Access Working Group. Available from: http://www.w3.org/wiki/SparqlImplementations, accessed 27 November 2011. 1 http://www.accessdata.fda.gov/scripts/cdrh/cfdocs/cfcfr/CFRSearch.cfm?CFRPart=8 20&showFR=1, accessed 28 February 2010. 2 http://www.rsna.org/Research/TQI/upload/Workshop-Summary-FINAL.pdf, accessed 17 March 2010. 3 Giger, QIBA newsletter, February 2010. 4 Giger M, Update on the potential of computer-aided diagnosis for breast disease, Future Oncol. (2010) 6(1), 1-4. BBMSC 14 of 14