Gernoth Grunst (St. Augustin, Germany) 2 Enabling Systems in Medicine Enabling Systeme in der Medizin Summary Zusammenfassung The essay introduces the training modules EchoExplorer and Cardio-Sim that provide an enabling system for echocardiography. The environment consists of practical sequential training units that build up a course. There are various forms of interactivity. A common feature of all modules is the combined use of digitized 2D or 3D ultrasound images with related views of a Virtual Reality model showing a beating heart. The exploratory scenarios allow forms of constructive learning that are to a large extent self-controlled. Die vorgestellten Trainingsmodule EchoExplorer und Cardio-Sim sind Elemente eines Enablingsystems für die Echokardiographie. Sie bieten dem Arzt aufeinander aufbauende Stufen praktischer Übungseinheiten, die unterschiedliche Arten der Interaktivität realisieren. Durchgängig ist dabei der kombinierte Einsatz von digitalisierten 2D oder 3D Ultraschall Daten und Virtual Reality Modellen des schlagenden Herzens. Die integrierten Explorationsszenarien erlauben ein weitgehend selbstgesteuertes, konstruktives Lernen. 1 Objectives The EchoExplorer [9] / Cardio-Sim[8] training environment for echocardiography is an illustration of a medical enabling system. Its objective is to support the acquisition of complex medical expertise. The example domain echocardiography requires the integrated application of a broad spectrum of cognitive and practical abilities. The entire range is covering diagnostic and anatomical knowledge, visual perception, and controlled handling of the ultrasound device. Thus multiple contents have to be made accessible in intuitive and related scenarios. The learning environment shall allow the active exploration of meaningful scenarios that present these contents mostly in nonsymbolic form. One of the research items of the developments has been to determine appropriate combinations of near realistic Virtual Reality models and medical image data that provide self-explaining Augmented Reality scenarios [1, 12]. 1 ZSfHD 3/99 Methodological approach The developments started with field studies of expert- / novice behavior in diagnosis and instruction. The studies applied techniques of immediate and video based interaction analysis [7]. The analyses allowed insights into pertinent Mental Models of skilled cardiologists as well as missing imaginations or typical misconceptions on the novice side. The details could be derived from verbal utterances as well as behavior patterns. The main advantage of this approach has been that we were able to identify relevant elements of expert knowledge and skill embedded in the context of their application. Such situations provide the optimal scenario for the acquisition of realistic expertise. On the other hand this model of apprenticeship learning is usually incompatible with the clinical reality. This holds especially for the initial states of learning. Therefore the analyses tried to identify important factors of insight based learning and to specify potential functional equivalents for apprenticeship learning. The main strategy of transfer was to organize interactive scenarios that allow constructive learning [2]. The role of a critical teacher in these learning scenarios should be reflected through various forms of (mainly visual) feedback. In the sense of Donald Schön [11] the idea was that the whole situation should “talk back” to the learner. Technically the findings are mapped onto combinations of interactive VR models and 3D image data. The VR elements are thought to impart the identified expert Mental Models of the dynamic anatomy and the way the heart is accessed via echocardiography. The related films and 3D data sets represent the reality of actual patients as it is documented via echocardiography. The explorable scenes shall allow autonomous inquiries and induce constructive learning. [2, 13] In formative and summative evaluation cycles the adequacy of the analyses and the technical transfer has been verified [4, 8, 12]. For example the anatomical detail of the VR models was redesigned after prototype tests with experts from different medical domains. 3 The training environment The development of the various enabling units started about eight years ago as a cooperation of the German National Research Center for Information Technology and Prof. Dr. D.A. Redel (Centre for Pediatry of the University Bonn, Department for Cardiology). First attempts to map the complex spatial logic of ultrasound onto simpler multimedia visualizations [3] showed that interactive 3D graphics were indispensable to design flexible visual explanations of the various standard views in echocardiography. This soon lead to the design of a static 3D reference model. Dynamic features and the various forms of interactivity were added after the first prototypes had ZSfHD 3/99 2 been used in realistic training sessions and the need of some form of dynamic anatomy became evident. The resulting interactive VR model could then be used in both training modules (the CD-ROM based course EchoExplorer and the simulator CardioSim). 3.1 EchoExplorer EchoExplorer is an interactive course that introduces students and cardiological novices into echocardiography. The CD-ROM contains three main modules (dynamic anatomy, examinations in echocardiography, and interactive echocardiography). The training module furthermore features a selfassessment component the “Quiz”. 3.1.1 Dynamic anatomy The first module of EchoExplorer (dynamic anatomy) displays the beating heart model as an explorable object. Several visualization modes can be chosen (surface view, transparent view, highlighted septum, transparent with blood flow). Fig. 1: The dynamic anatomy heart-model in transparent mode The learner can rotate the different versions of the 3D-model animation with the mouse and thus acquire flexible conceptions of the inner and outer heart-anatomy. The pertinent visual cues have to become automated match patterns that are later used in the application of echocardiography. In a further form of interaction with this scene the learner can manually steer the animation of the heart cycle with respect to the ECG curve. This way the timing of ventricle contraction and the opening of the valves can be explored in its relationship to the ECG. 3.1.2 Examinations in echocardiography In the next stage the various standard-views in echocardiography are introduced and related to the heart model. The learner can call up verbally commented animations showing the appropriate positioning of the transducer and the fine adjustment of these views. Fig. 2: The echocardiography module for standard positions, thorax view The animation is started as a demonstration of the correct transducer position with respect to the pertinent intercostal space. Reference object is a Virtual Reality model of the thorax (bone structure). The imaging segment is visualized and shown as a transparent triangle that is slicing the heart. The animation continues as a “dive” into the inner thorax showing the learner in detail where the transparent heart is cut by the ultrasound segment. Then the scene is rotated such that the transducer head moves to middle position in the top of the window. This reflects the logic of the transducer related imaging of real ultrasound and can be used as a module for the training of pertinent mental rotations [6]. The heart model is then decreased to the structures that are cut by the ultrasound segment. The resulting substructure of the beating heart model is a graphic match-pattern for the actual ultrasound image in this position. The animation then blends into an exactly fitting example video of a real ultrasound film. Fig. 4: the US image level with highlighted substructures. Fig. 3: levels of visual explanation: heart-model with plane, slice, 2D US image Further descriptions are related to substructures of the heart that can be seen in the actually covered region. In order to keep the perception focus on relevant structural and dynamic features the explanations apply a combination of graphical highlighting and verbal comments. The positioning process is thus decomposed into a kind of two step approach. First the transducer is roughly placed at pertinent positioning points. This action is guided by external anatomical landmarks of the chest and controlled by general, position dependent gestalts in the ultrasound image such as the typical pattern of a four-chamber view. Then the fine-tuning is based on control patterns for relevant substructures like the papillar muscles or the valves. The standard positions are related to certain subregions of the heart and necessary examinations of anatomical structures that can be accessed here via ultrasound. The selection of these areas is determined by the diagnostic task. But the choice of alternative views also has to reflect the ultrasound imaging features of the patient. For adipose patients for example some standard views do not produce useful images. Therefore cardiologists have to develop flexible strategies of adjustment i.e. intuitively switch to alternative standard positions that produce better image results for a specific patient. This ability is based on a graphic understanding of the heart and flexible conceptions of the ultrasound examination process. 3 ZSfHD 3/99 The module has only restricted forms of interactivity. It merely allows the selection, play, stop, and replay of animation sequences for particular standard positions. The rationale is to provide for the learner an introductory overview. 3.1.3 Interactive echocardiography The module “Interactive Cardiology” lets the user more actively explore the learning scenario. There is a choice between sets of ultrasound examples that represent healthy and pathological cases. For each case there are examples of standard views in the various ultrasound modes such as 2D, M-mode, qualitative and quantitative Doppler. The interface presents the ultrasound films in one window with related graphical explanations in an adjacent frame. The illustrations support the interpretation of the ultrasound films through virtual models on three levels. They are associated to three conceptual points of view. The first level (thorax) shows the scenario from the outside. This Virtual Reality scene includes the transducer in relation to the appropriate intercostal space of the thorax and thus links the conception of the ultrasound image to the perception of the patient under examination. ZSfHD 3/99 2 verbal explanations in this stage. He should now be able to relate the words not only to image details but also have a focussed spatial and anatomical understanding built up through the exploration of visual explanations on the levels “thorax”, “3D heart anatomy”, and “sliced heart structures”. 3.1.4 Selfassessment A very important element of any training is the assessment of achieved learning levels. In (ideal) apprenticeship situations the teacher accomplishes this duty. Interactive multimedia or VR training environments that intend to cover this objective instead have to provide for instruments of self-assessment. For this purpose EchoExplorer features the “Quiz” module. The learner can choose three test-levels with specific feedback. On level A he is presented with randomly selected model-slices and requested to choose the corresponding standard position from a selection-menu. Fig. 5: scenario of the combined images and VR models for interactive exploration The second level (3D heart anatomy) shows the transparent 3D-heart model and the enclosed visualization of the ultrasound segment. The user can turn this combined 3D object and thus inspect which substructures and areas of the heart are cut in the respective standard position. Here the ultrasound film is linked to the anatomical imagining of the heart. A click switches to the third level (sliced heart structures). As in the training module for standard positions, animations show the rotation of the 3D model such that the transducer head moves to middle position on top of the window. The heart model is then decreased to the actually sliced heart structures. Our evaluations showed that students extensively used this feature in order to exercise the spatial mapping of a “front view” of the heart model onto this rotated structural pattern. This pattern is aligned and can be synchronized with the real ultrasound film. The graphic dynamic gestalt of the anatomic model slice provides intuitive visual explanations for corresponding structures in the ultrasound. The learner can now continue to build up a more detailed understanding by addressing the mouse actions directly to the ultrasound window. By touching certain areas with the mouse the user activates descriptions in the form of standard abbreviations. Mouse clicks start detailed verbal comments on the physiological substructures and their appearance in ultrasound. We expect the learner to have more advantages of Fig. 6: the selfassessment module in test mode A If the learner is unable to give a correct answer he can call up a “tip” that facilitates the finding of an appropriate assignment. This module exploits the three levels of visual explanation that have been introduced in the “Interactive Cardiology” unit. Backward animations show a reverse rotation that ends with a front view of the transparent 3D-heart model and the ultrasound plane slicing it. This view lets the learner recognize the position and orientation of the transducer and thus narrow the possible range of standard views. This kind of support follows a specific elicitation strategy [10]. Such narrowing of a problem space is analytically shown to be one of the most effective ways to induce active mental processing, insights, and thereby learning. Thus the self-assessment module is not just an isolated test layer but closely related to the other learning scenarios. If test learners recognized gaps of knowledge in the “Quiz” they typically went back to the related training units and explored them in a more focussed manner. The level B shows real ultrasound films that have to be assigned to pertinent standard positions. The “tip” is realized by the presentation of the related model slice view. On level C the learner identifies substructures in a real ultrasound film. The “tip” support in this module is still insufficient. It only marks potentially relevant areas. 3.2 Cardio Sim In the Cardio-Sim (CS) unit the combined VR models and 3D data sets allow interactive explorations that build up visual expectation-patterns as they are required in the application of ultrasound in real heart examinations. For this purpose a virtual ultrasound transducer is used as interaction device. The transducer dummy is placed on pertinent points on a human model torso. The position, rotation, and tilt of the device is continuously checked and mapped onto a virtual transducer model on the screen. The spatial information is also used to slice associated 2D views out of the 3D model. Transparent presentations of the chest and the inner and outer structures of the heart provide visual cues for sweeps and rotations of the ultrasound plane. Reference planes of standard views can be used for guidance. The hand-eye-coupling and a visual understanding of the scenes are trained and merge into a Mental Model of the heart. This way the inspection of the anatomic model is closely linked to the handling of the imaging device. The next step from model learning to real diagnoses is the integration of real 3D or rather 4D (3D + time) ultrasound image data. These data sets are registered i.e. geometrically aligned with the anatomic model. The trainee can exploit the graphic orientation provided by the model in order to become acquainted with the interpretation, tuning and controlled adjustment of ultrasound images. The VR elements of the scene provide continuous visual feedback informing the trainee where the actual plane slices the heart and which anatomic structures should be seen. After extensive training in this “training wheel” mode the novice is able to switch off the orientation models and exclusively rely on the ultrasound image as visual feedback. Fig. 8: the merged VR model / 3D US data set feedback mode of Cardio-Sim Fig. 7: the Cardio-Sim simulator setting 5 ZSfHD 3/99 One of the cognitively demanding requirements of the visually controlled application of cardiac ultrasound is the necessity to continuously adapt to the transducer relative images. This requires mental rotations [6] that reflect the changing points of view in the various standard positions. Controlled sweeps in each of these diagnostically relevant positions need fluent adaptations to the anatomy of the heart. If a visual conception of this 3D structure is missing the diagnostician is unable to adjust the ZSfHD 3/99 2 transducer. If he can only rely on one inflexible standard imagining of the heart anatomy he consciously has to rotate and map the perceived structures onto this Mental Model. Our field studies showed that skilled cardiologists have build up dynamic Mental Models of the heart that are not related to fixed conceptual points of view. This allows immediate interpretations and thus a fluent eye-hand coupling. The exploration of the simulation environment allows the novice to actively build up this form of a dynamic Mental Model. The kind and degree of immersiveness [13] realized in this training module reflects the way that the cardiologist will experience diagnostic reality. A “fly through” the heart that would allow movement inside a structured scenario in this sense is less appropriate than a local constellation that the trainee can manipulate from outside. The skills that have to be activated in real diagnoses reflect the natural spatial relationship between the cardiologist and the patient. 4 Work in progress Adaptivity is a key issue in enabling systems that shall be used for learning and orientation on demand. This requires integrated concepts of knowledge-based evaluations with interactive 3-D graphics and animation. The Cardio-Sim environment shall in this sense be enhanced by adaptive modules that take into account the activities of the trainee. The most advanced form of analysis and feed back is based on adaptive evaluations of motion patterns [5]. These recognizable physical events have to be mapped onto cognitive orientation demands as they change during actual examinations. Furthermore the examination steps realized this way are related to the diagnostic context. Based on the evaluation of behavior protocols the system will provide context sensitive help animations that show the specific deviations from appropriate diagnostic steps and standard positions. 5 Generalization and lessons learned All developments of enabling systems require a detailed understanding of the competence and abilities that experts apply in problem solving. Furthermore it is necessary to have a clear conception of the way in which various layers of expertise build on each other. This is defining the spectrum and structure of training environments that attempt to cover real world expertise. Closely related to the contents of such enabling systems the designer has to be aware of the various cognitive modes that are involved in certain stages of effective problem solving. In the medical domain symbolic, visual, motor, and dynamic abilities, such as mental rotation or the combination of inferences and pattern matches must be acquired in a coherent and situated form. Medial presentation and interaction modes of enabling systems have to reflect these different forms of cognition. Certainly individual differences in cognitive and learning style are effecting learner-driven forms of interaction with the system. On the other hand enabling systems have to guarantee that basic contents are picked up by every trainee. Therefore introductory and more restrictive modules have to be included. Training modes that are to a higher degree learner-driven should support the active mapping of these contents onto realistic tasks. The most advanced kind of such environments are medical simulators that allow constructive learning. Even for advanced interactive learning environments there is no warranty that the intended results are achieved. Therefore various forms of assessment should be rendered possible. Typically controlled test tasks provide an authentic way to access achieved levels of expertise. The value of such assessments is even higher if even the learner himself can check the results. Adapted forms of correction and elicitation can be added to the self-assessment modules. This way the trainee gets focussed forms of feedback and support in the learning process. References [1] Berlage, Th. (1997): Augmented reality for diagnosis based on ultrasound images. In: Troccaz, J.; E. Grimson,, and R. Mösges, R. (eds.): Proceedings CVRMedMRCAS '97. (Lecture Notes in Computer Science Nr. 1205). Berlin: Springer. pp. 253-262. [2] Duffy, T.M., J. Lowick, & D.H. Jonassen (1993): Designing Environments for Constructive Learning. New York. [3] Fox, T. (1993): Kognitiv ergonomische Benutzerschnittstellen - Entwicklung interaktiver 3D-Visualisierungen und multimedialer Simulationen: Das Tutor-System COCARD zur Einführung in Ultraschall-Untersuchungen des Herzens. Sankt Augustin: GMD-Studie Nr.218. [4] Grunst, G.; T. Fox,, K.-J. Quast, & D.A. Redel, (1995): Szenische Enablingsysteme – Trainingsumgebungen in der Echokardiographie. In: Glowalla, U.; E.; Engelmann, A de Kemp, G. Rossbach, & E. Schoop, (eds): Auffahrt zum Information Highway. Kongressband Deutscher Multimedia Kongress '95, Heidelberg 11.13.6.1995. Berlin-Heidelberg: Springer, S.174-178. [5] Grunst, G., S. Trochim (2000): Enhanced Reality Trainingssysteme in der Medizin. Künstliche Intelligenz , Nr. 2 . - S. 28-29. [6] Metzler, J. & R.N. Shepard (1974): Transformational Studies of the Internal Representations of Three Dimensional Objects. In: Solso, R.L. (ed.): Theories of Cognitive Psychology: The Loyola Symposium. Hillsdale, NJ: Lawrence Erlbaum Associates. [7] O´Malley, S.W. C.E.O. Draper, and M.S. Riley (1985): Constructive Interaction: A Method for Studying Human - Computer - Human Interaction. Proceedings of INTERACT ´84, London, pp. 269-274. [8] Quast, K. (1997): Computerbasiertes Lernen in 3D-graphischen Szenen Entwurf, Realisierung und Evaluation einer Anwendung für die kardiologische Ultraschalldiagnostik. (Berichte der GMD Nr. 280). München: Oldenbourg, [9] Redel , D. A. & F. Hoffmann,(1998): EchoExplorer, Interaktive CD-ROM, München, Urban & Schwarzenberg [10] Rehbein, J. (1980): Hervorlocken, Verbessern, Aneignen. Diskursanalytische Studien des Fremdsprachenunterrichts. Bochum. [11] Schoen, D.A. (1990): Educating the Reflective Practitioner. San Franscisco,Oxford: Jossey-Bass Publishers. [12] Weidenbach, M., C. Wick, S. Pieper, K.J. Quast, T. Fox, G. Grunst and D. A. Redel (2000): Augmented Reality Simulator for Training in Two-Dimensional Echocardiography, in: Computers in Biomedical Research 33, 11 – 22, Academic Press. [13] Winn, W.D. (1994): A Conceptual Basis for Educational Applications of Virtual Reality. (Technical report No. HITL R-94-1). 7 ZSfHD 3/99 ZSfHD 3/99 2