TITLE Crisis managers making strategic decisions in the near real-life training environment of Pandora: evaluation framework for human performance and user experience. <Michele Cornacchia, Filomena Papa, Stefano Livi, Bartolomeo Sapio, Enrico Nicolò, Keith Strickland> Conference: eChallenges 2011, 26-28 October 2011, Florence, Italy Track: Technology enhanced learning and ICT Skills INTRODUCTION This paper describes the general evaluation framework to be adopted in the FP7 Project Pandora, a near real-life training environment for learning activities suitable for crisis scenarios. The main goal here is focused on setting up the most adequate framework to evaluate the performance of different levels of end-users, namely the single trainees, the group of trainees and the trainer. The Pandora environment is a very complex innovative system, providing its end-users with many technological supports for single and collaborative strategic decisions making. It responds to the needs of the traditional business continuity plans that do not adequately take to into account the many forces of human behaviour of individuals and groups as they are shaped by numerous intangible factors. Whether or not, at the end of a learning session, the trainees feel a perception of increased performance and/or self-efficacy about facing potential crisis situations, is the final purpose of the evaluation. Furthermore the user experience and the ICT acceptance of individuals moving and acting in a very innovative system which provides and takes into consideration also emotional responses, is also object of evaluation. In particular on this topic, regarding the work conditions under time pressure and/or the difficulties in information processing, group decision making is approached in terms of cognitive closure, the somehow irrational mechanism to eliminate ambiguity and reach definite conclusions. OBJECTIVES As previously introduced, the final performance in training sessions of trainees, group and trainer represents the final goal of this evaluation project. It basically applies a human factors approach which puts specifically in the centre of the consideration the end-users point of view, needs and expectations in relation with the complexity of Pandora. The main dependent variables chosen to describe such a performance are recognized in literature mostly from the models of user acceptance, user experience and managerial decision making under crisis. The effectiveness of training on the end-users’ side first and then some important process variables developed by the Pandora system (behavioural tendencies, crisis scenario, user profiling, emotional status, environmental conditions, physical set-up, didactical goals, recording, etc.) influencing the group decision making are supposed to give a wide-ranging accurate idea of whether or not learning is perceived like in a near real-life training environment and whether or not trainees feel themselves pragmatically more experienced when sessions are over. The effectiveness of training is therefore an objective measure of the learning performance, it is the extent to which crisis managers achieve a set of learning goals and apply procedures. Moreover, goals of evaluation are considered the matching between performance reports (automatically produced by the Pandora system) and self-evaluation, the quality of interaction during the training, the interpersonal or mediated communication among actors playing in the session (trainer to trainees; trainees to trainees; trainees to possible lower tactical level). METHODOLOGY Given that a scenario is the crisis or the emergency situation created by the Pandora system for the training session, the applicable methodology has to distinguish between three use cases for the potential deployment of the environment: the “single site training”, where trainees and trainer are in the same room with full technological capabilities; the “deployed training”, where trainees and trainer are in the same room but with reduced technological capabilities; the “distributed training”, where trainees and trainer are physically in different sites but share the same virtual environment. It is almost evident that the first two training configurations are very similar out of technological equipment, and that the third one differs consistently as it works in a virtual setting. It is then reasonable to expect close results from the first two and a significant distance of the last one, if we consider the variables model previously shortly mentioned. In this model the hypothesized differences existing between objective and perceived results, either achievable by direct measurements or by Pandora system itself, are modulated in accordance with the individual, group and trainer performances. In general it is possible to introduce the classification of “antecedent” variables and “process” variables in order to collect either objective or perceived sets of measures of effectiveness. For example, the cognitive “managing” and the physiological “readiness” are both antecedent variables and referred to individual; the perception of cognitive closure about having processed all information is a process variable, possibly applicable to both individual and group; self-efficacy and usability can be referred to the individual; social loafing in decision making to the group; information available and group dimension to the group; even the trainer attendance to the session may be considered as objective process variable; and so on. In general the three use cases set out for Pandora bring to require two different methods of investigation to access the expected results of improved performance in decision making and increased sense of confidence likewise said self-efficacy. On this purpose, preliminary studies (interviews to designated users, control group definition, in-between testing to single modules) can be applied to best fit the standard methods and tools of the human factors, usability and social research with the variables model of assessment. It is nonetheless crucial to set up the method with the schemes of qualified models like those provided for the emerging ICT acceptance, the tele-presence, the cognitive closure and managerial group decisions making in condition of high uncertainty. TECHNOLOGY DESCRIPTION Pandora is a special training system that provides appropriate metrics on the performance of a manager actively engaged in the strategic management of a crisis. The environment provides a realistic and complete scenario with near real-time action, coherent with that expected in a real-world situation. Realistic emotional statuses, through affective inputs (emotional statuses: fear, irritation, joy) and stress factors (pressure to decision, fragmentary information, short time to decide) are represented to different crisis managers belonging to different sectors. The main innovative aspects are in the realism and complexity of the simulated model. It can analyse behavioural aspects and report them during the training, it can model the crisis scenario and its evolution, it can create the user experience up to the reproduction of emotional status, it can create multiple interconnected training procedures, it gives a retroaction enabling the trainer and the trainees (through their decisions) at modifying the scenario on runtime, and other. The architectural system is composed of few main modules. The Behaviour Simulation and Modelling aims to create and update a profile for each trainee of features and actual behaviour as well as to provide indications on how to personalize the training itself. The Crisis Simulation and Modelling aims to combine the information coming from the previous sub-system and the knowledge about the critical, infrastructure domain in order to create a crisis scenario evolution to be simulated. The Environment and Emotion Simulation aims to effectively represent Non–Player Characters and environments for the simulation. At last, the Crisis Room is the place where the exercise really happens. As previously mentioned, it can be configured as complete in a fixed site, portable in a deployed site, or web distributed, in a virtual reality, according to the selected use case. CONCLUSIONS AND RECOMMENDATIONS The three use cases of the Pandora managerial training system for crisis scenario are here analysed in order to arrange a plan (the project ends on 31st December 2011) of performance evaluation by the end-users. The assessment aims at pointing out the major features of the Pandora complex environment by the viewing angle of the main actors involved, that is the trainees and the trainer. The dimension of both individuals and group specifically in the process of making decisions under crisis or emergency scenario is foremost central in the adopted human factors approach. The evaluation aims as well to verify if the project underlying hypothesis can be accepted, that is to which extent Pandora effectively bridges the gap between table-top exercises and real world simulation exercises, providing a near-real training environment at an affordable cost.