Michele CORNACCHIA
1
, Filomena PAPA
1
, Stefano LIVI
2
,
Bartolomeo SAPIO 1 , Enrico NICOLO’ 1 , Keith STRICKLAND 3
1
Fondazione Ugo Bordoni, Viale del Policlinico 147, Rome, 00161, Italy
Tel: +39 06 54801, Fax: + 39 06 54804406,
Email: mcornacchia@fub.it, fpapa@fub.it, bsapio@fub.it, nic@fub.it
2
Sapienza Università di Roma – Dipartimento di Psicologia dei processi di sviluppo e socializzazione, Via dei Marsi 78, Rome, 00185, Italy
Tel: +39 06 49917967, Fax: + 39 06 49917652, Email: stefano.livi@uniroma1.it
3 Emergency Planning College (Cabinet Office), The Hawkhills, York YO61 3EG, UK
Tel: +44 (0)1347 825027 Fax: + 44 (0)1347 822575
Email: keith.strickland@cabinet-office.x.gsi.gov.uk
Abstract: This paper describes the general evaluation framework to be adopted in the FP7 Project Pandora, a near real-life training simulated based environment for learning activities suitable for crisis scenarios. Pandora provides its end-users with much technological support to promote both single organisation and multiple organisation collaborative strategic decision making in response to a crisis. It takes into consideration emotional responses and provides significant stimuli to exert the decision makers capability to recognise, analyse and correctly assess a developing crisis simulation. User performance and acceptance of technology will be modelled within the evaluation for different levels of end-users; namely, single trainees, groups of trainees and indeed the trainer. In particular, the operating environment within the simulation will exert certain conditions upon the participants including critical time pressure and/or difficulties in information processing, group and collaborative decision making which is approached in terms of cognitive closure and the somewhat irrational mechanisms in play to eliminate ambiguity in order to reach definite conclusions.
Pandora is the FP7-ICT-2007-1-225387 project winner for the particular mixed call on ICT and Security, co-funded by the European Union under the Security Programme and monitored by REA (Research Executive Agency). It started in January 2010 and will end on 31 December 2011 [1]. Pandora is a simulated based training system for crisis managers that takes full account of human behaviour. It is a matter of fact that, when a catastrophic event occurs (e.g. extreme weather including hurricanes, floods drought etc., fire, explosions, earthquake, disease, deliberate acts, etc.), human reactions and the exactness of key decisions are often the most relevant factors in determining the efficacy of the response in order to prevent an emergency from becoming a disaster. Crisis situations occur frequently and they invariably require a multi-agency response [2], as impacts and consequences are usually wide ranging and are a risk to human health and life. Effective

management of a crisis is therefore a major issue and the training of strategic decision makers is therefore an important preparation to ensure a well managed response [3].
Currently there are two main methods for delivering strategic training [4]: table-top exercises and real world field exercises exercise. A table-top exercises is low cost and is relatively easily organised, but it cannot recreate the real atmosphere (e.g. stress, anxiety, confusion) and the pressure of making critical decisions in short timescales. On the other hand, simulation exercises in the field can be very effective, but such activity is hugely expensive in both time and resources and is not easily or quickly organised.
Pandora is designed to bridge the gap between table-top exercises and real-world simulation exercises, providing a near-real training environment at affordable cost in a controlled environment. The new training environment is provided in three deployment modes of training to provide single site, deployable and remote distribution for strategic level crisis managers (not tactical or operational), to exercise under simulated stressful conditions [5]. This paper focuses only on the first mode as that gives a clear picture of the capabilities that are invested within the Pandora product. It also provides a comprehensive
ICT solution, based significantly upon the analysis of both the end-users needs (trainees and trainer) and the learning/didactic objectives for strategic decision making training. In this respect, the evaluation framework proposed below aims to verify whether or not human performance and individual/group decision making in a simulated scenario of crisis is improved and to what extent. Moreover, in respect of the pressurised training environment, where information may not be complete and decisions are still required and quickly, group decision making is approached in terms of cognitive closure, and the mechanisms
(sometimes irrational or subjective) used to eliminate ambiguity and reach definite conclusions.
The user requirements have been determined from research conducted by the UK
Government with over 300 senior emergency response practitioners [4]. These user requirements have subsequently informed a system requirement document that details the functional capabilities that Pandora needs to encompass, with accompanying justification and prioritisation of individual attributes, in order to meet the user training requirements.
1.1
– Multi-Agency Working
Major Incidents/Emergencies are multi-agency by definition as there is unlikely to be any single agency that has all of the skills, knowledge and resources necessary to provide an effective response. Consequently, there is a need to promote collaborative working to deliver a combined and co-ordinated response that will unify the expertise and resources available across a wide partnership of organisations.
1.2
– User Training Needs
The main requirements that emerged from the user needs analysis are:
 to collectively train, practice and exercise strategic emergency management decision making across a range of different conditions to optimise both individual and group decision making processes;
 to undertake individual and collective training across all agencies that have a stake in emergency management (Public and Private Sectors);
 to undertake training to develop an in depth understanding of individual and group decision making behaviours;

 to conduct training that presents possible courses of action for consideration and selection by the decision maker(s);
 to conduct training that stimulates decision makers to develop options for consideration and subsequently select a preferred course of action with supportable justification;
 to provide training that will take account of scenarios contained in risk assessments at the local, regional and national levels;
 to undertake training in conditions that reflect as far as possible the real operating environment and the stress and dynamic tensions induced by the realism of the scenario;
 to develop confidence among individuals to adopt different decision making models under emergency response conditions;
 to collect objective training event data to support debriefing and wider learning processes.
Pandora builds its solution by developing a system capable of providing the following crucial functionalities:

Selection, modelling and monitoring the relevant human factors or psychological variables (e.g. personality traits, leadership style, background experience) that have been shown to have an influence in decision making under stress.
 Modelling the trainees’ actual behaviour/status with respect to the selected variables.

Personalisation of the training stimuli according to the trainees model.

Provision of appropriate metrics on the performance of an individual actively engaged in the strategic management of a crisis.
2.1
– The Pandora crisis training environment
The environment makes available a realistic and complete scenario with near real-time actions, coherent with those expected in a real-world situation. Realistic emotional statuses, through affective inputs (emotional statuses: fear, irritation, joy) and stress factors (pressure to make a decision, fragmentary information, short timescales for decisions to be made) are represented to different crisis managers representing different organisations but working towards a common purpose [6].
The main innovative aspects are in the realism and complexity of the simulation model.
Pandora can analyse behavioural aspects and report them during the training, it can model the crisis scenario and its evolution; create the user experience up to the reproduction of emotional status; create multiple interconnected training procedures; represent consequences within the simulation of trainees decisions.
2.2
– The architectural system
The architectural system is composed of 4 main modules. The Behaviour Simulation and Modelling [7] module aims to create and update a profile for each trainee capturing their features and actual behaviour as well as providing indications on how to personalize the training for individual trainees.
The Crisis Simulation and Modelling [8] module combines the information coming from the previous sub-system and the knowledge about the critical infrastructure domain in order to create the crisis scenario evolution that is to be simulated. The Environment and
Emotion Simulation [9] module aims to represent effectively the Non–Player Characters and environments within the simulation. Lastly, the Crisis Room [5] module is intended as the environment where the training event takes place.

This deployable mode in a fixed site is due to be set up at the UK Emergency Planning
College outside York, England, in a standard training room, in accordance with the basic end-users requirement and utilising “severe weather” crisis simulation scenario.
2.3
– Training Task
A group of trainees representing from public authorities and private agencies (e.g. Civil
Protection, Transport, Telecommunication, Police, Hospitals, etc.) will participate in training sessions. In the event that certain organizations are unable to be represented such entities will be simulated by the system (Non Players Characters). Each trainee, before participating in the training session, will be required to provide personal information and details in order to create personal profiles for each trainee (Pandora Kernel) covering their competences, attitudes and behavioral style (Behavioral Model). With these primary acquisitions the system will subsequently be capable of addressing the need to provide training that is personalized (Behavioral Planner). The real training process is contrarily created by a second module (Crisis Planner) which exploits either individual information retrieved from the trainee profile and or the knowledge base (Crisis Knowledge) within which the component of the working scenario reside [10].
In this way, Pandora will plan the training session dynamically by choosing, in relation with both the crisis progression and the reactions of each trainee, the most proper sequence of goals and required tasks to submit to the group or the general information that they will be given to share appropriately between themselves. A further important module (Emotion and Environment Synthesizer) contributes in parallel to the training plan by taking control of the primary stimuli modulation function and stress level generator. An additional module
(Trainer Support Framework) provides the trainer with the opportunity to monitor the temporal evolution of the session, i.e. the behavioral analysis of each trainee. The trainer can control in real time the didactic activities of both group and single individuals, for example, by intervening on their decision path by inserting new or conflicting information about the situation under discussion by the trainees. This capability allows the trainer to use judgment and experience to extract the best learning outcomes from the trainees.
As described in the previous sections, Pandora simulates and reproduces crisis scenarios for learning activities where individuals and individuals within a group are highly involved.
Thus, in such stressful and complex environments, psycho-social factors are highlighted requiring individuals to be trained to quickly and accurately accomplish their own tasks, and to efficiently integrate and coordinate their actions in stressful situations with other trainees [11]. In particular, the deleterious effects of stress on human performance are well documented and have been a focus of research in the psychological sciences although, even in simulation-based training, converging more on individual than group performance in decision making [12, 13]. In this case we use the term stress to indicate a process by which environmental demands (in our case time pressure during a threatening event) evoke a process in which perceived demand exceeds resources and that, in turn, results in undesirable physiological, psychological, behavioural, or social outcomes [14]. As a result, both individual and group level characteristics, processes and outcomes are involved, although some environmental characteristics will also be taken into account particularly in extreme training environments such as Pandora. In fact, Pandora, as a high-stress environment, includes specific task conditions (such as time pressure, uncertainty, high cognitive load and so on) that require specific responses such as the flexibility to adapt to novel and changing environmental contingencies [15].

At an individual level, stress influences both the psychological ability (e.g. memory, attention) and emotional and motivational ability, in that it reduces motivation and consequently the need for cognitive closure. This motivation indicates a tendency or an inclination, rather than a cognitive deficit, that is function of an individual weight of costs and benefits (or a lack) of a closure of the process by which people acquire and thus grasp information from the environment [16]. The need for closure has relevant implications both at individual and also group level [17, 18, 19] were decision making processes are involved.
The antecedents of the motivation towards cognitive closure can be found in those circumstances that highlight the perceived benefits and reduce the perceived costs of closure (i.e., situational uncertainty, time pressure, environmental noise, mental fatigue, boredom of a cognitive task). By contrast, the need to avoid closure may be instilled in those conditions that stress the costs of closure and the benefits of a lack of closure (i.e. accountability, fear of invalidity, evaluation apprehension). Moreover, the need for closure can also represent a stable individual disposition where at one polar extremity individuals may display a systematic proclivity to value closure positively, while, at the other pole, individuals may be predisposed to avoid closure in order to keep the situation open [17].
The need for closure [16] has widely ramifying consequences for social-cognitive phenomena at the intrapersonal, interpersonal and group levels of analysis [20]. Specialists in the matter identify two general tendencies: urgency and permanence. The urgency tendency represents an attitude to attain closure as soon as possible, in this case any other postponement of closure is experienced as troublesome. On the other side, the permanence tendency is the individual inclination to maintain the decision making process as long as possible, it is prevalent in the desire to perpetuate closure, giving rise to “freezing” past knowledge and safeguarding future knowledge.
These basic considerations lead us to identify a first view centred on the “need for closure” as depicted in Figure 1, where circumstances are the physical, environmental and psychological states, that increase the difficulties of information processing.
Figure 1: Antecedents and consequences of need for cognitive closure.
The motivation of cognitive closure may affect the way that individuals process information towards the formation, alteration or dissolution of knowledge. Such processes are typically embedded in a social-interaction context and they may affect significantly the way a person thinks about, feels about, acts towards, and even talks about others.
At a group level, decision making under stress becomes more centralized so that control and decision making is concentrated at higher levels of the group [12]. This centralization of authority in groups operating under stress, and the increased reliance on specific leaders to make decisions, is seen as an adaptive response to threat and uncertainty as it places responsibility in the hands of those that appear to better represent the group as a whole.
Moreover, on time pressure effects, some behavioural data [21, 22] shows that

individuals are less likely to take action as risk levels increase, and that time pressure doesn’t have a uniform effect on choice probability. Under time pressure individuals are more conservative at the lower risk levels but are more prone to take risks at the higher levels. It means in effect that they take more time to make specific decisions at moderate levels of risk and reduce their response time in general across all risk levels.
The performance measurement at individual and group levels, is currently driven by many theories and they all generally give many perspectives on how to approach the different playing situations. There are important implications within the Pandora environment concerning the simulation technique applied in the project [9]. This leads us to consider those frameworks that concentrate upon learning outcomes, defined as “persistent states making possible a variety of human performance”. From the training point of view, these are the steps that have to be negotiated in performing tasks explicitly designed for learning. Literature provides a further distinction about this in learning categories [13] by indicating the different implications for the nature of performance as well as the methods for measurement.
At the level of the individual trainee, the performance theory appropriate for Pandora measurements in SBT (Simulated Based Training) is the Learning Outcomes Framework.
This is where the outcomes of learning drive the effective performance, and different types of tasks require different types of learning outcome (cognitive, such as need for cognitive closure, skill and affective). This requisite in particular works well within Pandora, with a system conceived to monitor performance measures in an SBT environment (i.e. decision making under stress and trainee behavioural responses), by addressing a real time analysis of the learning outcomes involved in the task being performed and including the generation of corrective feedbacks.
At the level of the group and their processes (effectiveness and performance), the library of available literature is vast but the performance theory most appropriate [23] for
Pandora, seems to best fit the topics of the IPO (Input, Process, Output) models. The IPO model seeks to characterize the input variables (i.e. individual and team characteristics), the process variables (i.e. shared cognition, leadership, communication, coordination, decision making, and back-up behaviour), and outcome variables (i.e. performance outcomes, productivity, and satisfaction). By studying the relationships between the IPO variables it is possible to characterise the group work as dynamic and multidimensional as well as capturing the group performance [24] during practice activities even in SBT.
The two theories outlined are valuable tools for understanding performance in a complex training scenario like the one provided by the Pandora system. In order to enhance these outcomes, the critical aspects of performance must be observed and captured during end-user practice activities or, in the case of Pandora, specifically during the execution of those tasks designed to enable the demonstration/evaluation sessions. There are several methods available to achieve this goal including qualitative to quantitative, subjective (e.g., self-reports) to objective (e.g., task outcomes). It is nevertheless important to take into consideration the individual and group level that categorize performance measures in order to correlate them with end-user experience and capabilities (trainees level: basic, medium, advanced) [3].
In general, qualitative methods (e.g., the protocol analysis) seem to be particularly suited for the Pandora task (e.g., drives performance in near real-life world emulation and preliminary understanding) as these are easier to arrange and manage in a fixed site configuration. Conversely, quantitative methods are mostly suited to deep investigation approach when the drive outcomes have already been identified at both individual and group levels of performance. Considering that Pandora constitutes a novel approach to

training for complex situations and that it is actually aiming, at least initially, at a preliminary understanding of “what’s happened?”, it can be concluded that many methodologies from literature (e.g., Behavioural Observation Scales, Communication
Analysis, Event-based measurement, Structural Knowledge assessment, Automated
Performance Recording) are likewise applicable at individual and group levels of analysis.
The Critical Incident Technique in particular, generally used to elicit behaviours tied to competencies targeted for training, can be applied without fear of damaging either people or critical structures.
4.1
– User experience and technology acceptance issues.
As the fixed site deployment of Pandora is hugely ICT equipped and the end-users, namely trainees and trainer, are required to interact in some manner with a personal console specifically designed to monitor the scenario’s events/data flow and communicate each other, further user experience and user acceptance evaluations need to be carried out.
The user experience issues in the literature at least include:
•
Usability of Graphical User Interface (GUI), about that the literature defines specific metrics and techniques in [25, 26, 27].
• Perception of realism , namely the feel of being in the real crisis situation. As already shown in the previous sections, different features of the system are responsible of the realism perception and as well as they may influence the psychological processes in act.
• Perceived quality of the support provided for the system in use. The use of a new technology even though tacitly implies the acquisition of new skills, for the trainer to redesign and deliver the lesson, for the trainees to correctly use the system [28].
As for technology acceptance, the literature points out the UTAUT (Unified Theory of
Acceptance and Use of Technology) model [29], the four main constructs of which are recognised to play a significant role as direct determinants of the use behaviour, especially in the evaluation of contexts highly integrated by emerging ICT [30]:
•
Performance Expectancy , to which extent an individual believes that using the system will help him or her to attain gains in job performance.
• Effort Expectancy , easiness associated with use of the system; this is strongly related to the extent to which a person believes that using the system would be free of effort.
•
Social Influence , the impact other people may have on individual perception of utility.
•
Facilitating Conditions , about technical infrastructures to support the use of the system.
The assessment aims to point out the key features of the Pandora complex environment when viewed through the lens used by the main actors involved, i.e. the trainees and the trainer. The dimension of both individuals and groups within the process of making decisions under crisis or emergency scenario is foremost and central in the adopted human factors approach. The evaluation aims to verify if the underlying hypothesis of the project that Pandora effectively bridges the gap between table-top exercises and real world simulation exercises, providing a near-real training environment at an affordable cost, is validated. If the evaluation proves the concept, then Pandora will serve to increase the effectiveness of the strategic decision making processes used in response to a crisis or emergency situation.
[1] Pandora project website, http://www.pandoraproject.eu.

[2] Marecki, J., Schurr, N., Tambe, M., Scerri, P., (2009), Analyzing Dangers in Multiagent Rescue Using
DEFACTO, Safety and Security in Multiagent Systems, M. Barley, H. Mouratidis, A. Unruh, D. Spears, P.
Scerri, F. Massacci (Eds), Lecture Notes in Artificial Intelligence, vol. 4324, pp 241 -257.
[3] Cesta, A., Cortellessa, G., De Benedictis, R. and Strickland, K. Training Crisis Managers in Strategic
Decision-Making. In ERCIM News, N. 85:47-48, 2011.
[4] Deliverable D1.1 Part II: User Requirements Analysis and State of the Art, Chap 1.1 Civil Protection
Definitions, http://www.pandoraproject.eu.
[5] Deliverable D5.1: The emulated crisis room blueprint, http://www.pandoraproject.eu.
[6] Sayegh, L., Anthony, W. P., & Perrewé, P. (2004). Managerial decision-making under crisis: The role of emotion in an intuitive decision process. Human Resource Management Review, 14, 179-199.
[7] Cortellessa, G., D’Amico, R., Pagani,M., Tiberio, L., De Benedictis, R., Bernardi, G., Cesta, A.: Modeling
Users of Crisis Training Environments by Integrating Psychological and Physiological Data. In: IEA/AIE-1.
Proceedings of the Twenty-fourth International Conference on Industrial, Engineering and Other Applications of Applied Intelligent Systems, Siracuse, NY. Lecture Notes on Computer Science N.6704. 2011.
[8] Deliverable D3.2: Crisis module framework alpha release, http://www.pandoraproject.eu.
[9] Deliverable D4.1: Definition and modelling of the emotion simulation engine including the environment framework design, http://www.pandoraproject.eu.
[10] Celino, I., Dell'Aglio, D., De Benedictis, R., Grilli, S., Cesta, A. Ontologies, rules and linked data to support Crisis Managers Training. In IEEE Learning Technology Newsletter,13(1): 3-6, 2011.
[11] Brannick, M. T., Salas, E., & Prince, C. (1997). Team performance assessment and measurement:
Theory, methods, and applications. Hillsdale, NJ: Erlbaum.
[12] Driskell, J. E., & Salas, E. (1991). Group decision making under stress. Journal of Applied Psychology,
76(3), 473-478.
[13] Salas, E., Rosen, Michael A., Held, Janet D. and Weissmuller, Johnny J., Performance Measurement in
Simulation-Based Training: A Review and Best Practices, Simulation Gaming 2009 40: 328 originally published online 22 December 2008; http://sag.sagepub.com/content/40/3/328.
[14] Driskell, J. E., & Salas, E. (Eds.). (1996). Stress and human performance. Hillsdale, NJ: Erlbaum
[15] Driskell, J. E., Salas, E., & Johnston, J. (1999). Does stress lead to a loss of team perspective? Group
Dynamics: Theory, Research and Practice, 3(4), 291–302
[16] Kruglanski, A. W., Webster, D. M. (1996). Motivated closing of the mind: “Seizing” and “Freezing”.
Psychological Review, Vol.103, No.2, 263-283.
[17] Pierro, A., Mannetti, L., De Grada, E., Livi, S., Kruglanski, A.W. (2003). Autocracy Bias in Informal
Groups under Need for Closure, Personality and Social Psychology Bulletin, 29 (3), 405-17
[18] Livi, S., Mannetti, L., Pierro, A., Kruglanski, A.W., Kenny, D.A.(2005). Epistemic Motivation and
Groups’ Reaction to Change: Effects of Need for Cognitive Closure on Norms Transmissions in a Laboratory
Micro-Culture. CogSci 2005. http://csjarchive.cogsci.rpi.edu/Proceedings/2005/docs/p1308.pdf
[19] Livi, S., De Grada, E., Pierro, A., Mannetti, L., Kruglanski, A.W., Kenny, D.A. Bisogno di Chiusura e trasmissione di norme sperimentalmente indotte in piccoli gruppi . Giornale Italiano di Psicologia. 34, 3, 581-601.
[20] Webster, D. M., & Kruglanski, A. W., (1998). Cognitive and social consequences of the needs for cognitive closure. European review for social psychology, 8, 133-161.
[21] Dror, I., E., Busemeyer, J., R., Basola, B. (1999). Decision making under time pressure : An independent test of sequential sampling models. Memory & Cognition. 27 (4), 713-725.
[22] Finucane, M.L. (2008). Emotion, affect, and risk communication with older adults: challenges and opportunities. Journal of Risk Research. Vol.11, No.8, 983-997.
[23] Salas, E., Stagl, K. C., & Burke, C. S. (2004). 25 years of team effectiveness in organizations: Research themes and emerging needs. International Review of Industrial and Organizational Psychology, 19, 47-91.
[24] Guzzo, R. A., & Dickson, M. W. (1996). Teams in Organizations: Recent Research on Performance and
Effectiveness. Annual Review of Psychology, 47, 307-338.
[25] ISO/DIS 9241-11 (1998), Guidance on Usability. International Organization for Standardization.
[26] ISO 9126-1 (2000), Product quality Part1: Quality model. International Organization for Standardization.
[27] Bevan, N. and Macleod, (1994) Usability measurement in context, Behaviour & Information technology,
13,132—145.
[28] Gnisci A., Papa F., Spedaletti S. (1999), Usability Aspects, Socio-Relational Context and Learning
Performance in the Virtual Classroom: A Laboratory Experiment, Behaviour & Information Technology,
Taylor & Francis 1999,Vol. 18, N.6,pp. 431-443.
[29] Venkatesh, V., Morris, M. G., Davis, G. B., Davis, F. D., (2003). User Acceptance of Information
Technology: Toward a Unified View, MIS Quarterly vol. 27 No. 3, pp. 425-478, September.
[30] Cornacchia, M., Baroncini, V., Livi, S., (2008). Predicting the Influence of Emerging Information and
Communication Technologies on Home Life; J. Filipe and J. Cordeiro (Eds.): WEBIST 2007, Revised
Selected Papers, LNBIP 8, pp. 184-200, Springer-Verlag Berlin Heidelberg, 2008.