TITLE
Crisis managers making strategic decisions in the near real-life training environment of Pandora: evaluation
framework for human performance and user experience.
<Michele Cornacchia, Filomena Papa, Stefano Livi, Bartolomeo Sapio, Enrico Nicolò, Keith Strickland>
Conference: eChallenges 2011, 26-28 October 2011, Florence, Italy
Track: Technology enhanced learning and ICT Skills
INTRODUCTION
This paper describes the general evaluation framework to be adopted in the FP7 Project Pandora, a near
real-life training environment for learning activities suitable for crisis scenarios. The main goal here is
focused on setting up the most adequate framework to evaluate the performance of different levels of
end-users, namely the single trainees, the group of trainees and the trainer. The Pandora environment is a
very complex innovative system, providing its end-users with many technological supports for single and
collaborative strategic decisions making. It responds to the needs of the traditional business continuity
plans that do not adequately take to into account the many forces of human behaviour of individuals and
groups as they are shaped by numerous intangible factors. Whether or not, at the end of a learning session,
the trainees feel a perception of increased performance and/or self-efficacy about facing potential crisis
situations, is the final purpose of the evaluation. Furthermore the user experience and the ICT acceptance
of individuals moving and acting in a very innovative system which provides and takes into consideration
also emotional responses, is also object of evaluation. In particular on this topic, regarding the work
conditions under time pressure and/or the difficulties in information processing, group decision making is
approached in terms of cognitive closure, the somehow irrational mechanism to eliminate ambiguity and
reach definite conclusions.
OBJECTIVES
As previously introduced, the final performance in training sessions of trainees, group and trainer
represents the final goal of this evaluation project. It basically applies a human factors approach which puts
specifically in the centre of the consideration the end-users point of view, needs and expectations in
relation with the complexity of Pandora. The main dependent variables chosen to describe such a
performance are recognized in literature mostly from the models of user acceptance, user experience and
managerial decision making under crisis. The effectiveness of training on the end-users’ side first and then
some important process variables developed by the Pandora system (behavioural tendencies, crisis
scenario, user profiling, emotional status, environmental conditions, physical set-up, didactical goals,
recording, etc.) influencing the group decision making are supposed to give a wide-ranging accurate idea of
whether or not learning is perceived like in a near real-life training environment and whether or not
trainees feel themselves pragmatically more experienced when sessions are over. The effectiveness of
training is therefore an objective measure of the learning performance, it is the extent to which crisis
managers achieve a set of learning goals and apply procedures. Moreover, goals of evaluation are
considered the matching between performance reports (automatically produced by the Pandora system)
and self-evaluation, the quality of interaction during the training, the interpersonal or mediated
communication among actors playing in the session (trainer to trainees; trainees to trainees; trainees to
possible lower tactical level).
METHODOLOGY
Given that a scenario is the crisis or the emergency situation created by the Pandora system for the training
session, the applicable methodology has to distinguish between three use cases for the potential
deployment of the environment: the “single site training”, where trainees and trainer are in the same room
with full technological capabilities; the “deployed training”, where trainees and trainer are in the same
room but with reduced technological capabilities; the “distributed training”, where trainees and trainer are
physically in different sites but share the same virtual environment.
It is almost evident that the first two training configurations are very similar out of technological
equipment, and that the third one differs consistently as it works in a virtual setting. It is then reasonable to
expect close results from the first two and a significant distance of the last one, if we consider the variables
model previously shortly mentioned. In this model the hypothesized differences existing between objective
and perceived results, either achievable by direct measurements or by Pandora system itself, are
modulated in accordance with the individual, group and trainer performances. In general it is possible to
introduce the classification of “antecedent” variables and “process” variables in order to collect either
objective or perceived sets of measures of effectiveness. For example, the cognitive “managing” and the
physiological “readiness” are both antecedent variables and referred to individual; the perception of
cognitive closure about having processed all information is a process variable, possibly applicable to both
individual and group; self-efficacy and usability can be referred to the individual; social loafing in decision
making to the group; information available and group dimension to the group; even the trainer attendance
to the session may be considered as objective process variable; and so on.
In general the three use cases set out for Pandora bring to require two different methods of investigation
to access the expected results of improved performance in decision making and increased sense of
confidence likewise said self-efficacy. On this purpose, preliminary studies (interviews to designated users,
control group definition, in-between testing to single modules) can be applied to best fit the standard
methods and tools of the human factors, usability and social research with the variables model of
assessment. It is nonetheless crucial to set up the method with the schemes of qualified models like those
provided for the emerging ICT acceptance, the tele-presence, the cognitive closure and managerial group
decisions making in condition of high uncertainty.
TECHNOLOGY DESCRIPTION
Pandora is a special training system that provides appropriate metrics on the performance of a manager
actively engaged in the strategic management of a crisis. The environment provides a realistic and
complete scenario with near real-time action, coherent with that expected in a real-world situation.
Realistic emotional statuses, through affective inputs (emotional statuses: fear, irritation, joy) and stress
factors (pressure to decision, fragmentary information, short time to decide) are represented to different
crisis managers belonging to different sectors.
The main innovative aspects are in the realism and complexity of the simulated model. It can analyse
behavioural aspects and report them during the training, it can model the crisis scenario and its evolution,
it can create the user experience up to the reproduction of emotional status, it can create multiple
interconnected training procedures, it gives a retroaction enabling the trainer and the trainees (through
their decisions) at modifying the scenario on runtime, and other.
The architectural system is composed of few main modules. The Behaviour Simulation and Modelling aims
to create and update a profile for each trainee of features and actual behaviour as well as to provide
indications on how to personalize the training itself. The Crisis Simulation and Modelling aims to combine
the information coming from the previous sub-system and the knowledge about the critical, infrastructure
domain in order to create a crisis scenario evolution to be simulated. The Environment and Emotion
Simulation aims to effectively represent Non–Player Characters and environments for the simulation. At
last, the Crisis Room is the place where the exercise really happens. As previously mentioned, it can be
configured as complete in a fixed site, portable in a deployed site, or web distributed, in a virtual reality,
according to the selected use case.
CONCLUSIONS AND RECOMMENDATIONS
The three use cases of the Pandora managerial training system for crisis scenario are here analysed in order
to arrange a plan (the project ends on 31st December 2011) of performance evaluation by the end-users.
The assessment aims at pointing out the major features of the Pandora complex environment by the
viewing angle of the main actors involved, that is the trainees and the trainer. The dimension of both
individuals and group specifically in the process of making decisions under crisis or emergency scenario is
foremost central in the adopted human factors approach. The evaluation aims as well to verify if the
project underlying hypothesis can be accepted, that is to which extent Pandora effectively bridges the gap
between table-top exercises and real world simulation exercises, providing a near-real training
environment at an affordable cost.