Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
P
FTA
1
2
3
1
4
1
European Commission, JRC-IPTS, Seville, Spain, karel-herman.haegeman@ec.europa.eu
and elisabetta.marinelli@ec.europa.eu
2 European Commission, JRC, Brussels, Belgium, fabiana.scapolo@ec.europa.eu
3 ISIS, Rome, Italy, aricci@isis-it.com
4 Institute for Statistical Studies and Economics of Knowledge, Higher School of Economics, Moscow, Russia, sokolov@hse.ru
Keywords: Qualitative, Quantitative, barriers to combining methods, FTA
1
While the FTA, as a label, has been trying to bring together different disciplines that try to understand and shape the future, the divide between quantitative and qualitative practices in the field remains to a large extent unsolved. Yet, current trends in FTA combined with the increasing policy demand for robust evidence for decision making indicate that there may be a momentum for pushing forward the field in the direction of more methodological integration, increasing its relevance for policy, business and society. There is however a wide set of potential barriers to such more advanced mix of approaches. As this paper suggests, most of those seem to mainly relate back to epistemological, skill and trust issues.
Based on this assumption, a way forward is proposed in trying to overcome those barriers, which is translated into a research agenda and a list of policy priorities. The paper proposes to combine the shortterm treatment of ´symptomatic´ barriers with longer-term research and policy priorities that address the more fundamental barriers to method integration. The authors of this paper invite experts from within and beyond the FTA community to further discuss and complement some questions that are put forward regarding each of the above issues.
1
The authors would like to thank Ben Gardiner for commenting on a draft version of this paper.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
The methodological debate has been a relevant element of the FTA conference since its first edition in 2004. Whilst research has still plenty to explore about it, the range of methods and techniques has grown remarkably, offering opportunities for more tailored approaches, which benefit from each method’s particular strengths.
A key aspect of the methodological debate remains the combination of qualitative and quantitative methods, which is promising and controversial at the same time 2 . On the one hand, the combination of qualitative and quantitative approaches has the potential to produce rounded results, bringing together disciplines, within the FTA community, that have contributed differently to forward-looking studies and practices. On the other hand, whilst the synergies of the two approaches are acknowledged, some obstacles still prevent their full integration. This paper deals precisely with this conundrum and takes stock of the current situation and practices and identifies the barriers and the scope of a more extensive combination of methodologies.
Traditionally, FTA practitioners, and especially Foresight practitioners, have mainly concentrated on methods based on expert judgments (which generally presumes qualitative assessments except for Delphi methods, that produce quantitative assessments), with quantitative approaches used to provide background or introductory information for experts.
Quantitative methods were judged as not accurate in the long-run, due to their inability to cope with uncertainty and disruptive events (a crucial factor for FTA), to the unavailability of adequate data, and to their failure to correctly predict outcomes even within shorter horizons 3
.
On the other hand, a significant part of the community holds opposite points of view, whereby qualitative approaches should be considered as a second best option, to which we are somehow compelled to refer while we wait for adequate quantitative methods to extend their scope and power of analysis in those areas that are currently “non quantifiable". In other words, a (cultural) clash is in place between those coming from a quantitative vs qualitative school which, as will be clear throughout the paper, is hard to resolve.
Nonetheless, there is an emerging consensus which calls for the need to integrate both dimensions, for example, through a cross-checking of assumptions and findings of the same analysis carried out in parallel with a qualitative and a quantitative approach, or through the use of different methods/tools within the same foresight exercise (typically, using quantitative models for shorter time horizons, and qualitative approaches for longer ones, where storylines present higher credibility). Current trends indicate that such deeper understanding is becoming increasingly possible: the fast progress of the internet, the higher availability of
S&T and innovation indicators, the advances in quantitative methods (see Lombardo (2006) for an overview), the increasing demand for higher transparency and for more inclusive participatory approaches 4 , coupled with an increasing policy demand for robust evidence for decision making, may all facilitate better integration of quantitative and qualitative methods.
2 Needless to say, the choice of methods is inevitably affected by data availability
3
The most famous example is Jay Forrester’s model of the World dynamics (Forrester, 1971), which included variables such as population, pollution, economic growth and natural resources and predicted a global catastrophe which luckily has not come true.
4
Both new and established forms of governance are under scrutiny for more transparency and accountability, and within this context, there is an increasing demand for stakeholders' (and citizens) participation. Therefore, if FTA is to contribute to achieving new forms of participative governance, the way methodological transparency and robustness is achieved, is clearly very important. Notably, foresight may (and should) help in (i) imagining new, highly participatory, governance models, notably based on the extensive use of new and emerging functionalities enabled by on-going and future technological progress, and (ii) assess the impact of such new governance models on society at large.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
At the same time, the FTA community is aware that much more needs to be understood and research on this is needed in order to correctly combine different approaches, as many barriers still exist that impede a full integration of methods. In this paper we try to list and better understand some of these barriers, how some of these can be traced back to epistemological differences and how some other barriers are simply related to lack of skills and trust. To overcome this situation the paper proposes a way forward, addressing both short-term barriers (i.e. practical impediments to methodological advances) and longer-term ones (referred largely to the more fundamental epistemological differences between those looking at the future from a quantitative and qualitative point of view).
Although such methodological reflections originate from the inside of the FTA community, they are addressed to a broader audience. At the core of the paper, indeed, lies the awareness that other branches of social sciences, which have so far not been integrated in the community, could and should become part of it. It is in this sense an invitation to outsiders to join and share their views on this topic. To this end, each section concludes with some questions for further discussion during the conference. Some of those are also suggested to be taken up as further research priorities (section 5).
The paper is organised as follows: section 2 outlines briefly the different views within the
FTA community and specifies more the focus of the debate. Section 3 takes stock of the current level and practice of methodological combination, showing that, whilst there are signs of increasing methodological interaction, progress is still very slow. Section 4 digs further into the causes of the low (or slow) combination of methods, exploring the different barriers to mixing FTA methods. Section 5, identifies a research agenda and policy priorities that can help overcome those barriers. Section 6 provides an inroad across recent developments and applications of FTA methods, and of other discipline entering FTA that can provide a contribution in dealing with present and upcoming societal challenges. Finally, the last section highlights the opportunities that the FTA conference provides to move the debate forward.
Bridging qualitative and quantitative techniques in studying, shaping and anticipating the future has always been implicit in the FTA label, which encompasses communities with very different methodological foundations. These include (technology) foresight, which traditionally relies mostly on qualitative methodologies, technology assessment, and
(technology) forecasting, which relies predominately on the quantitative end of the methodological spectrum (see Cagnin and Keenan, 2008) 5 .
Although this may be seen as an oversimplification, the debate on the differences, commonalities and synergies between foresight and forecasting, or between quantitative and qualitative research and analysis methods, share many common features. To define the scope of our paper it is therefore important to first define clearly what we mean by FTA and by quantitative and quantitative methods.
Often FTA is understood to be equal to and limited to (technology) foresight. This is probably because the majority of the contributions to FTA conferences until today have come from the
5
Impact assessment also has a foresight component and could therefore also be considered to be (partially) part of FTA.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011 foresight community.
6 In this paper we understand FTA to be the umbrella term for the techniques applied by the different communities described by Cagnin et al (Cagnin and
Keenan, 2008) and introduced above. Keenan and Popper (Keenan and Popper, 2007) tried to define more concretely how FTA can be distinguished from other policy-support techniques, using six distinctive principles: future-orientation, participation, evidence base, multidisciplinarity, coordinated mobilisation of people and resources, and action orientation .
As not all techniques that are used by the communities under the FTA umbrella comply to the same extent with all six principles, those principles can also be used as criteria to measure the degree to which a technique can be considered to be part of the FTA toolbox
(Haegeman et al, 2010).
Quantitative and qualitative methods
In order to distinguish qualitative and quantitative methods, two main axes can be identified: o Type of data . Whereas quantitative methods rely on numeric information, qualitative ones mainly deal with data of different nature, including collective knowledge based on expert judgment o The analytical objectives . Quantitative methods deal with numerical measurements and combine empirical, deductive and experimental studies aimed at obtaining ‘objective’ results. They are applied to test hypotheses, identify numerical differences between groups, etc. Qualitative methods aim to explore meanings and deal with how experts understand present and future issues on the base of their knowledge, experiences and accept subjectivity. The objective nature of quantitative methods is however not absolute, as we will discuss later in the following sections of this paper.
Whilst there is an apparent interest in the methodological evolution of FTA, such interest is not fully consolidated in methodological guidelines or systematic reviews of the relevant literature. Although there have been many efforts on classification of methods and on the possible combination of qualitative and quantitative approaches either in handbooks or online tools 7 for the different stages of an FTA exercise (Eerola and Miles, 2011 and Saritas,
2007; Popper, 2008), the literature provides little evidence on systematic comparison and analysis about the consequences and outcomes of applying specific techniques in the course of FTA (Scapolo and Porter, 2008)
8
.
This absence of stock-taking analysis is also mirrored in the lack of guidance on how to evaluate foresight studies that combine the
6
Within this community foresight is also sometimes seen to be the wider umbrella that includes all methods related to anticipation. At the European Commission also the term Forward-Looking Activities (FLA) has already been used as an umbrella for foresight and forecasting (European Commission, 2010).
7 http://forlearn.jrc.ec.europa.eu/guide/0_home/index.htm
8
The EFMN (European Foresight Monitoring Network) conducted one of the few attempts to take stock of quantitative and qualitative foresight methods (the project is now continued by EFP, European Foresight Platform). The mapping identified only three quantitative methods (bibliometrics, modeling and simulation, trend extrapolation), and highlighted that they were most often combined with literature review, scenarios and expert panels. These conclusions however have to be interpreted with care, as there are sources of bias n the EFMN database (see Keenan and Popper (Keenan and Popper, 2007) for a critical assessment of the exercise). Remarkably, a similar mapping activity of cases from the forecasting communities is not available. The field of forecasting is however older and has reached a higher level of maturity, with advances in forecasting regularly being reported in several ISI indexed journals. The International Institute of Forecasters, which aims to develop and further the generation, distribution, and use of knowledge on forecasting, was founded in 1981, and many influential publications related to forecasting are even much older. Evidence stemming from the forecasting communities on cases combining qualitative and quantitative methods is rather limited, which could suggest that the need for combining methods may be less felt than in the foresight communities, which are younger and may still feel the need to prove the usefulness of foresight.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011 different methodologies. Yet, evaluation on methods is important in order to promote 'good practice' and to contribute to the questions of quality of FTA, in terms of process, content, and impacts (Scapolo and Porter, 2008).
Despite this lack of systematic investigation, there are clear signs of interest from the FTA community in mixing methods. A very preliminary review of the literature, allows us to identify some patterns of how qualitative and quantitative methods and data are combined in FTA: o Integration of methods with respect to bibliometric analysis of scientific papers and patents for FTA . Bibliometric studies have been widely used as a valuable source of data for FTA. They allow identifying emerging research fields that can offer disruptive technologies in 5-10 years ahead. The bibliometric analysis of frequency of keywords in scientific publications can be also used as a background for innovation forecasting models (see Cunnigham and Kwakkel (2011)) 9 . o Visualization of trends 10 .
Visual presentation of quantitative data provides experts with a powerful instrument to identify hidden trends on the basis of evidence based data, to monitor the convergence of adjacent technologies (Curran and Leker (2011), or to identify emerging S&T areas (see Lee et al (2010). The expert judgment based on the results of modeling can also contradict the results provided by the models themselves. o Quantitative models based on expert judgment. Any quantitative model has to be designed with the use of expert knowledge, if not its practical value will be very limited.
However, there is a wider room for the use of expert judgments as it allows increasing the outcome of quantitative models 11 . o Computer assisted methods.
Growing amounts of available detailed data related to S&T and permanently increasing performance of computer networks have been widening the background for new applications of quantitative methods 12 .
The slow progress in integration of methods in FTA lays down a question on possible barriers towards such integration, which we try to answer in the following sections.
The main barrier that needs to be overcome for a full methodological integration is epistemological in nature: there is a long-standing debate in social sciences (which is not confined to the FTA community) on the type of knowledge that qualitative and quantitative
9
For example, Shibata et al (2011) try to distinguish between incremental and radical innovations identification of emerging clusters on the basis of quantitative analysis of citations and keywords for a particular technology field. In this case the results have created a background for further qualitative studies. Furthermore, Jarvenpaa, Makinen and Seppanen (2011) analyse how bibliometric data can be used for distinguishing between groups of technologies that do and do not follow the linear model of innovation, which gives experts an opportunity to focus on particular technology areas using relevant qualitative methods.
10 This is well illustrated by R. Popper in Georghiou et al (2008).
11 Cunningham and van der Lei (2009) demonstrate how such an approach can be used for models providing support to making decision on selection of new technologies and discuss the issue of providing equilibrium between different groups of experts and stakeholders. Agami et al (2010) show how fuzzy logic can be used for trend impact analysis combining
“surprise-free” forecasts and consensus based experts’ judgments. Quantitative methods can also be used to process qualitative judgments for different purposes (scenario design – see Kemp-Benedict (2010), diagnosis of strategic trends –
Liebl and Schwartz (2010).
12 Thorleuchter et al (2010) demonstrate that the patent-based quantitative approach to cross-impact analysis for the identification of relationships between technologies can be used instead of, or in combination with, traditional qualitative methods based on literature reviews. Another application of computer-based methods is a participatory approach to scenario discovery (Bryant and Lempert (2010)).
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011 methodologies can produce and on the value of combining them 13
.
Suffice here to say that there is a strong opposition between those who argue that quantitative and qualitative approaches are incompatible (i.e. Sarantakos, 1993; Silverman, 1993) and those who argue the opposite (i.e. Sayer, 1992 (orig. 1984); Lawson 1996; Jick, 1979a; Olsen 2004; Howe,
1988, 1992). The latter group has highlighted that qualitative data and analysis are the glue that cements the interpretation of multimethod results (Jick, 1979a) or that triangulation (i.e. mixing of methods) specifically in social research is not aimed merely at validation but also at deepening and widening one´s understanding (Olsen, 2004); But despite these and many other 14 longstanding attempts to mix different types of methods, methodological combination is not the norm. The relevance of this divide to integration of FTA methods lies in that it can reveal possible barriers that go beyond FTA, and stem from more fundamental barriers between quantitative and qualitative approaches. Box 1 puts forward some research questions related to the more general barriers to mixing methods in social sciences, which may also have relevance to mixing methods in FTA.
Questions related to barriers to mixing qualitative and quantitative methods in social sciences, which may have relevance for the debate on mixing FTA methods o Do researchers themselves have an interest in integrating methods and findings?
15 o What is the impact of educational systems on the qualitative-quantitative divide?
16 o Do epistemological beliefs play a role in the field of study that students choose?
17
A layer of complexity is added to this methodological debate in the case of FTA. When it comes to FTA, the "future" is the object of analysis, and there is confusion as to what can be known about it. Until today the communities within FTA – foresight / forecasting / technology assessment / futures communities – are still perceived as independent communities that are competing rather than collaborating in looking at the future. In the context of qualitative and quantitative techniques, especially the marrying of quantitative modelling and foresight seems to be poorly developed. The idea that one can forecast or predict the future seems to be contradicting with the idea that it can be (jointly) shaped using foresight. Put it differently, the question is: can we know the future? Eerola and Miles (Eerola and Miles, 2010) look at
13 This is a longstanding debate. The interest reader is referred to Denzin, 1978; Webb et al (1966) and Jick T. D. (1979)
Campbell and Fiske (1959) and Campbell (1953, 1956), just to mention a few.
14 See e.g. Tashakkori and Teddlie, 1998; Johnson and Onwuegbuzie, 2004; Băban, 2008; Castro et al, 2010.
15 Bryman (Bryman, 2007) found indications that even in projects combining quantitative and qualitative methods data are not always brought together in the analysis. He sees a possible explanation in that researchers often do not intend to integrate findings, just because different data types often answer different research questions. In fact one often observes that, even those foresight exercises/projects that deliberately aim at mixing qualitative and quantitative approaches are mostly structured and organized in a way that does not facilitate truly interdisciplinary elaboration: qualitative and quantitative tasks are carried out by different teams (typically, different work packages in the EC Framework Programmes jargon), with some kind of bridging mechanism being then pursued. Rather than attempting to mediate/reconcile the two approaches ex post, integration between the two cultures should be promoted at the outset, creating multidisciplinary teams that interact since the very early stages of the foresight process.
16 Not only at the university, but also at earlier stages in education we are asked to make a choice between different options
(such as the divide between social, formal and natural sciences), that often implicitly also entails choosing for a more quantitative or qualitative training. In this regard Reiss and Bryman refer to the need for overcoming ´trained incapacities´
(Reiss, 1968; Bryman, 2007).
17 The research of Trautwein and Lüdtke (Trautwein and Lüdtke, 2007) explored why students in hard sciences tend to exhibit less sophisticated epistemological beliefs than students in social sciences. Their results support both the perspective of self-selection processes (students who believe more strongly in the certainty of scientific knowledge choose to study hard sciences) and the perspective of socialization effects (social sciences help students to acquire a critical stance as regards the
“truth” of scientific theories). Also, and despite the significant overall progress achieved in this area, one should not ignore that a significant share of students are still driven in their curricular choices by some form of “fear of maths”.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
FTA from this knowledge perspective, by arguing that knowing the future is about collecting knowledge about posits or possible future, by examining their plausibility and limits, their internal consistency and conformity with models and data, the extent to which they are consistent with expert judgement, their implications for action, etc. A wide variety of types of information about the past and the present can contribute to this and what is crucial is to understand the opportunities that different methodologies offer.
But understanding these opportunities seems not very straight forward. The historical difficulties for qualitative and quantitative approaches to FTA to collaborate effectively reflect deeply ingrained cultural differences that hinder good communication at the outset. Efforts to address this cultural clash have often focused on the adaptation of the methods and tools commonly used by the two communities, in the hope that adequate interfaces between the two could be found and thus facilitate operational collaboration. Indeed, such attempts have borne valuable fruits, such as e.g. through the increasing use of indicators as a possible bridge between models and storylines. When it comes however to promoting a new FTA culture that would inherently mix qualitative and quantitative approaches, such attempts have not been effective. Rather, one could paradoxically observe that the emergence of structured interfaces between narratives and numbers has led to the consolidation of the historical practice that sees the two communities working in isolation: each can safely remain in their own cultural realm, so long as a mechanism has been identified to ensure some level of communication.
It is our opinion that, to evaluate costs and benefits of different methodologies, some implicit misconceptions within the FTA community need to be addressed. Firstly, it is often assumed that models belong exclusively to the quantitative domain and have objective predictive power. Models are a simplified representation of reality and can be quantitative or qualitative, depending on the type of data they rely on. They may generate, as an output, informed estimates about the future. The latter, in the case of quantitative models, take the form of numbers with associated probability distributions, confidence intervals, etc. (depending on the model). However, regardless of the nature of the model, in the case of long-term horizons, such informed estimates have only limited predictive value. As clarified by Lüdeke
( Lüdeke, 2006), the value of models is not so much in their ability to tell us with a degree of certainty what will happen to society, rather in their ability to structure thinking on the bases of the information available, in other words a model is more valuable as an analytical rather than a predictive tool. This means that both quantitative and qualitative tools and techniques should be evaluated not so much against the accuracy of their prediction on the future, but against the assumptions upon which the model relies, the data (advantages and limitations) that has been used to devise the model, the alternatives (or lack thereof) among which the researcher had to choose in forming the model. In other words, when dealing with issues surrounded by risk, uncertainty and ignorance (as, typically, FTA does) the value of models is (at least) as much in the “process” as in the “output”.
Another common misconception associates “subjectivity” and “value judgement” to qualitative processes, but not to quantitative ones. Often, scientists (particularly hard scientists and technologists) tend to consider subjectivity, i.e. experts opinions, as a
“disturbance” that should be avoided or at least framed and assessed in order to reduce its distorting effects on the interpretation of evidence and its decision making powers. In other words, subjectivity is equated to randomness. There are two weaknesses in this point of view. First of all, regardless of the quantitative or qualitative nature of the approach followed, in any FTA exercise, there is value judgement involved. The distortion of such subjectivity can be reduced by the legitimacy of the subject making the judgement (be it the modeller choosing a functional form or an indicator, or a foresight practitioner identifying the right
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011 stakeholders to invite to the workshop). Secondly, and more to the core of the FTA and foresight community, when addressing the intrinsically uncertain challenge of devising possible futures, subjectivity, intended as expert/stakeholder opinion, is effectively a resource and as such should be fully and adequately exploited as one of the foresight aims is in general to envisage a wide and contrasted set of options that can be devised by eliciting experts knowledge. Whether one or the other of such options will eventually materialise will ultimately depend, among others, on the actions, decisions and behaviours of those who will forge the future, largely reflecting their individual and collective (evolving) preferences.
Capturing the subjectivity of FTA contributors, i.e. their valuable yet uncodified knowledge, is therefore of the essence.
18
Finally, the third misconception that needs to be addressed is the tendency to equate qualitative with participatory. This is clearly inaccurate, as participatory methods can (and in fact are) adopted in the framework of quantitative analyses as well, while, on the other hand, qualitative approaches have been adopted for many decades (e.g. descriptive scenarios) with no other involvement than that of the foresight experts. Acknowledging these aspects is essential to identify a common ground to develop mixed methodologies further.
The aforementioned epistemological barriers are reflected in and reinforced by the lack of researchers, practitioners and evaluators skilled in both quantitative and qualitative FTA approaches. This lack of skills to understand and interpret both approaches leads to lack of trust among practitioners of each approach. It is for instance not common for those who come from a strongly quantitative perspective to communicate the core of their work to anyone outside their narrow community. Such lack of trust again reinforces existing epistemological barriers, as illustrated in the graphic below.
Epistemological barriers
Lack of trust Lack of skills
This link between lack of skills and lack of trust does not only occur within the FTA community, but also outside of it, as it undermines trust in FTA methods by misplacing expectations of decision makers who should ultimately use the outcomes of FTA. For example, many policy and decision makers are eager to receive forecasts and predictions, which the FTA community is not set out to provide. Some additional causes for lack of trust are the lack of transparency of the foresight process, in terms of the methods and tools adopted, but also of the quality and clarity of the communication of the results, inappropriate foresight practice and misuse of specific methods and tools (e.g. models being used beyond the time horizon for which they have been originally conceived).
18 A good example is the contribution that FTA can provide to policy and decision makers in charge of the prioritisation of alternative technological options. The long term prospects of emerging technologies are usually scrutinized through the lens of e.g. Cost Benefit Analysis, or similar tools that require the valuation (or at least some form of quali-quantitative estimation) of a variety of factors including notably the social costs and benefits that can be expected from the diffusion of the technologies under scrutiny (how will their diffusion affect quality of life, the conservation of natural resources, landscape integrity, ecosystem services etc.). In the absence of market values for such goods and services, valuation practices commonly recur to Contingent Valuation Methods (CVM), which aim at eliciting information on the hypothetical dynamics of future preferences (individual and collective). The appraisal of the expected future performance (and the ranking) of alternative technological options therefore explicitly incorporates information that inherently reflects the subjectivity of social players, and its value and credibility thus enhanced.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
In literature many additional barriers have been identified in relation to the integration of qualitative and quantitative data and methods, many of which seem to be related to a certain degree to the epistemology-skills-trust triangle 19 . Box 2 contains some further questions as regards barriers to further integration of FTA methods.
Questions related to specific barriers in mixing qualitative and quantitative FTA methods o Do misconceptions between communities increase the espitemological barriers, or the other way around?
o Is there in fact an interest from all communities under the FTA umbrella and beyond in further integrating methods and practices? o Which other barriers exist towards integration of FTA methods (whether or not related to epistemological differences)?
In the next section, a way forward is proposed in trying to overcome those barriers, by proposing a research agenda and a list of policy priorities, taking into account the barriers described above.
This paper proposes to take a dual approach, combining the short term treatment of
´symptomatic´ barriers with longer term research and policy priorities that address the more fundamental barriers to methods integration.
One way to extend the application of quantitative methods in FTA is gradual integration of them into existing practices of FTA and their convergence with qualitative techniques. This gradual approach can relate to specific areas where combining FTA methods looks promising (e.g. scenario development and ex-ante impact assessment), or to specific sectors where quantitative approaches are traditionally stronger represented, such as energy 20 , transport, climate change, etc 21 . Below we sketch an agenda to overcome the main shortterm barriers
Incompatibilities of methods
The selection of methods in FTA remains largely an issue of choice that is context driven, as there are no 'recipe books' or attempts to better clarify the relation between context, content
19 For example, based on interviews with social scientists Bryman (Bryman, 2007) identifies eight barriers to integration of qualitative and quantitative data and methods, which can also be applied in the field of FTA: perceptions on the expectations of different audiences, methodological preferences of the (mixed methods) researcher, structure of the research project, different timelines for different method types, skill specialisms, the nature of the data, ontological differences, tendency of some journals to emphasise one type of research, lack of well-known best-practices.
20 Rossetti (Rossetti, 2010) suggests that the relatively stronger support to models in the field of energy may be due to the fact that energy is on the one hand sufficiently simple to be modeled but also too complex to be studied without any formalisation.
21 An example of such approach in the field of energy is the Energy Foresight Network, where both qualitative and qualitative methods were applied, aiming at building consensus on long term policy options identifying feasible pathways that combine short-medium term policy constraints with long term vision and implications as well as validate current policy options. ( www.efonet.org
).
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011 and approach of a FTA study. This is all the more the case when mixed methodologies are applied and where it is important to identify correctly the interface between qualitative and quantitative methods, highlighting their complementary and/or contradictory features. As highlighted in section three, it is hard to move the methodological debate forward given the uncertainty within the FTA community as to what is the methodological state of the art.
As a natural first step to tackle this issue, one could consider the following short term research agenda, in order to address incompatibilities 22 that hamper the practical combination of different methods:

Look at areas where convergence of quantitative and qualitative methods might bring potential benefits in the framework of FTA, such as scenario development combined with ex-ante impact assessment 23 , planning FTA Studies 24 , designing advanced models of macroeconomic forecasting 25 , anticipation of disruptive technologies 26 , use of quantitative methods for creation of a background for expert judgments 27 , forecasting in a technologically complex environment 28 , future market assessment 29 and the estimation of forecasting bounds of confidence and uncertainty.

Use new methods that are entering FTA to increase synergies between qualitative and quantitative approaches but also to strengthen the discipline (see section 6)

Develop some guidelines as to when it is appropriate to use quantitative vs qualitative.

Clarify the scope of different approaches (which method is more suitable for the shortterm/long-term).

See what is done in other fields about combination of methods (see sectoral studies, or other disciplines).

Analyse interfaces between qualitative and quantitative methods.
Misconceptions
In order to avoid getting stuck in the circle of epistemological barriers – lack of skills – lack of trust, avoiding misuse of methods and increasing transparency between all parties regarding
22 Addressing practical incompatibilities may also gradually change researcher preferences for specific methods in favour of combined approaches.
23 In the context of EU-policy-making, Valette (Valette in Rossetti, 2010) sees opportunities for foresight exercises that combine “ expert-based contrasted socio-economic and policy scenarios (qualitative part) and a mathematical quantification of the impacts of the alternative scenarios (quantitative part)
”. Also in regional cohesion policy some see possibilities for blending quantitative and quantitative approaches in ex-ante (and ex-post) impact assessment.
24 Use of quantitative methods, such as patent analysis, identification of emerging research fronts on the basis of citation analysis, etc has already proved its high efficiency for planning and implementation of FTA studies. It gives valuable information, for instance, for development of S&T Delphi topics, or identification of potential disruptive technologies that can create new markets or significantly change the existing ones. The same methods are helpful while identifying key experts in particular S&T fields (the most cited researchers, active inventors etc.).
25 Traditional economic models could be improved by taking into account and integrating some expert judgments with respect to evaluation of contribution of future technology development to economic growth (at the regional, national and/or global levels).
26 Combination of expert analysis and quantitative models can contribute to one of the key areas of FTA – identification of emerging disruptive technologies. It can be based on the integration of mathematical models providing wide-ranging semantic data-mining aimed at identification of potential properties of existing or emerging technologies that might be most relevant for future markets and expert evaluation of potential demand for technology-based innovations.
27 Identification of “outliers” (outstanding observations) by quantitative approaches could be further revised by experts as potential “wild cards”. Taking account of fast growing amounts of data available and new techniques of semantic search, it can bring unprecedented stock of knowledge to be analysed by traditional qualitative methods;
28 Quantitative modeling of life cycle of given technologies and assessment of their physical limits based on presumption of their incremental development can be integrated with identification of key drivers changing the contexts of technology application (see Garud 2008, pp. 163-164).
29 Modeling key characteristics of a particular technology can inform experts’ judgement on potential markets development.
Micro-level models could be developed and adjusted for the assessment of trends in particular technology fields. This information is of particular importance for roadmapping and some other qualitative methods. The synergy of both types of methods could also be used for anticipating emergency of new markets on the basis of analysis of customer preferences in product properties vis-à-vis foreseen technology development.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011 underlying assumptions and characteristics of both quantitative and qualitative approaches, it is required to increase not only the intrinsic ability of FTA to contribute to policy making, but also to gradually gain policy makers’ legitimacy and trust. Given the growing need for participation in decision making with the increasing policy demand for evidence-based options and their expected impacts, this is not a trivial aspect.
Unavoidably, policy makers and stakeholders will assign a higher plausibility to scenarios that somehow resonate with their own visions. On the other hand, it is those policy makers and stakeholders that will contribute to shaping the future through their decisions and their actions. As a result, the direct involvement of policy makers and stakeholders in the foresight process will not only increase its robustness, but it will also lead to higher acceptance levels and legitimacy. Participatory approaches, on the other hand, should also reflect the overall need of a finely tailored balance between quantitative and qualitative dimensions, and to this end directly involve forward-looking experts from both communities. This can be a first step towards increased understanding and increased mutual trust between different communities.
Questions related to overcoming short-term barriers in mixing qualitative and quantitative FTA methods o Which areas and sectors have a high potential for increasing transparency and convergence of quantitative and qualitative methods?
o What are appropriate uses of qualitative and quantitative methods? o What can be learned from other fields? o What are good practices in developing interfaces between qualitative and quantitative methods? o How can participatory approaches be designed to balance quantitative and qualitative dimensions? o How can researcher preferences for specific methods be changed in favour of combined approaches?
In order to overcome some barriers to the mixing of FTA methods, more time and effort would be required in order to have a good understanding on how to adequately address them. Below an agenda is proposed to overcome the main long-term barriers, mainly the lack of well-known good practices, lack of trust, and differences in beliefs.
Lack of well-known good practices
In the light of its ever-growing volume and variety of FTA practices, the establishment of a stock-taking of basic knowledge (including e.g. the relevant statistical evidence, existing scenarios and the corresponding outputs, coherent assumption sets, etc.) could be explored, together with the possible institutional and organisational setting required to run and maintain it.. Such setting should be consistent with the recognition that primary strategic FTA functions (steer, monitor, negotiate, seek consensus, link with policy circles) should by handled by a “centralized” body, while the implementation and day-to-day operation of FTA exercises must be carried out at the local/sectoral level.
Such stock-taking endeavour should compare FTA exercises according to their nature, scope, goals and span and should provide both methodological and evaluation guidelines.
Such guidelines should focus/assess in particular on the whole exercise by paying particular attention to its accuracy and uncertainty. The accuracy should be measured by checking whether the cause-to-effect mechanisms at play are represented in a robust manner, and not in terms of the probability that an envisioned future eventually comes true. In turn, uncertainty should be assessed with the primary aim of differentiating between the intrinsic
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011 variability of a given phenomenon that exhibits high sensitivity to small changes (e.g. networks congestion) and the uncertainty that derives from an insufficient knowledge of complex phenomena (e.g. climate change). Ultimately, what matters is that the methods and tools implemented provide a reasonably faithful representation of the systems being analysed, and that the intrinsic uncertainties associated to such representation are documented at best 30 . The current EFMN/EFP 31 resource could serve as a starting point for this endeavour. The creation of such common stock-taking repository would not only further the scientific development of the field, but could also push journals that currently emphasise one type of research to open up their application field and could stimulate the creation of new one.
Lack of trust
Stocktaking can aim to support the selection of an appropriate mix of qualitative and quantitative methods and tools in a given context, which can in turn play a major role in increasing transparency and trust between FTA communities and with FTA users (policy and decision makers, but civil society as well). A trade off is clearly necessary in identifying the appropriate mix of qualitative and quantitative approaches: the latter are known to be preferred when it comes to presenting results/findings of foresight exercises (figures and graphs speak better to the minds of most people, whatever their cultural background) than sheer narrative. However, the active involvement of non foresight experts in the elaboration of scenarios, assumptions, not excluding value judgments can hardly be carried out by recurring to purely quantitative frameworks, and calls for more techniques of a more
“conversational” nature such as e.g. multi-criteria analyses, for instance. The systematic adoption of such interactive tools also helps increasing the transparency (real and perceived) of the process, which in turn is an essential ingredient to build the required trust.
Another aspect of trust is that it derives from perceived credibility, which is intrinsically problematic when referred to long term, unverifiable representations of possible futures. Past performance is commonly considered as the most useful indication on which to build credibility and trust. For forecast exercises, past performance can be validated by comparing predictions to reality, or by comparing ex-ante impact assessment of alternative policy options against ex-post impact assessments. As foresight does not claim to predict the future, ex post validation of past foresight exercises is not very meaningful, and good practices should rather be sought by measuring, if at all possible, the improvement of the decision making process, or in assessing whether institutions (and private companies) that consistently adopt foresight approaches perform ultimately better th at those who don’t.
Differences in beliefs: closing the epistemological gap
In the long run the clash of cultures could be transformed into an asset for FTA, by developing the right skills to enhance the community’s work, the level of trust in FTA and, ultimately, its policy impact. In order to address the epistemological barriers and gain legitimacy and trust it is suggested to: o establish links and collaborate with the communities of mixed methods research in order to better understand the fundamental reasons and beliefs explaining choices in education, in research interests and method preferences o break down the barriers between quantitative and qualitative thinking at all education levels o give more importance to communication skills in order to allow for networking and crossfertilisation of ideas, research outcomes and processes within and beyond the FTA communities.
30 When it comes to evaluation approaches this is all the more important, as impact assessment is moving from "ex-post" evaluation towards interactive learning processes (Loikkanen et al. 2006).
31 EFP: http://www.foresight-platform.eu/ - EFMN: http://www.efmn.info/
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
Questions related to overcoming long-term barriers in mixing qualitative and quantitative FTA methods o Which evaluation approaches are needed in order to move forward mutual learning in selecting the best combination of methods in a given context?
o How can transparency and trust be increased within the FTA communities and with FTA users? o What can be learned from the general field of mixed methods research? o How can barriers between qualitative and quantitative thinking in education be tackled? o How can networking and communication skills within and beyond the FTA communities be improved?
Since its first edition the FTA conference devoted part of its attention to the development of new tools and methods, novel applications of existing methods and (new) disciplines entering FTA. In this section, we provide an account of recent developments and examples of implementation of methods either combining qualitative and quantitative ones, and new disciplines that are entering FTA contributing to better understand and tackling societal challenges and to provide evidence base knowledge for future policy making.
Undoubtedly, new technologies such as web 2.0 can be used by FTA to streamline operations by increasing the possibility of interactive participation of stakeholders, and by speed-up the provision of information and feedbacks. Yet, the use of collaborative tools, in particular such as social web platform, has been rather limited, not only in FTA practices, but in general to link policy and research.
Combining qualitative and quantitative methods: an example combining scenarios and modelling
The Millennium Ecosystem Assessment scenarios is an example of global environmental scenarios that integrates ecosystem dynamics and feedbacks. The objective is to provide decision-makers and stakeholders with scientific information on the links between ecosystem change and human well-being. The approach used was a combination of qualitative storyline development and quantitative modelling. The scenarios were able to capture aspects of ecosystem that are possible to quantify, but also those that are difficult and even impossible to be expressed in quantitative terms. The storylines covered many complex aspects of society and ecosystem, while models helped ensure consistency of the storylines and provided numerical information where quantification was possible. The study made some advances to explore possible futures of the linkages between ecosystem change and society, as through the combination of quantitative and qualitative methods, it was possible to cover a large number of ecological services and drivers of ecosystem change. However, in the scenario development it was possible to identify areas where analytical tools are relatively weak and can be improved. For quantification of ecosystem service scenarios, it would be necessary to advance in terms of model developments that further disaggregate services to local scales, address cultural and supporting ecosystem services and consider feedbacks between ecosystem change and human development (Alcamo, 2011).
How to ensure increased participation in FTA?
One of the challenges of FTA has, as one of the discipline that could contribute to governance 2.0, is to increase the implementation of tools and methods that allows for increased participation of stakeholder, experts and citizens to foster dialogue and transparency. One interesting example of a novel methodology based on citizen consultation and expert/stakeholder analysis, was developed in the context of a project titled CIVISTI 32
32 http://www.civisti.org/files/images/Civisti_Final_Report.pdf
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
(Citizen Visions on Science and Technology and Innovation, financed by the European
Commission Framework Programme 7 on Socio-economic Sciences and humanities programme. The aim of this project was to develop a list of new and emerging issues for
European S&T, including a set of policy options relevant for future European framework programme by involving citizens of seven European Member States, supported by the analytical capacity of experts and stakeholders. The methodology developed by CIVISTI was built on the interplay of Foresight and participatory Technology Assessment. The difference with the more classical Foresight approaches was in terms of the emphasis which was on the demand side (i.e. needs and trends of society and societal developments) rather than on the supply side (i.e. technological development and research disciplines). The methodological approach was divided in three steps and was carried out in seven European countries. It was mixing panels of citizens that developed their long-term view on needs, wishes, concerns, and challenges on the future, resulting of 69 visions for the future, with a group of European experts and stakeholders who analysed these visions and transform them into future research agendas. In order to do so and provide a list of recommendations for research agendas, a stream model widely applied in policy analysis was used (Kingdon,
2005). Finally, the list of recommendations was returned to the citizens groups to be validated and prioritised. The entire methodological approach was supported by an online web-tool that documented the process and outcomes of the various steps. This approach shows that it is possible to have a strong societal aspect in a Foresight by involving citizens and that it can be transformed into a more standardised approach. This type of methodological approach opens up possibilities for a more structured involvement of citizens in the definition of European future policies and agendas.
Tools from other disciplines
Strategic design
In terms of new tools from other disciplines that can enter the FTA field there is some evidence of the application of strategic design and or, design thinking to address societal challenges that are falling at the intersection of current knowledge. In fact, societal challenges have multiple owners, are evolving faster than in the past and have an overabundance of legacy issues. Nowadays what we need are methods that have the capability to address the 'architecture of the problems'. This would be a useful concept to apply to problem solving at governmental levels. Strategic design is the application of futureoriented design principles in order to increase organisation innovative and competitive qualities. At first sight, this method is more suitable for FTA at corporate and business level.
However, this method has been applied by the Finnish Innovation Fund 33 to develop holistic understanding of a challenge is issues related to ageing, education and sustainability. As it happens with Foresight, one of design's value propositions is that it offers a way of understanding problems and solutions in a feedback loop, rather than a linear relationship.
The main focus is on how to address complex questions and issues that are governing our societies and that are more likely to co-evolve with the understanding of the intrinsic elements that characterise these issues. The space where this co-evolution happens is the overlap between analysis and execution.
European societies dominant logic in the last century, saw always analysis and execution conducted by separate parties, e.g. by different departments in a same organisation up to completely separate entities. The drawback of such an approach is that it fails to deliver when the problems addressed are fuzzy or dynamic. Basically, growing groups of cutting edge practitioners of strategic design claims that by using strategic design it is possible to specify the systemic performance and thereby overlapping with what Foresight aims to do.
Strategic design propose itself as a capability to enhance innovation and it can be proven useful in contexts where the typical decision-making practices have been ineffective or failing at meeting the demands of the challenge ahead.
33 http://www.sitra.fi/en/About+Sitra/sitra.htm
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
Instead of having strategic design competing with Foresight, it could be interesting to explore synergies and complementarities between the two communities. As with FTA, also strategic design approach entails three facets: an open-mindedness that looks for insights in atypical places without favouring hard facts over soft evidence; a non-linear way of working which involves cycling between refining the understanding of the problem and searching for new solutions; and a concept of targeted focus that directs effort to the key moments in a project's development, thereby enabling the strategic designer to coordinate critical scales so that they can work in harmony (Boyer, 2011).
Strategic design posits that problems and solutions cannot be understood in their traditional
1:1 relationship. The complexity of grand challenges is so high that in some cases it is difficult to fully describe it. The linear approach of defining problems and crafting solutions risks becoming a pitfall, if the problem is ill-defined and it is deemed to stay so in a lack of a dynamic context. One possible way out would be to describe problems and solutions as existing in a continuous feedback loop where a fragment increase of our understanding of the problem enables better ideas towards its resolution. Having the impression of a possible solution (or elements of it) can inspire new questions about the problem space and that is where the cycle starts again. This cyclical way of working means somehow that there is a need to radically rethink the way challenges and problems are addressed.
Social network analysis
Social network analysis has attracted attention in the past years as it allows revealing relationships and links that make up various social processes. In science and technology this method was used to understand cluster of collaboration and diffusion networks by focusing on social relations among a set of actors. Nugroho and Saritas (Nugroho and Saritas, 2009) propose a framework to incorporate network analysis in foresight. They argue that network analysis can be used as a useful tool to analyse Foresight data, which are often complex to present and codify so to enable robust analysis. Network analysis has the advantage to reveal the structural features of the data and can inform the foresight process on emerging links or relationships, groups or clusters. The implication for Foresight methods is that network analysis can introduce a 'systemic' perspective in Foresight by emphasising relationships between actor, key issues and trends. Nugroho and Saritas (Nugroho and
Saritas, 2009) have demonstrated that network analysis provides benefits in the various phases of the Foresight process (i.e. scoping, participation, generation, action and evaluation).
Social scanning and prediction markets
FTA practitioners have to face many challenges in dealing with the future of complex issues.
It is therefore necessary to continuously explore new tools to support the expert based knowledge avoid biases, cognitive conservatism and strengthen the practice of the field as a whole. In an essay reconsidering the field of futures studies, Soojung-Kim Pang (Soojung-
Kim Pang, 2010) suggests that tools and methods such as social scanning and prediction markets, could be used to improve professional forecasting and foresight in an era of complex phenomena and disruptive events with high level of uncertainties. He argues that prediction markets would encourage, for example, participants of an expert group, to think more clearly about what impacts they think discrete events could have, what discrete events are needed to reach a particular future, how they believe different trends will interact. He also suggests that scanning should be made more visible, as it would contribute to understand more precisely pictures of possible futures, aggregate results into a broad view of what the future might entail. Finally, making scanning public would allow people concerned with complex phenomena, like geopolitics, climate change, or issues at the interplay between science and society, to keep track of the content of one's intellectual portfolio of knowledge.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
Although systemic integration of quantitative and qualitative methods in FTA is not a general case so far, there are many practical examples of successful combinations of both types of methods and the contributions to the FTA 2011 confirm such interest. The different and highly heterogeneous contributions of the FTA 2011 share a common bottom line: both quantitative and qualitative tools aim at better understanding possible futures and reducing uncertainty. This "reduced" uncertainty needs to be embraced and managed in order shape the future and prepare society for it. In this context, there is large scope for mixing different methodologies, with the awareness that only a careful methodological design can develop anticipatory intelligence from primary data.
In order to push forward the field of FTA in this direction, this paper has explored possible barriers hampering such development, and a way forward has been proposed. It aims at further stimulating the debate during the conference, mainly along the questions that were put forward at the end of each chapter and on the future needs regarding mixing methodologies. Ultimately it aims at increasing maturity of the field, transparency among the communities trying to understand and shape the future, and its impact on policy-making.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
Agami, N., Saleh, M., El-Shishiny, H. (2011) A Fuzzy Logic based Trend Impact Analysis method, Technology
Forecasting and Social Change 77 (2010), pp.1051-1060.
Băban A. (2008), Reconceptualisation of the division between quantitative and qualitative research methods,
Cognition, Brain, Behavior. Interdisciplinary Journal , Volume XII, No. 4 (December), pp. 337-343.
Boyer, B. (2011) Sketching at scale
– The Helsinki Design Lab Studio www.sitra.fi/en and www.helsinkidesignlab.org
Bryant, B.P., Lempert R.J. (2010) Thinking inside the box: A participatory, computer-assisted approach to scenario discovery, Technology Forecasting and Social Change 77, pp. 34-49.
Bryman A., Barriers to Integrating Quantitative and Qualitative Research, Journal of Mixed Methods Research,
Volume 1 Number 1, January 2007, pp. 8-22.
Campbell, D. T. (1953) A study of leadership among submarine officers. Columbus: Ohio State University Res.
Found.
Campbell, D. T. (1956) Leadership and its effects upon the group. Monograph No. 83. Columbus: Ohio State
Univer. Bur. Business Res.
Campbell, D. T., and Fiske D. W. (1959) Convergent and discriminant validation by the multitrait multimethod matrix, Psychological Bulletin, VOL. 56, No. 2, pp 81 -105.
Cagnin C. and Keenan M. 2008, Positioning Future-Oriented Technology Analysis, In Future-Oriented
Technology Analysis: Strategic Intelligence for an Innovative Economy, C Cagnin, M Keenan, R Johnson, F
Scapolo and R Barré (eds.). Berlin and Heidelberg: Springer Verlag.
Castro F. G., Kellison J. G., Boyd S. J. and Kopak A. (2010) A Methodology for Conducting Integrative Mixed
Methods Research and Data Analyses, Journal of Mixed Methods Research, September 20, 2010, vol. 4 no.
4, pp. 342-360.
Cinco, C. and Novotny, K. 'Impact 2.0 – new mechanisms for linking research and policy' Technical guidelines, version 2
Cunnigham, S.W., Kwakkel, J. (2011) Innovation forecasting: A case study of the management of engineering and technology literature, Technology Forecasting and Social Change, 77, pp. 346-357.
Cunnigham, S.W., van der Lei, T.E. (2009) Decision-making for new technology: A multi-actor, multi-objective method, Technology Forecasting and Social Change 76 (2009), pp. 1037-1050.
Curran, C.-S.; Leker, J.: Patent indicators for monitoring convergence - examples from NFF and ICT,
Technological Forecasting & Social Change, 78, pp. 256-273.
Denzin, Norman K. (1978) The Research Act, 2d ed., New York: McGraw-HI.
Eerola A. and Miles I. (2011) Methods and tools contributing to FTA: A knowledge-based perspective, Futures 43, pp. 265
–278.
European Commission 2010, European Forward Looking Activities, EU Research in Foresight and Forecast,
Socio-economic Sciences and Humanities, List of Activities 2007-2010, Directorate-General for Research,
Socio-economic Sciences and Humanities, EUR 24480 EN. Link: http://ec.europa.eu/research/socialsciences/pdf/eu-forward-looking-activities_en.pdf
Forrester, J.W. (1971): World Dynamics. Wright Allen Press
Georghiou, L., Harper, J.C., Keenan, M., Miles, I., Popper, R. (2008) The Handbook of Technology Foresight.
Concepts and Practices. Edward Elgar, UK.
Garud, R., Nayyar, P.R., Shapira, Z.B. (2008) Technological innovation: oversights and foresights, Cambridge
University Press.
Haegeman K., Harper J.C., Johnston R. 2010, Introduction to a special section: Impacts and implications of future-oriented technology analysis for policy and decision-making, Science and Public Policy, 37(1),
February 2010, pages 3
–6.
Howe K. R. (1988) Against the quantitative-qualitative incompatibility thesis, or, Dogmas die hard. Educational
Researcher, 17, 10-16.
Howe K. R. (1992) Getting over the quantitative-qualitative debate, American Journal of Education, 100, 236-256.
Jick T. D. (1979), Mixing Qualitative and Quantitative Methods: Triangulation in Action, Administrative Science
Quarterly, Vol. 24, No. 4, Qualitative Methodology. (Dec., 1979), pp. 602-61 1.
Jick T. D. (1979a) Process and lmpacts of a Merger: lndividual and Organizational Perspectives. Doctoral dissertation, New York State School of Industrial and Labor Relations. Cornell University.
Johnson R. B. and Onwuegbuzie A. J. (2004) Mixed Methods Research: A Research Paradigm Whose Time Has
Come, Educational Researcher, Vol. 33, Nr 7, pp. 14-26.
Keenan, M. and Popper R. (eds.) 2007. Guide to Research Infrastructures Foresight. Brussels: European
Commission.
Keenan, M. an d Popper, R. (2008), Comparing foresight ‘‘style’’ in six world regions, Foresight, VOL. 10 NO. 6
2008, pp. 16-38, Emerald Group Publishing Limited, ISSN 1463-6689.
Kemp-Benedict, E. (2010) Converting qualitative assessments to quantitative assumptions: Ba yes’ rule and the pundits wager, Technological Forecasting and Social Change 77 (2010), pp. 167-171.
Koivisto, R., Wessberg, N., Eerola, A., Ahlqvist, T., Kivissari, S., Myllyoja, J., & Halonen, M. (2008) Integrating
FTA and Risk Assessment Methodologies. FTA 2008 Conference Paper, Seville.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS

Fourth International Seville Conference on Future-Oriented Technology Analysis (FTA)
FTA and Grand Societal Challenges – Shaping and Driving Structural and Systemic Transformations
S EVILLE , 12-13 M AY 2011
Lawson T. (1996) Developments in Economics as Realist Social Theory, Review of Social Economy, pp. 405-
422.
Lee, P., Su, H., Wu, F. (2010) Quantitative mapping of patented technology – The case of electrical conducting polymer composite, Technological Forecasting and Social Change 77 (2010), pp. 466-478.
Liebl, F., Schwarz, J.O. (2010) Normality of the future: Trend diagnosis for strategic foresight
Futures 42 (2010), pp. 313-327.
Loikkanen, T., Kutinlahti, P . & Eerola, A. (2006) Towards an Integrated Framework of Impact Assessment and
Foresight Studies in Innovation Policy Analysis.
FTA 2006 Conference Paper, Seville.
Lüdeke M. K. B. (2006), Bridging Qualitative and Quantitative Methods in Foresight, Potsdam Institute for Climate
Impact Research, only available online, accessed 31/03/2011, http://www.amsforschungsnetzwerk.at/deutsch/publikationen/BibShow.asp?id=7832&sid=552935624&look=0&aut=L%FCdek e&gs=0&lng=0&vt=0&or=0&aktt=0&zz=30&mHlId=0&mMlId=0&sort=jahrab&Page=1 .
Nugroho, Y; Saritas, O. (2009) Incorporating network perspectives in foresight: a methodological proposal,
Foresight, vol. 11, issue 6, pp. 21-41
Olsen W K. 2004 Triangulation in Social Research: Qualitative and Quantitative Methods Can Really Be Mixed. In
Developments in Sociology , ed. Holborn, M., and Haralambos, Causeway Press.
Popper, R. (2008) How are foresight methods selected, Foresight, volume 10, N6, pp. 62-89.
Popper R. (2009), Mapping Foresight - Revealing how Europe and other world regions navigate into the future,
European Commission, Directorate-General for Research, Socio-economic Sciences and Humanities, EUR
24041 EN. Available at: http://ec.europa.eu/research/social-sciences/pdf/efmn-mapping-foresight_en.pdf
.
Popper, R., Keenan, M., Miles, I., Butter, M. and Sainz, G. (2007), ‘‘Global Foresight Outlook 2007: mapping foresight in Europe and the rest of the world. The EFMN Annual Mapping Report 2007’’, report to the
European Commission, University of Manchester/TNO, Manchester/Delft.
Reiss A. J. (1968) Stuff and nonsense about social surveys and participant observation. In H.S. Becker, B. Geer,
D. Riesman and R. S. Weiss (Eds.), Institutions and the person: Papers in memory of Everett C. Hughes,
Chicago, Aldine.
Rossetti di Valdalbero D. 2010, The Power of Science - Economic research and European decision-making: The case of energy and environment policies, Peter Lang, ISBN 978-90-5201-586-6 pb.
Sarantakos S. (1993) Social Research, Basingstoke, Macmillan.
Sayer A. (1992 (orig. 1984)) Method in Social Science: A Realist Approach, London, Routledge.
Shibata, N., Kajikawa, Y., Takeda, Y., Sakata, I., Matsushima, K. (2011) Detecting emerging research fronts in regeneration medicine by the citation network analysis of scientific publications, Technological Forecasting and Social Change 78 (2011), pp. 274-282.
Scapolo F. and Porter A.L., New Methodological developments in FTA, In Future-Oriented Technology Analysis:
Strategic Intelligence for an Innovative Economy, C Cagnin, M Keenan, R Johnson, F Scapolo and R Barré
(eds.). Berlin and Heidelberg: Springer Verlag.
Silverman D. (1993) Interpreting Qualitative Data: Methods for Analysing Talk, Text and Interaction, London,
Sage.
Soojung-Kim Pang, A. (2010) Future 2.0: rethinking the discipline, Foresight, vol. 12, issue 1, pp.5-20
Tashakkori A. and Teddlie C. (1998) Mixed methodology, Combining Qualitative and Quantitative Approaches,
Applied Social Research Methods Series, Volume 46, Sage Publications.
Thorleuchter, D., Van den Poel, D., Prinzie, A. (2010) A compared R&D-based and patent-based cross-impact analysis for identifying relationships between technologies, Technological Forecasting and Social Change 77
(2010), pp. 1037-1050.
Trautwein U. and Lüdtke O. (2007) Epistemological beliefs, school achievement, and college major: A large-scale longitudinal study on the impact of certainty beliefs, Contemporary Educational Psychology, Volume 32, Issue
3, July 2007, Pages 348-366.
Van den Ende J., Mulder K., Knot M., Moors E., Vergragt P. (1998) Traditional and Modern Technology
Assessment: Toward a Toolkit, Technological Forecasting & Social Change 58, nrs 1&2, pp. 5-21.
Van Zwanenberg, P., Ely, A. and Stirling, A. (2009) Emerging Technologies and Opportunities for International
Science and Technology Foresight, STEPS Working Paper 30, Brighton: STEPS Centre - http://anewmanifesto.org/wp-content/uploads/van-zwan-et-al-paper-30.pdf
Webb E., Campbell D.T., Schwartz R.D., Sechrest L. (1966) Unobtrusive Measures. Nonreactive Research in the
Social Sciences Chicago: Rand McNally.
T
HEME
3: C
OMBINING
Q
UANTITATIVE AND
Q
UALITATIVE
T
OOLS