CIFE CIFE Seed Proposal Summary Page 2013-14 Projects

advertisement
CIFE
Center for Integrated Facility Engineering
CIFE Seed Proposal Summary Page
2013-14 Projects
Proposal Title: Statistical Analysis of KPIs: the Missing Links in the VDC
Decision-making Process
Principal Investigator(s):
Calvin Kam (Civil and Environmental Engineering)
Sadri Khalessi (Statistics)
Martin Fischer (Civil and Environmental Engineering)
Research Staff: Devini Senaratna (Statistics)
Proposal Number (Assigned by CIFE):2013-03
Total Funds Requested: $72,867
First Submission?
Yes
If extension, project URL: NA
Abstract:
Key Performance Indicators (KPIs) help Architecture, Engineering and Construction
(AEC) project teams to make informed decisions. The Virtual Design and Construction
(VDC) Scorecard research shows that only 40% of AEC firms use KPIs to evaluate VDC,
as compared to over 80% in other industries such as textile. Other industries use KPIs
efficiently by analyzing correlations and providing insights for decision making. This
aspect is a ‘missing link’ in the AEC industry.
We will integrate statistical methods with KPIs to help AEC professionals in VDC
decision-making. Building upon 108 cases from the VDC Scorecard results, we will
conduct web-surveys to obtain VDC performance statistics to develop and enhance the
KPIs. We will use statistical methods including Structural-equation Modeling, and
Clustering to identify relationships between KPIs.
We will present to CIFE members, as the final product of this SEED proposal, 3
statistical models for benchmarking, decision-prioritization, and prediction, to enhance
VDC decision-making.
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
Statistical Analysis of KPIs: the Missing Links in the VDC
Decision Making Process
1. Motivation
Tracking KPIs and making timely decisions based on the KPIs have helped Architecture,
Engineering and Construction (AEC) firms better utilize Virtual Design and Construction
(VDC). Efficient utilization of VDC results in improved management of time, cost, and
resources. Figure 1 shows the relationship between, the number of KPIs tracked and
monitored by the AEC firms, and performance with respect to efficient VDC utilization
(measured by the VDC Scorecard Score). It can be observed that the VDC Score increases
as the number of KPIs tracked and measured increase.
Tools, such as the Characterization Framework, National Building Information Model
(BIM) Standard, Pennsylvania State University BIM Planning Guides, and CIFE’s VDC
Scorecard have been formulated due to the need for an evaluation framework using key
performance indicators (Kam, McKinney, Xiao & Senaratna, 2013).
Average VDC Scorecard Score vs. Number of
KPIs Tracked by AEC Firms
64%
52%
46%
37%
None
1 - 3 Metrics
4 -7 Metrics
8 or More
Metrics
Figure 1: Average VDC Scorecard Score versus number of VDC KPIs tracked.
1.1. Problem 1 - Limited usage of quantifiable KPIs/metrics by AEC firms
The first problem is that only 40% of the 108 projects scorecard using the VDC Scorecard,
tracked and monitored quantifiable KPIs (Kam, Xiao, McKinney & Senaratna, 2013),
despite the clear benefits of tracking and monitoring KPIs (Figure 1).
Kam, Fischer, Khalessi, Senaratna
2
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
Table 1: Usage of KPIs/metrics in AEC for VDC
Description
VDC in AEC
40%
Measuring Performance (Kam, McKinney, Xiao &
Senaratna, 2013)
KPI Analysis Software
None
Other Industries
80% in textile
Esin et al’s (2009)
In Manufacturing, KPI analysis
software greatly improved
enterprise performance
(Zevenbergen, 2006)
Other industries, such as manufacturing and information technology, use metrics or KPIs
more systematically and successfully. Esin et al’s (2009) study on the usage of KPIs in the
apparel industry pointed out that 80% had some form of quantifiable KPI/metric that was
measured, as opposed to just 40% for VDC.
It was inferred that though scorecards are available to track and measure BIM/VDC usage
and performance, a methodology to analyze correlations and give effective feedback and
recommendations is yet to be incorporated into the VDC Scorecard methodology. In other
industries, methodologies that instantly analyze a project’s status and provide feedback are
both available and effective. For example, in the manufacturing industry, Zevenbergen
(2006) explains how KPI analysis software greatly improved enterprise performance.
1.2.Problem 2: Missing links (or correlations) between KPIs
The second problem is that the present VDC Scorecard insights and recommendations (see
Figure 2 for an example) are based on independent analysis of the VDC Scorecard KPIs.
For example, if a KPI scored poorly with regards to VDC, feedback is given without
considering the impact of that KPI on other KPIs.
Figure 2: Spider diagram showing weaknesses and strengths of a VDC Scorecard project
(the red highlighted division is a drawback relative to the other divisions)
Kam, Fischer, Khalessi, Senaratna
3
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
Initial statistical analysis show, complex correlations between KPIs. These can be used to
provide useful feedback, such as forecasting performance, and benchmarking projects.
This is another motivation for this study. Statistical tests used to test correlations are the
parametric t-test, chi-squared test, Mann-Whitney test, and Spearman’s rank correlation
test.
Examples (Figure 3 and Table 2)
 Stakeholder Correlations: Statistical analysis demonstrates that one of the most
significant associations with project performance is, the involvement of
stakeholders in VDC. It was found that 95% of the top scored projects use BIM
models among multiple stakeholders to resolve engineering challenges.
 Funding: The magnitude of budget allocations on VDC did not have a significant
association with adoption of VDC, the efficient use of VDC technology, or even
project performance using VDC.
Table 2: Highly significant KPI correlations with Performance
Rank
KPIs (Metrics)
1.
2.
Having VDC Management Objectives
Stakeholders involved in VDC based decisionmaking process
3. Efficiency of VDC/BIM integrated Project-Wide
Meeting
*Significance at 10-8
35%
1 stakeholder
involved
46%
47%
51%
2 to 3
stakeholders
involved
4 to 5
stakeholders
involved
6 to 7
stakeholders
involved
Statistical Significance
(p-value)*
0.0000000000924
0.00000001469
0.000000236
65%
All stakeholders
involved
Figure 3: Average Performance Score versus the number of Stakeholders that Benefit
(Note increasing trend)
Kam, Fischer, Khalessi, Senaratna
4
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
1.3.Solution and Contribution to CIFE
Our solution to problems 1 and 2 is to develop 3 types of statistical models that use the
relationships between KPIs. These 3 types of models will motivate firms to monitor KPIs
and make better decisions based on the insights from these models. The three models are:
1. Statistical Model-based prediction: Models that can predict/ forecast
Performance, Adoption and Technology measures, based on lagging KPIs (see
methodology and Figure 4 for KPI-web model)
2. Prioritizing Models: A dashboard that prioritizes badly performing KPIs on an
easy-to-read scale (Red: immediate attention required; Orange: improvements can
be made; Green: things are under control)
3. Benchmarking and Ranking Models: Ranking and grouping projects based on
their features relative to industry norms. This can be used as a scoring method for
selecting Award Winning Projects.
Figure 4: A KPI web designed for a selected group of VDC Scorecard KPIs
Note: See Section 5 for proposed methods for dissemination of findings (workshops,
publications, other contributions)
2. Theoretical and Practical Point of Departure
2.1.The VDC Scorecard:
The VDC Scorecard was designed as a holistic, quantifiable, practical, and adaptable tool
to track, access, score projects and provide recommendations regarding the effective use
of BIM/VDC (Kam, McKinney, Xiao, Senaratna, 2013). At present, VDC Scorecard
researchers have collected and analyzed 108 unique case studies using this evaluation
framework and have obtained insightful information regarding the use of BIM/VDC in the
Kam, Fischer, Khalessi, Senaratna
5
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
AEC industry. This diverse data set with 108 projects covers projects from 13 countries,
11 facility types, and all 7 stages of the construction process.
The VDC Scorecard is holistic in nature as it assesses how much the project’s
performance is improved by the use of extreme social collaboration instead of focusing
only on capturing the performances of the creation and implementing the product model
(technology) of a project (Kam, McKinney, Xiao, Senaratna, 2013). The performance of
project objectives are quantifiably measured based on communication, cost performance,
schedule performance, facility performance, safety, project quality, and other objectives
established based on project needs (See Figure 5 for VDC Scorecard structure). The VDC
Scorecard is designed to adapt to changing VDC industry practices.
Figure 5: VDC Scorecard structure with average Area and Division scores resulting from
the 108 cases in 13 countries.
The VDC Scorecard is also practical in that the express version can be completed within
half an hour, and has the capacity to provide quick quantitative feedback (Kam,
McKinney, Xiao, Senaratna, 2013). Further, it is practical and realistic since the VDC
Scorecard has a concept called a “Confidence Level,” which quantifies the accuracy of the
VDC Scorecard based on the quality of responses obtained.
The most significant drawback of the VDC Scorecard at present is the lack of a
methodology that provides insightful and powerful information related to the project’s
performance. These include the benchmarking of projects, and identifying interdependencies between KPIs that can be used to provide detailed feedback for better
project management and decision making. The best data-driven methodologies with
Kam, Fischer, Khalessi, Senaratna
6
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
verifiable validation methods for such analysis are in the domain of statistics. Statistical
learning would potentially lead to continual evaluation and enhanced application of KPIs
in VDC.
Current statistical research on sample data give insightful information on how projects
with similar features group together, the relative position of a project respect to other
projects, and new KPIs that can help define better innovative measuring criteria for a
project’s success.
Feature Combination Score 2
Projects with similar features
will group together. For
example projects 1, 37, 56, 60
etc., group together
X and Y Axis: Feature
combinations, are numerical
statistical measures built
using a collection of KPIs
Feature Combination Score 1
Figure 6: Sample Data Analysis: Grouping of projects (numbered as 1, 2, etc.) into 4
clusters based on K-means clustering method
2.2.Dynamic Performance Monitoring and Management (DPMM): A Metric-based
Framework to Better Predict Project Success - Li, et al (2012)
The DPMM is a previously funded SEED project that developed a framework which
measures, explains and predicts management performance in a dynamic manner. Prior to
the development of the DPMM methodology, performance monitoring in general was
precedence-based and not client based, intuitively driven and not tactically driven, ad-hoc
and not systematic, and performance was statically assessed and not dynamically assessed.
The DPMM methodology used 20 metrics on quality, cost, schedule, sustainability, and
organization as performance metrics, and defined 7 client satisfaction metrics to integrate
the relationships between performance and client satisfaction. In the VDC scorecard detail
is given to other leading and lagging metrics, such as model usage and maturity,
information exchange between models, and specific stakeholder related information such
as attitudes on BIM/VDC usage. The VDC Scorecard has over 50 measurements,
summarized into 10 Divisions, and 4 Areas, and a final score (See Figure 5). Similar to the
VDC scorecard survey methodology and slightly in contrast to Bloom and Van Reenens’s
Kam, Fischer, Khalessi, Senaratna
7
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
(2006) research design surveys managerial practices methodology, this study included all
of the members of the design, construction, and owner team that affect the outcome of
project execution.
The DPMM methodology used self-reported data that were subjective in nature. The VDC
Scorecard has an independent scale and objective metrics. This study will include the
carefully selected client satisfaction metrics, to obtain additional information, which is the
focus of the DPMM methodology. Building upon the DPMM methodology idea, this
study will go further and explore the relationships between a web of metrics, instead of
only between performance and satisfaction metrics.
Figure 7: Performance metrics dashboard and client satisfaction metrics (Li, et al, 2012)
2.3.Statistical Methodology: Designing and benchmarking a platform to select
VDC/BIM implementation strategies - Alarcón, et al (2010)
The proposed statistical research to be carried out using the VDC Scorecard data is similar
to the first phase of the this study, which formulates a benchmarking methodology to
better facilitate VDC/BIM implementation strategies, in the AEC industry. Alarcón, et al
(2010) used a combination of three statistical methods - Data Envelope Analysis, Factor
Analysis, and Structural Equation Modeling.
Factor Analysis is commonly used to reduce the data into unobservable variables called
factors (See section 3). These factors can be used as the inputs for other analyses such as
the structural equation modeling or even multivariate regression.
Structural equation modeling was used by Alarcón, et al (2010) as a method to describe
and quantify the impacts of these implementation (and un-observable) factors on the
company’s and project’s outcomes. A similar methodology will be used to identify the
relationships between the BIM/VDC hidden factors and performance outcomes, as well as
Kam, Fischer, Khalessi, Senaratna
8
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
to identify relationships between leading and lagging metrics, in order to provide
insightful feedback.
2.4.Leading and Lagging Metrics (or KPIs)
The DPMM methodology developed their framework by studying the relationships
between customer satisfaction metrics and performance metrics. Another angle of viewing
the relationships between KPIs is to consider leading and lagging metrics. Leading
indicators are indicators that can be corrected (or actionable), while lagging indicators are
indicators that measure a situation where it’s too late to carry-out any actions. Metrics
such as revenues and number of accidents are lagging indicators, while metrics such as
customer satisfaction, number of BIM/VDC meetings, and safety measures are leading
variables. A Scorecard should contain both leading indicators that will have an impact on
the lagging variables, as well as the lagging variables, that will point to improvements for
future endeavors. It is, hence, important to create a holistic dashboard of leading and
lagging metrics –the VDC Scorecard provides just that.
3. Methodology
The final statistical products will be the following 3 types of Statistical Models
1. Statistical Model-based prediction
2. Prioritizing
3. Benchmarking and ranking
The methodology for building these models, to identify correlations is as follows (Figure
8):
Figure 8: Methodology and detailed tasks
Kam, Fischer, Khalessi, Senaratna
9
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
3.1.Preliminary Work
Using the DPMM methodology and Version 9 of the VDC Scorecard, we will collect
additional KPIs to better understand the metric or KPI-web and correlations between
KPIs. These include the following client satisfaction metrics used in the DPMM
methodology such as quality of management, quality of work and mutual trust and
confidence, as well as new KPIs for version 9 of the VDC Scorecard, such as percentage
of targets reached under planning and percentage of RFI on time (See Figure 9 for full
list).
Figure 9: List of additional KPIs /metrics
Next, some of the statistical methods were tested on a sample dataset to test the feasibility
of the study. These results showed positive signs towards designing the metric web.
Testing method: Cluster Analysis:
(See Figure 6)
Testing method: Factor Analysis:
Using a sample from the VDC Scorecard data, a Factor Analysis was carried out on the
ten divisions to identify the underlying or hidden factors that may contribute to refining
the VDC scorecard and its structure. The identified “Hidden” factors proved to give very
insightful results.
Some of these factors are:
 Factor 1: Highly loaded with Preparation (Correlation: 0.830), Process (Correlation:
0.742), Organization (Correlation: 0.822), Integration (Correlation: 0765) and
Qualitative (Correlation: 617) Scores. This factor could represent the hidden features
that contribute to “Qualitative and Social” aspects of the VDC/BIM process.
 Factor 2: Highly loaded with Objective (Correlation: 0.798) and Quantitative Scores
(Correlation: 0.660). This factor could represent the prior input needed to track
quantitative metrics (being Objective) and its outcome (Quantitative Performance).
Kam, Fischer, Khalessi, Senaratna
10
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
3.2.Survey Design
In order to obtain information on the additional KPIs/metrics and correlations related to
them, we will carry out web-surveys to obtain the data. The survey is to be carried out
using the online survey tool, survey monkey. It will involve responding to 14 questions,
which will take approximately 15 minutes. The web-survey will first be tested on 5
projects to verify technical soundness and coherence of the questions. The Survey is to be
carried out in two phases to obtain information from all stakeholders, in order to have an
independent final score. Figure 10 illustrates the plan to collect data:
Figure 10: Web-survey methodology
3.3.Model Building and Validation (to study correlation patterns)
The model building will be based on the 4 statistical methods:
1. Factor Analysis (Explained in Section 2)
2. Structural Equation Modeling (Explained in Section 2)
3. Cluster Analysis (Explained in Section 2)
4. Classification Trees
5. Other:
a. Principal Components Analysis
b. Canonical Correlations
c. Correspondence Analysis
Classification Trees:
This method can be used in selecting project types that associate the most with a particular
feature. For example, it can identify what are the characteristics of projects that did not
meet their schedule based objectives. It is commonly used in Market Analysis to find the
best group of customers to target. The type of classification tree method we will use is a
CHAID tree (Chi- squared Automatic Interaction Detection).
Kam, Fischer, Khalessi, Senaratna
11
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
Others: Principal Component Analysis can be used to rank projects. Canonical
Correlation Analysis is used to identify links between groups of project features, which is
in turn used to forecast project performance. Finally, Correspondence Analysis is used to
identify relationships between categorical features.
3.4.Validation
Validating statistical models is essential to ensure good decision making using KPI
correlations. Validation will be carried out using two methods:
1. The data-driven validation of the results will use selected test data. The test data
are to be collected by the CEE 212A class as their class project for the course.
2. Validation will also be carried out by using expert opinion of CIFE members and
industry personnel.
The models will be refined until there is agreement between the models, the test data and
expert opinion.
3.5.Dissemination of Results
To disseminate results, we will carry out two workshops to educate AEC professionals on
the value of the findings, and how to use the insights in decision making (See section 5).
The first workshop will demonstrate initial findings on correlation patterns. The second
will be related to using the insights for decision making. We will complete one journal
publication, and contribute towards McGraw and Hill’s Smart Market Report and the
National BIM Standards.
4. Relationship to CIFE Goals
This study will encourage the efficient use of KPIs in the AEC industry and to improve
the benefits of BIM/VDC, so as to facilitate its usage in the AEC industry. The final
product will include project-specific KPIs/metrics or benchmarks that identify and
quantify both lagging and leading KPIs/metrics. In addition, the study involves innovative
statistical modeling methods, such as Factor Analysis and Structural Equation Modeling,
to forecast and monitor BIM/VDC performance.
4.1 Intellectual Merit, Relevance and Value
This study involves complex multivariate statistical methodology, and comprises of a
multidisciplinary team. The statistical methodology will add a novel perspective to the
design, analysis and interpretation of KPIs, and to understand the relationships between
them. Further, this team comprises of individuals with expertise in statistics, VDC, as well
as non-AEC industries such as manufacturing and production management. This will
further add to the innovativeness and intellectual merit of the study. This study is directly
related to ongoing CIFE research on the VDC Scorecard and will integrate the ideas and
outcomes of the previous SEED project on the DPMM methodology. Finally, current data
analysis carried out on a sample of the VDC data, especially the clustering nature of the
Kam, Fischer, Khalessi, Senaratna
12
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
cases, provides a sign of the feasible nature of carrying out in-depth analysis using the
proposed additional survey data. Also See Section 5 on “Dissemination of Findings”
5. Industry Involvement
5.1.Dissemination of Findings
We will disseminate the findings of this study by:
1. Publications: Results will be publicly available on the VDC Scorecard website
(vdcscorecard.stanford.edu/) and we will write one journal publication
2. Workshops: We will offer two workshops (November ’13, February ’14). They
will be conducted by Kam, Fischer and Khalessi, and participation will be possible
even remotely via “gotowebinar”.
3. Contributions: The findings of this research will be submitted to the National
BIM Standards, as well to the McGraw Hill Construction Market Report.
5.2.Data Collection, Validation and Missing Value Imputation
The additional data will be collected from the existing VDC Scorecard cases. This
includes both CIFE member organizations and other industry contacts. Results validation
will be carried out using two methods, a statistical data-based validation and an expert
opinion based validation. For the purpose of validating the recommendations, CIFE
members, stakeholders of the projects scored using the VDC Scorecard, students from
CEE 112/212, and AEC experts will be required to express their opinions on the
feasibility of the results, based on the characteristics of a few test-cases. If missing values
are observed, expert opinion may be required, along with statistical data imputation
methods, to fill in the data gaps.
6. Research Plan, and Schedule
6.1.Milestones
Task
Number
1.
2.
3.
4.
5.
6.
7.
Task(s)
Tentative
Deadline
August 1st 2013
Initial Preparation: Pilot Literature Review,
Statistical Analysis, Initial Survey Forms.
Survey Implementation
October 30th 2013
Data cleaning and checking for model assumptions November 15th 2013
Workshop I
November 2013
Workshop II
February 2013
Final Analysis: Factor Analysis, Structural Equation
February 7th 2013
Modeling, Clustering and, related results validation
Final documentation of publication materiel
February 15th 2013
performance feedback methodology.
Kam, Fischer, Khalessi, Senaratna
13
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
Documentation
Results Validation
Statistical Analysis
Data Cleaning
Survey Implementation
Survey Design
Tasks
Preliminary Work
6.2.Project Plan
July
August
September
October
November
December
January
February
6.3.Risks and contingencies:
1. Response rate: Due to time issues, it is expected that the response rate could be as
low as 30%. Data imputation or bootstrapping will be used to mitigate this risk.
2. Validating results for projects with rare and unique characteristics will be difficult,
and carrying-out validation using “matching” will not be logical for all cases. It is
possible to archive these cases and test them at the follow-up stage of this project.
7. Next Steps
As a next step, the process of developing the 3 statistical model proposed in the SEED
proposal will be automated. Though, this SEED Proposal will explore the relationships
between the existing KPIs, as VDC is a rapidly developing field, these relationships need
to be analyzed and refined. The project team will continue to offer this research to CIFE
members, and, additionally, seek external funding grants and joint collaborative research
with other universities. The findings of this research will be submitted to the National
BIM Standards, as well to the McGraw Hill Smart Market Report (See sections 3 and 5).
Kam, Fischer, Khalessi, Senaratna
14
Statistical Analysis of KPIs: the Missing Links in the VDC Decision Making Process
References:
1. Abeysekera, S. (2006) Multivariate methods for index construction, UNSTATS,
United Nations Statistics Division.
2. Alarcón, L., Mourgues, C., O’Ryan, C., and Fischer, M. (2010) Designing a
Benchmarking Platform to select VDC/BIM Implementation Strategies. Proceedings
of the CIB W78 2010: 27th International Conference –Cairo, Egypt, 16-18
November.
3. Bloom, N. and Van Reenen, J. (2007) "Why do management practices differ across
firms and countries?", Journal of Economic Perspectives, Vol 24, No 1, Winter 2010,
203-224.
4. Bloom, N. and Van Reenen, J. (2006) "Measuring and explaining management
practices across firms and countries", NBER Working Paper No. 12216, May, 2006.
5. Esin, C., Von Bergen, M., and Wüthrich, R. (2009) Are KPIs and Benchmarking
actively used among organisations of the Swiss apparel industry to assure revenue?
PricewaterhouseCoopers AG, Publication.
6. Gao, J. (2011) A Characterization Framework to Document and Compare BIM
Implementations on Projects, PhD thesis. Stanford, CA: Center for Integrated Facility
Engineering (CIFE), Dept. of Civil and Environmental Engineering, Stanford
University.
7. Kam, C., McKinney, B., Xiao, Y., Senaratna, D. (2013) The Formulation and
Validation of the VDC Scorecard, CIFE Publications. Stanford, CA: Center for
Integrated Facility Engineering (CIFE), Dept. of Civil and Environmental
Engineering, Stanford University.
8. Kam, C., Xiao, Y., McKinney, B., Senaratna, D. (2013) The VDC Scorecard
Evaluation of AEC Projects and Critical Findings, CIFE Publications. Stanford, CA:
Center for Integrated Facility Engineering (CIFE), Dept. of Civil and Environmental
Engineering, Stanford University.
9. McGraw and Hill (2012) The Business Value of BIM. McGraw and Hill, Smart
Market Report
10. Li, W., Fischer, M., Schwegler, B., Bloom, N., Van Reenen, J. (2012) Dynamic
Performance Monitoring and Management: A Metric-Based Framework to Better
Predict Project Success. CIFE SEED Proposal. Stanford, CA: Center for Integrated
Facility Engineering (CIFE), Dept. of Civil and Environmental Engineering, Stanford
University.
11. Zevenbergen, J., Gerry, J., and Buckbee, G., (2006) Automation KPIs Critical for
Improvement of Enterprise KPIs. Journees Scientifiques et Techniques.
Kam, Fischer, Khalessi, Senaratna
15
Download