Session T2D
TEACHING DESIGN OF EXPERIMENTS AND STATISTICAL ANALYSIS OF
DATA THROUGH LABORATORY EXPERIMENTS
Stacy Gleixner1 , Greg Young 2 , Linda Vanasupa3 , Yasser Dessouky4 , Emily Allen5 and David Parent 6
Abstract  A new laboratory course at San Jose State
University, Advanced Thin Film Processes, integrates
fabrication of thin films with design of experiment and
statistical analysis of data. In the laboratory section of this
course, students work through six multi-week modules that
increase in the complexity of design of experiment and
statistical analysis of data. The six modules have been
developed with a standard format that includes learning
objectives, background on the specific thin film process,
theory of design of experiment principles, instructor notes,
dry lab exercises, experimental plan worksheets, and
assessment tools. While the modules were developed for a
semiconductor processing class, they can easily be
implemented in other engineering classes. The modules
have been developed with a robust framework that allows
the instructor to teach design of experiments and statistical
analysis of data along with the specific engineering
principles related to their class.
Index Terms  teaching design of experiment,
semiconductor processing laboratory class, statistics
experiments, DOE
OVERVIEW OF M ICROELECTRONICS PROCESS
ENGINEERING PROGRAM
San Jose State University has initiated an innovative
undergraduate degree in microelectronics process
engineering (µProE).
The program evolved from
conversations with industry partners about their demand for
microelectronics process engineers. This demand was seen
in a range of industries where the term microelectronics
processing is meant to generalize anyone using
manufacturing processes similar to those of the
semiconductor industry.
These include traditional
semiconductor companies that manufacture computer chips
as well as companies that package microelectronic products;
semiconductor equipment manufacturers; data storage
companies; flat panel and other display manufacturers;
chemical and pharmaceutical companies using micro-fluidic
devices; and industries making micro-electrical mechanical
devices such as sensors and actuators. Process engineers are
those that maintain, monitor, evaluate, and optimize the
manufacturing processes in these industries. The skill set
needed to successfully perform these job functions requires a
broad base of engineering fundamentals. Companies usually
generate successful process engineers by hiring graduates
from a range of engineering majors and training them in
house for the needed process engineering skills not covered
in their major. Our program was developed in order to give
engineering students the fundamentals needed to be effective
process engineers immediately upon graduation.
The program combines existing classes in chemical,
materials, electrical, and industrial and systems engineering
along with three new laboratory classes in semiconductor
processing. Figure 1 shows the engineering curriculum for
the µProE degree.
A detailed description of the
development of the program and curriculum is published
elsewhere [1].
MatE
E.O.M.
Props
Materials
Structures
EE
Materials
Characterization
Device
Physics
µProE
ISE
ChemE
Intro to IC
Proc. & Design
Engr. Stat.
Thermo
Adv. Thin
Film
Process.
Micro.
Mfgr.
Methods
DOE
SPC
Transport
Reactor
Kinetics
FIGURE. 1
CURRICULUM FOR A BACHELOR’ S DEGREE IN MICROELECTRONICSPROCESS
ENGINEERING
1
Stacy Gleixner, Dept. of Chemical and Materials Engineering, San Jose State University, One Washington Square, San Jose, CA 95192-0082,
gleixner@email.sjsu.edu
2
Greg Young, Dept. of Chemical and Materials Engineering, San Jose State University, One Washington Square, San Jose, CA 95192-0082,
glyoung@email.sjsu.edu
3
Linda Vanasupa, Materials Engineering Department, California Polytechnic State University, San Luis Obispo, CA 93407, lvanasup@calpoly.edu
4
Yasser Dessouky, Industrial and Systems Engineering, San Jose State University, One Washington Square, San Jose, CA 95192-0085,
ydessouk@email.sjsu.edu
5
Emily Allen, Dept. of Chemical and Materials Engineering, San Jose State University, One Washington Sq uare, San Jose, CA 95192-0082,
elallen@email.sjsu.edu
6
David Parent, Electrical Engineering, San Jose State University, One Washington Square, San Jose, CA 95192-0084, dparent@email.sjsu.edu
0-7803-7444-4/02/$17.00 © 2002 IEEE
November 6 - 9, 2002, Boston, MA
32 n d ASEE/IEEE Frontiers in Education Conference
T2D-1
Session T2D
Figure 1 emphasizes the interdisciplinary nature of the
µProE engineering education. The faculty involved in the
development of the curriculum and the microelectronics
processing lab are also interdisciplinary: coming from the
chemical, materials, electrical, and industrial and systems
engineering departments.
Part-time instructors from
industry have also contributed to course development.
The three µProE classes shown in Figure 1 were added
to the engineering curriculum to specifically target the
education gaps cited by our industry partners. They include
extensive theory and laboratory work related to the
chemistry and physics needed to understand the
manufacturing of microelectronics. The classes also build
skills in teamwork, communication, and the use of statistics
in manufacturing. They are all hands-on laboratory classes
that are typically team taught by faculty from different
disciplines. In addition to µProE majors, the classes are
taken as electives by electrical, chemical, materials,
mechanical, industrial, and computer engineering majors.
The first class in the series is Introduction to Integrated
Circuits Processing and Design (MatE/ EE 129). This class
operates as a fictitious company where students rotate
between being on manufacturing and research teams [2]. In
this class, students learn a basic overview of
microelectronics processing through coursework and the
hands-on manufacturing of NMOS transistors. Teamwork
and communication skills are taught and extensively utilized
in this class.
Feedback from our industry partners
emphasizes that these skills are critical to process engineers
who work with a range of other engineers and technicians
every day.
Microelectronics Manufacturing Methods (MatE/EE
167) teaches the advanced microelectronics theory needed
by process engineers including limitations on current
materials and devices and the use of modeling in process
optimization. There is a strong emphasis on the skills
needed to monitor and optimize the manufacturing process,
specifically statistical process control (SPC) [3]. Students in
MatE/EE 167 have already taken MatE/EE 129. Like
MatE/EE 129, students alternate between manufacturing and
research teams in the lab. This class produces CMOS
circuits and extensively utilizes SPC to track and optimize
their manufacturing process.
Students in MatE/ChE 166: Advanced Thin Film
Processes have already taken MatE/EE 129. They have a
basic understanding of the microelectronics process. This
class goes into depth on the materials and chemical
engineering principles needed to understand the deposition,
patterning, and etching of thin films. Deposition and etching
of thin films are the most utilized step in the overall
manufacturing process and one of the most critical. The
laboratory sections of the class place a strong emphasis on
using design of experiments (DOE) and statistical analysis
of data. The statistics and DOE laboratory modules
developed for this course are discussed in detail in this
paper.
STATISTICS AND DOE EXPERIMENTS
Statistics and DOE were cited as skills our industry
collaborators felt were weak in engineering graduates they
were currently hiring. The employers also felt these skills
were critical to a process engineer’s success. A report
generated by the Semiconductor Research Corporation on
Microelectronics Manufacturing Education listed statistics
and DOE as two of the top skills needed for new college
graduates to be hired as process engineers [4]. There are
frequently stand alone classes on statistics and design of
experiments (which our µProE majors take to add depth to
their skills). However, integrating the statistics and DOE
directly into a laboratory class stemmed from a desire for the
students to gain a more hands-on understanding of statistics.
Six lab modules were designed to provide learning
experiences in a range of basic topics in DOE and statistical
analysis of data. Table I lists the modules and the schedule
for a fifteen week semester. The laboratory experiments are
open ended, where teams of students design their own
experiments (to answer given questions). The modules
increase in complexity of both the DOE and statistics
covered as well as the experimental planning required of the
students.
The first week of each lab module is reserved as a dry
lab session. (Module 1 is solely a dry lab exercise.) During
this week, the fundamentals of the DOE or statistical
analysis of data needed for that module are taught. Students
gain mastery of these skills by working through given dry
lab
exercises
(numerical
problems
related
to
microelectronics process engineering). Then as a team they
design an experimental plan to be carried out in the
following weeks. The plan should be based on a specific,
assigned question and generate the needed data to be
analyzed according to that mo dule’s DOE and statistics
principles.
TABLE I
DOE AND STATISTICAL ANALYSIS OF DATA MODULES USED IN THE
LABORATORY CLASS
Lab Module
Module 1: Introduction to Statistics
Module 2: Determining the Precision
of a Measurement Technique
Module 3: Determining the Variation
Within a Process
Module 4: Design of a Single Factor
Experiment & Analysis of Variance
Module 5: ANOVA of a Single Factor
Experiment With Blocking
Module 6: Design of a Two-Level Full
Factorial Experiment
# of Weeks
1
2
3
3
3
3
0-7803-7444-4/02/$17.00 © 2002 IEEE
November 6 - 9, 2002, Boston, MA
32 n d ASEE/IEEE Frontiers in Education Conference
T2D-2
Session T2D
M ODULE FORMAT
A standard format was developed for each module. The goal
of this was to create a full laboratory class that could easily
be taught by other faculty at San Jose State or exported to
other universities. It is also expected that some or all of the
modules will be useful in integrating DOE and statistical
analysis of data into a range of engineering laboratory
classes. Each module consists of:
• Laboratory Handout- This is a comprehensive
handout given to the students explaining the lab theory
and experiment. More details on this are given below.
• Dry-lab Tutorial- This is a PowerPoint presentation the
faculty can utilize during the first week of the lab
module to teach the needed DOE and statistical analysis
of data. It contains numerical examples designed
specifically for microelectronics process engineering.
• Dry-lab Exercises- These are numerical problems
specific to microelectronics process engineering that
were designed to give the students practice in the DOE
and statistical analysis of data principles specific to that
module.
• Experimental Plan Worksheet- Students fill out a
detailed plan for their experiment during the first week
of the module (the dry lab week). Prior to beginning in
the lab, the students review their experimental plan with
the instructor. Aspects such as proper control of
variables, use of a statistically valid number of samples,
and time management to accomplish the entire
experiment in the allotted lab sections are emphasized.
• Student Feedback Survey- Each lab module includes a
student feedback survey to assess whether the students
felt the lab taught them the desired principles and if they
view those principles as useful to manufacturing.
For each laboratory module, there is a detailed lab
handout that is given to the students. The handout includes:
• Learning Objectives- The handout provides the
students and faculty with a detailed list of learning
objectives for that module. The list utilizes Bloom’s
taxonomy and represents skills that can be assessed in
the lab’s outcomes. They include technical skills on the
theory and laboratory equipment as well as mastery of
the DOE and statistical analysis of data principles.
Table II lists the learning objectives for Module 4:
Design of a Single Factor Experiment and Analysis of
Variance as an example.
• Microelectronics Theory- Any information needed to
complement the lectures and textbook in order for the
students to fully understand the microelectronics
process being studied is presented in this section.
• Statistics Theory- This section contains a detailed
description of the statistics and DOE. This includes
explanations of the equations as well as numerical
examples.
•
•
Experimental Goals- Depending on the module,
student teams are assigned lab equipment or specific
questions to investigate. This section guides them in the
factors they need to consider and control when
designing their experiments.
Lab Outcome- This section details the outcome
expected from the laboratory experiment. This is
typically a formal written laboratory report or an
engineering memo. The content required in the reports
is specified. The required data analysis is also clearly
detailed.
TABLE II
EXAMPLE OF LEARNING OBJECTIVES USED IN MODULE 4: DESIGN OF A
SINGLE FACTOR EXPERIMENT & ANALYSIS OF VARIANCE .
#
1
2
3
4
5
6
7
8
9
Learning Objectives
Write clear objectives and statement of problem for an
experiment.
Identify controllable and uncontrollable factors in an
experimental set-up.
Choose a factor for a single factor ANOVA based on
expected outcome and given time and equipment
constraints.
Determine the appropriate levels to be researched for a
factor based on expected outcome, equipment control,
metrology precision, and time constraints.
Design an experiment using proper replication,
randomization, and control of variables.
Calculate the variation between levels using a sum of
squares method.
Calculate the variation using an F-test.
Plot data of all the levels to show variation between
levels.
Organize technical information into a clear and
concise formal laboratory report.
M ODULE CONTENT
Module 1: Introduction to Statistics
Module 1: Introduction to Statistics is solely a one week dry
lab exercise. There is a tutorial session in which the students
learn (or review) mean, median, mode, standard deviation,
and variance. A one sample T-test (to determine if the mean
of a process is within standard or given specification) is
taught. Students are introduced to the concept of a P value
(the smallest level of significance that would lead to the
rejection of a hypothesis) and how to use this value to
determine if their results are statistically significant. The lab
handout details the needed equations along with numerical
examples. The students then perform a dry lab experiment
where they statistically analyze data from oxidation of
silicon wafers. (he students are all familiar with this data
and process from their Introduction to IC Processing class.
0-7803-7444-4/02/$17.00 © 2002 IEEE
November 6 - 9, 2002, Boston, MA
32 n d ASEE/IEEE Frontiers in Education Conference
T2D-3
Session T2D
Module 2: Determining the Precision of a Measurement
Technique
students statistically analyze their results to determine the
variability within a run and from run to run.
Module 2: Determining the Precision of a Measurement
Technique is a variation of a laboratory experiment
developed by Vanasupa et al. [5]. The difference between
precision and accuracy are taught. Students are then given
the skills to statistically determine whether a piece of
equipment is precis e. This includes simple calculations of
the precision (3σ). Students also use an F-test and a paired
T-test to compare the variance and mean (respectively) of
two processes. In the dry lab portion of the exercise,
students practice analyzing experimental results using these
DOE principles. The microelectronics process engineering
problem they analyze is related to quantifying the difference
between two measurement techniques for oxide thickness.
For the experiment portion of the lab, the student teams
are assigned two pieces of equipment that both measure the
same property. We use equipment that measures oxide or
metal thickness. The students develop an experiment to
determine the precision of each piece of equipment. They
develop standard operating procedures to attempt to improve
the precision and repeatability of the tool. The students
statistically compare the precision of the two tools.
This lab is critically important in that it introduces to
students that errors in their experiments come not only from
the processing but also (and sometimes most significantly)
from the measurement tool.
It also emphasizes the
importance of repeatability and having a standard operating
procedure. Students begin to build the statistical skills
needed to determine whether the differences between two
sets of measurements are statistically significant. This
corrects the notion that a lot of students have that if the
measurement readings are not exactly the same, one (or
both) of them are wrong.
Module 4: Design of a Single Factor Experiment and
Analysis of Variance
In Module 4: Design of a Single Factor Experiment and
Analysis of Variance, students are taught the skills needed to
properly design a complex experiment. The dry lab exercise
steps them through the process of designing a robust
experiment, see Table III [6].
Analysis of variance
(ANOVA) is taught so the students have the statistical tools
to compare multiple levels of a variable and determine if
changing the variable has a statistical impact on the
outcome. In the dry lab experiment, the students apply the
ANOVA when analyzing the results of photoresist thickness
following soft bakes that vary in time. They are able to
determine whether time is a statistically significant variable
in the experiment.
Students then design a single factor (one variable)
experiment. They are assigned a microelectronics process
and asked to determine if a chosen variable has an impact on
the outcome. Students show final mastery of the concept by
utilizing ANOVA in their final report to prove whether their
variable was statistically significant.
TABLE III
STEPS USED TO P ROPERLY DESIGN AN EXPERIMENT [6]
#
1
2
3
4
5
Module 3: Determining the Variation Within a Process
In Module 3: Determining the Variation Within a Process,
students begin to use the statistics and design of experiments
they have been learning in the other modules to evaluate
results from complex sample sets, that is multiple lots (runs)
that each contain multiple samples (wafers). In the dry-lab
portion of the exercise, students review the F-test and how it
can be applied to multiple runs and samples. The two
sample T-test is taught as a way of comparing the mean
between two different lots. The students gain mastery of
these principles by statistically analyzing given results of
varying photoresist thickness (multiple runs and multiple
wafers each run).
Students then plan and carry out an experiment to
statistically evaluate the repeatability of a process. Each
team is assigned a microelectronic process (oxidation,
photoresist spinning, metal evaporation, plasma etching...).
They design a process flow through the equipment and
repeat that for two sets of wafers. In their final report,
6
7
8
9
DOE Steps
Develop a problem statement.
Determine all the variables that impact the process.
Determine which are controllable.
Choose a factor to investigate (one that may impact the
problem in step 1). Choose levels of the factor that are
significant (theorized to be measurably different).
Control all the other variables.
Develop an experimental order that randomizes the
sample runs and provides for replication.
Develop a plan to carefully monitor your experiment.
Determine the steps needed to statistically analyze the
results.
Review the experimental plan with the instructor.
Carry out the experiment and analyze the results.s
Module 5: ANOVA of a Single Factor Experiment With
Blocking
The DOE skills learned in Module 4 are expanded on in
Module 5: ANOVA of a Single Factor Experiment With
Blocking. In the previous lab modules, it was emphasized to
the students to hold all process parameters other than the
one(s) being studied as constant (controls). In a real
manufacturing setting, this is not always possible.
Uncontrollable changes can happen such as a material lot
changes or experiments are assigned to a different (but
supposedly equivalent) tool. Blocking is a way of grouping
and statistically analyzing the data after such a switch
occurs.
0-7803-7444-4/02/$17.00 © 2002 IEEE
November 6 - 9, 2002, Boston, MA
32 n d ASEE/IEEE Frontiers in Education Conference
T2D-4
Session T2D
In the dry lab exercise, students learn and utilize
advanced ANOVA calculations that require blocking. They
practice those skills in analyzing given photolithography
data (resolution of the lithography) for variations in
lithography process parameters as well as runs that varied
the photoresist supply. They then generate a detailed DOE
for an experiment of their own on an assigned
microelectronics process.
Blocks are purposefully
introduced, such as assigning different anneals to different
furnaces or using different sources of the raw materials. The
students utilize the complex ANOVA calculations to
determine if the factors they were studying (the variables
that they purposefully changed) as well as the blocks had a
statistical impact on the results.
CONCLUSIONS
Six laboratory modules were developed for a core course in
the microelectronics process engineering program
(MatE/ChE 166: Advanced Thin Film Processes). The
overall goal of this course is to teach the fundamentals of
thin film processing alongside design of experiment and
statistical analysis of data.
While designed around
microelectronics process engineering, the modules could be
used to integrate DOE and statistical analysis of data into a
range of engineering laboratory classes. The modules were
developed using a standard framework that will make them
portable to other classes and universities. Contact the
authors for a copy of the modules.
ACKNOWLEDGMENT
Module 6: Design of a Two-Level Full Factorial
Experiment
In Module 6: Design of a Two-Level Full Factorial
Experiment, a more realistic scenario of a complex
engineering experiment is presented, one where there are
multiple factors that may affect the outcome and they should
be investigated simultaneously because they may also
influence each other.
In the dry-lab exercises, the
calculations for 2k ANOVA are taught. These show how to
mathematically compare results when you vary two different
factors (such as time, temperature, pressure) and each factor
is tested at “k” different levels (such as high, medium, and
low).
The students practice their ANOVA skills by
statistically comparing the results of plasma etching when
the power and gas pressure are changed.
Using the steps in Table III, students design a complex
experiment for an assigned microelectronics process. Their
experiment investigates whether two variables in the process
have an impact on the final thin film produced. During the
following two weeks, they carry out the experiment. Their
mastery of 2k DOE is assessed when they use the ANOVA
to analyze the results of their experiment.
M ODULE ASSESSMENT
The classes in the µProE program were designed around a
comprehensive set of program learning objectives. Each
class has their own set of learning objectives (on the order of
100 specific learning objectives) that target specific program
objectives. Details of the extensive plan for assessment of
the program and class learning objectives has been published
elsewhere [7]. Assessment of each laboratory is also built
into the module. This is done through evaluation of the
student performance on the dry-labs, the development of
their experimental plans, and the management of their
experiment in lab. Student mastery of the DOE and
statistical analysis of data is assessed in how the students
communicate the results in their laboratory reports.
Assessment in the form of student feedback is also collected
through student surveys at the end of each lab module.
The curriculum development for these modules was
accomplished with the support of the National Science
Foundation (DUE#9952707). The laboratory has been
enhanced by a Challenge Grant from the Society of
Manufacturing Engineers, as well as financial and in-kind
support from Intel Corporation, Applied Materials, Novellus
Systems, Cypress Semiconductor, Advanced MicroDevices,
Project Consultants and Quintel Corporation. Much of the
work of the laboratory development was done by Irene
Wibowo as part of her Master’s project in Industrial and
Systems Engineering. Neil Peters, the Microelectronics
Processing Lab Engineer, did an extensive amount of work
setting up and running the laboratory sessions.
REFERENCES
[1]
Allen, E., Gleixner, S., Young, G., Parent, D., Dessouky, Y. and
Vanasupa, L., "Microelectronics Process Engineering: A NonTraditional Approach to MS&E," Procs. Materials Research Society,
684E, 2001, online GG5.1.
[2]
Muscat, A.J., Allen, E.L., Green, E.D.H and Vanasupa, L.S.,
“Interdisciplinary Teaching and Learning in a Semiconductor
Processing Course,” J. Eng. Educ. 87, 1998, 413.
[3]
Parent, D., Dessouky, Y., Gleixner, S., Young, G. and Allen, E.,
“Microelectronics Process Engineering Program at SJSU”, Biennial
University/Government/Industry Microelectronics Symposium Proceedings, 2001, 14th Biennial University/Government/Industry
Microelectronics Symposium, 2001, 128.
[4]
SRC Competitiveness Foundation Workshop on Microelectronics
Manufacturing Engineering Education, Semiconductor Research
Corporation, Research Triangle Park, NC, 1991.
[5]
Vanasupa, L., Smith, H., Gleixner, S.H., Young, G., Allen, E.L.,
“Dealing with Variations in Measurements and Processes:
Experiments for an Undergraduate Laboratory”, Procs. Materials
Research Society, 684E, 2001, online GG5.8.
[6]
Montgomery, D.C., Design and Analysis of Experiments, John Wiley
& Sons, New York, NY, 1997.
[7]
Young, G., Gleixner, S., Parent, D., Dessouky, Y., Allen, E. and
Vanasupa, L., “Course Assessment of the Microelectronics Process
Engineering Program at SJSU”, to be published in the American
Society for Engineering Education Annual Conference Proceedings,
2002.
0-7803-7444-4/02/$17.00 © 2002 IEEE
November 6 - 9, 2002, Boston, MA
32 n d ASEE/IEEE Frontiers in Education Conference
T2D-5