Performance-Based Services
Acquisition Training Project
Design Treatment
December 7, 2005
Prepared for:
Cynthia Allen
Gaile Garner
Opal Enterprises, Inc.
Prepared by:
Kristin Schmidtfrerick
Jeremy Cannella
Kate Imholt
George Mason University
EDIT 730: Analysis and Design of
Hypermedia Environments
Dr. Kevin Clark
Table of Contents
1.0
Purpose of this Design Treatment .................................................................... 3
2.0
Introduction and Project Background .............................................................. 3
2.1
PBSA ......................................................................................... Error! Bookmark not defined.
2.2
Opal Enterprises, Inc. .................................................................................................................. 3
3.0
Needs Analysis ................................................................................................... 4
3.1
Client Expressed Needs ............................................................................................................... 4
3.2
Analysis Methodology................................................................................................................. 4
3.3
Desired Status: "What Should Be" .............................................................................................. 5
3.4
Current Status .............................................................................................................................. 7
3.5
Discrepancies and Identified Issues............................................................................................. 9
4.0
Task Analysis.................................................................................................... 10
4.1
Task Analysis Methodology...................................................................................................... 10
4.2
Master Design Chart.................................................................................................................. 11
4.3
Learning Objectives................................................................................................................... 14
5.0
Instructional Design Approach ....................................................................... 18
5.1
Course Design Overview........................................................................................................... 18
5.2
Lesson Overview ....................................................................................................................... 21
6.0
Technical Design Approach ............................................................................ 24
6.1
SCORM ..................................................................................................................................... 24
6.2
Accessibility .............................................................................................................................. 24
6.3
Authoring and Delivery Environment ....................................................................................... 24
6.4
Multimedia ................................................................................................................................ 25
6.5
Prototype Storyboards ............................................................................................................... 25
7.0
Formative Evaluation Methodology ................................................................ 27
8.0
Recommendations ........................................................................................... 28
8.1
Initial Development Focus......................................................................................................... 28
8.2
User Navigation......................................................................................................................... 29
9.0
References and Data Sources ......................................................................... 29
9.1
Theory and Practice ................................................................................................................... 29
9.2
Background................................................................................................................................ 29
9.3
Content ...................................................................................................................................... 30
9.4
Audience.................................................................................................................................... 31
DESIGN TREATMENT
Page 2 of 31
1.0
Purpose of this Design Treatment
The Design Treatment presents the project needs and task analyses and details the solution
created to address the identified needs. This document contains the Introduction and Project
Background, the Needs Analysis, the Task Analysis, the Instructional Design Approach, the
Technical Design Approach, the Formative Evaluation Methodology and
Recommendations.
2.0
Introduction and Project Background
2.1 Performance-Based Services Acquisition (PBSA)
Performance-Based Services Acquisitions (PBSA) are a government mandate. This mandate has
resulted in acquisition organizations being in a state of transition as they undertake a new way of
doing business. Performance-Based Services Acquisitions (PBSA) is “structuring all aspects of
an acquisition around the purpose of the work to be performed with the contract requirements set
forth in clear, specific, and objective terms with measurable outcomes as opposed to either the
manner by which the work is to be performed or broad and imprecise statements of work”
(Federal Acquisition Regulation, 2.101). This is a change from traditional acquisitions that have
indicated how contractors were to fulfill the contract. Although Federal Acquisition Regulations
(FAR) mandate "what" must be included in a performance-based contract, there are no standards
on "how" to meet those standards. No courses are currently available that address this deficit
between what and how.
Government wants PBSA but knowledge gaps lead to problems translating the requirements into
the PBSA contract, particularly with regard to corporate/private sector risk and arising contract
issues. Part of the knowledge gap comes from the fact that the government acquisition
community is undergoing a massive demographic turnover, as a large part of the workforce will
soon become eligible for retirement. These older workers are having a hard time transitioning to
Performance-Based Services Acquisitions from the traditional acquisitions models and newer
employees are unfamiliar with the process.
Knowledge gaps also exist on the private-sector contractor side. Contractors are impacted by the
risk inherent in executing a contract, and a poorly constructed contract can increase that risk
significantly. If contractors perceive high risk in an acquisition, they will adjust costs upward or
even avoid bidding altogether, resulting in the government not obtaining the best price or best
solutions for its needs. The goal of performance-based acquisitions is to mitigate risks for all
parties.
2.2 Opal Enterprises, Inc.
Opal Enterprises, Inc. is an e-learning service provider in the discipline of federal acquisition
online courses. The federal government has a statutory requirement to ensure that all federal
contract requirements are written as performance-based acquisitions to the maximum extent
possible. Opal Enterprises, Inc. has requested the development of an interactive course module
to teach the performance-based acquisition process using current instructional technology
methods that are appropriate and challenging to adult participants from the target audience.
In addition to delivering instructional content, Opal Enterprises, Inc. intends to offer the
proposed online training system as a premier service to the government and contractor
DESIGN TREATMENT
Page 3 of 31
communities. Opal Enterprises, Inc. prides itself on its ability to produce training and instruction
for adult learners and focuses on professional development rather than secondary or tertiary
education.
3.0
Needs Analysis
The purpose of this Needs Analysis is to present the results of a needs assessment and analysis
performed for Opal Enterprises, Inc.'s performance-based services acquisition (PBSA) training
project. A needs analysis examines the current status of a goal or problem and compares it with
the desired status. The result of this comparison is a description of the discrepancies between
"what is" and "what should be." The discrepancies can then be prioritized and addressed through
the development of solutions (Altschuld & Witkin, 2000).
3.1 Client Expressed Needs
The genesis of this project came from the identification by Opal Enterprises, Inc. of a gap in the
execution of performance-based service acquisitions by the federal acquisition workforce. As
stated by Opal Enterprises in their project proposal, "the government has a knowledge deficit
within its federal acquisition workforce and functional area expertise with regards to the ability
to write, present, and evaluate performance based acquisitions." The proposal further identifies a
specific problem with regards to the skill of writing performance-based acquisition statements of
work requirements.
Related to the primary problem is the issue of the large target audience. The project proposal
indicates a desired audience of individuals involved in the acquisition process in government
(federal, state, and local), business, and academic settings. In addition, the target audience
encompasses a wide age range: 20 through 55 years.
3.2 Analysis Methodology
The methodology followed to perform this needs analysis is adapted from Altschuld and Witkin's
(2000) From Needs Assessment to Action. Elements of the methodology are the level of need
assessed and the phases of the need assessment.
3.2.1 Levels of Need and Target Groups
Altschuld and Witkin (2000) identify three levels of need: the primary level (individuals who are
the direct recipients of solutions), secondary level (individuals who delivery services to the
primary level individuals) and tertiary level (resources and inputs into the solutions). In keeping
with the author's guidance, this needs assessment focuses on the primary level groups, in this
case those directly involved with the acquisition process. In addition, the secondary and tertiary
levels of need are also considered.
3.2.2 Needs Analysis Phases
The three phases of a needs analysis are pre-assessment, assessment, and post-assessment
(Altschuld & Witkin, 2000). The pre-assessment phase consists of identifying the purpose and
major need areas (in this case, the client's expressed needs), determining data sources and
collection methods, and establishing a plan. To fulfill these requirements, the GMU Design
Team completed a performance analysis and produced a Briefing Report. The Briefing Report
DESIGN TREATMENT
Page 4 of 31
identified drivers and barriers to the design process and was used to guide the second and third
phases of the needs assessment.
The second phase of the needs assessment involves the collection and analysis of needs related
data. The project goals are analyzed to determine desired status, current status and discrepancies
between desired and current status. This needs assessment was performed using a combination
of qualitative and quantitative data collected from client/Subject Matter Expert (SME) interviews
and extant publicly-available data. The Performance Analysis identified target audience
members as another desired data source. However, due to the difficulty in obtaining access to
these individuals, the client decided to forego this data source.
The final phase of the needs assessment is the post-assessment. During the post-assessment
phase, the identified discrepancies are prioritized. This prioritized list of needs will be used in
subsequent steps of the design process to inform the Task Analysis and Treatment.
3.3 Desired Status: "What Should Be"
The first task in the needs analysis was to determine the desired status of each of the client's
expressed needs. Desired status describes the "what should be or ought to be" condition for each
element in the analysis (Altshuld & Witkin, 2000). Using the findings of the performance
analysis and instructional design best practices, the GMU Design Team broke the client
expressed needs into three elements for analysis: Performance-Based Service Acquisitions;
Audience; and Environment, Situation, and Technology.
3.3.1 Performance-Based Service Acquisitions: Desired Status
The primary resources for gathering data on the desired status of Performance-Based Service
Acquisitions have been publicly available reports and statistics as well as discussions with the
SME from Opal Enterprises. The GMU design team has gained an understanding of the PBSA
process. This outline of the PBSA is an overview of the desired outcomes for PBSA and a more
thorough task analysis is also included in this report.
The desired status is for federal government contracting officials to design and implement a
PBSA contract. Federal government agencies have mandated implementation of PBSA at
varying percentages of their total dollar amounts of acquisitions. For example, “the Under
Secretary of Defense for Acquisition and Technology directed that 50 percent of service
acquisitions be performance based by the year 2005” (RAND, Ausnik, Camm, et al., 2005).
An expert at writing PBSA contracts follows a process of several steps as outlined in the Seven
Steps to Performance-Based Service Acquisitions. The PBSA contracting process involves
gaining an understanding of the problem to be addressed and using market research to
understand possible solutions, creating the Statement of Work (SOW) or Performance Work
Statement (PWS), selecting a contractor and managing the contract. These major steps require a
wide range of knowledge, skills and abilities and it is important that those designing PBSA
maintain those elements as well as understand the greater context of PBSA.
From a skills and abilities perspective, team building, communication and writing skills are
critical for the creation of a PBSA contract. The first step of the PBSA process involves
determining the team that will be working together to create the PBSA. Though important for the
entire process, this step in particular relies heavily on team building and communication between
team members. The second step involves being able communicate and understand the desired
DESIGN TREATMENT
Page 5 of 31
results of the acquisition process in terms of the objects or services that the program staff is
requesting. Writing skills are critical to the implementation of the PBSA contract, in particular,
for the fourth step in the PBSA process, writing of the Statement of Work. Clear writing serves
to minimize contract disputes that can occur due to misunderstanding of the contract. It also
serves to create the distinction between traditional contracting where the contractor is told how to
implement the contract and PBSA where the contractor is simply told what to provide.
The knowledge required to complete the steps for PBSA includes the understanding of
regulations governing federal acquisitions, understanding of penalties and incentives that can be
written into a PBSA, and the risk implications that these penalties and incentives create.
Government contracting officials should also understand the role of performance based standards
and how to design these balancing the government’s interest with the feasibility for the
contractor. The also includes an element of risk for the contractor if incorrectly designed.
Understanding of Federal Acquisition Regulations is key to avoiding violations of these policies
and to underscore the importance of clear writing in PBSA. Additionally, by understanding the
use of penalties and incentives and standards, and the implied risks for contractors and the
government, government officials are able to write more effective PBSA contracts. Without
understanding of the risk faced by the government and contractors in PBSA, contracts can
become burdensome in their implementation.
3.3.2 Audience Analysis: Desired Status
The desired status of the target audience, the federal government acquisition workforce, is to
possess the knowledge, skills, abilities, and attitudes to successfully execute performance-based
services acquisitions. An additional facet of the desired status is that the target audience be able
to fully use and engage in the solutions is produced from this design process.
3.3.3 Environmental, Situational, and Technological Analyses: Desired Status
3.3.3.1 Environment/Situation
As described in Step 1 of the Seven Steps to Performance-Based Services Acquisitions, the
desired environmental/situational status is primarily collaboration among
“contractors…program, financial, user and even legal offices (2005). Ideally, program,
financial, user and legal staff would be located within the same building, if not the same floor, in
order to encourage an integrated solutions team environment. Also necessary for successful
performance-based service acquisitions are the need for management buy-in and clearly defined
work processes and policies.
3.3.3.2 Technology
A single computer baseline, containing the same software and version of software, for all
Contracting Specialists within the federal government would be desirable in order to ensure the
delivery and effectiveness of the training solution. As the proposed training is to be offered to
the federal government acquisition workforce through the Internet via a Learning Management
System (LMS), all users must have Internet access in order to receive the proposed training.
DESIGN TREATMENT
Page 6 of 31
3.4 Current Status
Having established the desired status for the need elements, the GMU Design Team analyzed the
collected data to determine each element's current status. The current status describes the "what
is" condition as observed by analysis and interpretation of available data.
3.4.1 Performance Based Service Acquisitions: Current Status
Based on needs outlined by Opal Enterprises and information gathered from public sources, the
current status of PBSA contracting has been examined. This area of analysis will need continued
review and additional data to fully determine the current condition.
Though mandated by federal regulations, not all agencies have yet reached their percentage goals
for implementing PBSA within their contracting. The Department of Commerce exceeded their
goals for implementing PBSA in FY 2003 and FY 2004; however, the Department of State failed
to reach their goals for PBSA implementation in FY 2003.
A variety of issues have been expressed that may shed light on why goals for implementing
PBSA have mot been met. Some contracting officials felt that they “received insufficient training
before being asked to implement PBSA practices” (RAND, Ausnik, Camm et al., 2005). The
lack of understanding of certain elements of the PBSA contract seems to be perceived by the
current audience. For example, Opal Enterprises has expressed that the government contracting
officials often have difficulty writing performance-based Statements of Work. The particular
challenge is incorporating appropriate penalties and incentives into the SOW. According to a
study conducting on Air Force contracting, “one of the biggest challenges associated with
implementation of performance-based service contracts as defined by FAR Part 37 is defining
measurable performance standards” (RAND, Ausnik, Baldwin et al. 2005).
Though government contracting officials seem to be aware of FAR regulations, there is an
additional layer of agency-specific regulations that seem to be an impediment to implementing
PBSA. According to a DOD focus group, pay caps and political complications were preventing
the incentivizing of contracts, not allowing them to become PBSA (Focus Group).
3.4.2 Audience Analysis: Current Status
3.4.2.1 Trends
The trend in government of doing more with less applies to the acquisition workforce.
According to the GAO, the total federal government-wide acquisition workforce has decreased
by 22% in the ten years from 1992 to 2002.
3.4.2.2 Occupation
The federal acquisition workforce is composed of twenty four occupations ranging from logistics
management to contracting to quality assurance to property disposal. In keeping with the
findings of the performance analysis, the audience analysis focused on one occupational code
category: the Contract Specialist, GS-1102. This series includes multiple grade levels 5 through
15. At specific levels, (5, 7, 9, 11, 12, and 13) the requirements of the position (job duties and
responsibilities, knowledge required, and guidelines) change. As of FY 2004, there were 26,936
individuals in this occupational series, the largest of the twenty four occupational groups.
DESIGN TREATMENT
Page 7 of 31
3.4.2.3 Turnover
The acquisition community continues to face a significant turnover issue. In FY 2004, 15% of
Contracting Specialists were eligible to retire, with more than half expected to reach retirement
age in the subsequent decade. During FY 2004, the GS-1102 series experienced a loss rate of
9%.
3.4.2.4 Demographics
The GS-1102 series employees are majority female. Average age is 46.90 years with a range
between 20 and 55 years. With the impending retirement wave, the average age of the audience
may change. The average age of internal hires is 45 years, and for external hires is 34 years. No
other demographic data (e.g. native language, special needs) were available. Furthermore,
statistics from the Office of Personnel Management show that 6.71 percent of federal
government white-collar workers are disabled.
3.4.2.5 Education
As with demographics, the increasing turnover of Contracting Specialists is causing a shift in the
educational background of the audience. College graduates make up 67% of the current total
population. However, of all persons (internal and external) hired as Contracting Specialists, 79%
have college degrees. An even higher percentage of external hires, 88%, possess a college
degree. Of the current workforce members who are college graduates, more than two-thirds
possess business, law, or public administration degrees. Post-graduate degrees comprise 36% of
total degree holders.
3.4.2.6 Experience
Of those new to the Contracting Specialist series, just over one half (51%) were hired from
outside the government. Another 45% were recruited from inside the government. Specific data
on other experience types and levels has been difficult to obtain.
3.4.2.7 Attitudes
Some indication of audience attitudes has been obtained through focus group transcripts. In a
report on the Acquisition Reform Focus Group (1997), several attitude issues were identified.
Chief among them was a reluctance in both government and contractor culture to make the
necessary change in organizational culture. The target audience didn't see the need or value in
using performance-based contracting methods. One respondent stated that "getting folks to go
performance service contracting has been a problem since 1981" so it would appear that
resistance is long standing and deep seated (Acquisition Reform Focus Group, 1997 p. 33).
3.4.2.8 Summary
The target audience for this project is a diverse group facing many challenges. The acquisition
workforce is shrinking in size with an increasing percentage retiring and being replace by new
hires. The acquisition profession comprises many occupations and each occupation contains
multiple levels of knowledge, skill, and responsibility. The target audience skews female and
older than the general population, though the average age may drop due to the anticipated
turnover. The target audience is highly educated with two-thirds of the population possessing
college degrees. This is expected to increase with nearly nine of every ten new hires being
DESIGN TREATMENT
Page 8 of 31
college graduates. Experience is reasonably evenly divided between government and nongovernment background. Finally, there is indication in the existing population of resistance to
change, though the expected population turnover may also impact this.
3.4.3 Environmental, Situational, and Technological Analyses: Current Status
3.4.3.1 Environment/Situation
Due to lack of access to federal employees from differing government agencies, determining the
current environment/situation for each federal agency is not feasible at this time. However, a
member of the GMU Design Team can verify that at the United States Patent and Trademark
Office (USPTO), under the Department of Commerce, contracting specialists, financial, user and
legal staff are located within separate buildings and floors.
3.4.3.2 Technology
Each government agency may have their own requirements for “baselined” computers; therefore,
operating system/software that one agency uses will not be the same at a different government
agency. The USPTO alone has eleven different computer baselines within the organization (C.
Pham, personal communication, November 28, 2005). It is presumed that all federal agencies
have Internet access, and would therefore be able to receive training via LMS.
3.5 Discrepancies and Identified Issues
Discrepancies are the difference between the desired status and the current status. Discrepancies
are "the very heart" of needs assessment (Altschuld &Witkin, 2000, p. 83). The identification of
discrepancies and related issues provides an indication of the causes behind needs and suggests
possible solutions.
3.5.1 Performance Based Service Acquisitions
Discrepancies between the desired status and the current status seem to center on the inability of
government contracting officials to design particular elements of the PBSA. The writing of the
SOW, and in particular, performance measurement portion of the SOW has been indicated to be
problematic. Further data on the knowledge of regulations, understanding of penalties and
incentives, and communication skills needs to be gathered to further determine where
discrepancies may lie.
3.5.2 Audience
The discrepancies and related identified issues between the desired status and the current status
of the target audience are:
•
The target audience is well educated adult learners with wide ranging ages. In addition,
high expected turnover is directly impacting the knowledge base and experience of the
target audience, and may change the currently observed demographics. In addition, it is
expected that a percentage of the audience will have some type of disability. A need
exists for a solution that accommodates the requirements of adult users with varying
abilities and experiences.
DESIGN TREATMENT
Page 9 of 31
•
Due to reduction in the size of the population, the target audience has limited resources to
devote to training or other solution interventions. A need exists for solutions that can be
used with minimum disruption to the daily activities and responsibilities of the learners.
•
There is indication that there is a resistant attitude toward learning the new tasks and
culture required for performance-based service acquisitions. A need exists for a solution
that positively impacts user attitudes toward PBSA.
3.5.3 Environmental, Situational, and Technological Analyses: Current Status
The discrepancies between the desired environmental/situational/technological statuses are as
follows:
4.0
•
Staff members may be located on different floors, different buildings, or different
city/stat/country.
If this is the case, collaboration tools such as email,
synchronous/asynchronous discussion boards, teleconferencing, etc. may lessen the
effects of numerous staff locations.
•
Multiple computer baselines (such as operating systems, software, etc.), not only
between federal agencies but within federal agencies, creates a challenge when
delivering Web-based training. There is a need for training that can be received on
various platforms and through multiple browsers.
Task Analysis
The purpose of this Task Analysis is to present the results of a task analysis performed for Opal
Enterprises, Inc.'s performance-based services acquisition (PBSA) training project. A task
analysis provides a description of the activities required to complete a specific job, activity, or
outcome. Task analysis provides a blueprint for the further development of learning objectives,
instructional strategies, and assessments (Jonassen, Tessmer, and Hannum, 1999).
4.1 Task Analysis Methodology
The methodology followed to perform this needs analysis is adapted from a method found in
Jonassen et al. (1999) Task Analysis Methods for Instructional Design. The task analysis method
employed is a subject matter content analysis method known as the master design chart (MDC).
According to Davies (1976), a master design chart is a method for developing a course
curriculum through analysis of content areas and related behaviors. In this MDC, the behaviors
are organized using Gagné's taxonomy of knowledge. Each taxonomic level is broken out into
specific sub-behaviors. The second axis contains a break out of the subject matter into content
headings, in this case categorized by the overarching tasks of the performance-based service
acquisitions' "seven steps." Finally, numbers are placed in the cells of the body of the table
representing the intersection of a behavior with a content item. In the MDC used for this
analysis, the numerical rankings indicate importance of the content item/behavior dyad. If there
is no significant interaction between a behavior and a content item, a dash is used. As a task
analysis tool, the MDC focuses on the task as performed in the work environment. The learning
objectives that follow then use this task analysis to determine student outcomes in terms of the
learning environment.
The benefits of using a master design chart are that it provides a graphical depiction of the
decisions made in development. The chart may also reveal the assumptions upon which the
DESIGN TREATMENT
Page 10 of 31
course is based (Davies, 1976). It is also useful for identifying and emphasizing the structure of
the content, expressing relative importance of tasks, guiding creation of learning objectives, and
suggesting objectives sequencing (Jonassen, et al. 1999). Finally, according to Davies (1976),
"[a]s a communication vehicle of the curriculum, the design chart is probably without equal (p.
174)."
4.2 Master Design Chart
A master design chart for performance-based service acquisitions is presented below. The
source of the task descriptions is derived from the "Seven Steps to Performance-Based Services
Acquisition." The key for the numerical rankings is as follows:
3
High importance
2
Moderate importance
1
Baseline importance
-
Not applicable
Conduct
Conduct as a team
Maintain
Choose
Apply carefully
Appraise
Analyze
Create
Create link to acquisition
Organize
Use
Evaluate
Describe
Describe in terms of results
Appreciate
Verbal
Explain
Problem-Solving
Identify
Intellectual
Attitude
BEHAVIOR
Senior management support
1
-
-
-
1
-
-
-
-
2
-
-
-
-
-
-
-
Senior management involvement
1
-
-
-
1
-
-
-
-
2
-
-
-
-
-
-
-
Multi-disciplinary expertise
1
-
-
-
-
1
-
-
-
-
-
1
1
-
-
-
-
Roles and responsibilities
1
1
-
-
-
-
-
-
-
-
-
2
2
-
-
-
-
Rules of conduct
1
1
-
-
-
-
-
-
-
-
-
2
2
-
-
-
-
Empowerment of team members
1
-
-
-
-
-
-
-
-
1
-
1
-
-
-
-
-
Stakeholders
2
-
-
-
-
1
-
-
-
-
-
-
-
-
-
-
-
Consensus
1
-
-
-
-
-
-
-
-
2
-
-
-
-
-
-
-
Knowledge base
1
-
-
-
2
-
-
-
-
-
-
-
-
-
-
-
-
Team Incentives
1
1
-
-
-
1
-
-
-
-
-
-
1
-
-
-
-
CONTENT
Integrated Solutions Team
DESIGN TREATMENT
Page 11 of 31
Conduct
Conduct as a team
Maintain
Choose
Apply carefully
Appraise
Analyze
Create
Create link to acquisition
Organize
Use
Evaluate
Describe
Describe in terms of results
Appreciate
Verbal
Explain
Problem-Solving
Identify
Intellectual
Attitude
BEHAVIOR
Mission and performance objectives
1
1
-
-
-
-
-
-
-
-
1
-
-
-
1
-
-
High level desired results
1
1
-
2
-
2
-
-
-
-
-
-
-
-
1
-
-
Success indicators
1
1
-
-
-
2
-
-
-
-
-
-
-
-
1
-
-
Current status
1
1
-
-
-
-
-
2
-
-
-
-
-
-
1
-
-
Market research
1
-
1
-
-
-
-
-
-
-
-
-
-
-
1
-
-
Public-sector counterparts research
1
-
2
-
-
-
-
2
-
-
-
-
-
-
-
-
-
Private-sector company research
1
-
2
-
-
-
-
2
-
-
-
-
-
-
-
-
-
One-on-one industry meetings
1
-
1
-
-
-
-
-
-
-
-
-
-
1
-
-
-
Existing contracts research
1
-
1
-
-
1
-
-
-
-
-
-
-
-
-
-
-
CONTENT
Problem or Need
Private-sector & Public-sector Solutions
Performance Work Statement (PWS)
-
Desired outcomes
1
1
-
-
-
1
-
-
1
-
-
-
-
1
-
-
Outcome analysis
1
3
3
-
-
-
-
-
2
1
-
-
-
2
1
-
-
Performance analysis
1
1
3
-
-
-
-
-
2
1
-
-
-
2
1
-
-
Measurable performance standards
1
2
-
-
-
3
-
-
-
-
-
-
-
-
1
-
-
Acceptable Quality Levels (AQLs)
1
2
-
-
-
3
-
-
-
-
-
-
-
-
1
-
-
Importance ("so what")
1
2
-
-
-
-
-
1
-
-
-
-
-
1
1
-
-
Outcome analysis matrix
1
1
-
-
-
-
-
-
-
3
-
-
-
-
1
-
-
Performance Work Statement
1
-
-
-
-
-
-
-
-
3
-
-
-
-
1
-
-
-Requirements
1
1
-
-
-
-
-
-
-
1
-
-
-
-
1
1
-
-Measurable performance standards
1
1
-
-
-
-
-
-
-
1
-
-
3
-
1
-
-
-Quality Assurance Surveillance Plans (QASPs)
1
1
-
-
-
-
-
-
-
2
-
-
3
-
1
-
-
-Performance incentives
1
1
-
-
-
-
-
-
-
1
-
-
1
-
1
-
-
-Process descriptions and "how" statements
1
1
-
-
-
-
-
-
-
1
-
-
1
-
1
-
-
-Contractor problem solution
1
1
-
-
-
-
-
-
-
1
-
-
1
1
1
-
-
-
-
2
-
-
-
-
1
1
-
-
-
1
1
1
-
-
Final checks
Statement of Objectives (SOO)
DESIGN TREATMENT
Page 12 of 31
CONTENT
Elevator message
Conduct
Conduct as a team
Maintain
Choose
Apply carefully
Appraise
Analyze
Create
Create link to acquisition
Organize
Use
Evaluate
Describe
Describe in terms of results
Appreciate
Verbal
Explain
Problem-Solving
Identify
Intellectual
Attitude
BEHAVIOR
1
1
-
-
-
-
-
-
-
-
-
-
1
-
1
-
-
Scope
1
1
-
-
-
-
-
-
-
-
-
-
-
1
1
-
-
Performance objectives
1
3
-
-
-
-
-
-
-
3
-
-
3
-
1
-
-
Government and contractor objective sharing
1
1
-
-
-
-
-
-
-
-
-
-
1
-
1
-
-
Constraints
1
1
-
-
-
-
-
-
2
-
-
-
-
1
1
-
-
Project Background
1
1
-
-
-
-
-
-
-
1
-
-
1
-
1
-
-
Final checks
1
1
2
-
-
-
-
1
1
-
-
-
1
1
1
-
-
Performance Measurement and
Management
Success determinants
1
-
-
-
-
-
-
-
1
-
-
-
-
-
-
-
-
Commercial quality standards
1
-
-
-
-
1
-
-
-
-
-
-
2
-
-
-
-
Contractor metrics proposal
1
-
-
-
-
-
-
-
-
-
-
-
1
-
-
-
-
Contractor quality assurance plan proposal
1
-
-
-
-
-
-
-
-
-
-
-
1
-
-
-
-
Success measures
1
-
-
-
-
1
-
-
-
-
-
-
-
1
-
-
-
Contractual language for negotiated changes
1
-
-
-
-
-
-
-
-
-
-
-
1
-
-
-
-
Contract-type order of precedence
1
-
-
-
-
-
1
-
-
-
-
-
-
-
-
-
-
Incentive-type contracts
1
-
-
-
-
-
-
-
-
-
-
-
3
-
-
-
Award term
1
-
-
-
-
2
-
-
-
-
-
-
1
1
-
-
-
Other incentive tools
1
-
-
-
-
-
-
-
-
-
-
-
3
2
-
-
-
Profit as motivator
1
-
-
-
-
-
-
-
-
-
-
-
-
-
-
-
1
Management by relationship
1
-
2
-
-
-
-
-
-
-
-
-
-
3
1
-
-
Solution competition
1
-
2
-
-
-
-
-
-
-
-
-
1
-
-
-
-
Downselection
1
-
1
-
-
1
-
-
-
-
-
-
1
2
-
-
-
Due diligence
1
-
1
-
-
-
1
-
-
-
-
-
1
2
-
-
-
Oral presentation and communications
1
-
1
-
-
-
-
-
-
2
-
-
1
-
-
-
-
Past Performance
1
-
-
-
-
-
-
-
1
-
-
-
-
3
-
-
-
Best-value evaluation
1
2
-
-
-
3
-
1
-
-
-
-
1
3
-
-
-
Source selection rules
1
-
-
-
-
-
2
-
-
-
-
-
1
-
-
-
-
Contractor Selection
DESIGN TREATMENT
Page 13 of 31
Conduct
Conduct as a team
Maintain
Choose
Apply carefully
Appraise
Analyze
Create
Create link to acquisition
Organize
Use
Evaluate
Describe
Describe in terms of results
Appreciate
Verbal
Explain
Problem-Solving
Identify
Intellectual
Attitude
BEHAVIOR
1
-
-
-
-
-
-
-
2
-
-
-
-
2
-
-
-
Roles and responsibilities
1
-
-
-
1
-
-
1
-
-
-
-
-
1
-
-
-
Contract performance management
accountability
Kick-off Meeting
1
-
-
-
-
2
-
-
-
-
-
1
1
-
-
-
-
1
-
-
-
-
-
-
-
-
-
-
-
1
-
-
-
-
Contract Performance Improvement Working
Group
Right questions
1
-
-
-
-
-
-
1
1
-
-
-
2
3
1
-
-
1
-
-
-
-
-
-
-
-
1
-
-
1
-
1
-
-
Contractor performance report
1
-
-
-
-
-
-
1
-
1
-
-
-
1
1
-
-
CONTENT
Solutions' conflict of interest
Performance Management
4.3 Learning Objectives
Based on the seven steps outlined for performance-based services acquisition and the results of
the master design chart, a series of terminal and enabling objectives have been developed. The
terminal objectives align with each step of the acquisition process while the enabling objectives
are critical areas necessary for mastering the terminal objectives.
STEP 1: Establish an Integrated Solutions Team
Terminal Objective: The learner will be able to assemble a team when presented with the
requirement of conducting a performance-based acquisition.
Enabling Objectives:
1. The learner will be able to create support within senior management for the performancebased acquisition.
2. The learner will be able to generate involvement from senior management for the
performance-based acquisition.
3. The learner will be able to select multi-disciplinary experts appropriate for the specific
PBSA and who are among the best in their field.
4. Given the need to establish an integrated solutions team, the learner will be able to define
the roles and responsibilities for team members.
DESIGN TREATMENT
Page 14 of 31
5. Given the need to establish an integrated solutions team, the learner will be able to
develop rules of conduct.
6. Given the need to establish an integrated solutions team, the learner will be able to
explain the importance of empowering team members.
7. The learner will be able to identify stakeholders dependent on the content of the specific
PBSA.
8. The learner will be able to recognize the importance of nurturing consensus within the
integrated solutions team.
9. The learner will be able to explain the importance of maintaining the human knowledge
base over the project life.
10. The learner will be able to explain the importance of correlating program mission to team
members’ performance through incentives.
STEP 2: Describe the Problem that Needs Solving
Terminal Objective: The learner will be able to describe the problem that the performance-based
acquisition intends to solve.
Enabling Objectives:
1. The learner will be able to describe the relationship between the acquisition and the
agency’s mission goals.
2. The learner will be able to define the desired results of the acquisition in an overarching
statement of purpose.
3. The learner will be able to select the criteria for measuring successful accomplishment of
the desired results.
4. The learner will be able to describe the current level of performance of the activity which
the acquisition is intended to impact.
STEP 3: Examine Private-sector & Public-sector Solutions
Terminal Objective: The learner will be able to explain the importance of conducting (as a team)
market research to identify, analyze and describe private and public-sector solutions to PBSA.
Enabling Objectives:
1. The learner will be able to identify, compare and contrast PBSA solutions among
counterparts in other government agencies to analyze best practices and lessons learned.
2. The learner will be able to describe how to conduct research on private-sector
commercial offerings, capabilities and practices.
3. The learner will be able to appraise the effectiveness of one-on-one meetings with
industry leaders.
DESIGN TREATMENT
Page 15 of 31
4. The learner will be able to identify and choose existing PBSA contracts to see if an
existing contract is available to meet agency requirements
STEP 4: Develop Performance Work Statement (PWS) or Statement of Objectives (SOO)
Terminal Objective: The learner will be able to develop a description of the requirement of a
PWS or SOO in terms of results or outcomes, measurable performance standards, and acceptable
quality levels.
Enabling Objectives (PWS):
1. The learner will be able to determine the desired outcomes to satisfy the requirement of
the PBSA.
2. The learner will be able to describe how to conduct an outcome analysis to determine
what tasks must be accomplished to arrive at the desired outcomes.
3. The learner will be able to describe how to conduct a performance analysis to identify
performance standards and determine acceptable deviation levels.
4. The learner will be able to identify and explain the essential inputs, processes, and
outputs during job analysis.
5. The learner will be able to create a matrix identifying the results of the analysis.
6. The learner will be able to create a performance work statement that describes the
requirements in terms of results, rather than processes.
7. The learner will be able to create and use measurable performance standards and quality
surveillance plans.
8. The learner will be able to determine when provisions for reductions of fees or price are
acceptable.
9. The learner will be able to identify and create provisions for performance incentives
where appropriate.
10. The learner will be able to conduct a thorough inspection of the PWS to determine if any
requirements are non-essential and eliminate them.
11. The learner will be able to conduct a thorough inspection of the PWS to search for any
process descriptions or “how” statements and remove them, thereby allowing the
contractor to craft innovative solutions.
Enabling Objectives (SOO):
1. The learner will be able to describe the process for creating opening statement that
identifies how the acquisition relates to the agency’s program or mission need.
2. The learner will be able to explain how defining the scope in a SOO will assist
contractors in understanding the size and range of the services needed.
3. The learner will be able to create performance objectives in the SOO that will define
measurable, mission-related objectives.
DESIGN TREATMENT
Page 16 of 31
4. The learner will be able to explain the importance of the government and contractor
objective-sharing to meet the same goal.
5. The learner will be able to identify any constraints, such as existing policies, directives
and standards that may hinder the contractor in their ability to propose innovative
approaches and solutions.
6. The learner will be able to describe the importance of including background information
and current environment of the program in order for contractors to perform due diligence.
7. The learner will be able to conduct a final inspection of the SOO to evaluate the content
and eliminate any content that is non-essential.
STEP 5: Decide How to Measure & Manage Performance.
Terminal Objective: The learner will be able to develop a plan for the measurement and
management of contractor performance in the performance-based acquisition.
Enabling Objectives:
1. The learner will be able to apply commercial quality standards as part of the quality
measurement of the performance-based acquisition.
2. The learner will be able to describe the reasons for allowing the contractor to propose the
metrics and the quality assurance plan for the performance-based acquisition.
3. The learner will be able to select the minimum number of measures on which to judge the
success of the performance-based acquisition.
4. The learner will be able to use contractual language for negotiated changes to the metrics
and measures in the performance-based acquisition.
5. The learner will be able to apply the contract-type order of precedence as regulated by
law to the performance-based acquisition.
6. The learner will be able to apply the appropriate type of incentive contract based on the
needs of the specific performance-based acquisition.
7. The learner will be able to select the appropriate award term based on the needs of the
specific performance-based acquisition.
8. The learner will be able to use incentive strategy tools incorporating multiple
performance incentives based on the needs of the specific performance-based acquisition.
9. The learner will be able to recognize the importance of profit as motivation for the
contractor in a performance-based acquisition.
10. The learner will be able to conduct management by relationship in a performance-based
acquisition.
DESIGN TREATMENT
Page 17 of 31
STEP 6: Select the Right Contractor.
Terminal Objective: The learner will be able to select the best contractor to implement the
performance-based acquisition based on limiting the risk to the government and the contractor’s
ability to achieve the performance goals outlined in the acquisition.
Enabling Objectives:
1. The learner will be able to select a method for limiting the competitive pool to those
contractors most likely to offer a successful solution.
2. The learner will be able to design a plan for contractors to perform due diligence.
3. The learner will be able to use oral presentations to communicate with bidding
contractors during the selection process.
4. The learner will be able to evaluate the likelihood of success based on the past
performance of the contractor.
5. The learner will be able to use best-value evaluation in the source selection process.
6. The learner will be able to assess solutions for issues of conflict of interest.
STEP 7: Manage Performance.
Terminal Objective: The learner will be able to demonstrate management of the contractor’s
performance of the performance-based acquisition.
Enabling Objectives:
1. The learner will be able to evaluate the roles and responsibilities of members of the
integrated solutions team.
2. The learner will be able to organize accountability for managing contract performance.
3. The learner will be able to use the formal kick-off meeting to add the contractor to the
integrated solutions team.
4. The learner will be able to apply the Six Disciplines of Performance-Based
Management™ to the contract management process.
5. The learner will be able to evaluate contractor performance in a Contract Performance
Improvement Working Group.
6. The learner will be able to identify questions emphasizing performance improvement for
the performance-based acquisition.
7. The learner will be able to describe the contractor’s performance for past performance
reporting requirements.
5.0
Instructional Design Approach
5.1 Course Design Overview
To address the gaps identified in the needs analysis, the solution proposed by this design
treatment is a multimedia hypermedia instructional course on the seven steps of the performanceDESIGN TREATMENT
Page 18 of 31
based services acquisition process. This design was selected based on the results of the needs
analysis and task analysis. Specific benefits of each design element are addressed below. An
overview of the instructional design approach for each lesson follows.
5.1.1 Multimedia Approach
The instructional course will use multimedia: on-screen text, voice narration, audio cues, still
graphics, and animation. The rationale for the use of multimedia is threefold. First, the multiple
methods of presentation will accommodate the assumed multiple learning styles (e.g. auditory
vs. visual) of the diverse audience as described in the needs analysis. Second, the complexity of
the topic will require the depiction and explanation of information via spatial and temporal
methods. Third, to promote involvement of the adult learners as proscribed by andragogical
model of learning (Knowles, 1990), the course will use scenario-based practice and assessment.
The use of multimedia will enhance the authenticity of the scenarios.
5.1.2 Hypermedia Development and Distribution Environment
A hypermedia (Internet/World Wide Web) development and distribution environment has been
chosen for this course. A Web-based hypermedia environment will accommodate the target
audience learners in their diverse physical locations and work environments. Hypermedia will
provide the learner locus of control and self-paced requirements of the andragogical learning
model (Knowles, 1990). Finally, use of a Web-based hypermedia development and distribution
environment fulfills the capabilities and goals of the client organization, Opal Enterprises, Inc.
5.1.3 Organization and Presentation
A content and task analysis determined that the course should be organized according to the
seven step process for performance-based acquisitions. While there are some over-arching task
similarities between lessons, the content is overwhelmingly procedural and sequential. It is
believed that the learners will benefit most from maintaining the structure of the seven step
process rather than reorganizing the content according to some other schema. Depth of coverage
of each topic will be determined from the importance ratings in the master design chart
developed in the task analysis. To accommodate learners from both government and private
industry, the course will contain two versions of each module. As recommended by the
performance analysis, this design treatment focuses on the federal government contractor
audience perspective. See Figure 5-1 for a flowchart of the course sequence.
5.1.4 Course Introduction
The course introduction module will serve a variety of functions. When learners begin the
course, they will be presented with an animation sequence, perhaps of a variety of types of
services that can be purchased through performance-based acquisitions. At the end of the
sequence, learners will be asked to select their role as a government acquisitions employee, a
government program employee or a contractor. Learners will then be presented with background
information on performance-based acquisition such as the regulation of increased use in
government contracting and the general overall philosophy of PBSA. After being provided with
this general information, the purpose of the training course and the terminal objectives for the
course will then be presented based on the learner’s selection of their role.
DESIGN TREATMENT
Page 19 of 31
Figure 5-1: Course Sequence
DESIGN TREATMENT
Page 20 of 31
The introduction module has been designed to address the first two events of instruction as
outlined by Gagné, “gaining attention” and “informing the learner of the objective” (Smith &
Ragan, 1999, p. 114). Specifically, the use of the animation sequence and asking the learner to
select their role, are designed to gain the learner’s attention. For a procedure such as
implementing performance-based services acquisition, Smith and Ragan suggest that this can be
done either through asking a question requiring application or by demonstrating application of
the procedure (1999, p. 216). Since it is not possible to present an entire situation demonstrating
PBSA within the confines of the introduction, the use of the animation serves the purpose of
gaining attention in the same way as an unusual picture which Smith and Ragan suggest for
introducing a concept (1999, p. 182). Animation also serves to arouse interest in the topic while
the need to implement PBSA for their jobs serves as motivation for many adult learners.
Within the background information, the point that PBSA is a government requirement further
strengthens the motivation. The background information also helps to stimulate “recall of
prerequisite learning” (1999, p. 114), Gagné’s third event of instruction. Despite their diverse
backgrounds in government acquisitions, all learners should have a general understanding of the
fact that PBSA is mandated by the federal government and that it is a type of acquisition, distinct
from traditional methods. Presenting the objectives for the entire course serves to prepare
learners for the material and to set expectations (1999, p 116).
5.2 Lesson Overview
Each of the main content lessons will follow the expanded events of instruction as described by
Smith and Ragan (1999). This approach, an elaboration of Robert Gagne's events of instruction,
includes four major instructional events: an introduction, body, conclusion, and assessment.
Each event, and its implementation in the present design, is described following. See Figure 5-2
for a flowchart of the lesson sequence.
5.2.1 Introduction
The lesson introduction will serve to engage the learner's attention on the lesson, provide
information on the purpose of the lesson, motivate the learner, and present an overview of lesson
activities. Each lesson will begin with a brief multimedia animation displaying content related to
the lesson topic to aid the learner in attending to the upcoming instruction. Following the
animation, the learner will be presented with the purpose and objectives of the lesson and told
why the information in the lesson is important to them. Finally, an outline of lesson activities,
along with average expected completion times, will be displayed. The events in the introduction
are intended to stimulate interest, expectancy, and motivation in the learner. The introduction
fulfills the first two steps of Gagne's (1992) events of instruction (gaining attention and
informing the learner of the objective). It also provides the "need to know" element of Knowles'
(1990) andragogical model of learning.
5.2.2 Body
The body of each lesson will begin by reviewing relevant prior knowledge to facilitate learner
recall of previously learned information. Because this course is being developed for learners of
differing experience levels, additional hyperlinked information and learning aids will be
provided as needed to provide access to definitions and other basic concepts to scaffold learners
of minimal prior experience in the content domain.
DESIGN TREATMENT
Page 21 of 31
Figure 5-2: Lesson Sequence
DESIGN TREATMENT
Page 22 of 31
Next, the learner will be presented with alternating instruction and practice activities for each
lesson’s enabling learning objective or sub-objective. The instructional content associated with
each enabling learning objective will provide content information and examples using
appropriate media and instructional strategies to focus attention. Following the content
presentation, the learner will participate in a guided practice activity. The learner will perform
the actions in the corresponding learning objective. Scaffolding will be provided via help
prompts, reference material, and the like. The alternating instruction/practice design of the
lesson body incorporates Gagne's (1992) third through sixth events of instruction (stimulating
recall of prerequisite training, presenting the stimulus material, providing learning guidance, and
eliciting the performance). In addition, the additional scaffolding provided in the lesson supports
the variation of learner's experience identified in the andragogical model of learning (Knowles,
1990).
5.2.3 Conclusion
Following all of the instructional and practice sections, each of the practice sessions will be
combined into a summary of the subtopic practices. Each of the subtopics will be linked together
and when combined, the subtopic practices will form a scenario demonstrating the entire step of
the module. Here the learner will receive feedback on their choices from the practice sessions.
Feedback will include specific suggestions to improve the learner’s choices in the practice
sessions and will direct the learner to review instructional screens as needed.
This portion of each module serves to provide feedback, the seventh event of instruction (p. 114).
Whereas each practice session elicits performance, the informative feedback provided gives
“learners the opportunity to consider information about the appropriateness of their responses
during practice” (p. 119). By summarizing the feedback in one section, learners will be able to
visualize how the subtopics presented during the module form a cohesive step in the PBSA
process. They will also see how the choices they have made impact later substeps in the process.
5.2.4 Assessment
After receiving feedback from the practice sessions and having the opportunity to review specific
subtopics as needed, learners will be assessed on their understanding of the information and
process presented in the entire module. This assessment will be similar to the scenario presented
in segments during the practice sessions but will be unified as in the feedback section of the
module. Learners will be asked to respond to the scenario as if they were performing a step of a
performance-based services acquisition. As in the practices, learners will respond to elements of
the scenario based on the subtopics of the step by selecting elements based on information
provided in the scenario, identifying appropriate material, or providing written answers and
explanations. The score on the assessment will be based on a rubric with a passing score to be
determined by Opal Enterprises. Even with a score equal to or greater than the determined
passing score, predetermined critical tasks must be completed correctly in order for learner’s to
successfully complete the module. These critical tasks more successful completion must also be
determined by Opal Enterprises.
This method of assessment was chosen based on the need expressed by Opal Enterprises for
learners to be assessed by producing information rather than answering multiple choice
questions. This ties in to Smith and Ragan’s statement that “The ultimate assessment for a
procedure is the learner’s ability to correctly apply the procedure” (p. 221). By having each
DESIGN TREATMENT
Page 23 of 31
module’s assessment serve as a segment of a scenario, the learner will be able to complete a
performance-based services acquisition in its entirety during the instruction and will be assessed
directly on completion of this PBSA.
Each module will conclude with a review of the learning objectives of the entire module and
with the learner receiving feedback from their module assessment. Should the learner not
complete the module successfully, he or she will be informed and required to repeat the module.
Feedback provided should be specific enough to direct learners to which specific subtopic should
be reviewed but not such that it provides the answer to the learner.
This conclusion serves to address Gagné’s ninth event of instruction, enhancing retention (p.
114). It also addresses Smith and Ragan’s element to provide final feedback to learners and
allowing them to seek remediation (p. 222). By providing feedback in the manner addressed
above, learners will be informed “of what specific component of knowledge has not been
mastered” (p. 222).
6.0
Technical Design Approach
6.1 SCORM
The Sharable Content Object Reference Model (SCORM) is a design, development, and delivery
standard for e-learning created by the U.S. Department of Defense and the Advanced Distributed
Learning Initiative. The goal of the SCORM standard is to provide guidelines that result in
Web-based training that is interoperable and reusable. The latest version of the SCORM standard
is SCORM 2004. Opal Enterprises has expressed a need for the PBSA course to conform to the
SCORM standard. The design and development of the content of the PBSA course as described
in this design treatment will be consistent with the SCORM content design requirements. The
technical requirements of the SCORM (e.g. XML manifest files) is beyond the scope of this
effort.
6.2 Accessibility
The audience analysis shows that 6.71 percent of federal government workers have some form of
disability. Therefore, it is expected that some members of the target audience for the PBSA
course will have a disability that may required the use of assistive technology for interaction with
a personal computer. Furthermore, Opal Enterprises has expressed a requirement for the course
development to comply with the requirements of the 1998 Amendment to Section 508 of the
Rehabilitation Act. These “Section 508” standards describe design and development standards
to accommodate individuals with disabilities or who rely on assistive technology. In
consideration of these needs, the PBSA course will be designed to be consistent with the Section
508 standards.
6.3 Authoring and Delivery Environment
Opal Enterprises has identified an authoring tool (Apixel) and delivery environment
(Blackboard’s Learning Management System) for the PBSA course. Upon initial review of these
applications, it is believed that they will support development of the course as described in this
design treatment. Adjustments of the design may be required to meet the requirements of the
authoring and delivery tools.
DESIGN TREATMENT
Page 24 of 31
6.4 Multimedia
Based on the diverse audience characteristics and the expectancies of adult learners, use of
multimedia is warranted. The use of multiple methods of content delivery (visual, auditory, and
textual) will accommodate different learning styles. Use of dynamic and interactive multimedia
(e.g. animation) will add realism to practice exercises, provide engagement opportunities, and
shift locus of control towards the learner. Types of multimedia and interactivity will be subject
to the constraints of the authoring tool and delivery environment.
6.5 Prototype Storyboards
The prototype storyboards presented in Figure 6-1 through Figure 6-4 represent an initial
concept for inclusion of the instructional and technical elements discussed in the previous
sections. The storyboards show the major components of the design strategy: student type
selection, main course menu, typical instructional screen, and assessment feedback screen. They
are "wireframe" storyboards, focusing on placement of functional elements in the course's user
interface. Further development of storyboards to address look-and-feel and other issues will take
place in future design and development steps.
Figure 6-1: Audience Selection
DESIGN TREATMENT
Page 25 of 31
Figure 6-2: Main Menu
Figure 6-3: Typical Instructional Screen
DESIGN TREATMENT
Page 26 of 31
Figure 6-4: Assessment Feedback
7.0
Formative Evaluation Methodology
A formative evaluation of the proposed design of the training package will be a critical step in
the design process, as the content area is new to the GMU Design team, and this instruction is
expected to be disseminated among many government agencies and the private sector (Smith &
Ragan, 2005, p. 328). There are four stages of formative evaluation that the GMU Design team
has been and/or anticipates conducting: design reviews, expert reviews, learner validation and
ongoing evaluation.
7.1.1 Design Reviews
The GMU Design team has received feedback at each stage of the design (performance analysis,
needs analysis and task analysis) from the EDIT 730 course professor, instructional design
students, and the client. Much of this feedback has been incorporated into the revised versions of
each document, which will be used as a basis for the design of the instructional content. As the
development of the instructional prototype occurs, further design reviews will be forthcoming.
Certain changes from the design review have been included in the final analysis and treatment.
One such adjustment included revising the distinction between business goals and learning goals.
This change was made as many of the original business goals do in fact have a great deal of
impact on the learners and the instruction. Another suggestion was to include a brief definition of
PBSA in order to provide additional background information to non-SME readers. This has been
added to the introduction section of the design document.
The design team did not to make certain changes recommended by the design review. These
included securing audience members from the federal government that write, present, and
evaluate performance based services acquisitions. Due to limited access to federal government
DESIGN TREATMENT
Page 27 of 31
employees, this was not feasible but a great deal of information addressing the concerns of the
government workforce about PBSA was obtained from publicly available resources. A
suggestion that was also made was to further distinguish between “Performance Measurement
and Management” and “Performance management” in the Master Design Chart. The design team
determined that it was not necessary to alter this in the Master Design Chart because though the
headings are similar, the content addressed in each section is quite distinct.
7.1.2 Expert Reviews
The instructional design prototype of the PBSA modules will undergo expert review by the SME
(also our client), who will review instructional content for accuracy, consistency of perspective,
practicality of examples and practice exercises, and appropriateness of the instruction and
instructional strategies (Smith & Ragan, 2005, p. 329).
7.1.3 Learner Validation
The GMU Design team anticipates conducting several “field tests” with individual learners and
small groups to ensure the proposed instruction will meet the needs of the learners. Smith and
Ragan (2005, p. 330) recommend designers should attempt to answer the following questions:
•
Do the learners understand the instruction?
•
Do learners know what to do during the practice and tests?
•
Can learners interpret graphics in the text?
•
Can the learners read all of the textual material?
This list is not exhaustive, but rather provides a starting point for supplementary questioning.
7.1.4 Ongoing Evaluation
It is anticipated by the GMU Design team that the formative evaluation process will not cease
even after the instructional design and content has been implemented by the government and
private sector. Potential content changes/updates, skills of the learners, or even environment
could impact the effectiveness of the instructional goals and objectives.
8.0
Recommendations
8.1 Initial Development Focus
The GMU Design team recommends focusing on step 4 of the Seven Steps to PBSA for the
initial prototype development. This recommendation stems from the criticality of tasks in this
module as displayed in the Master Design Chart from the Task Analysis Treatment, and from
feedback received during the formative evaluation of the task analysis.
Additionally, as the client has requested that each module encompass approximately 30 minutes
of instructional time, it is possible that step 4, the most complex module, will need to be divided
into two smaller modules. The design team suggests that this can be done most effectively by
dividing the instruction between the writing of a Statement of Objectives and the writing of a
Performance Work Statement. Though similar, the distinct nature of these objectives can be
divided into smaller modules to reduce the instructional time of the step 4 module.
DESIGN TREATMENT
Page 28 of 31
8.2 User Navigation
During the needs and task analysis activities, Opal Enterprises indicated a preference for a
mostly linear approach to user navigation in the course. This preference has been incorporated
into the design as presented in section 5.0 of this document. However, research indicates that
adult learners typically prefer to have more control over the conduct of their educational
experiences (Knowles, 1990). In consideration of this finding, GMU Design Team recommends
that a more non-linear approach to course navigation be adopted. A more non-linear approach
may result in greater user satisfaction with the course and may also support future use of the
course as a reference and/or refresher resource.
9.0
References and Data Sources
9.1 Theory and Practice
Altschuld, J.W. & Witkin, B.R. (1999). From needs assessment to action: Transforming needs
into solution strategies. Thousand Oaks, CA: Sage Publications, Inc.
Davies, I.K. (1976). Objectives in curriculum design. Maidenhead, UK: McGraw-Hill Book
Company (UK) Limited.
Gagné, R.M., Briggs, L.J., & Wager, W.W. (1992). Principles of instructional design. Fort
Worth, TX: Harcourt Brace Jovanovich College Publishers.
Jonassen, D.H., Tessmer, M., & Hannum, W.H. (1999). Task analysis methods for instructional
design. Mahwah, New Jersey: Lawrence Erlbaum Associates.
Knowles, M. (1990). The adult learner: A neglected species, 4th ed. Houston: Gulf Pub. Co.
Lee, W. & Owens, D. (2004). Multimedia-based instructional design. San Francisco: Pfeiffer.
Smith, P.L. & Ragan, T.J. (1999). Instructional Design, 2nd ed. Hoboken, NJ: John Wiley &
Sons, Inc.
Smith, P.L. & Ragan, T.J. (2005). Instructional Design, 3rd ed. Hoboken, NJ: John Wiley &
Sons, Inc.
9.2 Background
Acquisition Workforce Status of Agency Efforts to Address Future Needs. (2002, December).
Retrieved September 18, 2005 from http://www.gao.gov/new.items/d0355.pdf
Information Technology Management Reform Act (now the Clinger-Cohen Act) of 1996, 40
USC 1401(3). Retrieved September 18, 2005 from
http://www.cio.gov/Documents/it_management_reform_act_Feb_1996.html
Progress in Implementing the Services Acquisition Reform Act of 2003. (2005, February).
Retrieved September 18, 2005 from http://www.gao.gov/new.items/d05233.pdf
Opal Enterprises, Inc. (2005). Retrieved September 17, 2005, from
http://www.opalenterprises.com
Report on Competencies for the Federal Acquisition Workforce - Executive Summary. (2003,
December). Retrieved September 18, 2005 from http://www.fai.gov/pdfs/CompReptExecSum01-08-04FINAL.pdf
DESIGN TREATMENT
Page 29 of 31
9.3
Content
Acquisition Solutions, Inc. (2004, July 30). Advancing a performance-based service acquisition
culture throughout the department. Retrieved October 15, 2005, from the U.S.
Department of Commerce, Office of Acquisition Management and Financial Assistance
Web site: http://oamweb.osec.doc.gov/docs/PM_2005-01_Attachment_APBSA_Assessment_Report.pdf
Federal Acquisition Regulations, 48 C.F.R § 37 (2005). Retrieved September 10, 2005, from
http://205.130.237.11/far
Federal Acquisition Regulations, 48 C.F.R § 2 (2005). Retrieved September 10, 2005, from
http://205.130.237.11/far
General Services Administration. (2004). Seven steps to performance-based services
acquisitions. Washington, District of Columbia: Andrecs, L., Burnette, T., Canales, M.,
Costello, A., McKee, M.. Rogin, R., Temperley, B., & Wise, J. Retrieved September 5,
2005 from AcqNet Web site:
http://www.arnet.gov/Library/OFPP/BestPractices/pbsc/library/SevenSteps_execversion.
pdf
National Aeronautics and Space Administration. (2005, September). Performance-based
contracting. Retrieved October 15, 2005, from
http://ec.msfc.nasa.gov/hq/library/perfba.htm
Office of Management and Budget. (1998). A guide to best practices for performance-based
service contracting. Washington, District of Columbia: Office of Federal Procurement
Policy. Retrieved September 5, 2005 from AcqNet Web site:
http://www.acqnet.gov/Library/OFPP/BestPractices/BestPPBSC.html.
Office of Management and Budget. (1998). A report on the performance-based service
contracting pilot project. Washington, District of Columbia: Office of Federal
Procurement Policy. Retrieved September 5, 2005 from AcqNet Web site:
http://www.acqnet.gov/Library/OFPP/PolicyDocs/pbscpilpro.html
Office of Management and Budget. (2003). Performance-based service acquisition: Contracting
for the future. Washington, District of Columbia: Office of Federal Procurement Policy.
Retrieved September 15, 2005, from AcqNet Web site:
http://www.acqnet.gov/Notes/0703pbsa.pdf
Office of Management and Budget. (2003). Performance-based contracting for language
training services: Performance-based service contracting (PBSC) work statement.
Washington, District of Columbia: Office of Federal Procurement Policy. Retrieved
October 30, 2005, from AcqNet Web site:
http://www.acqnet.gov/Library/OFPP/PolicyDocs/pbscltsv.html
RAND (2001). Performance-based contracting in the Air Force: A report on experiences in the
field. Retrieved October 31, 2005, from
http://www.rand.org/pubs/documented_briefings/DB342/DB342.pdf
DESIGN TREATMENT
Page 30 of 31
RAND (2002). Implementing performance-based services acquisition (PBSA): Perspectives
from an air logistics center and a product center. Retrieved October 31, 2005, from
http://www.rand.org/pubs/documented_briefings/2005/DB388.pdf
U.S. Army Quartermaster Corps. (2005) Department of the Army prototype (Part I) performance
work statements for full food service (FFS), dining facility attendant (DFA) service, and
management and food production (M&FP) service performance-based service
acquisition. Fort Lee, Virginia: Army Installation Management Agency. Retrieved
November 10, 2005, from
http://www.quartermaster.army.mil/ACES/publications/pubs/contract/pws.pdf
9.4 Audience
Cahlink, G. (2001). Drained: The Defense Department’s debilitating loss of critical workers.
Govexec. Retrieved on September 10, 2005 from
http://www.govexec.com/features/0201/0201s1.htm
Farrell, L.P. (2002). Acquisition workforce nears crisis point [Electronic version]. National
Defense. Retrieved September 10, 2005 from
http://www.nationaldefensemagazine.org/issues/2002/May/Acquisition.htm
Federal Acquisition Institute. (2005). Report on the federal acquisition workforce. Retrieved
September 10, 2005, from http://www.fai.gov/pdfs/fapis0405.pdf
Gruber, A. (2003). Report lauds acquisition workforce training efforts. Govexec. Retrieved on
September 10, 2005 from http://www.govexec.com/dailyfed/0103/011703a1.htm
Office of Management and Budget. (2005). Policy Letter 05-01. Retrieved September 12, 2005,
from http://www.whitehouse.gov/omb/procurement/policy_letter_05-01.html
Office of the Undersecretary of Defense for Acquisition, Technology, & Logistics (1997).
ODUSD-AR acquisition reform focus group performance based service contracting
(depot & installation). Retrieved September 10, 2005 from http://www.acq.osd.mil
Office of the Undersecretary of Defense for Acquisition, Technology, & Logistics
(1997).Acquisition reform focus group performance based service contracting training
issues final report. Retrieved September 10, 2005 from http://www.acq.osd.mil
Office of the Undersecretary of Defense for Acquisition, Technology, & Logistics (1997).
ODUSD-AR acquisition reform focus group performance based service contracting
(multiple groups). Retrieved September 10, 2005 from http://www.acq.osd.mil
Vacante, R.A. (2003) Providing the DoD acquisition, technology and logistics workforce more
control over its training. Logistics Spectrum. Retrieved September 10, 2005 from
http://www.findarticles.com.
DESIGN TREATMENT
Page 31 of 31