iSa EVALUATION PLAN
TEAM BLUE
Blue Team
May 6, 2008
In order for our project to be successful, iSa must be evaluated on a
regular basis to make sure that we are adhering to the four main goals of our
project: quality, cost, time, and scope. And in order to properly evaluate the
project we have to set aside some given criteria for success. Below are the
detailed criteria necessary for our project’s success.
iSa Limitations
In order to properly evaluate iSa, then we also have to understand the
scope of the project. While iSa is designed to help people with a speech disorder
communicate better in society it is not intended to solve all communications
problems.
iSa is NOT capable of:
1) Solve all communication problems
Evaluation Criteria
Our project will be broken up into 4 phases: phase 0 (project inception),
phase 1 (product prototyping), phase 2 (product design), and phase 3 (product
out years).
For project phases 0 and 1, the success of our project will be gauged
through weekly group meetings and by consulting the CS 410 instructor and
expert panel. During these two phases the success of the project will be closely
monitored and necessary changes will be made to ensure its success. Overall
success in phase 0 will be also determined by a successful SBIR proposal and
completion of additional required documentation. Once we have received our
initial SBIR grant, then we can proceed with phase 1. In phase 1 the main criteria
for success are to implement a functional prototype and to secure a phase 2
SBIR grant.
Project phases 3 and 4 are mainly concerned with being able to mass
produce the iSa system and support it throughout the product lifetime. Once we
have secured the phase 2 SBIR grant, then we can proceed with phase 2. In
phase 2 our main task is to improve on the functional prototype and apply
lessons learned in order to create a final product. More emphasis will be put on
marketing and product distribution, while our research and development costs
will go down. Success in phase 2 will be determined by the overall commercial
success of our product. In phase 3 the main focus will be on the product out
years, and our continuing support of the product. Again the success of phase 3
will be determined by our business success and how well we market and sell iSa.
2
Blue Team
May 6, 2008
Each phase will be evaluated based on the evaluation checklist for that
phase. Major evaluations will occur every 3 months or as deemed necessary by
the PM. The evaluation checklist consists of five subcategories. They are as
follows:
1) Personnel Factor: each member of the team will be evaluated based on
technical, communication, management (if applicable), and overall impact
to the group.
2) Administrative Factor: scheduling and task breakdown analysis is
included in this evaluation factor.
3) Financial Factor: the scheduled budget for the current phase will be
evaluated for cost efficiency, and funding allocation.
4) Marketing Factor: the current target market and an updated list of
competitors will be validated.
5) Technical Factor: the components of the software, hardware and overall
system will be evaluated for completion, testing, and integration.
Phase 0
Phase 1
Phase 2
Phase 3
Evaluation Checklist
(Phase 0, 1, 2 or 3)
Technical
Factor
Personnel
Factor
Marketing
Factor
Administrative
Factor
Financial
Factor
3
Blue Team
May 6, 2008
Phase 0 Breakdown
The main concern in phase 0 was to come up with a feasible project idea.
After initial discussions and planning we decided on creating a drowning
detection system. After determining our project concept we then had to complete
all of our deliverables in order to assure that we had a idea that worth continuing
with. The work done in preparation of the Feasibility and Milestone presentations
has refined our idea so much that we have determined that it can and should be
done, and that we are ready to prototype the system in order to test it. The
overall goal of phase 0 is to accomplish the following deliverables and to
eventually secure a phase 1 SBIR grant.
Deliverables
Feasibility Presentation
Milestone Presentation
Proposal Presentation
SBIR Phase 1 Proposal
Product Website
Personnel Factor
Weekly meetings will need to be conducted so that not only a good project
idea can be determined, but that we can collate all of our individual
research and apply it to the overall progress of the group. Personnel
success in this phase is based around the following:
1) Good communication between group members about what
needs to be done, and when it will get done.
2) Good collaboration is necessary so that the work is spread
evenly and that it can all be tied together seamlessly
3) Each member must contribute equally and have work of a high
enough quality so that the group does not suffer.
Administration Factor
Specific scheduling of tasks and the breakdown of the work is assigned by
the project manager. Meetings must be scheduled and properly led so that
the group knows and understands its tasks. The project manager must
also facilitate the group members so that they have the materials
necessary to complete their tasks. The project manager must ensure that
all work is done by the time required and that it is of high enough quality to
be considered finished. Specifically for phase 0, success is determined by:
4
Blue Team
May 6, 2008
1) Assignment of the research to determine the necessity and
feasibility of our project and the associated documentation by
the project manager and its completion.
2) Assignment of all of the required deliverables (presentation work
and plans) and their completion in the proper timeframe.
3) The ability of the project manager to integrate the team’s work
so that the project can proceed to the next phase.
Financial Factor
Determination of overall and projected costs is important in the success of
a project. For phase 0 there are no associated costs, so the bulk of the
financial work involves estimating the future costs of the project and
determining the overall bottom line of the project. For this phase success
is determined by:
1) Accurately estimating the associated hardware, software, and
personnel costs for the upcoming phases.
2) Preparation of the financial documentation necessary for the
SBIR proposal.
Marketing Factor
As with the financial considerations, since there is no product to market
yet, we must consider the implications of our product in the market place.
We have to examine our competitors and look at their product’s
capabilities and make sure that our product isn’t lacking. We also need to
gauge the market’s desire for a product like ours and see if our product
will be marketable. Marketing success in this phase is determined by:
1) Correct competition assessment (hence the completion of the
competition matrix)
2) Contacting potential customers and accurately gauging their
interest in our product.
Technical Factor
Understanding the technical implications of our project is necessary so
that we can actually make a product. If the technology necessary is not
understood the product cannot be made and then the project fails. In
phase 0 since there is no product yet, but only an idea, research must be
done so that we can determine what technology will allow us to achieve
our goals. For phase 0, success is determined by:
5
Blue Team
May 6, 2008
1) Understanding the technological barriers and limitations for our
project, and finding technologies that overcome these.
2) Researching these technologies and making sure that these will
overcome the project limitations
3) Making sure we have the support of the technology’s
manufacturers so that we can effectively prototype the product.
Phase 1 Breakdown
Now that we have come up with a feasible project idea it is time to test it.
We have determined that the product can be done (personnel and technical
factors) and we have determined that it should be done (financial and marketing
factors), so now it is time to create a prototype. Assuming that we have a Phase
1 SBIR grant, the money must be managed so that we can properly prototype
and test our product and then begin to refine and market it. Completing the
Phase 1 deliverables will help us to pursue a Phase 2 SBIR grant.
Deliverables
iSa Prototype
SBIR Phase 2 Proposal
Personnel Factor
Since we are now required to create a prototype of our product, our
personnel requirements have changed from phase 0. In addition to the
personnel requirements in phase 0, we might also need to hire new
personnel (specialists) and increased cooperation and collaboration is
necessary while we integrate the various parts of the product. Personnel
success in phase 1 will be determined by:
1) Having the personnel in order to effectively create a product
prototype, and hiring more if necessary.
2) Increased cooperation during the integration phase in the
product prototype.
Administration Factor
The project manager’s role now shifts from collating everybody’s ideas to
keeping everyone on the same page. The project manager has to make
sure that the software and hardware specialists have enough money/tools
in order to create a prototype. The manager also has to relay between the
hardware / software specialists and the financial / marketing specialists
6
Blue Team
May 6, 2008
and the so that they have accurate cost estimates and accurate product
release dates. In phase 1 administrative success will be determined by:
1) Proper delegation of tasks between group members while
creating the prototype.
2) Making sure all group members understand the needs of others,
so that delays can be avoided, and all group members work can
progress in a timely manner.
3) Making sure that if there is a delay, that its cause can be
resolved and that all group members are informed so that their
work can reflect the delay.
Financial Factor
Having secured a Phase 1 SBIR grant, we now have $100,000 to proceed
with. This will contribute to personnel salaries, implementation cost, and
marketing cost to name a few. This money has to be wisely spent and
properly distributed between group members so that everyone can
accomplish their tasks. In the same sense, the money has to be
accurately accounted for, for business purposes (taxes etc…) and costs
will also need to be accounted for the next SBIR proposal. Ultimately we
should be working towards securing a phase 2 SBIR grant. Success will
be measured by:
1) Making sure that the appropriated funds are adequately
distributed in order to be able to successfully create a prototype.
2) The money must be properly accounted for and all expenses
recorded for tax and SBIR proposal 2 purposes.
3) Areas must be identified where we can cut costs for phase 2.
Marketing Factor
Since we are now working on a product, we can reasonably try to market
it. Now that we have a prototype the marketing expert has a decent
understating of what our product does and does not do, and it can be
marketed accordingly. Also as part of an effective marketing / testing
strategy, we can offer to beta test our system at various watermarks, and
use our success stories as more marketing fuel. Marketing success in
phase 1 can be determined by:
1) Marketing potential beta testing to various water parks should
allow for real world testing and provide great marketing
examples for future marketing campaigns.
2) The marketing tools should accurately reflect what iSa is and
what exactly it does.
7
Blue Team
May 6, 2008
3) Marketing success will result in increased interest and future
purchase of our system.
Technical Factor
Now that we are prototyping our system, this is where a large bulk of the
technical work is done. The hardware we are using needs to be tested
individually so that we can determine that it is actually working. We must
also test the software to make sure that it does exactly what we want.
Then once we integrate everything we have to test the whole system
again and make sure everything works correctly. Our product testing will
involve various strategies including design and code reviews, module
testing, integration testing, and finally beta and end-user testing. Once our
product is tested then we can find hardware / software errors that we can
then fix. After the prototype has been tested to satisfaction we can begin
to refine the product for phase 2, which is our eventual goal. In order to
measure our success for this phase the following must be met:
1) A working prototype must be created.
2) The prototype must pass all of the required testing in order to be
considered a successful trial.
3) After we have thoroughly tested the system it must be
successfully demonstrated to be working.
4) Lessons learned should be documented so that refining the
prototype will be easier in phase 2.
(See the iSa Product Evaluation section for testing strategies)
Phase 2 Breakdown
Now that we have successfully prototyped the iSa system and secured
funding for phase 2 it is time to refine the iSa system further and mass produce it.
Also now that we have a working product, marketing and financial factors have a
higher priority and our business has to become self sufficient.
Deliverables
A Fully Functional iSa System
Personnel Factor
Since our iSa system has been refined and is now in production, our
personnel needs have changed from phase 1. We will now need
marketing experts, good accountants, and a lawyer to make sure that the
8
Blue Team
May 6, 2008
business will stay afloat. The engineers will now shift their duties primarily
to system upkeep, and will focus on further refining it, rather than
developing it. Success will be measured by the following:
1) Keeping the required staff that is necessary to support our
product throughout its development, refinement, and out years.
2) Hiring more staff to manage and keep the business running.
Administration Factor
The project manager’s role is similar to that in phase 1. The project
manager needs to make sure that the tasks are assigned and that each
team member does their work. This includes making sure that the system
engineers refine the iSa system and make it ready for deployment, as well
as making sure that the marketing and financial people are taking care of
their end of the business. Success is met by doing the following:
1) Proper delegation of tasks between group members while
making iSa ready for deployment.
2) Making sure all group members understand the needs of others,
so that project delays can be avoided, and all group members
work can progress in a timely manner.
3) Making sure that if there is a delay, that its cause can be
resolved and that all group members are informed so that their
work can reflect the delay.
Financial Factor
Having secured a Phase 2 SBIR grant, we now have $750,000 to proceed
with. With this money we need to become self sufficient. That money will
help cover salaries and start-up costs, but it needs to be handled
efficiently and correctly so that we do not run out of cash on hand. Costs
must be analyzed so that it can be determined where costs can be cut,
and where the business is successful. Financial success can be
determined by:
1) Proper accounting techniques are used to track and manage the
money supply.
2) The money must be adequately distributed so that the business
does not suffer in any way.
3) Cost cutting measures must be taken so that the business can
be self sufficient and remain in the black.
Marketing Factor
Now that our system is in production it is available for sale. So we need to
steadily increase our marketing and come up with new strategies. Good
9
Blue Team
May 6, 2008
marketing will be the key to keeping our business alive. Success will be
determined by the following:
1) The marketing for iSa accurately depicts what our system does
in an easy to understand way.
2) Gaining new customers.
3) Retaining our old customers and making sure that if they need
another system installed, they will come to us again.
Technical Factor
Since iSa has been prototyped successfully and has been properly tested
and evaluated it is time to fix whatever went wrong with the prototype and
to improve and refine the current design. After the improvements have
been made then we will need to retest the system in the same way we did
for phase 1. After successful testing has been accomplished then we to
prepare to produce the system as a whole. Success in this area will be
concluded by the following:
1) The prototype must be refined to production level quality.
2) The Production model must successfully pass all testing
measures.
3) All design modifications and lessons learned should be
documented for the upcoming product out years.
(See the iSa Product Evaluation section for testing strategies)
Phase 3 Breakdown
Now that we have a successful product on the market, the main focus is to
maintain a stable growing business. We need to aggressively pursue more
customers, while refining our product and providing support to old customers.
Deliverables
A Working iSa System with Product Support
Personnel Factor
Now that our system is in production and it works, the main focus with
personnel is to have enough people to maintain / refine the system, and
enough people to keep the business running. If there is substantial
business growth, or major upgrades / improvements to the system then
more people will be need to be hired to maintain the business and help
support the product. Personnel success will be judged by:
10
Blue Team
May 6, 2008
1) Keeping the required staff that is necessary to support our
product throughout its development, refinement, and out years.
2) Hiring more staff to manage and keep the business running as
necessary.
Administration Factor
The project manager’s role is the same as in phase 2. The project
manager needs to make sure that the tasks are assigned and that each
team member does their work. This includes making sure that the system
engineers refine the iSa system and improve it as necessary, as well as
making sure that the marketing and financial people are growing the
business. Success is met by doing the following:
1) Proper delegation of tasks between group members while
making iSa ready for deployment.
2) Making sure all group members understand the needs of others,
so that project delays can be avoided, and all group members
work can progress in a timely manner.
3) Making sure that if there is a delay, that its cause can be
resolved and that all group members are informed so that their
work can reflect the delay.
Financial Factor
After phase 2 our business should be self sufficient. If it is not, then other
means of securing funds should be pursued (i.e. grants). Also over the
lifetime of the business and our iSa system, cost cutting will be important
to maintaining profits, and it may be wise to start looking into LEAN or Six
Sigma practices to help maintain them. Financial success will be
measured by the following:
1) Proper accounting techniques are used to track and manage the
money supply.
2) The money must be adequately distributed so that the business
does not suffer in any way.
3) Cost cutting measures must be taken so that the business can
be self sufficient and remain in the black. LEAN and Six Sigma
opportunities should be investigated.
Marketing Factor
Marketing strategies are similar to those in phase 2. We will want to
pursue more and more customers in order to grow the business. We will
11
Blue Team
May 6, 2008
also need to get customer feedback so that we can improve our system
over its lifespan so that it can stay innovative and fresh. It would also be
prudent to explore possible alternative markets for our product, so that we
can maximize sales. Success is measured by the following:
1) New customers must be gained and old customers must be
retained.
2) Feedback from customers must be obtained so that we can
improve the current iSa system.
3) Alternative markets must be found to increase profits.
Technical Factor
Now that we have overcome most of the technical challenges in just
creating something that will work, the bulk of the technical work in phase 3
will be to upgrade and add features to the current system. Feedback
should be taken from existing and potential customers to see what should
be added (if anything) and then all additional changes and releases should
be tested as before. Product support also has to be provided over the
lifetime of the business. Success will be measured by:
1) There must be well organized and correct documentation for the
iSa production model.
2) The system engineers should make upgrades and
improvements as necessary (taking into account customer
feedback) and they should be tested as before.
3) Customer support must be a priority in order for us to retain
customers.
(See the iSa Product Evaluation section for testing strategies)
iSa Product Evaluation
The major challenge of this entire project is overcoming the technical difficulties
of creating a drowning detection system the way we are. We will be using
hardware in a way it has never been used before, and we will also be using it in a
complex environment (water, concrete, etc). Getting this hardware to connect
with the software will also be a specifically difficult thing to accomplish, however it
can be done. Building the iSa prototype will be difficult at first, and we will no
doubt have many errors in our system, so in order to find them and fix them we
need to develop an effective testing strategy.
BDUF and Design Reviews
12
Blue Team
May 6, 2008
Starting with the overall design of the system (hardware and software) a
Big Design Up Front (BDUF) needs to be set forth so that a lot of basic
flaws are avoided. After the initial design has been made then there needs
to be a series of design reviews that go over every single part of the
design so that nothing is missed. Then once the system is being designed
the code for the software needs to be reviewed periodically (this can be
done through pairs programming) so that any errors in the design of the
software can be caught and fixed before they become major problems.
Total system review will make all team members more knowledgeable
how iSa should work and it allows the whole team to do their jobs more
effectively.
Module Testing
Once each individual piece of the design (The software, hardware
components, etc) has been implemented, there needs to be an automated
testing strategy for each piece. Building up an automated set of trial cases
for a module allows you to make sure that your module is working as it
should. If an error is found that is not a test case it can then be added and
that error can be prevented in the future. Also when a change is made to a
module it helps identify errors that were introduced by accident.
Integration Testing
In the integration testing a combination of top-down and bottom-up testing
would be used to make sure all of the modules work together as intended.
This will test for validity and effectiveness in regards to all other existing
pieces. This step is to ensure that all components of the project continue
to work well in relation to one another.
Performance Testing
This type of testing is necessary that our software interacts well with the
hardware. We will need to make sure that the software we make can be
handled well by the computer system. Some things to check are the size
of the software when installed, the amount of main memory and/ or
secondary storage it requires and the demands made of the operating
system when running within normal limits or the response time. Testing
the performance of the sensor hardware and the software it communicates
with under the worst conditions will help prove the reliability of the
software.
Stress testing and Recovery Testing
13
Blue Team
May 6, 2008
In this type of testing we basically try to break the system and see how
well it handles the increased stress. We will put abnormal demands upon
the software, such as increasing the rate at which it is asked to accept
data or the rate at which it is asked to produce information in order to see
if it will crash. More complex tests may attempt to create very large data
sets or cause the software to make excessive demands on the operating
system. After we have strained the software as much as possible we will
then deliberately interrupt the software in a number of ways by taking the
hard disc off line or even turning the computer off, to ensure that the
appropriate techniques for restoring any lost data will function.
Beta Testing
This is where the system will be released to the actual end users and
tested in their pools. Alpha releases will be done at ODU and they will be
tested thoroughly for errors. Once the alpha release has been tested
successfully then the errors that have been made known will be fixed and
a beta version will be released. We will then release the beta version to
schools where speech impaired kids can test this device. From these tests
and polls, we will evaluate the usability and effectiveness of our product.
Alpha and Beta testing plan
The following will be checked during testing:
Product functionality
1) It will use voice syntheses.
2) It will allow for pre-recorded phases.
3) It will have customizable short hand
4) It will provide customizable themes
Product’s user-friendliness
1) It will have a graphical user interface
2) It will have a touch screen for ease of use
3) It will have a user friendly help menu
User Documentation
1) Each tester will be given some personal instruction
2) Each tester will have a user manual to figure out how to use the
system
14