軟體測試簡介
All about quality !!
1
Testing
Testing is not important in school
homework
 Testing is not important in research work
to produce experimental statistics for
publishing paper
 Testing is not important in a research
prototype
 Testing, however, is very important for a
commercial products.

Why testing?

Let’s look at some well-known software defects
台灣高鐵售票系統：系統上線之後當機連連，系統無
法應付突然湧現的購票人潮 （no stress testing)
 台灣彩卷系統：類似的問題
 2000-2005- 巴拿馬國家癌症中心─5個病人接受過量的
迦瑪射線照射死亡。15人引發了嚴重的併發症
 2003 ─軟體造成美國東北部及加拿大停電。5000萬人
受影響，3人喪生
 2000 美國海軍飛機墜落，4人喪生 （控制軟體問題）
 1997 韓國空難，225人喪生 （雷達控制軟體問題）
 1995 美國航空在哥倫比亞撞山159人喪生（導航軟體
問題）

3
Statistics
2002- 美國國家標準局報告─軟體品質每年
造成595億美元的損失 （0.6％ GDP）
 國內又如何？

 台北縣政府校務行政系統案例
 台師大林口校區學務系統案例
4
Quality Control and
Quality Assurance
Software is a product.
 Good quality product requires 品質控管
quality control (QC) and quality assurance
(QA) 品質確保

5
Manufacturing in other discipline

在製造業的世界裡面



產品品質(quality)是很重要的一個因素
當你投入非常多成本與原物料時你永遠希望你生產出
來100個產品有100個都是可以賣的
QC (Quality Control) 品質控管



製造過程從設計到生產可能有數十道到幾百道程序
每一道程序都可能影響到後面的品質
如何透過製造流程的改善(process improvement)來提
高良率，一直都是製造工業的重要課題
6
How to improve quality?

在製造過程中如果能夠能夠使用機器，最
好的改善品質的方式就是自動化。機器不
容易犯錯而且可以不斷地重複單調無聊的
工作。機器會出錯的時候通常是由於製造
機器的磨損，必須重新校正。
 Ex.

日本的步進馬達精確度超高的秘密。
Question
 當某部分的工作不能由機器來做的時候，製造
過程如何做到品質控管？
7
Assembly line in manufacturing
8
生產線
每個生產線員工只做一件夠簡單到不容易
出錯的工作
 生產線員工不需要夠聰明
 生產線將複雜的產品製造切割成許多小步
驟可以在短時間以及最少的技術內可以完
成
 發現產品的問題並不難.

9
Question?



Is coding a design process or a manufacturing process
Is Code a design or a product?
Again, the analog breaks
More questions
 If coding is not a design process, we should be able to
hire many 生產線女工 to control our quality
 If coding is a design process, bad news, errors are
inevitable because programmers are always doing
complicated design jobs, not simple and easy job.
10
Facts

Now, you should know why software has
so many bugs or notorious for being
flawed
11
The QC story
產
品
缺
陷
率
software
其他工程製造
QC 改進時程
12
A Joke
有天晚上我參加一個老同學的喜宴，
同桌的人有律師、會計師，
只有我一個是電腦業界的人。
為了引起話題，我說我羨慕其他行業
的人，像是律師、會計師等，在
學校學的那一套終生都受用，每
年就算有新法條也不會改太多。
不像我們電腦界的人，每天都在K新
的技術手冊，新產品新技術隨時
大翻新，改朝換代的速度讓人措
手不及，所以學電腦的人看起來
都比較蒼老，因為太辛苦了。
我話剛說完，一位會計師馬上回我一
句：「你錯了，其實我們最羨慕
你們電腦界的人，因為沒有任何
行業的消費者像你們電腦界的消
費者一樣好欺負。」
頓時我成了眾矢之的，
這些電腦使用者的抱怨全部集中到我身上
來。
其中一位仁兄還舉了個有趣的例子，
他說：
「如果傢俱業和電腦業一樣，世界會變成
什麼樣子？」以下是他講的故事：
如果傢俱業跟電腦業一樣，比如說我到傢
俱店買了一張桌子，搬回家往地板上一放，
啪啦一聲桌子就塌了。這時候我不會生氣、
不會罵人，我會先自己檢查一下出了什麼
錯。
13
我會先檢討自己，
是不是我做錯了什麼，
或是我對桌子的使用不夠熟悉。
於是我會去買書來看
(書名可能是《快快樂樂學修桌子》、
《21天學會修桌子》、《修桌子技巧
與實例》、
或是《修桌子的聖經》)。
要是書看得不太懂，
我會再花錢去報名上修桌子的課程。
學完之後還是修不好，
我會請其他比較懂得修桌子的朋友來
幫忙。
最後沒有辦法，
終於我打電話給原先的傢俱行，
(可能還要購買《技術支援方案》)，
結果他們跟我說：
「唉呀！你買到的是搶鮮版啦！
本來就應該有問題的」。
於是我恍然大悟原來是自己的錯，
我就再去買一張「正式版」的桌子。
回家一擺還是啪啦又塌了！
修了半天還是有問題，
再請傢俱行的技術人員來做仔細檢查，
最後終於發現問題的所在──
「我家的地板和桌子不相容」，
又是我自己的錯，
於是我得趕快幫家裡的地板升級......。
等一切都忙完了，
桌子可以使用了，
我趴在桌上寫字，
心裡充滿了成就感，
我很得意地跟網友分享我修理桌子的經驗，
並暗自慶幸自己在科技的潮流上沒有落
伍......。
14
Software Quality
如同前面所說的，由於軟體產業的生產工
具只有人，，人注定會犯錯，很多人還會
犯很多的錯
 不要相信 programmers
 This is why testing is so important in
matured software industry.

15
一座橋蓋好了，需要大量的測試嗎？
16
How do you test a DVD player
17
How do you test a software
18
The complexity of software testing
一般而言軟體有複雜的介面 (including user
interface, network interface, file interfaces,
etc. ）
 一般而言軟體測試必須考慮更多的情境

 Correct
execution paths（正確的路徑）
 Incorrect execution paths （不正確的路徑）

有許多應用軟體的測試其實是一門困難的
學問理論，技術，實做，程式能力的要求
都不低
19
SDLC (Software Development Life
Cycle)



需求分析 Requirement analysis (marketing research)
軟體規格寫作 Software function/performance specification
分析與設計 Analysis and Design (QA 在這時候就要參與專案）





程式實做與單元測試 Coding and unit testing (by programmers)
整合測試 Integration testing (by programmers/QAs)
Alpha testing (by QAs)




Most software functions and features are basically completed
All functions are tested, no functions will be added beyond this point
Serious flaws (high severity) are solved and addressed (show stopper)
Beta testing (by Beta users)




分析，選定PL，選定 platform, database, network protocols.
定義軟體功能模組，模組之間的關連，合作，介面
sub-serious bugs are all fixed
Test plan has been completely executed
Bug discovering rate is lower than bug fixing rate
軟體釋出 Release




Bug discovering rate is lower than bug fixing rate for a long period of time.
The version after fixing bugs has been regressively tested (regression test)
Quality is formally proved by QA team
All documents are ready
20
Common software defects
70% occurs in design and hard to correct
 Software specifications are not precise
 Software is complicated
 Coding errors (20%)
 Function changes which affects other
components
 3rd party software is flawed

21
Work (person-days)
 Early detection of errors
Error occurs
Jan 13
Error found
Feb 13
Error found
Mar 13
10
20
Time
Work proceeds
at (say) 10 persondays per month
10 person-days of work
has been done assuming
the error is not there.
Now this must be redone.
If error found this late,
20 person-days must
be redone
22
What to test for a QA (要測什麼？）

完整性 (completeness)









軟體是否具備軟體規格書，設計文件中所描述的功能
與性能
正確性 (correctness)
可靠性 (reliability)
相容性 (compatibility)
效率（efficiency)
可使用性 (usability)
可攜性 (portability)
可變比例性 (scalability)
易測性 (testability)
23
迷思

軟體有問題是測試人員的錯
 軟體測試只是一種有效的提高軟體品質手段，
但無法百分之百解決軟體品質的問題

軟體測試技術要求不高，比程式設計容易
 兩種人無法類比，程式設計能力好的人，可能
可以更勝任軟體測試。
 自動化測試常需要程式設計高手
24
就業市場概觀

美國的軟體產業現況
 Big
software company has their testing team
 Small software companies which do not have
resources to test can outsource to 3rd party
testing company

台灣的軟體產業現況
software company like 趨勢科技Trend
Micro has QA team
 Many software companies use programmers
as testers – a sad fact!
 Big
25
Company Organization
Developers
Marketings
QAs (testing teams)
26
Management Hierarchy
Director of
development
Director of QA
Test manager
testers
Test automation
Team manager
Test automation
programmers
Test facilities
manager
Developer
manager
Developer
manager
Test database
programmers
programmers
programmers
27
Important reasons to make QA team and
developer team equal and separated


只有極少數處女座的 programmers 才會有品質（完美主
義）的概念
叫 programmer 寫完程式，自己找bugs 就如同叫法官判
完案之後，承認自己的判案是錯誤的



如果他真的找出很多bug ，他決不敢聲張與邀功，因為這代表自
己程式寫太爛
找的bug越多，表示待完成的工作越多，沒有人會自己找碴，讓自
己無法喘息
上述「人」的因素使得



QA 必須在軟體品質上與 programmer 站在對立面
QA 的升遷管道必須是抓到越多 bug，功勞越高，QA才會努力找
碴
QA 不能歸 developer team 管轄，這就像是司法不能在行政部門
底下是一樣的
28
Marketing
恐怖平衡
QA (testers)
Developer
29
軟體測試V模型
軟體功能需求
回歸測試
驗收測試
以此作為驗收測試依據
軟體規格說明書
系統測試
以此作為系統測試依據
分析與設計文件
以此作為整
合測試依據
軟體模組設計書
整合測試
單元測試
程式實做
30
Smoke Tests

A smoke test is a subset of the test cases
that is typically representative of the
overall test plan.
 Smoke
tests are good for verifying proper
deployment or other non invasive changes.
 They are also useful for verifying a build is
ready to send to test.
 Smoke tests are not substitute for actual
functional testing.
31
http://www.stellman-greene.com
Load testing
To exercise the system to the maximum
load that is specified in specs.
 The goal is to test whether the system
meet the requirement

33
Stress testing

The goal of stress testing is to exercise a
system beyond the load in specs to see what
it can happen.
 require considerable cost and efforts.
 often require you to implement a system to
test the system.
 Load testing is necessary but stress testing is
optional.
 E.g., a online game server may limit the
users to prevent system from crash
Let’s back to the reality

Most vendors do not possess a separate QA
team
 They can fake test plan and report
 So? What can we do with the vendor?


Ask for a bug report system (issue tracking system)
Examine the test cases and sampling




Smoke test
Extended test
Critical test cases
Watch the bug report history
35
If you are hired to build a dog
house,
what ability you should
Have?
36
Different Scale of Software
Development
If you are hired to build a
Taipei 101
- What skills and talent do
you need to have?
37
Usability Test
38
A Common story


各機關不斷的引入各式各樣的軟體系統
系統可能內部開發, 可能外包，以外包為例



通常由廠商撰寫規格書
最低價得標通常是夢靨的開始
當系統上線時， 資訊人員通常首當其衝的面對各種狀況




廠商將使用人員當測試人員
使用人員對於使用上的不方便諉因於自己對軟體的不了解，而不
敢提出建言
發包單位有結案的壓力
測試不完全，廠商與使用人員一路使用一路修正錯誤。幸運的話,
共體時艱，不幸運的話彼此怨懟，進入最可怕的驗收夢靨
 主要原因之一:沒有
usability test
39
Some facts

Most programmers do not possess the
knowledge in UI design
 Programmers often think UI is not a difficult part,
which is wrong in practice, especially in
commercial products
 Change UI design (e.g., making UI more friendly)
often means writing a lot more code, which is not
expected.
 Programming tasks are often spec-oriented, not
user oriented.
40
41
The failed story
42
Why Microsoft lost its markets?
UI
 UI
 UI
 UI
 …..

43
Goals of usability tests




Performance -- How much time, and how many steps,
are required for people to complete basic tasks? (For
example, find something to buy, create a new account,
and order the item.)
Accuracy -- How many mistakes did people make? (And
were they fatal or recoverable with the right information?)
Recall -- How much does the person remember
afterwards or after periods of non-use?
Emotional response -- How does the person feel about
the tasks completed? Is the person confident, stressed?
Would the user recommend this system to a friend?
44
Some usability test method

Think aloud protocols

Users are asked to say whatever they are looking at,
thinking, doing, and feeling, as they go about their
task.
 Observers at such a test are asked to objectively take
notes of everything that users say, without attempting
to interpret their actions and words.
 Test sessions are often audio and video taped so that
developers can go back and refer to what participants
did, and how they reacted. The purpose of this
method is to make explicit what is implicitly present in
subjects who are able to perform a specific task
45
Eye tracking
46
Eye tracking pattern
47
Investigation room
48
Observations
Most 行政資訊系統 I have used are
difficult to use, learn, and remember
 Poor exception handling
 Function-oriented is a common

49
Source Code Quality
程式碼鑑賞

程式設計有藝術的成分在嗎？
 軟工的研究顯示
研究顯示，高手與庸手的表現有極大的差異，
而且往往是一個數量級的差異 （an order
of magnitude, 10^1)
and GRANT
- SACKMAN, EROKSON,
 另外一個研究顯示
 成為高手與庸手與年資經驗沒有絕對關係
Yes
我們認為寫軟體有藝術的成分在
 藝術的東西

 有時只能意會不能言傳
有學過畫畫等藝術的東西嗎？請問你怎麼
學的？
 上過藝術鑑賞這種課程嗎？
 通常藝術的傳授一定會配合藝術的鑑賞

AND？
你看過多少別人的程式碼？（教科書的片
段程式碼不算）
 你知道什麼樣的程式碼是好的？
 你知道什麼樣的程式碼是壞的？
 我們先來看看壞的

Typical symptoms of bad code
Difficult to understand
 Difficult to change (changing at one place
results in changes in many other places)
 Hardwire the solutions (for example,
instead of coming out a nice algorithms
and data structures, many if is used)
 Source code is tightly coupled because
the excessive use of global variables.
 Poor design that is difficult to extend.

Reasons for writing bad code

CS curriculum


In CS major, main focuses are problem solving such
as algorithms, techniques (networking, multimedia).
writing high quality code is not encouraged or viewed
as an important topic among students
We teach students how to
assemble gears to solve a
problem but do not
educate them how to produce
quality gears and measure
quality of the gears
What is a good design?
Comfortable?
The truths are

Clean and simple interface, replaceability,
modularity, composablity
Their importance is not obvious
in Software development,
particularly if you never has
chance to maintain a software
Coupling (耦合）

Most software engineering methods (e.g., OO) focus on
create beautiful design so that







Change one thing does not involving changing many other
things
Replace/substitute one thing does not involving modifying many
other things
Extend one thing does not involving rewriting many other things
High coupling is bad
We pursuit low coupling
OO is a technique to decouple the system architecture
The concept of coupling will be introduced in the future.
Remember this word
Again

Why writing low-coupled source code is
not as easy as other engineering?
 Software
is invisible and source code is not
geometric related that you can judge from
your eyes
 The dependency and coupling are not
obvious and easy to identify. Consider a poor
programmer writing thousand lines of code for
you.
 Software behaviors are dynamic
Reasons for writing bad code

Experienced programmers may write bad
code
 Tight
schedule and limited budget
 Demo-oriented programming
 Not enough time to do design
 They accustomed themselves to write bad
code if no adequate software engineering
processes has been established in their
working environment
Motivation

What is a bad code smell?




No formal definitions and descriptions have been
established so far
Sometimes, it is not easy to explain why a piece of
code is bad to beginners
This means “bad” is not measurable quantitatively
Understand what is bad can lead to a better
understanding of what is good. For example, why
using some design pattern is good and what can be
improved?
Spaghetti Code
Lasagna code

Lasagna code is used to describe software
that has a simple, understandable, and
layered structure. Lasagna code, although
structured, is unfortunately monolithic and
not easy to modify. An attempt to change
one layer conceptually simple, is often
very difficult in actual practice.
Ravioli code


In ravioli code, each of the components, or
objects, is a package containing some meat
or other nourishment for the system; any
component can be modified or replaced
without significantly affecting other
components.
The ideal software structure is one having
components that are small and loosely
coupled; this ideal structure is called ravioli
code.
Now, let’s look at some spaghetti code and
give a critics
Top 100 signs that I may be writing
spaghetti code
I have no idea where this constant is defined
I have printf stmts littered throughout my code.
3.
There is an error but it isn't handled, and I can't find it
to figure out what's wrong.
4.
Passing variables directly to 635 functions and using
'global' on 453 functions
5.
I have to modify the same function 10 times because I
have not moved it into a function...yet.
6.
I find that rewriting is quicker and easier than
modifying the original design.
7.
You have a library of small functions that you wrote
yourself, and will cut'n'paste them into almost every
project you work on... Whether they're needed or not...
Now let’s continue and win your credits
1.
2.
Top 100 signs that I may be writing
spaghetti code- (CONT)
8.
9.
10.
11.
12.
Every time you want to change a variable you have to use grep to
find it ;)
you have so many "for" loops you've declared a variable for each
letter in the alphabet.
there are enough "if" statements to make you scroll to the right.
You see comments in the code that read something like:
[Your Name: Don't ask me how or why this works, I know it looks
overly complex and it appears there IS a simpler solution, but this
works, it fixed a bug from the 'simpler code'...just leave it be...don't
touch it, I've hired a voodoo witch doctor to place a curse on
anybody who alters this code...trust me, just leave it alone! ]
you know you're digging through a bowl of the stuff when, as a
regular tool, you're using the windoz's search feature to find
functions, variables and other bits of code.
Top 100 signs that I may be writing
spaghetti code- (CONT)
13.
14.
15.
16.
17.
18.
19.
20.
having to "peel" your way through while/for loops and a chunks of IF
and ELSE statements because the original author (probably in all his
haste) couldn't be bothered to use brackets. The code works but not
always for the right reasons.
use every imaginable variation of the word "foobar" in variable names
You sit down, open your program, and wonder where to begin with
debugging... (ooh boy, I know the feeling of that!)
You wrap it up and realize that there are 20 funtions in the include
files which are never called, although at some point, they were all
required.
Changing database is (e.g MS SQL to MySQL) requires the rewrite of
SQL statments.
You realize that you are changing global variables with functions
inside include files
You create a separate class for every single function.
You have no idea what a CRC Card is ...
Again, let’s back to reality

Source code quality can be manifested

When you ask vendor to do some change, it takes a
lot of time to complete it.
 And, it gets worse and worse…..
 Until the original programmer leave
 They don’t want to earn the maintenance contract
 Someday, you prepare new money for a new vendor
to rework (transferring the data is a basic requirement)
 There is definitely no vendor dare to promise a task to
repairs other vendor’s system
75
My advises!
This is the most difficult part in software
engineering! Little can be done about it.
 See it as a power to put more investment
on software (The principle for FORD
manufacturing)
 Like hardware, software cannot last
forever (especially the small companies
that produces MIS systems)

76
Test Driven Development
For those of you who are
programming in your work
77