Lips-Control Assistive
System for Communication
Presenter: Wei-Min Chang
Advisor: Dr. Shih-Chung Chen
Date: 2006/10/25
Bio-Medical Electronic Center. Institute of Electrical Engineering
Southern Taiwan University of Technology
1
Outline
1. Motivation
2. Purpose
3. Paper Review
4. Materials and Methods
5. Experimental Results
6. Conclusions
7. Future Works
8. References
2
Motivation
• Many computers’ input devices are designed for
normal persons, so these devices are unsuitable for
the disables.
• Many researchers develop many auxiliary devices for
the disables ,but these auxiliary devices still have
many defects when the disables use them in real life.
3
Purpose
• We hope to realize a set of lips-control assistive
system for communication by labview without
wearing any auxiliary devices for the disables with
cerebral palsy or basket case.
• “Morse Code-Based Mouth Controlled Input Device
with Fuzzy Recognition for the Severe Spinal Cord
Injuries” had developed a stonking communication
system by mouth-control.
4
Paper Review
• Pham The Bao, Jin Young Kim, Seung You Na, “FAST MULTI-FACE DETECTION IN
COLOR IMAGES USING FUZZY LOGIC”, Proceedings of 2005 International
Symposium on Intelligent Signal Processing and Communication Systems.
5
Paper Review
• Anagnostopoulos C., Anagnostopoulos I. ,Vergados D. , Papaleonidopoulos I. , Kayafas
E., Loumos V. and G. Stasinopoulos,” A Probabilistic Neural Network for face
detection on segmented skin areas based on fuzzy rules”, IEEE MELECON 2002,
May 7-9,2002, Cairo, EGYPT.
3
Paper Review
• Ching-Hsing Luo, Chung-Min Wu, Shu-Wen Lin, Tsan-Hsun Huang, Cheng-Hong Yang,
Ming-Che Hsieh, Shih-Chung Chen and Chih-Kuo Liang, “Mouth-Controlled Text Input
Device with Sliding Fuzzy Algorithm for Individuals with Disabilities”, IEEE
instruement and measurement 2005 (submitted).
• Chung-Min Wu and Ching-Hsing Luo, “Morse code recognition system with fuzzy
algorithm for disabled persons” Journal of Medical Engineering & Technology 2002, Vol
26
• Wu, C. M., Luo, C. H., Lin, S. W., Chen, S. C., Hsieh, M. C., Chao, C. T., & Tai, C. C.
(2002). “Morse code recognition system with adaptive fuzzy algorithm for the
disabled”. Journal of Medical and Biological Engineering, 22(4), 205-210.
……
Fuzzy Algorithm
Controller
Goal
Input of Hardware
(morse code)
4
Materials and Methods
• Equipment of Experiments
a. Computer : intel pentium4 2.4G
b. SDRAM 512mb
c. High resolution digital color ccd : CH-926 HK 1/3”
d. Color image acquisition card : NI PCI-1411
e. Format of image transmission : NTSC
• Software of Experiments
LabVIEW 7.1
Vision assistant 7.0
LabVIEW PID toolkit
4
Materials and Methods
• Schematic Diagram of System Structure (New
(Old Version)
Version)
( Image processing algorithm )
Image processing
algorithm
user
Parallel port
ccd
McTin
PC
Ps2
morse code
Morse code
System
algorithm
utilizing
fuzzy theory
( Fuzzy algorithm,
Calling API)
Database
functions
4
Materials and Methods
• Technology of Kernel Processing
• 1. Image Processing
Acquiring images
from CCD
(RGB)
Face tracking
and detection
algorithm
Allocation
and extraction
of lips images
Processing and
recognition of lips
images
2
Materials and Methods
• Technology of Kernel Processing
• 2. Fuzzy Theory
• Conventional or crisp sets are binary. An
element either belongs to the set or doesn't.
• A question of a degree of association.
Everything is a matter of degree !
2
Materials and Methods
• Technology of Kernel Processing
• 2. Morse Code Communication Protocol
i.e. Character “A”
dot space dash space
Next Character
n(k)
dot-dash space
Protocol:
Character space
(1) dot space:dash space=1:3 (Tone space)
(2) dot-dash space:character space=1:3 (Silent space)
2
Materials and Methods
• Technology of Kernel Processing
• 2. Morse Code Communication Protocol
t
t
PB
t
t
PS
Long sound
NM
Threshold
Short sound
2
Materials and Methods
• Technology of Kernel Processing
• 2. Fuzzy algorithm Applied in Mctin System
fT
Xk +
ek
-
Fuzzy Algorithm
yk
+
yk-1
*2
Ik
e’k +
Z-1
Tk
Controller
2
Materials and Methods
• Technology of Kernel Processing
• 3. Windows API
• The Microsoft Windows application programming interface
(API) provides building blocks used by applications written
for Windows .
• You can provide your application with a graphical user
interface; display graphics and formatted text; and manage
system objects such as memory, files, and processes.
2
Materials and Methods
• Technology of Kernel Processing
• 3. Windows API
Open
notepad file
Send?
Yes
Paste
End
No
Start
keybd_event
Send
FindWindowA SetWindowPos
Notepad file
open? Yes
No
Paste
End
Auto open
notepad file
2
Example
6
Materials and Methods
• Verification of fuzzy recognition algorithm by software
Test1: Data of expert
Taking the data analysis for experts in reference[7] . Experimental method is ditto .
使用參考文獻[7]中所使用的專家數據。實驗方式同上
Test2: Data of a cerebral palsy
Taking the data analysis for a teenager with cerebral palsy in reference[7] . Experimental method
is that random reading that 20 data, 200 numbers every time, ten times in operation .
使用參考文獻[4]中所使用的一名十幾歲的腦性麻痺患者數據。實驗方式為隨機讀取這20筆資料，每次共200筆數據，共進行
十次
• Verification of fuzzy
Test3: Data of normal
recognition algorithm by human
20 20~30 year-old normal person who input long-short sound data array which divided into
“long” 、”short” 、”long” 、”short”….Number of data is 100. (Long and short sounds are 50
and
50 respectively ).
由二十位20~30歲的正常人,以按鍵的方式，每人輸入固定的長短音資料組，分別為一長、一短、一長、一短…共100筆資料
(長、短音各50筆)
Test4: Fuzzy1 & fuzzy2
One disable with Spinal injury who input long-short sound data array which divided into
“long” 、”short” 、”long” 、”short”….by open and close mouth. Number of data is 20. (Long 4
Experimental Results
• Verification of fuzzy algorithm by software
Test1.Testing of the data analysis for experts
nth
Number of
right/data
correct
rate (%)
Dash
Dot
Lm (ms)
Sm (ms)
1
197/200
98.50
1
249
63
2
198/200
99.00
2
274
72
3
197/200
98.50
3
263
99
4
196/200
98.00
4
304
57
5
198/200
99.00
5
259
92
6
197/200
98.50
6
318
61
7
191/200
95.50
7
365
102
8
197/200
98.50
8
197
60
9
200/200
100.00
9
256
68
10
200/200
100.00
10
394
114
Average
± sd
No.
98.55±1.2
6
The data analysis for experts
Result of experiment of the data analysis for experts
6
Experimental Results
• Verification of fuzzy algorithm by software
Test2. Testing of the data analysis for a teenager with cerebral palsy
Dash
Dot
Lm(ms)
Sm(ms)
1
619
163
2
645
117
3
677
110
4
812
79
5
733
75
6
634
73
7
755
125
8
969
139
9
749
134
10
1238
146
No.
nth
Number of
right/data
correct
rate (%)
1
198/200
99.00
2
197/200
98.50
3
198/200
99.00
4
199/200
99.50
5
198/200
99.00
6
197/200
98.50
7
193/200
96.50
8
198/200
99.00
9
200/200
100.00
10
199/200
99.50
Average
± sd
98.85±0.9
4
The data analysis for a teenager
Result of experiment The data analysis for a teenager
with cerebral palsy
with cerebral palsy
6
Experimental Results
• Verification of fuzzy algorithm by human
Test3.Testing of normal ( FUZZY1)
nth
Number of
right/data
The longto-short
ratio ± sd
correct
rate (%)
1
97/100
3.48±0.88
97.00
2
98/100
6.59±2.18
98.00
3
94/100
2.61±0.76
94.00
4
99/100
3.85±0.87
99.00
5
97/100
6.71±1.73
97.00
6
97/100
3.46±0.74
97.00
7
96/100
7.27±2.59
96.00
8
93/100
8.34±4.48
93.00
9
98/100
3.75±0.68
98.00
10
98/100
4.89±0.98
98.00
11
98/100
3.34±0.46
98.00
12
97/100
5.58±2.21
97.00
13
98/100
2.51±0.49
98.00
14
95/100
4.24±0.96
95.00
15
84/100
2.50±1.26
84.00
16
94/100
3.47±0.96
94.00
17
99/100
4.40±0.92
99.00
18
55/100
1.58±0.31
55.00
19
97/100
3.67±0.97
97.00
20
95/100
3.52±0.74
95.00
Average
± sd
93.95
±9.74
Result of experiment of normal
6
Experimental Results
• Verification of fuzzy algorithm by human
Test4.Testing of disables (FUZZY1)
Fuzzy1
nth
Number of
right/data
The longto-short
ratio ± sd
correct
rate (%)
1
17/20
3.65
±1.74
85.00
2
18/20
3.83
±2.26
90.00
Avera
ge
± sd
87.5
±3.54
x-axis :Number
y-axis :Time (ms)
6
Experimental Results
• Verification of fuzzy algorithm by human
Test4.Testing of disables (FUZZY2)
Fuzzy2
nth
Number of
right/data
The longto-short
ratio ± sd
correct
rate (%)
1
19/20
3.47
±1.56
95.00
2
19/20
3.54
±1.54
95.00
Avera
ge
± sd
95 .0
±0.00
x-axis :Number
y-axis :Time (ms)
6
Conclusions
• Case 18
• The effect of system performance by adding
fuzzy algorithm
• The effect of light on system.
• Fuzzy1 & fuzzy2
6
Future Works
• Testing for the disables.
• To modify image processing algorithm
• Function test of the remote control system of electrical
home appliances.
• Mouse control
• Testing for the effect of system performance by adding
fuzzy algorithm
17
References
• “ 認識fuzzy-第二版,” 王文俊, 全華科技, 2001
• LabVIEWTM PID Control Toolset User Manual
• LabVIEWTM Fuzzy Logic for G Toolkit Reference Manual
• “LabVIEW & Microsoft 的整合實例(I),” 陸光中/蕭子健, 高立圖書, 2004
• LabVIEWTM Using External Code in LabVIEW User Manual
• Ching-Hsing Luo, Chung-Min Wu, Shu-Wen Lin, Tsan-Hsun Huang, Cheng-Hong Yang,
Ming-Che Hsieh, Shih-Chung Chen and Chih-Kuo Liang, “Mouth-Controlled Text Input
Device with Sliding Fuzzy Algorithm for Individuals with Disabilities”, IEEE
instruement and measurement 2005 (submitted).
• Chung-Min Wu and Ching-Hsing Luo, “Morse code recognition system with fuzzy
algorithm for disabled persons” Journal of Medical Engineering & Technology 2002,
Vol 26
• “應用於重度脊髓損傷患者之摩斯碼模糊辨識嘴控輸入系統,” 國立成功大學電機系, 吳崇民, 博
士論文, 2004
18
Thanks for Your Attention!!
19
Results of Experiments
Performance testing of image processing and fuzzy algorithm
Test3. performance testing
Data of expert
Data of cerebral
palsy
Image
processing
Number
Average (ms)
Sd (ms)
105
24.03
7.73
Number
Average (ms)
Sd (ms)
105
23.13
8.72
Number (Frame)
Average (ms)
Sd (ms)
105
271.87
23.63
6
Results of Experiments
Timing diagram
Image processing
Status of lip
Processing of fuzzy
algorithm
6
Results of Experiments
Verification of image recognition algorithm by human
Test3.正常人數據測試
專
敘述統計
個數
104
104
VAR001
有效的 N (完全排除)
平均數
.02403
標準差
.007731
敘述統計
VAR00 1
有效的 N (完全排除)
個數
10 6
10 6
平均數
.0 23 13
標準差
.0 08 71 9
p
敘述統計
VAR001
有效的 N (完全排除)
個數
104
104
平均數
271.86538
標準差
23.632731
敘述統計
VAR001
有效的 N (完全排除)
個數
106
106
平均數
.02109
標準差
.008880
影
影自
輸
表6. 正常人數據測試實驗結果
6