Technion - Israel
Summer 2002
MMI
Man Machine Interface
Ohad Stein
Sefi Epel
1
Technion - Israel
Summer 2002
Introduction
•It is known for years that stress, press, fear, happiness and many
other feelings, can be measured by several techniques .
•Based on this knowledge, we would like to build an
interface, that will enable people to communicate without
using any muscle, and without making any physical effort.
•Our interface, will enable communication by being a “mindreader”. Detection of voltage deflections measured from
subjects arms, will enable us to estimate the subject’s will.
2
Technion - Israel
Summer 2002
How can we do it?
•There are several ways to “speak” with the machine:
•One way, is to measure the resistance of the skin, to the
passage of a very small electric current.
•The main disadvantage of this technique is that it is hard
to calibrate the system. The measured values changes
between different subjects, and between different times
with the same subject.
•Our way was to measure the change in DC potential
across neurones of the autonomic nervous system
connected to the sensori-motor strip of the cortex.
•In this way – we will detect a signal only after causing it.
3
Technion - Israel
Summer 2002
Part 1: Experimental background
•In order to gain some basic knowledge about the MMI abilities we have done
some basic experiments, in several different ways.
•On the following slides, 3 basic experiments are shown.
•The first and second experiments have a strong emotional effect. We did
them in order to have a basic indication about the ability of building a MMI.
•The third experiment do not have any emotional aspect. The subject should
try to control the voltage deflections measured from his arm.
•The third experiment is actually the prototype for our project.
4
Pictures experiments
•Exp #1: “Pictures Experiment” – several pictures where shown to the subject,
changes on the skin voltage ratio where measured according to the pictures
emotional effect.
•Be aware that the fall of the voltage ratio is a short time after the
“strange/unusual” picture is shown.
5
Technion - Israel
5
10
Summer 2002
15
20
25
30
Seconds
6
Technion - Israel
5
10
Summer 2002
15
20
25
30
Seconds
7
Technion - Israel
Summer 2002
Voice experiment
•Exp #2: “Voice Experiment” – several words were announced to the
subject, different voltage ratio was measured according to the words
emotional effect.
•The words that did not affect the subject were words like “silence”, ”beautiful”,
”sunrise”, and the words that did affect the subject were “war”, “accident” and
etc.
•Note that in this experiment the result can be affected by screaming,
changing tone, speaking quietly and etc.
8
Technion - Israel
Summer 2002
The person heard 5 different good/bad words. Here are the results:
GOOD
5
GOOD
10
BAD!!!
GOOD
15
20
GOOD
25
30
Seconds
9
Technion - Israel
Summer 2002
The person heard 5 different good/bad words. Here are the results:
GOOD
5
BAD!!!
GOOD
10
15
GOOD
20
GOOD
25
30
Seconds
10
Technion - Israel
Summer 2002
Letters experiment
•Exp #3: “Letters Experiment” – several letters were shown to the subject, one
by one . Different voltage ratio was measured according to the picture
emotional effect.
•Exp #3 is a much more complicated experiment. The emotional effect is
dependent on the assistant decision and concentration.
11
Technion - Israel
Summer 2002
•Based on Exp# 3 – we will try to prove the ability of the man to write
down a word without using a pen, pencil, keyboard or anything else. Just
by measuring the voltage ratio on his arm ! ! !
•The red letter shown in the following graphs, is the letter that the subject
was waiting to be shown.
12
Technion - Israel
Summer 2002
M
Q
6
K
11
F
17
T
22
C
27
33
Second
0
M
Q
6
K
11
F
17
T
22
C
27
33
Second
13
Technion - Israel
Summer 2002
B
0
O
6
B
0
L
11
O
6
S
16
L
11
W
22
S
16
U
27
W
22
33
Second
33
Second
U
27
14
The experiments success
•As expected, not all the experiments succeeded.
•There were some reasons that caused it:
1. The training session was not long enough.
2. The session training was not led by Bio-feedback
experts.
3. Not always the system was calibrated.
4. The low-frequency signal is not reduced by the
system.
•We had about 70% successful experiments.An experiment
is characterized “successful”, if a simple algorithm can
detect which simbol the subject chose.
15
Technion - Israel
Summer 2002
In this slide, you can see the importance of reducing the low-frequency signal.
T
0
Q
5
K
11
M
16
C
22
F
28
33
Second
16
Technion - Israel
Summer 2002
Here the calibration system is missing, you see the “initializing” effect.
B
0
O
6
L
11
S
16
W
22
U
27
33
Second
17
Technion - Israel
Summer 2002
In this slide it is hard to detect the effect.
M
0
Q
6
K
11
F
17
T
22
C
27
33
Second
18
Technion - Israel
Summer 2002
In this slide there isn’t any effect.
5
10
15
20
25
30
Seconds
19
Technion - Israel
Summer 2002
Part 1 Summary
•Basic experiments indicate that there is a linkage between the emotions and
the voltage deflections measured from our skin.
•Emotions can be evoked by pictures and sounds.
•Emotions can be controlled by the subject decision.
20
Technion - Israel
Summer 2002
Part 2 – Bio-Feedback
•The meaning of “Bio-Feedback” is that man can “learn” his reflexes, can
control them, and can use them as a way of therapy.
•Every hospital uses, nowadays, biofeedback experts in the pain clinics, in
order to help the people help themselves.
•The technique is based on a long training session, led by experts, in order to
learn the “patient emotionalism”, and the special biofeedback way that can
be useful for him.
21
Technion - Israel
•
Summer 2002
There could be some differences between patients:
1. In the recording site.
2. In the detection way.
3. In the psychological ways of the training.
4. In the time of training session which is needed
.
22
Technion - Israel
Summer 2002
•In order to improve our biofeedback abilities, and after consulting with some
experts from the academy and from the medicine area, we have built 3
different train tools for the training.
1. A balloon. The balloon radius is affected by the measured voltage ratio,
of the subject skin. The subject should try to increase the balloon radius.
23
The balloon radius
changes , due to
voltage ratio
change
24
The balloon radius
changes , due to
voltage ratio
change
25
The balloon radius
changes , due to
voltage ratio
change
26
Technion - Israel
Summer 2002
2,3. A beeper and a colored column. The column height and the beeper tone
are the feedback in this tool.
27
28
29
30
31
Technion - Israel
Summer 2002
Part 3 – Visual Basic
Real-Time implementation
•In this part of the project we have build an real-time user-interface,
for our implementation.
•Stage #1 – a matrix is shown to the subject. 5 lines of 5 letters each.
•Each line become bold for 5 seconds.
•The subject choose one of the 25 letters, therefore he will “like” one line
more than the others (the one that includes the chosen letter…)
•The following slides will show the interface and the effect of the “chosen
line” to our mind…
32
Technion - Israel
Summer 2002
The user interface
33
Technion - Israel
Summer 2002
Line 1
3
Line 5
Line 4
6
9
Line 3
12
Line 2
15
18
Second
s
34
Technion - Israel
Summer 2002
Line 1
3
Line 5
Line 4
6
9
Line 3
12
Line 2
15
18
Second
s
35
Technion - Israel
Summer 2002
Line 1
3
Line 5
Line 4
6
9
Line 3
12
Line 2
15
1
Secon 8
ds
36
Technion - Israel
•
Summer 2002
We can see, that the voltage ratio measured from the subject skin, is
affected from the bold line.
How does it work?
•
There are some stages on the interface process:
1. Calibration. On the first 15 seconds the system calibrates.
37
Technion - Israel
Summer 2002
2. After calibrating, the system starts with bolding each line in a random order
for 5 seconds.
The screen appearance is shown in the following slide.
38
Technion - Israel
Summer 2002
We can see here the
bolded line.
Line #3
39
Technion - Israel
•
Summer 2002
3. After choosing a line, by the changes of the voltage ratio on the subject
arm, we stop bolding lines and start bolding letters on the chosen line. Now,
the subject will be able to choose the wanted letter and print it in the right
place. The screen will look as followed.
40
Technion - Israel
Summer 2002
Bolding letters one
by one, in random
order.
41
Technion - Israel
Summer 2002
The chosen letters are
written here.
42
Technion - Israel
Summer 2002
Summary
•The main target was to be able to write a whole word in the real-time
implementation. We have not completely reached this target.
•In accuracy of 70% we are able to write one chosen letter.
•The real-time implementation functionality is not completed.
•We can point at some areas for improvement:
1. A much longer training session.
2. Try to use other Bio-feedback tools.
3. The algorithm should solve some problems that our
algorithm did not solve, like reducing the low-frequency
signal.
43