1999 Systems Engineering Capstone Conference • University of Virginia ERICA, INC. CAPSTONE 1998-1999 Student team: Dayna Balcome, Peter K. Bokach, Justin Petersen Faculty Advisors: Michel A. King, Department of Systems Engineering Client Advisors: Professor Thomas E. Hutchinson, Chris Lankford ERICA, Inc. P.O. Box 615 Ivy, Virginia 22945 E-mail: ericainc@esinet.net KEYWORDS: Eye, Eye gaze, Lie detection, Advertising, Database design ABSTRACT The main goal of this year's Capstone team was to explore potential new markets for the ERICA (EyeGaze Response Interface Computer Aid) system as well as continue to serve its current clients. Dayna Balcome studied ERICA’s potential as a non-invasive deception detection device. She tested the system using the Control Question Test interrogation procedure on 28 subjects while the system measured the subjects’ pupillary size over time. Pupillary response was scored using the techniques described by Lubow and Fein (1996), resulting in correct classification of 75% of the subjects. Peter Bokach’s research explored the potential use of ERICA in the advertising industry. The purpose of his project was to find a correlation between dwell times in Trademark LookZones and recognition supported by Starch Readership Reports. Justin Petersen designed and developed a database system to handle the saving and retrieval of data for the ERICA application, GazeTracker. This system has decreased the workload required of the ERICA programmers for future versions of the application thereby increasing development efficiency. Together, these projects demonstrated new and better ways in which the versatile ERICA system can be utilized. INTRODUCTION TO ERICA, INC. The ERICA project began in 1984 under the direction of Dr. Thomas Hutchinson, the Calcott Professor of Systems Engineering at the University of Virginia. Dr. Hutchinson sought to design a system that would enable disabled individuals to use computer systems in order to communicate effectively. ERICA allows users to control a computer by emulating mouse functions with their eyes. Over the past 15 years, the project has grown in size and scope, and the project was made into a company, ERICA Inc., in 1994. Dr. Hutchinson is the President of this corporation and Chris Lankford, a graduate student at the University of Virginia, is Vice-President. In recent years, ERICA, Inc. has been exploring the uses of the ERICA system in applications other than as a communication aid for the physically handicapped. This year’s ERICA Capstone team, under the guidance of Michel King, the technical advisor, aimed to develop a foundation of experimental data that would support these alternative uses for the ERICA system and sought to develop a database that would increase development efficiency for the ERICA programmers. THE ERICA SYSTEM The ERICA system, using its software and hardware capabilities, tracks the user’s eye and achieves the function of an eye-controlled mouse. The system tracks movement of the eye by monitoring optical features which may be translated to a gazepoint, or a point of regard. This represents the location of the user’s gaze on the screen. In this way, the input from the user's eye allows the ERICA system to execute an unlimited number of functions. In particular, the ERICA system is configured to record the point of regard and ocular characteristics, such as pupil area, as a function of time and position on the computer display. The ERICA system can also grant complete control over the computer using only the eye (“A User’s Guide to ERICA” 1999). The system hardware, through the use of a light emitting diode (LED) reflects harmless infrared light off of a mirror positioned towards the user’s eye. Figure 1 shows the ERICA hardware mounted beneath the computer monitor. 99 Erica, Inc. Capstone 1998-1999 Figure 1 The Camera projects infrared light onto the mirror which reflects it into the user’s eye. The eye absorbs some of the infrared light and re-emits some from the retina. This phenomenon illuminates the pupil, which is then called the bright-eye. The position of the bright eye moves in the camera image field, following the motion of the eye. The corneal surface reflects an intense point of light, which remains stationary as the pupil position changes with the user’s gaze on the screen. This point, called the glint, serves as a reference for determining the user’s gaze point or focal point during the program he or she uses. Figure 2 shows how this harmless infrared light illuminates the eye. Glint Bright Eye Figure 2 Image of the eye with the Glint and Bright Eye. The vector distance between the center of the pupil and the center of the glint provides necessary information for tracking the user’s gaze points on the screen and changes in pupil diameter. The ERICA software then analyzes these images to determine gaze position. In this way, disabled individuals who are physically unable to operate a mouse or the keyboard, can use a computer by simply focusing on the screen (“A User’s Guide to ERICA” 1999). In addition, the ERICA system is completely non-invasive, that is, it requires no physical attachment to the subject. ERICA’s capability to measure and track changes in eye position and pupil size over time implies that the system may have other applications outside of its benefits for the physically disabled. DECEPTION DETECTION Problem Definition This project was based on past research that has determined that pupil dilation can be a physiological 100 indicator of fear or anxiety (Ford 1996), and pupil dilation increases with the workload imposed on an individual’s mental processes (Kahneman and Beatty 1966). It was hypothesized that by tracking an individual’s pupillary response in a lie detection experiment, the ERICA system can detect increases or decreases in stress levels or mental processing. Because the ERICA system is physically noninvasive, it can minimize undue stress in the test subject. In addition, according to Ford (1996), the pupils are governed by the autonomic nervous system and thus are out of the liar’s control. Therefore, it would be very difficult for a subject to devise countermeasures to this system. Experiment Regardless of the interrogation format that is used, the theory behind all lie detection techniques is the same, namely, the physiological response occurring with deception is different from the physiological response to telling the truth. In testing ERICA as a lie detector, the Control Question Test (CQT) format was used as it is the technique most commonly used when investigating specific issues. It centers on two types of yes/no questions: relevant and irrelevant. Working with Sandra Williams, a fourth year Electrical Engineering student, we modeled our experiment after that described by Bradley, MacLaren, and Black (1996). In this case, one group of the test subjects committed a mock crime, others were informed of the relevant facts about the crime, and the third group remained innocent (Bradley 1996). Fifteen male and fifteen female undergraduate students at the University of Virginia were used as subjects, and five female and five male subjects were assigned to each of the three groups. Each subject signed a consent form and was tested individually. Two technicians were present during the testing process; one acted as the interrogator while the other was the assistant who worked with the subject to ensure his or her understanding of the proceedings. Each subject was given instructions as to what his or her condition was, i.e. innocent, innocent informed, or guilty, and the instructions described the actions they were to carry out as part of the mock crime. Guilty subjects were told to go into Office 101f in Olsson Hall and steal $7.00 out of a desk. The innocent-informed were merely told of the crime and that they were to be interrogated, and the innocent were only told that a crime had been committed and that they were going to be questioned. A blank gray slide was presented to the subject to stare at while he or she answered the questions, and the ERICA System monitored the test subjects’ pupillary response during each question. Participants were told they would receive $10 if they were found innocent. 1999 Systems Engineering Capstone Conference • University of Virginia The entire set of ten questions was presented 3 times to each subject during the interrogation. There were four relevant questions paired with a similar but irrelevant question, and the order of the irrelevant and relevant questions in each pair was randomized each round. Additionally, two irrelevant questions were posed regarding the subjects’ first and last names. An intro slide as well as a 10 to 20 second break between rounds was added after testing the first 19 subjects per the recommendation of a professional polygrapher. The sampling rate for pupil diameter was approximately 28 samples per second, and pupil size was measured in pixels. Y pupil diameter is often inaccurate due to eye rotation. Therefore, only X pupil diameter was used for analysis. A simple scoring algorithm (Lubow and Fein 1996) was applied to the average X pupil diameter and standard deviation on each slide. Each subject was awarded two scores from 0 to 6, one score for mean pupil size and one for standard deviation. Each subject could earn up to 2 points per round. They received a 2 if the largest pupil diameter (or standard deviation) in the round occurred on a guilty question, a 1 if the second largest pupil diameter (or standard deviation) occurred on a guilty question and a 0 otherwise. Ideally, innocent or innocent-informed subject should score a 4 or below; Guilty subjects should score 5 or higher. Results and Conclusions Using the mean scoring system, the ERICA system identified 78% of the guilty subjects correctly, 67% of the innocent-informed correctly, and 80% of the innocent correctly. Overall, the system correctly classified 75% of the subjects, while the average accuracy of current polygraph procedures is only 70%. Figure 3 shows the results of this algorithm. Results of Scoring Algorithm The difference in mean scores is significant at the 98% confidence level for guilty versus innocent subjects and at the 82% confidence level for guilty versus innocent informed, but the difference between innocent and innocent informed is probably not significant (34% confidence level). The estimated minimum sample sizes required for 95% confidence in the observed differences, if the observed differences are significant, would be 5 subjects for the guilty vs. innocent comparison, 9 for guilty vs. innocent informed, and 79 for innocent vs. innocent informed. In this experiment, after discarding one guilty subject’s data due to testing error, there were 19 subjects in the guilty vs. innocent comparison and 19 in the guilty vs. innocent informed comparison. Thus, the sample sizes were adequate to support our conclusions concerning the significant differences between these groups. The 20 subjects in the innocent vs. innocent informed comparison did not provide a large enough sample size to assume that the differences in the observed mean scores are statistically significant. Further statistical analysis was based on normalized X diameters for all subjects. This was accomplished by dividing each subjects’ X diameter observations by their total mean X diameter over all questions. Several possible relationships in the data were investigated including statistical comparisons of each question’s mean and standard deviations to every other question’s data for each subject and statistical comparisons of overall mean and standard deviation for all questions and all subjects. However, Figure 4 illustrates the strongest results apparent in the experimental data. It seems that there is a difference in the mean pupil response between irrelevant and relevant questions. The p values for this relevant versus irrelevant question comparison were 0.050 for the guilty subjects’ response, 0.009 for the innocent subjects’ response, and 0.084 for the innocent informed subjects’ response. 7 6 Score 5 4 3 2 1 G 0 0 5 10 15 Subject # 20 25 30 I I-I Figure 3 The scoring algorithm. Guilty subjects should score 5 or above; innocent and innocent informed should score 4 or below. 101 Erica, Inc. Capstone 1998-1999 Mean Comparison 1 .01 1 .005 Pupil Mean 1 0.995 0.99 0.985 0.98 0.975 0.97 Relevant G I I-I Irrelevant Question Type Figure 4 Mean Comparison by Subject Group of Normalized Pupil Size for Relevant compared to Irrelevant Questions Unfortunately, however, in order to prove that normalized X diameters are a successful deception detector, a comparison across subject groups for relevant questions and irrelevant questions would have to produce low p values. When these comparisons were made, the following p values (listed in Table 1 and Table 2) resulted: P Value Guilty Innocent Innocent 0.77 * Inn-Inf 0.80 0.94 Table 1 P-Values for Relevant Question Comparison P Value Guilty Innocent Innocent 0.76 * Inn-Inf 0.83 0.99 Table 2 P-Values for Irrelevant Question Comparison The p values are lower for the comparison between Guilty and both Innocent and Innocent Informed subjects for both question types, however they only indicate an approximate 15-25% confidence level. Thus, we conclude that a comparison of the mean normalized X diameter for a subject’s response to relevant questions versus the mean normalized X diameter for a subject’s response to irrelevant questions is not a useful technique for detecting deception. The comparison of the standard deviation of the normalized X diameter for a subject’s response to relevant questions versus the standard deviation of the subject’s response to irrelevant questions was also not useful. Recommendations for Future Study The experiment documented in this report shows that ERICA is capable of performing as a deception detection device. However, we believe that further 102 analysis would lend additional insights. For instance, the manner in which the questions were labeled irrelevant or relevant could have affected our results. For this reason, further analysis should include a comparison of guilty, innocent, and innocent-informed responses to each question to determine whether some questions were more useful than others in detecting deception. As time was a serious constraint in the analysis of the results, this type of analysis could not be conducted. This project not only contributes to the potential of ERICA, Inc. to compete in the lie detection industry, but it also contributes a significant amount of research to the study of pupillometry, an area that is new to all fields of research. Future ERICA researchers or Capstone team members may consider testing ERICA’s capabilities to perform successfully using the another interrogation format, the General Knowledge Test (GKT), as Lubow and Fein did in 1996. In addition, future projects should continue to work with a representative from the CIA’s Department of Technical Services in order to project ERICA’s presence as a marketable deception detection device further into the lie detection industry. ADVERTISING Problem Definition This project sought to determine the correlation between dwell times using visual fixations determined by the ERICA system’s software and recognition supported by Starch Readership Reports as applied to print advertisements. Advertising effectiveness is determined by readers’ abilities to recall print advertisements and aspects of each advertisement that depict the advertiser’s name, brand, or company logo. Since companies spend large and ever-increasing amounts of money to analyze print advertisements each year, a faster and more costeffective analysis tool would be beneficial to advertising agencies, advertising research companies, businesses, and consumers. Therefore, proving that ERICA is an effective system for predicting the success or failure of advertisements may have a significant social, cultural, and economic impact within the United States and throughout the world. By providing a more cost-effective and expedient method to analyze print advertisements, advertisements will reach the general public sooner and at less cost to companies. Consequently, consumers may be able to enjoy products sooner and at lower cost. 1999 Systems Engineering Capstone Conference • University of Virginia Experiment To prove the ERICA system is an effective tool for analyzing advertisements, we compared the data collected using ERICA with pre-tested data on identical advertisements. The pre-tested data is supplied by Starch Readership Reports, an industry standard for analyzing advertisements. An example of a Starch Report for Cover Girl is provided in Table 3. In the table, % Noted means the percentage of readers that remember previously viewing the advertisement, % Associated means the percentage of readers that remember the logo or advertiser’s name, and % Read Most means the percentage of readers that read most of the body text. Advertisement % Noted Cover Girl 69 % Associated % Read Most 67 47 Table 3 Starch Report Scores Because our pre-tested data was obtained from female readers viewing Glamour magazine advertisements, we tested seventeen female test subjects viewing twenty advertisements, such as the one shown in Figure 5. Figure 5 Thermasilk Advertisement After observing the correlation between a test subject’s dwell times in ERICA LookZones and recall as measured by Starch Readership Reports, we assessed the effectiveness of ERICA as an analysis tool for the advertising industry. Results and Conclusions There is a great deal of quantitative data that can be utilized to support the claim that ERICA is an effective analysis tool. When comparing each test subject individually with all of the advertisements, analysis revealed that a strong negative correlation (=-0.855, p=0.02) exists between the Starch Associated data with high scores and the Average Time in Trademark LookZones for all test subjects. Trademark LookZones refer to LookZones around the logo and advertiser’s name within an ad. As the average test subject spends more time looking at the brand or advertiser’s name, she is less likely to remember the name of the brand or advertiser. Several possible explanations for this observation are: Some advertisements may already be familiar to some readers, especially if they currently use that product. Thus, such readers may decide to spend their time looking at other aspects of the ad, rather than looking at a brand name or logo that they are already familiar with. In such cases an increase in a reader’s Time in Trademark LookZones might be correlated with a lower likelihood of her making an association. When an advertisement is unfamiliar to the reader, she may spend more time on the advertiser’s name because it is foreign to her. Then, when asked to recognize the specific brand or name in question, she may still be sufficiently unfamiliar with that advertiser’s name that she cannot associate it with the product. Readers may recognize advertisers’ names or brands that are connected to the theme or general concept of the entire advertisement. For example, a reader may spend minimal time reading the “Wonderbra” label; but, if she spends a great deal of time looking at the model in lingerie she may associate the entire picture with the advertiser Wonderbra. These suggestions are only some of the possible explanations as to why the negative correlation exists. Additional research will be required to determine all of the contributing factors. Although our experiment does not provide insight into the causal mechanism, we conclude that Time in Trademark LookZones, as measured by ERICA, is proven to be a negative indicator of advertiser or brand association. When coupled with additional information concerning causal factors, this should help ad agencies and ad researchers predict the successfulness of print advertisements. Recommendations for Future Study Our recommendations for follow-on research are: Extend our research and analysis of Starch Readership Reports in order to look for other significant correlations within the data available. Test a larger number of subjects and use different publications other than Glamour in order to differentiate the types of advertisements. 103 Erica, Inc. Capstone 1998-1999 Contact other advertisement research agencies and test the ERICA system against their testing methodologies. Explore the use of ERICA to analyze television commercials and quantitatively assess their effectiveness to sell products or services. opened and display the user’s direction of gaze as it is used. We have provided convincing evidence as to the effectiveness of ERICA to analyze print advertisements. Therefore, we hope that additional time and resources will be expended to fully realize the potential of ERICA in the advertisement analysis industry. INCREASING DEVELOPMENT EFFICIENCY THROUGH DATABASE DESIGN Problem Definition The employees at ERICA have begun to take the roles of consultants to outside customers. A company or individual who has a specific need for the system requires particular demands of ERICA, Inc. These employees then analyze the request and custom fit the system to the needs of the customer. The original design of the ERICA system was found to be inefficient. The software employed a file system in which all recorded data was saved in an assortment of file extensions. Each time the developers needed to access the information, they had to sort through a large number of file types. Additionally, the system required an intense level of code for each new type of data retrieval. Whenever a customer submitted a request concerning a specific type of data analysis, the employees at ERICA had to painstakingly hard code the analysis options into the code of the ERICA software. This process proved to be long and tedious. One responsibility of the ERICA, Inc. Capstone Project Team was to improve the ease of development of an ERICA software application, called GazeTracker. This program has proven to be the most development intensive application for the ERICA programmers. GazeTracker is responsible for two functionalities using the ERICA system. First, the “Slide Show” function observes a user’s gaze response to a slide show of still images. It allows for the complete recording and playback of where the user is looking as different images are displayed on the computer screen. Figure 6 includes a picture of a print advertisement after it has been viewed in a “Slide Show” analysis. A line shows where the viewer was looking on the advertisement over time. Second, the “Application Analysis” functionality allows for the recording of gaze direction as other Windows applications are used. On playback, GazeTracker can display each application as it is 104 Figure 6 “Slide Show” analysis example In order to improve the ease of development of GazeTracker, we designed and developed what is called an object-relational database system. This system provides an object-oriented programming environment to save and retrieve data to and from a relational database. That is, it allows the ERICA programmers to send an object (i.e. application data member) to be saved without having to worry about how or where it is saved. The object-relational system breaks this object down into smaller data types that can be saved in a relational database. The database behind the GazeTracker application is divided into two database files representing the two functionalities of GazeTracker and a section of code used to access them. For the purposes of this document, the Slide Show database, BmpBase, deals with the storing and organization of data relating to the Slide Show functionality of GazeTracker. Similarly, the application database, AppBase, deals with data relating to the application functionality in GazeTracker. The combination of BmpBase, AppBase, and the code used to access them is appropriately called GazeBase. In order to complete this project, we utilized the help of Jason Rudolph, a Computer Science major working with ERICA, Inc. He assisted us by writing the code in Visual C++. Database Design and Development The database design and development consisted of four major stages. In the first phase, specification and requirements, we identified what data were to be stored in the database. In the second stage, system architecture, we organized the major components of an 1999 Systems Engineering Capstone Conference • University of Virginia object-relational database system. In the third stage, we created the database on paper with a series of design documents. Finally, we implemented the design of the database in Microsoft Access 97. The following sections will explain these stages in more detail Specifications and Requirements. This stage of the design allowed us to create a list of variables that were to be stored in the database and to understand the relationships between the variables. This work consisted of learning how GazeTracker works, learning the code behind the application, and interviewing the ERICA personnel about their expectations of the database system. Systems Architecture The database architecture defines the main components of the database system and how they interact. The object-relational database system is composed of three components -- the relational database, its encapsulating code, and the GazeTracker application. The reason for using this type of database system is as follows. A relational database alone cannot store user-defined types of variables such as those used in GazeTracker. Each instance of a class must be broken down into their member variables to be saved. Then, when the information is retrieved from the database, the variables must be recreated using the stored member variables. Figure 7 provides a pictorial representation of the Database system. As can be seen in the figure, there are no direct links between GazeTracker and the databases. The application contains instances of classes that contain member variables that must be saved and retrieved from a database. The encapsulating code (named Communication Component in Figure 7) performs these communications between the application and database. System Design We created four design documents that would help us develop the database system. The first document, the Class Object Model, gave a pictorial representation of the data in GazeTracker. It included the variables organized by class and the relationships between the classes. The Relational Layout showed the relationships between these data variables, as they were to be included in the database. The third document, the Data Dictionary, provided definitions for each table and field in the database. Finally, the Data Characteristics Document indicated whether or not each field was a required field, gave field lengths for text data types, and identified all primary and foreign keys. System Development After the first iteration of the database design, using Microsoft Access 97 and the original versions of the Relational Layout, Data Dictionary and Data Characteristics Document, we developed the first version of BmpBase. With each design iteration thereafter, we reflected any necessary changes in the Access database. After a number of iterations, we completely implemented the current version of BmpBase. While we implemented the database, Jason Rudolph wrote the encapsulating code to support the communication between it and GazeTracker. We are currently developing AppBase and its associated encapsulating code to support the data saving and retrieval for the Application Analysis functionalities of GazeTracker. Results and Conclusions Prior to this project, ERICA Incorporated employed a complicated and inefficient file saving system using numerous file extensions to save different types of data. The system was complicated because it caused the software developers to keep track of a large number of file types at one time. It was inefficient because it required a large amount of code for the saving and retrieval of data in each new version of the software. This system prohibited the ERICA programmers to maintain development efficiency. In an effort to begin solving this problem, we have produced the backbone of the data storing in the GazeTracker application. The object-relational architecture allows for easy data storing and retrieval. The project has also produced a significant amount of documentation that will allow future database developers at ERICA, Inc. to easily maintain and revise the system. Figure 7 Object-Relational DB System Architecture. 105 Erica, Inc. Capstone 1998-1999 CAPSTONE PROJECT CONCLUSION BIOGRAPHIES During the 1998-1999 school year, the Capstone Team members, Dayna Balcome, Peter Bokach, and Justin Petersen, worked to expand the ERICA system’s potential to compete in various markets. We succeeded in introducing ERICA into the field of deception detection. In particular, the Department of Technical Services at the Central Intelligence Agency and the U.S. Department of Defense are interested in our work. We also demonstrated the system’s capability to analyze print advertisements and improved the marketability of the system in the advertising industry. The Gaze-base project improved the flexibility of ERICA, Inc.’s software so that ERICA, Inc. employees can respond to new customer requirements more efficiently. Dayna Balcome is a fourth-year Systems Engineering major from Charleston, SC. Her principal contribution to the project was in the area of developing the ERICA system as a deception detection device. Ms. Balcome has accepted a position with Ernst & Young LLP in New York, NY, and will begin training in the Management Consulting division in August. REFERENCES Bradley, M.T., V.V. MacLaren, and M.E. Black. 1996. “The Control Question Test in Polygraphic Examinations with Actual Controls for Truth.” Perceptual and Motor Skills, 83: 755-762. DeMars, Geneva. 1997. “Lie Detection: An ERICA Application.” Undergraduate Thesis. University of Virginia. DeSanti M., and J. Gomsi. 1994. "A comparison of object and relational database technologies." Object Magazine 3.5 (January): 51, 56-57. Ford, Charles V. 1996. Lies! Lies!! Lies!!! The Psychology of Deceit. American Psychiatric Press, Inc., Washington DC. Kahneman, D. and Beatty, J. 1966. Pupil Diameter and Load on Memory. Science, 154: 1583-1585. Lubow, R.E. and Fein, Ofer. 1996. “Pupillary Size in Response to a Visual Guilty Knowledge Test: New Technique for the Detection of Deception.” Journal of Experimental Psychology: Applied. 2, No.2: 164-177. Rational Rose. “Integrating Object and Relational Technologies.” Rational Rose Web Site. 1998. <http://www.rational.com/support/techpapers/object_re lational.html#What_Creates_Need_ORI> Roper Starch Worldwide Inc. “Starch Readership Report.” Glamour. September 1998. “A User’s Guide to ERICA.” 1999. Property of ERICA Inc. 106 Peter Bokach is a fourth-year Systems Engineering major from Columbus, Ohio. His principal contribution to the project was determining the effectiveness of ERICA as an advertising tool and comparing his test results with an industry standard, Roper Starch Reports. Mr. Bokach has accepted a position with the Management Consulting division of PricewaterhouseCoopers in Los Angeles, CA. He will begin working in September of 1999. Justin M. Petersen is a fourth-year Systems Engineering major from Chapel Hill, NC, concentrating in Computer Information Systems. His principal contribution to the ERICA Capstone Project was in the area of database system design and implementation. Mr. Petersen has accepted a position with the Atlanta, GA office of Ernst & Young Consulting LLP and will begin working in September of 1999.