UNS A-6 Trainer Effectiveness

Calhoun: The NPS Institutional Archive Theses and Dissertations Thesis Collection 1981 A model to measure bombardier/navigator performance during radar navigation in device 2F114, A-6E weapon system trainer. Mixon, Ted R. Monterey, California. Naval Postgraduate School http://hdl.handle.net/10945/20611 i..'<.t;'.' M't HJ^ MvV,' ,*.l';l'^^•• ^vl;!<v;i;' - ¥0.' .OL /oî-'9t NAVAL POSTGRADUATE SCHOOL Monterey, California THESIS A MODEL TO MEASURE BOMBARDIER/ NAVIGATOR PERFORMANCE DURING RADAR NAVIGATION IN DEVICE 2F114, A-6E WEAPON SYSTEM TRAINER by Ted R. Mixor L March 1981 Thesis Advisor: W. F. Moroney Approved for public release; distribution unlimited. T199180 6 UNCLASSIFIED SCCUHITY CLAtSlFICATlOM Of THIS ÂCC (Wham Omm enl«#«tf) READ INSTRUCTIONS BEFORE COMPLETING FORM REPORT DOCUMENTATION PAGE I. 4. mtPour NUMSCM TITLE (mtd 2. OOVT ACCCtSION NO. Subtltit) TYPE OF REPOBT 5. A Model to Measure Bombardier/Navigator Performance During Radar Navigation in Device 2F114, A-6E Weapon System Trainer 7. WeClPtEHT'S CATALOG NUMBER i- AUTMO»»f«> k PERIOD COVERED Master's Thesis March 1981 (. RERFONMINS ORG. RCRORT NuMSCR • COMTWACT OM GRANT NLMSCRCl . Ted R. Mixon » RCMFORMINS OnOANlZATtON NAME ANO AOOMCSI 10. RROORAM ELEMENT. PROJECT TASK AREA 4 WORK UNIT NUMBERS ' Naval Postgraduate School Monterey, California 93940 1 I. CONTROLLING OFFICE NAME ANO ADDRESS 12. M^rrh 1QR1 Naval Postgraduate School Monterey, California 93940 14. MONITOMINC AGENCY NAME REPORT DATE IS. NUMBER OF PAGES 301 * AOORESS<i/ «ll«r«n( ham Cati«rollln« OlUcm) IS. SECURITY CLASS, (al (Ma tiôn) Unclassified tSa. DECLASSIFICATION/ OOWNCRAOINC SCHEDULE l«. DISTRIBUTION STATEMENT (at tttl» JtapMl) Approved for public release; distribution unlimited. 17. DISTRIBUTION STATEMENT It. SUPPLEMENTARY NOTES IS. KEY WORDS (Comtlmf (ol (ft* mhttrmel an rawmrtm tlâ II mttfH In A-6 Flight Training A-6E Weapon System Trainer Aircraft Simulator Training Devices ABSTRACT (Camtlmf on r«v«r«« alM II 30, II tllllmrmtl fra« ff«por*) n«e«««arr ">* I4amtlfr kr MacM numbmr) A- 20. Sloe* nacmmamr mt4 Simulator Training Simulator Performance Bombardier/Navigator Performance Human Performance Pprfnrm;:inr-<^ i^mmlllr *r M«efe w— fc»rj This thesis modeled a performance measurement system for the Bombardier/Navigator (B/N) Fleet Replacement Squadron student during low level radar navigation flight in the Navy A-6E Weapon System Trainer. The model was designed to determine student skill acquisition measures for the purpose of providing information for decision-making by the squadron instructor and training manager. Model formulation methodology DO FORM ,1 AN 7) 1473 EDITION OW NOV •• S/N 0tO2>OI4'6«01 t I IS OBSOLETE UNCLASSIFIED StCUBITY CLASSIFICATION OF THIS PAGE (Whm- Of Kniarad) < UNCLASSIFIED euiT^ euitM*>«e*T»»ii o» Two m^%%n^tm ntm ««it— (continued) was based on a literature review of aircrew performance measurement from 1962-1980 and an analytical task analysis Over 50 currently accessible candidate of the B/N's duties. measures were listed and a proposal was made for a competitive exercise (Derby) using A-6E fleet aircrews flying preprogrammed routes to establish performance standards using the candidate measures. Multivariate discriminate analysis was recomnfended for measure reduction. A sequential sampling decision model selected for evaluation provided fixed decisional error rates for successful training termination decisions and utilized both objective and subjective performance measures. Several display formats were recommended for use in debriefing. 20 19 (continued) Performance measurement Performance criteria Performance tests Measurement Pilot performance Pilot performance measures Pilot performance measurement Aircrew performance Aircrew performance measurement Objective measurement Student measurement Student achievement Training evaluation Evaluation Form D 1473 Jan 73 i/N 0102-014-6601 Sequential testing Quantitative methods Proficiency Pilot proficiency Instructor pilot Task analysis System analysis Time Line Analysis Individual differences Skill Flight skills Navigation skills Complex skill acquisition Navigator UNCLASSIFIED 1 iteuMi^v cuAUiriCATiOH o^ »•<• ^•atr«*«" o*** t»»»«»«<i Approved for public release; distribution unlimited A Model to Measure Bombardier/Navigator Performance During Radar Navigation in Device 2F114, A-6E Weapon System trainer by Ted R. Mixon Lieutenant, United States Navy B.S., United States Naval Academy, 1974 Submitted in partial fulfillment of the requirements for the degree of MASTER OF SCIENCE IN OPERATIONS RESEARCH from the NAVAL POSTGRADUATE SCHOOL March 1981 ^. OUOtEV KNOX I 'SflAftV NAVAJL POSTGrtAOlisT' MONTEREY. CALIf a39«. ABSTRACT This thesis modeled a performance measurement system for the Bombardier/Navigator (B/N) Fleet Replacement Squadron student during low level radar navigation flight in the Navy A-6E Weapon System Trainer. The model was designed to determine student skill acquisition measures for the purpose of providing information for decision-making by the squadron instructor and training manager. Model formulation methodology was based on a literature review of aircrew performance measurement from 1962-1980 and an analytical task analysis of the B/N's duties. Over 50 currently accessible candidate measures were listed and a proposal was made for a competitive exercise (Derby) using A-6E fleet aircrews flying preprogrammed routes to establish performance standards using the candidate measures, Multivariate discriminate analysis was recommended for measure reduction. A sequential sampling decision model selected for evaluation provided fixed decisional error rates for successful training termination decisions and utilized both objective and subjective performance measures. recommended for use in debriefing. Several display formats were TABLE OF CONTENTS I. A. B. C. II. 12 INTRODUCTION 13 BACKGROUND 1. Simulation and Performance Measurement 13 2. Review of Previous Studies 17 SUBJECTIVE AND OBJECTIVE PERF0Rr4ANCE MEASUREMENT- 22 1. Introduction 22 2. Subjective Performance Measurement 23 3. Objective Performance Measurement 27 4. Summary 31 A-6E TRAM AIRCRAFT AIvID ITS MISSION 32 1. A-6E Performance Specifications 32 2. Mission 33 3. Scenarios 34 4. Summary 3 5 STATEMENT OF PR0BLEI4 37 A. PROBLEM STATEMENT 38 B. THE IMPORTANCE OF OBJECTIVE PERFORMANCE MEASUREMENT 39 1. Policy Guidance 40 2. Establishment of Standards 41 3. Effective Use of Simulators 42 4. Effectiveness of Instructors 42 5. Effectiveness of Training 43 III. IV. 6. Skill Identification and Definition 44 7. Instructional Systems Development- 45 47 iMETHODOLOGY A. PROCEDURE 47 B. ASSUriPTIONS 49 1. Simulator Fidelity 50 2. Pilot and B/N Relationship 50 3. Literature Review Results 51 4. Mathematical Description of Behavior 51 5. Experience and Skilled Behavior 52 THE FUNDAMENTAL NATURE OF PERFORMANCE EVALUATION A. B. MEASUREMENT CONSIDERATIONS 53 54 1. Definition and Purpose of Measurement 54 2. Types of Measures 57 3. Levels of Measurement 66 4. Measure Transformations 68 5. Accuracy of Measurement 69 6. Reliability of Measurement 72 7. Validity of Measurement 75 8. Selection of Initial Measures 79 CRITERIA CONSIDERATIONS 81 1. Definition and Purpose of Criteria 81 2. Types of Criteria 81 3. Characteristics of Good Criteria 3 3 4. Other Criteria Characteristics 83 5. Establishment of Criteria 84 6 C. D. V. 6. Sources of Criterion Error 85 7. Measures of Effectiveness 85 8. Selection of Criteria 88 PERFORMANCE MEASUREJMENT CONSIDERATIONS 90 1. Subjective Versus Objective Measures 90 2. Combining Measures 91 3. Overall Versus Diagnostic Measures 93 4. Individual Versus Crew Performance 95 5. Measures and Training 96 PERFORMANCE EVALUATION CONSIDERATIONS 97 1. Definition and Purpose 97 2. Types of Performance Evaluation 99 3. Accuracy of Evaluation 4. Evaluating Individual and Group Differences 5. Characteristics of Evaluation THE NATURE OF THE BOMBARDIER/NAVIGATOR TASK 101 — 102 103 104 A. INTRODUCTION 104 B. TASK CONSIDERATIONS 107 1. Task Definition 107 2. Classification of Tasks 107 3. Task Relation to Performance and System Purpose 103 C. RADAR NAVIGATION TASK ANALYSIS 110 1. Background 110 2. Previous A-6 Task Analyses 113 3. Current Task Analysis for Performance Measurement 118 4. VI. A-6E WST-PERFORiyiANCE MEASUREMENT SYSTEM A. B. C. VII. B/N Skills and Knowledge WST CHARACTERISTICS B. 137 137 1. General Description 137 2. Performance Evaluation System 14 PERFORMANCE MEASUREMENT SYSTEMS 14 5 1. Systems Analysis 146 2. Allocation of Functions 150 3. System Criteria 154 4. System Implementation 156 CURRENT PERFORMANCE MEASUREMENT IN THE WST RESULTS AND DISCUSSION A. 127 CANDIDATE MEASURES FOR SKILL ACQUISITION 158 166 167 1. Method of Measurement 167 2. Measure Segment 167 3. Scale 182 4. Sampling Rate 182 5. Criteria 182 6. Transformation 182 7. Currently Available 183 8. Accessible 183 ESTABLISHMENT OF PERFORMANCE STANDARDS 18 3 1. Radar Navigation Maneuver 18 3 2. Simulator "Intruder Derby" 184 C. MEASURE SELECTION TECHNIQUES 185 D. EVALUATION METHODS 18 6 - VIII . E. INFORMATION DISPLAYS 190 P. IMPLEMENTING THE MODEL— 198 SUr4MARY 200 A. STATEriENT OF PROBLEM 200 B. APPROACH TO PROBLEM 201 C. FORMULATION OF THE MODEL- 203 D. IMPACT TO THE FLEET 204 BIBLIOGRAPHY 206 APPENDIX A A-6E TRAM RADAR NAVIGATION TASK LISTING 221 APPENDIX B GLOSSARY OF TASK SPECIFIC BEHAVIORS 23 6 APPENDIX C A-6E TRAM RADAR NAVIGATION TASK ANALYSIS 240 APPENDIX D RADAR NAVIGATION MISSION TIME LINE ANALYSIS 280 SEQUENTIAL SAMPLING DECISION MODEL- 287 APPENDIX E: INITIAL DISTRIBUTION LIST 298 LIST OF TABLES I. NAVIGATION MANEUVERS AND SEGMENTS 36 II. A TAXONOMY OF MEASURES 59 III. LEVELS OF MEASUREMENT 67 IV. COMMON MEASURE TRANSFORMATIONS 7 V. PERFORMANCE EVALUATION DECISION MODEL FOR THE INSTRUCTOR 98 VI. CLASSIFICATION OF BOMBARDIER/NAVIGATOR BEHAVIORS 121 VII. TAXONOMY OF SKILLS REQUIRED 124 VIII. A-6E WST EVENT RECORDING SELECTION 14 4 IX. HUMAN AND I4ACHINE CAPABILITIES AND LIMITATIONS 151 X. CURRENT B/N TASKS GRADED IN WST CURRICULUM 160 XI. CANDIDATE PERFORMANCE MEASURES 168 XII. RADAR NAVIGATION PERFORMANCE SUMMARY 197 10 LIST OF FIGURES 1. Methodology Flow Diagram 48 2. A-6E Crew-System Network 105 3. Skill Acquisition and Performance Evaluation Model 4. Device 2F114, A-6E WST 5. A-6E WST Performance Evaluation System 141 6. Typical B/N Flight Evaluation Sheet for WST 163 7. Sequential Sampling Decision Model for TP Navigation Task 189 8. Time Activity Record Display 192 9. Time-on-TP Management Display 193 10. Fuel Management Display 194 11. Radar Navigation Accuracy Display 196 El. Hypothetical Sequential Sampling Chart 290 — 132 -138 11 I. INTRODUCTION Since 1929, when Edwin A. Link produced the first U.S.- built synthetic trainer designed to teach people how to fly, flight simulation has witnessed substantial advances in simulation technology and increased incorporation of these devices into both military and civilian aviation training programs. A recent addition in 1980 to this advance was the A-6E Weapon System Trainer (WST) , device 2F114. A sophisticated flight simulator with a six degree of freedom motion system, the A-6E WST was designed to provide the capability for pilot transition training, Bombardier/Navigator (B/N) transition training, integrated crew training, and maintenance of flight and weapon system proficiency in all non-visual elements of the A-6E Carrier Aircraft Inertial Navigation System (CAINS) mission. The development of high-fidelity flight simulation has been accompanied by advances in aircrew performance measurement systems, which are ideal for the simulator training environment, and have been widely implemented and the subject of extensive research in all three military aviation communities. The purpose of this thesis is to design a system to improve current performance measurement techniques for the B/N Fleet Replacement Squadron (FRS) student by the development and application of a performance measurement system that 12 incorporates the advantages of both objective skill acquisition measures and subjective instructor measurement. A . BACKGROUND While simulation has been a popular component of many aviation training systems for over forty years, objective performance assessment had not been incorporated until some fifteen years ago. This section will discuss the current state of flight simulation, aircrew performance measurement, and provide a brief review of previous navigator performance measurement studies. 1. Simulation and Performance Measurement Simulation is the technique of reproducing or imitating some system operation in a highly-controlled environment. Modern flight simulators have evolved from simple procedure trainers into devices that represent specific aircraft counterparts, and imitate or duplicate on- board systems and environmental factors. The two main purposes of flight simulators within the training environment are training and evaluation. Training is designed to improve performance and some means of providing feedback to the student is needed to indicate the adequacy of his behavior, and ought to provide guidance for the correction of inappropriate response patterns. Evaluation involves testing and recording the student's behavior in the performance examination situation. [Angell, et al., 1964]. The reasons for using the simulator as an integral part of a military flight training program were examined by 13 Tindle [1979], Shelnutt, et al. [1977] , and Roscoe [1976] . [1980], North and Griffin Some basic justifications for simulator training include: (1) Simulation provides training in skill areas not adaptable to an actual training flight because of technological or safety considerations. (2) Crews master sJcills in the aircraft in less time after learning those skills in a simulator. (3) The cultivation of decision-making skills is an instructional objective calling for situational training that may be carried out safely only in a simulated tactical environment. (4) Simulators are effective for training crewmembers of varying experience and expertise in a variety of aircraft for a number of flight tasks. (5) Greater objectivity is obtainable for measuring student performance by using controlled conditions and automated performance measurement features in the simulator than in the aircraft. . (6) Instructors are not distracted by operational constraints in the simulator and are more available for teaching and evaluation roles. These considerations are by no means exhaustive, but they do indicate the utility of simulators in flight training programs, especially in evaluating student performance. This thesis is addressed primarily to the problem of measuring B/N performance during a radar navigation training flight while in the A-6E WST. The performance of a B/N is the exerted effort (physical or mental) combined with internal ability to accomplish the A-6E mission and its functions. Some development of performance measurement definitions and goals is necessary because the problem of assessing B/N performance 14 , during a training program is obviously but a segment of a broader topic - the measurement of human behavior. Glaser and Klaus [1966] defined performance evaluation as the assessment of criterion behavior, or the determination of the characteristics of present performance or output in The importance of defining terms of specified standards. performance evaluation is paramount to any training assessment situation, as it gives common ground to operationally describing the human behaviors that make up performance itself, and identifies behavior elements that may be measured by either objective or subjective means. An expanded discussion of performance measurement and evaluation can be found in Chapter IV. The purposes for assessing performance by the application of standard objective measurement operations were stated by Angell, et al. [1964], and Riis [1966]: (1) Achievement - to determine the adequacy with which an activity can be performed at the present time, without regard, necessarily, for antecedent events or circumstances. (2) Aptitude - to predict the level of proficiency at which a person might perform some activity in the future if he were given instructions concerning the activity. (3) Treatment efficacy - to observe the effects upon performance of variation in some independent circumstances such as (a) instructional techniques, (b) curriculum content, (c) selection standards, (d) equipment configurations, or the li]<:e. The flight simulator training environment allows for special applications of the above as found in Danneskiold [1955] Angell, et al. [1964], Riis [1966], Glaser and Klaus 15 [1966], . Farrell [1974], Shipley [1976J and McDowell , [1978] : (1) Diagnostic - determine strong and weak areas of student proficiency. (2) Readiness - determine operational readiness of an aviation unit. (3) Discrimination - assess performance to provide information about an individual's present beliavior as compared to other individuals. (4) Selection - of persons for promotion or advancement or placement. (5) Learning rates - determining the rate at which learning takes place. (6) Management subsystems (7) Evaluation - of training devices in terms of effectiveness and transf er-of-training. of an entire training program and its - The above goals of performance measurement represent some of the major reasons why assessment of student performance is important in training program simulators. Most importantly, measurement provides FRS instructors and training officers with the information needed to make correct decisions [Obermayer, et al. , 1974; Vreuls and Wooldridge, 1977]. Per- formance measurement does not in itself replace the decision- maker in the FRS, but instead provides complete and necessary information of an objective nature to the appropriate evaluator (instructor or training officer) , so that more accurate and reliable decisions can be made concerning student progress within the training syllabus. If instructors and training officers utilize the potential of a performance measurement system, a more effective and efficient training program would be a result. 16 , Review of Previous Studies 2, Most of the literature on aircrew performance measurement in the last forty years has primarily concentrated on the pilot crewmember [Ericksen, 1952; Danneskiold, 1955; Smode et al., 1962; Buckhout, Mixon and Moroney, 1981] 1962; Obermayer and Muckler, 1964; . The first comprehensive evaluation of techniques used in flight grading was by Johnson and Boots [1943] , who analyzed ratings given by instructors and inspec- tors to students on various maneuvers throughout stages of training. One result showed correlations between grades assigned by different raters to the same subject as being very low. This result of low observer-observer reliability when using subjective ratings will be discussed in the next section. The earliest studies involving the radar navigation performance of a crewmember other than the pilot were a pen and pencil radar scope interpretation experiment by Beverly [1952] , and two Air Force radar bombing error projects by Voiers [1954], and Daniel and Eason [1954]. The first study was concerned with constructing a suitable test for the measurement of navigational radar scope interpretation ability of student aircraft observers. The latter two studies were concerned with identifying perceptual factors which contributed to cross-hair error during bomb runs of a radar bombing mission and with comparing the components of simulated radar bombing error in terms of reliability and sensitivity to practice, respectively. A similar follow-up study on radar scope 17 interpretation and operator performance in finding and identifying targets using a radar was performed by Williams, et al. [I960]. These four studies represent most of the research of non-pilot radar navigation performance measurement prior to 1965. This fact is not surprising, due mainly to the early role played by the observer in very simplified and pilot- oriented aircraft as compared to today's specialized navigator in complex, computer-oriented aircraft. Since navigation is a primary duty of any aviator across a spectrum of aircraft types, some helicopter pilot and copilot studies are of some value to review. Nap-of-the-Earth (NOE) flight is a Helicopter visual-dominated low level mission where altitude and airspeed are variable in close proximity to the ground. Some navigational performance measures utilized in these studies were; number of turn points found, probability of finding a turn point, and route excursions beyond a criterion distance [Fineberg, 1974; Farrell and Fineberg, 1976; Fineberg, et al., 1978; Smith, 1980]. Low level visual navigation flights in helicopters were also studied in some detail, again with pilot performance being the main concern [Lewis, 1966; Billings, et al., 1968; Sanders, et al. , 1979]. Two rather novel investigations of Anti- Submarine Warfare (ASW) helicopter team performance using the content and flow of team communications during simulated attacks were done by Federman and Siegel [1965] and Siegel and Federman [1968] 18 All team communications were recorded, classified, and compared to target miss distance as an effectiveness measure. Although they found some types of team communication to correlate highly with mission success, their method was highly impractical for the operational situation due to the large number of personnel needed to play back and classify the communication types. Nevertheless, the results from this research indicate the value of using crew communication as a measure of crew performance. Several fixed-wing studies with navigation as the primary mission are also of interest to the current study. Schohan, et al. [19 65] and Soliday [197 0] used the Dynamic Flight Simulator for several Low Altitude High Speed (LAHS) missions designed to investigate pilot and observer performance during turbulent, lengthy (3-hour) flights. [1972] Jensen, et al. did several studies investigating pilotage errors in area navigation missions for the Federal Aviation Administration. These three studies are significant in that numerous navigational accuracy performance measures were used to assess pilot (or observer) performance. After 1970, due to the increased complexity of many modern aircraft, more research was directed toward individual aircrew members, and not just the pilot. Among the aircraft investigated were: P-3C, F-4J, A-7, F-106, B-52, C-141, C-130, KC-135, and the C-5 [Matheny, et al., 1970; Vreuls and Ober- mayer, 1971; Obermayer and Vreuls, 1974; Geiselhart, et al., 19 . 1976; . These studies are unique in Swink, et al., 1978]. that defining and assessing aircrew performance by other than the subjective ratings, as commonly used for decades, became a technological challenge requiring new analytical and empirical approaches An Air Force fighter-bomber, the F-lllD, was designed and built during the late 1960 tical capability of the A-6E. 's with virtually the same tac- With a two-man side-by-side cockpit arrangement, this land-based aircraft is the closest counterpart to the A-6E for the radar navigation air interdiction mission. Two experiments using the F-lllA flight simulator were performed mainly for equipment configuration effects on pilot performance [Geiselhart, et al. Geiselhart, et al., 1971]. , 1970; Research by Jones [1976] examined the use of the F-lllD flight simulator as an aircrew performance evaluation device. Unfortunately, these F-111 studies do not specifically address the issue of how to measure navigator performance during radar navigation, but do provide some information on measuring performance in an aircraft with a similar mission and almost the same crew interactions as the A-6E. Only one experiment known to this author has been conducted using an A-6 configured simulation. [1970] Klier and Gage investigated the effect of different simulation motion conditions on pilots flying air-to-air gunnery tracking tasks in the Grumman Research Vehicle Motion Simulator 20 (RVMS) They concluded that simulator motion need not be a faithful reproduction of real-life motion in order to provide essential motion cues. Saleh, et al. [1980] performed an analytical study for two typical tactical combat missions representative of the A-6E and A-7E aircraft to determine significant decisions which are made in the course of accomplishing mission objectives. The results of this study provide information regarding the decision type, difficulty, and criticality and can be used in identifying the critical areas in which aircrew decision-aiding may significantly improve performance. Finally, a study by Tindle [1979] concluded that the integration of the A-6E WST (device 2F114) into the FRS training program would be more cost-effective than using the A-6E WST as an addition to existing training programs. This study also concluded that aircrew performance measurement in the A-6E WST was vital for more effective use of the simulator. This section has presented a brief review of aircrew performance measurement studies in the literature. The potential value in reviewing the literature lies in uncovering the analytical and empirical approaches taken in measuring aircrew performance, noting both the significance and practicality of those approaches. Since previous research on actual B/N performance during the radar navigation air interdiction mission appears to be nonexistent, extrapolations from other aircrew performance measurement studies is important and necessary to the current study. 21 What has worlced and been practical for other aircraft and aircrews in the way of performance measurement certainly applies to the A-6E B/N, keeping in mind the definitions and goals of the A-6E B/N, performance measurement, and the A-6E mission. B. SUBJECTIVE AND OBJECTIVE PERFORMANCE MEASUREMENT 1, Introduction Traditionally, all aircrew performance measurement in the Navy, Air Force and Army has been assessed by an instructor pilot or navigator using a subjective rating scale which places the student in one of several skill categories based on norm-referenced testing. More recently, objective methods of evaluating performance have been developed and implemented in both the simulator and in-flight environments. Subjective and objective methods are not dichotomous but represent a continuum of performance measurement. At one extreme there exists the strictly personal judgement and rating of performance, and on the other end of the continuum is a completely automated performance measurement and assessment system. This section will define and describe the elements of each method together with the advantages and disadvantages associated with each. The approach taken in this study will be to integrate the use of automatic performance measurement within the A-6E training environment while still exploiting the advantages of using the instructor as a component of the measuring system. 22 2. Subjective Performance Measurement Subjective measurement can be defined as an observer's interpretation or judgement between the act observed and the record of its excellence with an almost complete reliance placed on the judgement and experience of the evaluator [Cureton, 1951; Ericksen, 1952] . Simply stated, subjective measurement is qualitative in nature as what is being measured is observed privately 1973; [Danneskiold/ 1955; Knoop and Welde, Roscoe, 1976; Vreuls and Wooldridge, 1977]. Through an introspective process, the "expert" instructor judges the performance level demonstrated by a student whether or not agreed-upon standards of performance have been applied [Billings, 1968; McDowell, 1978]. a. Advantages of Subjective Measurement The advantages of using subjective performance measurement methods have been well-documented throughout the literature. Instructor ratings in the past have been the least expensive of all evaluation methods [Marks, 1961]. This decisive advantage has been eroded in recent times by severe shortages of military aircrew in both operational and training units. Marks [1961] also pointed out that ratings forms are constructed easily and quickly, and the administration of the ratings system requires no physical arrangement. [1978] McDowell recently concluded that a subjective performance measurement system for many complex tasks such as flying were easy to develop, gave the rating instructor high face validity since 23 he is usually an acknowledged expert, and contained specific feedback of a type important in the training situation usually not found in objective performance measurement systems. Rat- ings are still used because they meet the needs of training management without seriously intruding into the instructor pilot's operational capabilities [Shipley, 1976]. One important use of an instructor to subjectively grade a student is to motivate the student through selective reinforcement [Prophet, 1972; Carter, 1977] . Sometimes an overly positive or negative grade by the instructor in the appropriate area of desired performance improvement for the student serves as a catalyst in the student's attitude toward self- improvement. Some studies have shown that some degree of high reliability can be achieved between instructor ratings [Greer, et al., 1963; Marks, 1961]. Brictson [1971], in a study of over 2500 carrier arrested landings, reported measures derived from the Landing Signal Officer (LSO) grades to be highly correlated with objective estimates derived from a weighted combination of wave-off s, bolters, and the particular wire engaged. Similar high correlations between raters were also found by Waag, et al. [1975] for undergraduate pilots flying seven basic instrument maneuvers in the Air Force Advanced Simulation in Undergraduate Pilot Training (ASUPT) facility, b. Disadvantages of Subjective Measurement High reliability between instructors using subjective rating methods is generally not the case and therein 24 lies the foremost disadvantage of the subjective performance measurement method. Knoop [197 3 J reported two instructor pilots (IP's) subjective ratings were each correlated with certain objective performance measures, but the objective measures themselves were determined to be not highly correlated with skilled or proficient operator performance. Ericksen [1952] reviewed numerous flight studies involving pilot training between 1932 and 1952 and concluded that subjective grading involved a lack of reliability and inconsistent differentiation between students. Danneskiold [1955] found observer-observer correlations no higher than .47 for three Basic Instrument Check maneuvers, while a more objective test had observer-observer correlations of .86. The training and evaluation skills of the instructor evolve primarily from their personal experiences in the highly complex aircraft and simulator environment. Establish- ing adequate standards of performance, or criteria, is a major problem in all flight training. Knoop and Welde [197 3] found lack of agreement between pilots on the specific criteria for successful performance of certain aerobatic maneuvers, due largely to the and proficiency. differences in examiner knowledge, experience, It was also found that the same maneuver may be flown satisfactorily in a number of different ways. Other research has explored the criteria problem which is inherently part of subjective performance measurement [Cureton, 1951; Danneskiold, 1955; Marks, 1961; McDowell, 1978]. 25 Even rating . . . , , methods were inadequate when used as criteria to validate alternative methods of measurement and evaluation [Knoop and Welde, 1973J An instructor must be able to process large quantities of information during a simulator session. Subjective grading competes with this capability of the instructor, and may prevent perception and evaluation of all the relevant dimensions of task performance during [Knoop, 1968; a training mission Roscoe, 1976; Shipley, 1976; Carter, 1977; Vreuls and Wooldridge, 1977] Several other factors which contribute to subjective aircrew rating variances are discussed below: (1) A tendency of raters to be more lenient in evaluating those whom they know well or are particularly interested in [Smode, et al., 1962; Bowen, et al., 1966]. (2) The observations tend to accumulate on one or two points in the rating scale, usually near the central or average point. This phenomenon contributes toward a lack of sufficient discrimination among students 1962; [Smode, et al. Shipley, 1976]. (3) Instructor and student personalities interact to yield a result which does not reflect true performance [Marks, 1961] (4) A tendency for the ratings on specific dimensions to be influenced by the rater's overall impression of the student's performance - the "halo effect" 1962; Glaser and Klaus, 1966]. 26 [Smode, et al . . . (5) . . An unrelated problem with the simulator may influence the evaluation outcome [Jones, 197 6] (.6) Evaluation results are dependent upon the attitude, concern, and values of the instructor, thus a natural personal bias is introduced into the performance observation [Smode, et al. 1962; Knoop and Welde, , 1973; Jones, 1976] (7) Instructors have different concepts of the specific grading system in regard to the flight parameters involved, knowledge tested, weights to be assigned, and ranges of qualifying categories (8) [Knoop and Welde, 1973] A tendency to actually rate others in the opposite direction from how the rater perceives himself on the particular performance dimension has been found [Smode, et al. , 1962] . (9) Ratings tend to become more related along different dimensions when they are made closer to each other in time than ratings having a larger interval of time between observations [Smode, et al. (10) , 1962] Unless the simulator has a playback capability, a permanent record of the performance is lost when subjective ratings are used [Forrest, 1970; Gerlach, 1975]. 3. Objective Performance Measurement Objective measurement is defined as observations where the observer is not required to interpret or judge, but only to record his observations [Cureton, 27 1951] . While subjective judgement is more qualitative in nature, objective measurement demands that what is being measured be observed publicly and with a quantitative result [Knoop and Welde, 197 3; Roscoe, 1976; Vreuls and Wooldridge, 1977; McDowell, 1978] . Objective measures demand that performance be evaluated in terms of criteria which are relatively independent of the observer, have consistent interpretations, and a high degree of observerobserver reliability [Ericksen, 1952; Danneskiold, 1955; Marks, Smode, et al., 1962]. 1961; The first systematic use of an objective grading method was by Miller, et al. Objective measures were collected [1947]. during a single week from over 8,000 students in four different phases of pilot training. Objective observations by way of a prepared checklist reduced variability attributable to the observer and correlations as high as .88 were found between instructors observing a student during the same flight. In most cases, higher observer-observer reliability has been found when objective measures are used [Angell, et al., 1964; Forrest, 1970] . The measures are free from personal and emotional bias of the instructor, as well as judgemental bias that are characteristic of subjective measurement. a. Advantages of Objective Measurement Most advantages of objective performance measurement appear to contrast the disadvantages of subjective measurement. By having a computer process large amounts of continuously varying information, the instructor is freed to 28 concentrate on those aspects of student performance which resist objective measurement, and to devote more attention to the primary duty of instructing [Krendel and Bloom, 1963; Knoop and Welde, 197 3; Vreuls and Obermayer, 1971, Vreuls, et al., Development of performance criteria could be made 1974]. on the basis of permanently recorded objective measures [Forrest, 1970; Angell, et al . , 1964]. A system of data collection would provide records and transcriptions of individual and crew performance in practice missions to identify particularly effective or ineffective behaviors for later analysis in the event of an aircraft accident et al. , 1964] [Forrest, 1970; Angell, . Objective measurement enables timely and diagnostic information of consistent weaknesses in performance [Angell, et al. , 1964; Knoop and Welde, 1973]. Instructional methods may be modified as performance results indicate their effectiveness, or laclc of it. Students attaining desired achievement levels may also be identified earlier within the training syllabus. Several researchers postulate the quantification of skill learning rates 1973]. Bowen, et al. [Angell, et al., 1964; Knoop and Welde, [1966], even found objective measures motivated pilots to actualize their skills in overt performance measurement. They speculated that the "heightening of performance is due to intrinsic motivation (personal desire to achieve) , social motivation (pressures from group to demonstrate proficiency) , and a focusing of attention on each 29 particular performance which encourages the pilot to actualize the knowledge and skills which he possesses." b. Disadvantages of Objective Measurement If objective performance measures are so much more desirable than subjective ratings, why have they not been incorporated into more training situations? The major reason lies in the fact that they are much more expensive than subjective methods and some aircrew tasks are difficult to automatically record and computer grade. Whenever a number of simultaneously occurring tasks such as communication/ procedures, and the application of knowledge are present during a particular task, measuring and quantifying the operator behavior involved becomes a complex and inherently difficult task in itself. Smode [1966] found that when flight instructors are required to monitor and record performance information during a flight task, some resentment against the objective measurement method occurred due to the large amount of instructor attention required. This instructor monitoring and recording of student performance information has since been replaced by automatic digital computers. The same report also concluded that detailed analyses of aircrew tasks into perceptual-motor tasks, procedural tasks, and decision-making distorts reality as all three are very interrelated. Objective tests are rigidly constrained by given hardware that has to be physically arranged, programmed, and is subject to equipment malfunctions. Since objective tests require more structure than subjective 30 testing, instructors are given very little choice in what they must do and "there is a certain natural resentment against the regimentation of setting up and observing this event at this time [Smode, 1966]." 4. Summary Subjective and objective performance measurement of aircrew has been defined, compared, and contrasted for strengths Subjective testing is universally feasible, and weaknesses. minimizes paperwork, allows for instructor flexibility, is easy to develop and administer, and is inexpensive. Objective testing minimizes instructor bias, eases grading due to automatic data collection, storage and dissemination, improves performance standardization, and produces a high inter-rater reliability. Each method has its merits individually, but when used together in a cohesive and synergistic combination, improvements can be made in aircrew performance measurement and assessment. Angell, et al. [1964] stated, "There are some areas in which the human observer can make more subtle judgements and more sophisticated evaluations than can any electromechanical instruments. . . the human observer/teacher should not be an adjunct, but rather an integral part of the total measurement system." This report will attempt to use this observation in the design of a system to measure A-6E B/N performance during radar navigation in the simulator. This section is concluded by a listing of items the examiner should evaluate, as determined by Knoop and Welde [1973]: 31 (1) Ability to plan effectively. (2) Decision-making capability. (3) Sensorimotor coordination and smoothness of control. (4) Ability to share attentions and efforts appropriately in an environment of simultaneous activities. (5) Knowledge and systematic performance of tasks. (6) Confidence proportionate to the individual's level of competence. (7) Maturity; willingness to accept responsibility, the ability to accomplish stated objectives, judgements, and reaction to stress, unexpected conditions, and aircraft emergencies. (8) Motivation (attitude) in terms of the manner in which it affects performance. (9) Crew coordination. (10) Fear of flying. (11) Motion sickness. (12) Air discipline - adherence to rules, regulations, assigned tasks, and command authority. C. A-6E TRAM AIRCRAFT AND ITS MISSION 1. A-6E Performance Specifications The A-6E aircraft is a two-man, subsonic, twin engine medium attack jet aircraft, with side-by-side seating for the pilot and B/N. Designed as a true all-weather attack aircraft using a sophisticated radar navigation and attack system, the aircraft can accurately deliver a wide variety of weapons without the crew ever having visually acquired the ground or the target. Capable of carrying a pay load of up to 8.5 tons, it is the only carrier-based aircraft in the Tactical Air 32 (TACAIR) wing capable of penetrating enemy defenses at night, or in adverse weather, to detect, identify and attack fixed or moving targets. The TRAM (Target Recognition, Attack, Multisensor) configured A-6E aircraft has a completely integrated computer navigation and control system, radar, armament, flight sensors, and cockpit displays that enable the aircraft to penetrate enemy defenses at distances approaching 600 nautical miles in radius while at an extremely low altitude. 2 . Mission The mission of the A-6 "Intruder" is to perform high and low altitude all-weather attacks to inflict damage on the TACAIR recognizes three primary enemy in a combat situation. missions to accomplish the objective of successfully waging war [Gomer, 1979]. Counter Air (CA) , The missions are: Close Air Support (CAS), and Air Interdiction (AI) . CAS is air action against hostile ground targets that are in close proximity to friendly ground forces, requiring detailed integration of each air mission with the battle activities and movements of those forces. CA operations involve both offensive and defensive air actions conducted to attain or maintain a desired degree of air superiority by the destruction or neutralization of enemy air forces. AI missions are conducted to destroy, neu- tralize, or delay the enemy's military potential before it can be brought to bear against friendly forces, usually at far distances not requiring detailed integration of air and ground activities. The AI mission was selected in this study 33 because it is representative of those missions frequently performed in the A-6 community. Analysis of all three primary TACAIR missions was beyond the scope of this study. Saleh [1980] defined a mission as the aggregate scenarios, maneuvers, and segments that constitute successful employment of the system. The starting point in determining criteria for performance measurement and suggesting what specific and clearly identifiable operations of the B/N should be examined in greatest detail is an operational definition of the man-machine mission [1963] [Smode, 1962; Vreuls, 1974]. McCoy further stated that in order to judge the effectiveness of any element of a man-machine system, it must be judged in terms of contribution of the element to the final system output, which is the ultimate objective of the man-machine system. It is with these criteria in mind that the AI mission definition is used to limit performance measurement of the B/Z^ in the A-6E WST. 3. Scenarios Analysis for comprehensive performance measurement begins with a complete decomposition of the mission into smaller parts for which activities and performance criteria are more easily defined [Vreuls, 1974; Connelly, 1974; Vreuls and Cotton, 1980J . Any mission may be described in terms of a scenario, or intended flight profile or regime. A performance measurement standards tri-service project [Vreuls and Cotton, 1980] , classified military aviation into ten scenarios or 34 flight regimes: tion, (3) euvers, (8) Transition/Familiarization, (1) Formation, (6) (4) Instruments, Air Combat Maneuvering, Ground Attack, (9) (5) (7) (2) Naviga- Basic Fighter Man- Air- to- Air Intercept, Air Refueling, and (10) Air Drop. Scenarios may be further subdivided into maneuvers by identifying natural breakpoints using time, position, or definitive portions requiring computation of different performance measures or changes in required operator skill level. of maneuvers are take-off, climb, navigation. Examples landing, and point-to-point Segments are subdivisions of maneuvers that con- tain groupings of those activities that must be accomplished in performing the maneuver. Table I (adapted from Vreuls and Cotton [1980]) contains possible maneuvers and segments for the navigation scenario. The present study selected point-to-point navigation using radar terrain mapping to further narrow the scope of the effort and to tailor the performance measurement aspects of the A-6E mission toward the tasks of the B/N. 4 . Summary The mission of the A-6 for the current study has been defined as Air Interdiction which is further subdivided into point-to-point navigation using radar terrain mapping. Success- ful accomplishment of the radar navigation point-to-point segments reasonably infers some degree of overall Air Interdiction mission accomplishment, which is the overall objective of the A-6 man-machine system. Operationally dividing the overall A-6 35 mission into flight maneuvers enables practical performance measurement as the operator skills required (and measured) vary from segment to segment. Within each segment, measurement is conceivably possible at two levels: (1) measurement of the total man-machine system outputs for comparison to expected mission goals, and (2) measurement of human operator activity in relation to system outputs [Vreuls, 1974]. TABLE SCENARIO Navigation I: NAVIGATION iMANEUVERS AND SEGMENTS I4ANEUVERS SEGiMENTS Point-to-Point Flight Dead Reckoning Contact (visual) Inertial Radar Terrain Mapping Nap-of- the- Earth Very Low Map Interpretation Airways -Radio VOR TACAN ADF- Source: Off Airways -Radio Area Navigation Over Water LORAN Celestial Global Positioning System Vreuls and Cotton [198 0] 36 II. STATEMENT OF PROBLEM The need for aircrew performance measurement and assessment has long been recognized across all aviation communities. Per- formance measurement produces information needed for a specific purpose, such as the evaluation of student performance or the identification of aircrews needing training. Unfortunately, the assessment of aircrew proficiency in those skills associated with advanced flying training still depends largely on subjective evaluations by qualified instructor pilots (IPs) and instructor B/Ns (IB/Ns) , supplemented with analytically- derived somewhat objective mission performance metrics, e.g., bombing scores [Obermayer, et al., 1974]. An economically acceptable means of objectively measuring behavioral skills in the operational or crew training environment has continued to be a critical problem in the FRS, due mainly to a "nice to have" and nonessential outlook towards any performance measurement scheme other than the traditional "always done this way" method of subjective ratings. A Department of Defense review of tactical jet operational training in 1968 commented: "The key issue underlying effective pilot training is the capability for scoring and assessing performance ... in essence, the effectiveness of training is dependent upon how well performance is measured and interpreted [Office of Secretary of Defense, 1968]." 37 Despite the key issue of scoring and assessing performance being identified, current aircrew performance measurement by- IPs and IB/Ns during simulated missions in the A-6E WST is all subjective in nature, although the 2F114 simulator has a More current objective performance measurement capability. details on current aircrew performance measurement by instructors in the A-6E WST will be given in Chapter VI. A. PROBLEM STATEMENT Current student performance measurement and assessment in the A-6E WST by an instructor is entirely subjective in nature. The A-6E WST has the capability to objectively measure student performance, but is not being utilized in this fashion. Meas- uring performance is the key to training effectiveness. Objective measurement for aviation training programs has been Effective performance measure- prescribed by higher authority. ment by using objective methods is vital to establishing performance criteria, the effective utilization of the simulator, instructor effectiveness, aircrew skill identification and definition, and the Instructional Systems Development systems approach to training. (ISD) The problem that must be addressed is designing a performance measurement and evaluation system for the B/N during radar navigation that will incorporate the characteristics of objective performance measurement and still retain the judgement and experience of the instructor as a valuable measuring tool. This thesis will focus upon 38 , performance measurement as a system with definable components that interact and produce information necessary for the successful identification of the skill level of the student in regards to navigating the A-6E aircraft. As a result, the A-6E WST can be utilized more effectively, instructors can become more effective in teaching students critical combat skills, and students can complete FRS training being identified at a minimum skill level and "mission ready" for full-system radar navigation in the A-6E aircraft. B. THE IMPORTANCE OF OBJECTIVE PERFORMANCE MEASUREMENT A number of factors have contributed to the emerging role of objective aircrew performance measurement in both actual flight and simulators of military aviation units. Generally, this role has developed through an increased awareness of the advantages associated with objective measurement, and the several basic disadvantages of the subjective evaluation method, as outlined in Chapter I. The remainder of this section will outline both potential and actual necessities for objective performance measurement of aircrew in the training environment. Beginning with a study of Department of Defense policy toward aircrew performance and evaluation methods, the benefits of objective performance measurement are discussed in regards to: standards establishment, increased simulator, instructor, and training effectiveness, aircrew skill level identification and definition, and lastly, ISD requirements. 39 : 1. Policy Guidance Several studies offer guidance with respect to the issue of using more objective measurement techniques in air- This guidance supports the development and crew training. utilization of objective performance measurement as an adjunct or complement to current subjective ratings. In 1968, the Department of Defense review previously cited found that "subjective evaluation was the technique in general use in training programs observed" and had been since before World War II [Office of Secretary of Defense, on to comment, 1968]. The study went "Judgement and experience can be helped by quantitative analytical methods" and that the application of such methods serves three purposes (1) They make it necessary to identify the standards of performance desired for each of the many events the pilot must learn. (2) They determine how many practices or trials a student must accomplish, on the average, to meet the desired standard. (3) They tell the manager how much improvement he normally may anticipate with each additional practice or trial. This study concluded: "The services should apply objective evaluation techniques where currently feasible in parts of existing training programs ..." and "where valid performance data in -aircrew training programs can be recorded and stored, quantitative analytical methods should be used to assist the commander in making decisions concerning revising and adjusting the course." 40 A Study by the Comptroller General of the United States (General Accounting Office) in 1973 to the Congress on the use of flight simulators in military pilot training programs stated, "Simulators could also be used to more accurately measure pilot proficiency by using systematic grading procedures." A lack of standardized grading instructions which did not show performance tolerances for the Navy was noted. Conclusions reached were: Objective grading of pilot proficiency using simulators would provide more consistent and accurate results for many phases of flight training and eliminate the possibility of human bias and error associated with the current evaluation method simulator grading accurately evaluates pilot proficiency for certain flight maneuvers. . 2. . . Establishment of Standards The performance criteria, or standard, is a statement or measure of performance level that the individual or group must achieve for success in a system function or task [Office of Secretary of Defense, 1968] . When performance standards are established on the basis of subjective experience and expertise, the result in most cases will be inadequate. When standards are set too low, some risk is incurred with degraded system effectiveness. is the result [Riis, 1968; Campbell, McDaniel, 1980] When set too high, costly overtraining 1966; Office of Secretary of Defense, et al., . 1976; Deberg, 1977; Rankin and The establishment of a standard or baseline of performance is an important result of objective performance measurement. The Department of Defense review in 1968 stated. 41 "Reliable measures of pilot performance against validated standards is the keystone for determining how much instruction and practice is required to attain desired levels of skills [Office of Secretary of Defense, 1968]." Even though the importance of performance standards is recognized, some concern by the A-6 FRS instructors has occurred about establishing operational standards for aircrew performance, due to possible misuse, incorrect adaptation in the training program, or insufficient assessment before implementation et al. 3. , 1976]. [Campbell, This issue will be addressed in Chapter VII. Effective Use of Simulators Objective performance measurement increases the effective use of simulators. When performance measures and criteria are defined, inputted, and monitored by an automatic system requiring little instructor intervention, other training and teaching functions of the simulator may be used by the instructor; thus an increase in the effective use of simulators occurs [Danneskiold, 1955; Knoop, 1968]. 4. Effectiveness of Instructors The major impact of an effective measurement method on the instructor during a simulator mission would be to free him from monitoring dials. Cathode Ray Tubes (CRTs), and lights for aircrew performance measurement and evaluation. Due to the complexity of the A-6E WST, the simulator instructor is humanly unable to monitor and interpret in real-time all pertinent performance information during a training mission. 42 Objective measurement techniques would relieve the instructor of these monitoring duties, enabling more time for teaching and evaluating those aspects of human performance that are inaccessible by objective methods [Smode and Meyer, 1966; Knoop, 1968; Vreuls, 1977; Charles, 1974; Kemmerling, et al., 1978, Semple, et al . , 1979]. 1975; Carter, When performance standards are established by objective methods, instructor judgements can be made more reliable and valid by confining the instructor's judgement to evaluating performance without the additional burden of establishing and adjusting personal standards [Office of Secretary of Defense, 1968]. Efficiency of instructor utilization may be achieved by allowing instructors more flexibility in identifying and assisting students who are found to be deficient from objective performance measurement feedbacJc of criterion levels reached [Carter, 1977; Deberg, 1977; Kelly, et al., 1979]. Such objective information might also provide instructors diagnostic information about their own performance as a teacher after seeing patterns of strengths and wea]^nesses in their students 1979] [Kelly, et al., . 5. Effectiveness of Training Many factors influence simulator training effectiveness, including: simulator design, the training program, students, instructors, and the attitude of personnel towards the simulator [Tindle, 1979] Smode and Meyer [1966] . , in a review of Air Force pilot training, concluded: "The development 43 of objective scores to be used in simulator training would represent a major step toward improving the effectiveness of pilot training programs." Other notable results of objective measurement can also contribute to increased training effectiveness. The real-time feedback of performance measurement to the student is essential to the fundamental concept of knowledge of results, a prerequisite to any learning process. Quantitative feedback, in turn, allows the student and instructor to determine the student's individual strengths and weaknesses in performing the mission, which may then be concentrated on by the instructor for remedial training [Smode and Meyer, 1966; Obermayer, et al., 1974, Deberg, 1977; Pierce, et al., 1979; Kelly, et al., 1979]. Carter, 1977; Modifications in training methods, course content, and sequence of course material could be more accurately assessed by the FRS training officer [Vreuls and Obermayer, 1971; Pierce, et al Kelly, et al., 1979]. . , 1979; Student progress within a training program can be more accurately monitored, culminating with the introduction to the fleet of an "operationally capable" or "mission ready" aircrew member at minimum cost [Riis, 1966; Campbell, et al., 1976; Pierce, et al., 1979]. 6. Skill Identification and Definition The employment of objective measures in simulator training will enable the identification and definition of critical combat skills of mission ready aircrews. The precise definitions of "current" and "proficient" and the quantitative 44 . major problem measurement of these concepts continues to be a in both training and fleet environments today [McMinn, 1981] Objective performance measurement requires the definition of "proficient" as a prerequisite to quantification of performance [Pierce, et al., 1979]. 7. Instructional Systems Development Instructional Systems Development is currently being applied to military flight training systems. The approach requires extensive analysis of the specific training to be accomplished, the behavioral objective for each task to be trained, and the level of proficiency required [Vreuls and Obermayer, 1971]. In support of ISD, measures and a measurement system are necessary to: (1) in their operational environments, instructional standards, (3) perform analyses of systems (2) establish quantitative provide an index of achievement for each behavioral objective, and (4) evaluate alternative instructional content, approaches, and training devices [Vreuls and Obermayer, 1971; Obermayer, et al., 1974; Deberg , 1977; Prophet, 1978; Kelly, et al., 1979]. When a state-of-the-art flight simulator is available to an ISD team, it should be the basic medium around which the course is organized [Prophet, 1978]. [1976] Campbell, et al. applied the ISD process to the design of an A-6E aircrew training program and used the A-6E WST for a large part of student training. The study concluded that "difficulty was experienced in applying the ISD process to the development 45 of Specific Behavioral Objectives (SBOs) and criteria test statements, due to the lack of documented quantitative standards of performance." In a review of U.S. ing program development, Prophet Navy fleet aviation train- [197 8] reviewed the A-6E ISD application as well as three other major ISD efforts for various aircraft. The results of that study concluded that one "... major shortcoming was in the area of performance measurement and evaluation," and recommended measurement as a possible future area for improvement to the ISD model. The need for incorporating objective performance measurement methods has been addressed. The methodology for the introduction of objective performance measurement into the A-6E WST for B/N performance will now be discussed. 46 III. A. METHODOLOGY PROCEDURE The methodology used in formulating a model to measure B/N performance during radar navigation in the A-6E WST was based on an extensive literature review and an analytical task analysis of the B/Ns ' duties. proach taken in this report. Figure 1 illustrates the ap- After selection of the mission, scenario, and segment of interest, the review concentrated on aircrew performance measurement research, which emphasized navigation, training, and skill acquisition. A model was then formulated to show the relationship among student skill acquisition, performance evaluation, and the radar navigation task. This hybrid model, discussed in Chapter V, was improvised by the author specifically to illustrate difficult concepts of aircrew performance measurement and evaluation. The literature review identified different approaches taken in using performance measurement from a systems point of view, some of which were integrated and applied to the current situation. An in-depth task analysis of the B/N was performed with the purpose of generating candidate performance measures for operator behavior. Skills and knowledge required to perform the radar navigation maneuver were identified and a mission time line analysis was conducted to identify tasks critical to performance. A model was formulated of the A-6E crew-system 47 Select A-6E Mission, Scenario and Segment \/ \/ \/ Review Literature on Aircrew Perf. Methods /Measures 1 2k Reviev/ Generic Aircrew PMS Concepts aL Perform B/N Task Analysis ± Formulate Model of Measurement and Skill Acq. \^ Identify Skill/ Knowledge Required for each Task -^ \/ Review Current Grading of B/N and WST PMS Identify Candidate Measures for Navigator Training Perform MTLA to Identify Critical Tasks /Measures J^ Formulate Model of Crew-System Network 3^ Identify Candidate Measures for Radar Navigation \/ Candidate Select Measures for Skill Acquisition ^' Formulate PMS Model for B/N Figure 1. Methodology Flow Diagram. 48 interactions to illustrate the complexity involved in measuring B/N performance. After defining the purpose of measuring B/N performance, candidate performance measures were identified for possible use in measuring B/N performance. Candidate performance measures from the literature and the task analysis were compared and measures were selected that met the criteria of face validity, ease of use, instructor and student acceptance, and appropriateness to the training environment. These candidate m.easures were then compared to current B/N student performance measurement and generic performance measurement systems. The result was a performance measurement system for the B/N during radar navigation in the A-6E WST. Evaluation models were then investigated; a sequential sampling decision model was selected for B/N performance evaluation. B. ASSUMPTIONS This section will present some underlying assumptions that are necessary for performance model development and implementation, beginning with a discussion on the necessity of the A-6E WST to realistically duplicate the A-6E CAINS aircraft in both engineering and mission aspects. The unique role of the pilot during the radar navigation mission in the A-6E WST is discussed with respect to his contribution to measuring the B/N's performance. A discussion of the literature review in respect to the relationship between results from pilot studies and navigator performance is presented, followed by a discussion of the need for the existence of a mathematical 49 . relationship between measurement and operator behavior. Finally, an assumption is stated concerning the relationship between motivated and experienced aircrew and high skill levels. 1. These discussions follow. Simulator Fidelity It is assumed that the A-6E WST represents to a satisfactory degree those elements of the A-6E CAINS aircraft such that the A-6E WST aircrew is confronted with a realistic "duplication" of the operational situation and that the aircrew should be required to perform as they would in actual aircraft flight. Given this assumption, training and performance evaluation can be effectively achieved in the simulator for most B/N activities. 2 Pilot and B/N Relationship The A-6E effectiveness in terms of crew-system output is a function of pilot and B/N crew coordination. Because of the major role of the B/N's activities in achieving the desired mission success during A-6E radar navigation, it is assumed that any variability within the A-6E system that can be measured and attributed to the pilot will be small. In effect, the pilot's function within this mission, scenario and segment will be to "fly system steering," which, for the most part, is the result of the B/N's performance as a navigator and systems operator. 50 . . 3. Literature Review Results Most of the literature in the area of aircrew performance measurement has for the most part concentrated on the pilot for performance measurement and evaluation. For similar missions, scenarios and aircrew tasks, it is assumed that what was a significant result in terms of performance measurement for a pilot will be much the same result as that for a navigator. This assumption does not include the psychomotor domain of human operator performance entirely, but does draw some parallels from pilot research results to the naviga- Although each position within the aircraft is somewhat tor. different, many similarities are assumed to exist in terms of operator output, man-machine system output, and measures of effectiveness 4 Mathematical Description of Behavior It is assumed that a mathematical relationship exists between some aspects of operator behavior and performance measurement and evaluation. Most likely, for the multidimensional aspects of behavior, a multi-descriptive mathematical result would best describe that behavior in valid and reliable terms. Objective performance measurement relies for the most part on numerical and statistical analysis of operator and system outputs. Thus, this assumption is necessary for the utilization of objective performance measurement to measure and evaluate the B/N's control movements. 51 5. Experience and Skilled Behavior When properly motivated and presented with a realistic simulated flight mission with the representative flight tasks, highly experienced ("fleet qualified") aircrews are assumed to exhibit skilled behavior of an advanced stage or high level that is characterized by minimum effort and consistent responses ordinarily found in actual aircraft flight for the same mission. The demonstrated performance of highly skilled aircrew, under this assumption, allows for the establishment of performance standards from which comparisons can be made to populations of aircrew that are less than highly skilled. Both the problem of motivated behavior and establishment of performance standards will be discussed in Chapter VII. 52 IV. THE FUNDAMENTAL NATURE OF PERFORjyiANCE EVALUATION This section is not intended to be a definitive expo- sition on performance measurement and evaluation theory. However, certain basic concepts of performance measurement and evaluation need to be defined and explained so that a common understanding of subsequent chapters will occur with minimum confusion. This material is approached with a logical time-dependency sequence, beginning with measurement theory, and ending with some desirable characteristics of a total performance measurement and assessment system in the training environment. Four major areas of performance evaluation will be discussed in this section: measurement considerations, criteria considerations, performance measurement considerations, and performance evaluation considerations. The main purpose is to show that measurement and criteria are needed before the evaluation process begins. Measurement considerations include the definition and purpose of measurement, types of measures, levels of measurement, transformations, measurement accuracy, reliability and validity of measurement, and the selection of initial measures for man-machine performance. The area of criteria considerations addresses the definition and purpose of criteria, types of criteria, characteristics of criteria, establishing criteria, sources of criterion error, measures 53 . of effectiveness, and selection of criteria. Performance measurement considerations include other aspects of performance measurement such as: subjective versus objective measures, combining measures, overall versus diagnostic measures, individual versus crew performance, and training measures. The last area of this section, performance evaluation considerations, shows how evaluation depends upon measurement and criteria, and discusses the definition and purpose of performance evaluation, types of evaluation, accuracy of evaluation, evaluating individual and group differences, and the characteristics of evaluation. The reader already familiar with the above material may wish to skip ahead to the next section. Others not familiar will need the theory to aid understanding of subsequent chapters A. MEASUREMENT CONSIDERATIONS 1. Definition and Purpose of Measurement Measurement is information about performance for a specific purpose, such as whether or not a student is "mission ready" to navigate a particular aircraft 1980] . [Vreuls and Cotton, Unfortunately, this definition leaves open the serious question of quantification; just what and how do you measure and then transform the raw data into useful information? In elemental measurement theory, measurement involves the assignment of a class of numerals to a class of objects, where the 54 class of objects becomes human behavior, the class of numerals must be defined, and some sort of rules for assigning the numerals to the objects must exist [Lorge, 1951; Forrest, 1970] Measurement then becomes an abstract concept of "mapping" a class or set of numerals to a class or set of human behaviors or performance, but this concept then becomes more quantifiable in nature. All measurements are estimates of the true value or actual amount of the human behavior possessed at a given point in time [Smode, et al., 1962]. The difficulty in the measurement of human behavior increases when the important aspects of the behavior being measured are more qualitative than quantitative in nature [Glaser and Klaus, 1966]. Meas- urement requires the action of observation, where behavior is noticed or perceived and recorded. Glaser and Klaus [1966] and Lorge [1951] noted that some observations can be made directly, involving perceptions of the behavior or of the behavior's properties, where other observations can only be estimated from inferences about the behavior, or its properties from its effects on other system components. to measurement, as outlined by Forrest [197 0] , The steps include: (1) Determine the specific object or aspect to be measured. (2) Locate or expose the particular object or aspect to view. (3) Apply a measurement scale. (4) Express the results as a dimension. 55 Measurement must precede the activity of performance evaluation, which is the process of interpreting the results of measurement and comparing them to an established standard. Measurement in the training context serves a variety of functions which emphasize either achievement (present knowledge or skill level) or prediction (expected performance under specified conditions) . Several specific purposes of training performance measurement as outlined by Smode, et al. Buckhout and Cotterman [1963] and Obermayer [1971], Farrell [1962], Glaser and Klaus [1966] , [1974], and Vreuls, Vreuls , et al . [1975] are enumerated below: (1) Prediction of future performance of a student for specified future operational setting. (2) Present performance evaluation of knowledge, skill level, or performance level of a student. (3) Learning rate evaluation at several points in a training program to provide a basis for judging a student's present stage of learning for subsequent advancement to the next training phase. (4) Diagnostic identification of strengths and weaknesses of a student so that additional training may occur. (5) Training effectiveness resulting from the nature and extent of training syllabus or course material changes. (6) Criterion information necessary for defining what constitutes successful or proficient performance. (7) Functional requirements for future training equipment. C8) Selection and placement of the student with an achieved level of proficiency to a position or mission with a required minimum proficiency level. 56 a . 2. Types of Measures Measurement is a process of producing raw data in the form of measures (parameters or variables) as a result of observing human behavior in a man-machine system. Measures are quantities which can take on any of the numbers of some set and which usually vary along a defined continuum, or scale [Knoop, 1968] . The classification of measures is commonly done by using the characteristics of the measures themselves. Using taxonomies developed by Smode, et al et al. [1964] , and Vreuls and Obermayer . [1962], Angell, [1971] , measures may be grouped into several major classes with a collection of like measures within each class. The major classes are listed and briefly defined below: (1) Time periods in output or performance. (2) Accuracy or correctness of output or performance. (3) Frequency of occurrence or the rate of repetition of behavior. (4) Amount achieved or accomplished in output or performance. (5) Quantity used or resources expended in performance in terms of standard references. (6) Behavior categorization by observers (subjective measurement) (7) Condition or state of the individual in relation to the task which describes the behaviors and results of that behavior on system output. This classification produced approximately 83 measures within the seven major classes and are listed completely in Smode, et al. [1962] . A more recent taxonomy by Mixon and Moroney 57 . [1981] . . grouped objective only aircrew performance measures by the following six major classes: (1) Physiological outputs from the operator. (2) Aircraft systems (3) Man-machine system output within the operating environment (4) Time periods in output or performance. (5) Frequency of occurrence or the rate of repetition of behavior. (6) Combined overall measures. , instruments or equipment. These measures were obtained from an extensive literature review of aircrew performance measurement spanning the years 1962-1980. class. Table II lists over 180 measures within each major It is interesting to note that all of these measures were obtained from actual aircrew performance field or laboratory results, in contrast to previous listings of proposed or candidate measures. Some measures listed in Table II will be used as candidate measures for B/N performance during radar navigation in the WST (see Table XI) a. Distributions of Measures The process of measuring continuous and discrete human behavior results in a sample of measures that are estimators of the actual operator behaviors in the system being examined. The usefulness of the measures becomes apparent when they are used to describe the behavior as "good" or "bad" when compared to a reference or standard measure (criterion) Statistical techniques are used to transform raw measures into 58 . TABLE II: A TAXONOMY OF MEASURES PHYSIOLOGICAL OUTPUTS Biochemical analysis (blood) Cardiovascular Communications/Speech Electroencephalogram (EEG) Electromyogram (EMG) Eye movements Finger tremor Galvanic Skin Response (GSR) Metabolic rate Perspiration weight loss Pupillography Respiration Temperature Time of sleep Urinary catecholomines Visual Evoked Potential (VEP) AIRCRAFT SYSTEMS ADI displacement Aileron Aircraft gross weight Angle of attack Approach glideslope display error Approach localizer display error Automatic Direction Finder (ADF) Autopilot vertical tracking error Ball angle CDI error Collective Control stick Cyclic DME Elevator Engine Pressure Ratio (EPR) Engine RPM Flaps Flight director error Fuel flow Heading L Source: Mixon and Moroney [1981] 59 TABLE II (Continued) AIRCRAFT SYSTEMS (Cont'd) Landing gear Pedal (helicopter) Radar altimeter error Rotor RPM Rudder Speed brake Tail rotor position Throttle Thrust attenuator Thumbwheel inputs Trim MAN-MACHINE SYSTEM OUTPUT Acceleration ACM plane of action (X^Y^Z) Aircraft/boom oscillations Airspeed Altitude Altitude (pitchout) Altitude (radar) Approach angle error Approach center line error Approach glideslope error Approach range Checklist errors Circular bomb error Closing speed Course error Crosstrack error Deviations from ideal flight path Dip to target error (ASW) Distance traveled Dive angle at bomb release Drift Emergency detections Energy Management Index (EMI) Ground speed Ground track Landing aim point Landing attitude Ldg. dist. to ideal touchdown point 60 TABLE II (Continued) MAN-MACHINE SYSTEM OUTPUT (Cont'd) Ldg. dist. to runway threshold Ldg. height at runway threshold Landing result (flare, bolter, .. Lateral acceleration Lateral velocity Mach number Maneuvering rate Miss distance (ASW) Navigational accuracy (LAT/LONG) Pitch Pitch/roll coordination Pointing angle advantage Position estimation Positional error (formations) Power Prob. of finding turn point Prob. of target acquisition Prob. of target detection Procedural errors Range at target detection Range a-t target identification Range at target recognition Rate of information processing Ratio: carrier accidents Ratio: carrier bolters Ratio: carrier bolter rate Reaction to an event Roll Sideslip Takeoff position error Torque Tracking error Turn errors Turn rate Vertical acceleration Vertical velocity Yaw TIME Combined total seconds of error Defensive time 61 TABLE II (Continued) TIME (Cont'd) Lead time Offensive time Offensive time with advantage Opponent out of view time Ratio: offensive/defensive time Reaction time to an event Time Time estimation Time of event Time of task exec-ution Time on criterion Time on target Time to acquire target Time to criterion Time to detect target Time to envelope Time to first kill Time to identify target Time to recover from unusual attack Time to turn Time within criterion Time within envelope Time within flight path Time within gun range Time within missile range FREQUENCY No. No. No. No. No. No. No. No. No. No. No. No. No. No. 62 of of of of of of of of of of of of of of aircraft ground impacts collisions (formations) control losses control reversals correct decisions correct responses correct target acquisitions correct target classifications correct target detections correct target identifications correct trials course corrections crossovers errors per trial TABLE II (Continued) FREQUENCY (Cont'd) No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. No. of of of of of of of of of of of of of of of of of of of of of of of errors to criterion false target detections false target identifications gun hits/kills incorrect control inputs incorrect decisions lost target contacts (ASW) missile hits/kills overshoots refueling disconnects qualifying (criterion) bombs scorable bombs successful unusual attack rec. taps (secondary task) target detections (no fires) target hits target kills target misses times inside criterion times off target times outside criterion turn points found turns to assigned heading COMBINED OVERALL MEASURES Good Stick Index (GSI) Landing Performance Score (LPS) Objective Mission Success Score (OMSS) Trials to criterion 63 . . useful forms of information for the comparison or evaluation Since all measures are, in effect, random variables process. in a statistical sense, it is important to determine the family, or distribution population, that characterizes the particular measure being examined before performing any statistical operations. Two such example distributions would be the Exponential (time measures) and the Normal or Gaussian (accuracy measures) Each distribution has preferred statistical summary estimators that use the generated measures (data) in order to best estimate the actual or true operator performance dimension at hand. b. Error Measures Accuracy, or error measures, are of special interest in aircrew performance measurement due to the obvious relationship between operator error and aircraft accidents. These unique measures are usually expressed as a measurable deviation of a variable from an established or arbitrary reference point, and have been of great interest to the aviation accident investigation community [Hitchcock, 1966; Ricketson, 1974] Chapanis [1951] dichotomized errors as basically constant or variable; constant errors indicated the difference between a statistical estimator of a quantity and the true, or expected value, and variable errors are described by a statistical estimator of measure spread or dispersion. That study concluded that variable errors indicated the actual instability of the man-machine relationship and thus were more of a contributing factor to accidents. 64 : c. Aircrew- Aircraft Measurement A descriptive structure for flight crew performance measurement relating system performance and human behavior to segments of maneuvers which constitute a flight mission was recently developed by Vreuls and Cotton [1980] earlier work by Benenatti, et al. Vreuls, et al. [1973] , [1962], Knoop derived from [1968], and The structure states that a measure . must have meaning only when specified fully by the following five determinants (1) Measure segment. (2) System state variable (3) Sampling rates for continuous variables. (4) Desired values. (5) Transformations. (s) and their scaling. Measure segments are any portions of flight for which the desired behavior of the system follows a lawful relationship from an unambiguously defined beginning to end. Segments are related closely to specific behavioral objectives (from the ISD training approach) , are possibly but not required to be the same as a maneuver segment, and defined explicitly by start/stop logic. discussed (measures) System state variables, as previously , are scaled so that their entire dynamic range is represented without information loss. Scaling will be discussed along with desired values and transformations later in this section. Sampling rates are the temporal frequency at which a measure is recorded or observed by the 65 . measurement system. One sampling rate guideline is to record data faster than the natural frequency response for the specific axis in which the measurement is being made, although others are proposed [Vreuls and Cotton, 1980] 3 . Levels of Measurement In examining the nature of performance measurement, four levels can be distinguished. The level of scale determines the amount of information resulting from the measurement and the mathematical and statistical operations that are per- missible within that level. [1951], Siegel [1956], Table III is adapted from Lorge and Smode, et al . [1962], and lists each level and the applicable characteristics associated with A brief description of each level as analyzed by Smode, each. et al. [1962] a. is discussed below: Nominal measurement scales have units being measured placed into classes or categories. Units placed in the same class are considered equivalent along some dimension or in some respect. b. ranJc order. Ordinal measurement scales have units assigned a Nominal categories are now ranked with respect to each other in terms of an "amount possessed" of the quantity being measured, with judgements assessing the amount possessed by the units involved. RanJcings can be composed of unidimensional or multidimensional variables. In the latter case, a composite judgement or ordering is performed which essentially places a multidimensional variation on a unidimensional linear scale. 66 0) G .. M rc 1 c -H •H 0) 4-4 j-i r^ rH LH (U (U -H (U T3 w ^ CO CQ <3 CO E-t CO CO U H H M S ft K < W E^ Oi CO S -P W >, (U .* 0) 0) U -H cn CJ 0) M 4J G CO -H •H U C a 3 O" 0) 0) G (U -H (U CO -M G HU QJ i-l 4J M iH CO (U CO G C 4J (T3 oj T3 a; O 0) <U Oi -H g -H O C --H rd -P (d cn (U .* iq Cfl 3 CO -H +J 4-1 (T3 -H C O U" (D -P G U-i Oi-P QJ T3 G 0) 0) G 13 fa M fe4 -H SU U O S C 0) 3 CO CO CT G 0) ra QJ td T! -P S-i Q; CO fc4 M S S CO Q) CO ^H CO -H 4-> -H -P Q> -P 03 e 03 Q4-P 03 -P M CO •4-1 (U 03 4-1 (H ^-1 G U C -P 0) fC CO 04 Oi 03 i^ 04 M-i •H -H •H c OJ M C U U G CPH-( cn 4J CO -P CO <N 03 2 O4 CTl a >i cn G G w ^ D < Eh UU H D Oi < &H S CO K eu E^ D <^ o S OJ Pc5 E-« CJ a CO M CO nj en G 0) CU 05 S^ 3 OJ 3 M en c •H ^X S-i cn -U -' to U-4 4J D e u • II U H G - -^^ 3 cn r-l -H (T3 CU -P -P M X M O >i OJ jj X •H X (d S-l 03 03 1! rH •H - X O e CO II - X -73 G g G 0) •H 03 O '^ QJ + 0) G G ^03 3. S H ^ -H 0) 04 u 0) •H II CO Q) (T3 G -H X X o -^ --^'-'•H G M-l U-l G +J a, u O e G - X 3 -H n3 CO in CO 4J fO >i c •H > -p <; CQ >, u CO U-l -p c c G U (TJ T) vH >i-H -P -P Q) 0) -P CO T3 -H MH x: C H M Q) •H C CO a -H X5 fd 5-1 G 0) CO •H +J CD M 0) -P ^3 03 tn S fO M 3 ^3 -H S 13 T3- S-l 0) -H TS > OJ V^ 0) cr 4J (U h-3 < 2 H 5 3 0) -P u 04 a Q >i 3 to -P O -P G G cr 04 -P Tj 4-1 0) Q) 03 -H S-l M >i-H (U > G iT> -P -H OJ (U ^ -H CO G G •H 03 -P Jh -TO <D > U 04 S-i O 67 •H QJ CO QJ M N QJ >i-H (T3 4-1 QJ QJ -P H e 03 M 3 QJ Cr -P QJ CO QJ V s-i M 3 r-t ^3 QJ -G -H T3 -P CO > M -H X! U 5 03 QJ cn O4 O QJ H Z H G M C -H Q cn OJ T3 G QJ QJ <D QJ -P •H 03 G 4J 73 03 < > Eh 4-1 T3 0) - -P 03 -H T3 i-i 4-1 '73 4-1 C -H 4-4 •H rH -H e 03 13 S-i •H 4J T3 5-1 > > CO -P -HO -H ^ QJ •H >i +J CO >i-H G -P G -H •H rH Q hJ <: 4J ij-l (1) W U 4J kj <: u H CO « 2 MO 04 M S Eh W < 2 uw M Ch CO O rH >i >1 iH Eh 2 O 3 O CO . c. Interval measurement scales have units being measured in equidistant terms. In addition to an indication of not only rank order or direction, there is also an indication of the amount or size of difference between units on the Since the zero point is usually arbitrary, it does scale. not represent complete absence of the property being measured. d. Ratio measurement scale is an extension of an interval scale with a natural, absolute zero point, and represents the highest measurement level. It is with this level that the most powerful statistical tests of significance are available when evaluating performance measures. The determination of a parameter or measure of performance should include the units of scaling, i.e., knots (air speed) , and to 64000 feet to 640 (pressure altitude) Without a clear definition of the scaling units, the improper use of statistical operations or tests may occur, causing the measurement of performance to provide irrelevant or erroneous results. 4 . Measure Transformations Measures are observed, recorded, and usually subjected to transformation, which is any mathematical, logical, or statistical treatment designed to convert raw measures into usable information [Vreuls and Cotton, 1980] . When measures are discrete, the transformation may be the actual value, absolute value, or a tolerance band. When measures are continuous, transformations may include means, variances. 68 frequency content, departures from norms, or several others as listed in Table IV, The relationship between the distribution of a measure or estimate and transformations is well known to statisticians For a given popula- but sometimes not very clear to others. tion distribution, there exists unbiased and consistent estimators (transformations) that will best describe the true value or quantity that is being estimated. transformations are not applicable for and in a sense are useless. to Mood, et al. [1974] a Indeed, some given population, The interested reader is referred for a detailed analysis of applicable transformations for a known population distribution. 5. Accuracy of Measurement Measurement produces measures which are sample estimators of the true value or actual am.ount of the quantity possessed. Accuracy of measurement refers to how close an estimator is to the true value. All measurement systems are subject to accuracy problems as discussed below. a. Measuring aircrew behavior is confounded by the systematic influence of the total operating system, since the measures obtained are frequently determined to some extent by the performance of other components in the system [Glaser and Klaus, 1966J. b. Any statistic, or known function of observable random variables that is itself a random variable, whose values are used to estimate a true quantity, is an estimator of the 69 TABLE IV: COMMON iMEASURE TRANSFORjyiATIONS TIME HISTORY MEASURES Time on target Time out of tolerance Percent time in tolerance Maximum value out of tolerance Response time, rise time, overshoot Frequency domain approximations Count of tolerance band crossing Zero or average value crossings Derivative sign reversals Damping ratio AMPLITUDE-DISTRIBUTION MEASURES Mean, median, mode Standard deviation, variance, range Minimum/maximum value Root-mean- squared error Absolute average error FREQUENCY DOMAIN MEASURES Autocorrelation function Power spectral density function Bandwidth Peak power Low/high frequency power Bode plots, fourier coefficients Amplitude ratio Phase shift Transfer function model parameters Quasi-linear describing function Cross-over model BINARY, YES/NO MEASURES Switch activation sequences Segmentation sequences Procedural/decisional sequences Source: Vreuls and Cotton [1980] 70 the true quantity and may be subject to biased or inconsistent properties. For all distributions of measures that have been identified during measurement, there exists at least one unbiased and consistent estimator that will be closer to the true value of the quantity being measured than any other estimator. Using the incorrect estimator for a known population will, in effect, result in avoidable measurement inac- curacies [Krendel and Bloom, 1963; Mood, et al . , 1974]. There is no simple way to assure measurement accuracy, but several techniques to improve the accuracy of measurement may be incorporated and follow. c. Scope of Measurements Accuracy will increase as a result of increasing the inclusiveness or completeness of the measures to include all relevant behavior and system components. Skilled performance in an aircraft normally involves complex behaviors that require a wide scope of measurement to accurately estimate the existing true performance level d. [Smode, 1962]. et al., Number of Measurements For most situations, the greater the number of observations involved in deriving an estimator, the closer the estimator wil be to the true value [Mood, et al . , 1974]. Increasing the number of observations when there is a large variability in the individual measurements will reduce the variability and minimize the effects of random or chance variations which may occur from measurement to measurement 71 . [Smode, 1962]. et al., . Large sample sizes also are desirable when establishing standards or references (criteria) when evaluating performance [Krendel and Bloom, 1963] e. Controlled Conditions of Measurement By insuring the desired measurement conditions are controlled, accurate measurement may be improved. This may be done by defining those factors which are to be present and varied systematically, maintaining uniformity of conditions during measurement in order to reduce bias and unwanted variability, and insuring the intended measurements are ta>:en correctly [Smode, et al., 1962]. 6 . Reliability of Measurement Reliability refers to the accuracy of measurement or the consistency or stability of the recorded and statistical data upon repetition [Glaser and Klaus, 1966; Knoop and Welde, 1973; Grodsky, 1967; ThorndiJce, 1951]. When the dispersion, or spread, of measures obtained from one individual on a particular task is large, the measures lack reliability, and any statistics that are formed from the measures that are used in evaluation will be incapable of differentiating consistently among individuals who are at different skill levels. If the measures are precise, resulting in a statistic that is stable, an individual would receive exactly the same evaluation score each time his performance was measured [Glaser and Klaus, 1966; Thorndike, 1951] 72 Computation of Reliability a. Thorndike [1951] cautioned: "There is no single, universal, and absolute reliability coefficient for a test. Determination of reliability is as much a logical as a statistical problem." Several methods of approximating reliability have since been proposed or utilized in aircrew measurement. Smode, et al. [1962] classified reliability expressions as either absolute or relative and suggested using the standard error of measurement (absolute measure of precision) , coefficient of internal consistency (uses single set of observations) ment over time) , , coefficient of stability (measure agree- and the coefficient of equivalence (agreement between two like measures) as statistical computations of reliability. Glaser and Klaus [1966] dichotomized reliability assessment into two methods: test-retest and alternate form, and recommended computing reliability by using the statistical correlation coefficient. Winsorization and trimming Some recent statistical techniques - may provide a better reliability approximation than was previously possible. Winsorization and trimming involve removing the effects of a large variability in a measure sample, with virtually no loss in information [Dixon and Massey, 1969] . These techniques would appear to be quite useful for reliability calculations, although their actual use in aircrew performance measurement has not yet been demonstrated. 73 b. Sources of Unreliability The sources of measurement accuracy problems, as previously discussed, are also sources of unreliability. Other sources inherent in the measurement of human behavior, which hinder reliable measurement of aircrew performance, are from Glaser and Klaus [1966] and Thorndike [1951], and include the following: (1) The environment in which performance is being measured influences measurement variability. Differ- ences in weather, amount of turbulence, wind direction and velocity, unexpected noise, equipment malfunction, and extreme temperatures are factors contributing to unreliability. (2) Equipment used for measurement or personnel participating in performance assessment are sources of unreliability. (3) The complexity of the behavior being evaluated influences unreliability. Since the behavior being measured and evaluated involves many dimensions of performance, and any individual's skill level may fluctuate considerably from one dimension to the next, each component dimension is susceptible to all previously discussed sources of unreliability that have been described above. (4) The performance of the individual being assessed fluctuates as a function of temporary variations in the state of the organism. Some factors frequently involved that decrease measurement reliability are: 74 individual motivation, emotional state, fatigue, stress, test-taking ability, and circadian rhythm. c. Improving Reliability of Measurement In any training situation, some degree of reliability in the measure of the ability being trained is necessary if any evidence of improvement is required. By reducing chance factors or variability in measurement, reliability can be improved. The techniques for improving the accuracy of measurement, as previously discussed, also contribute towards improving reliability. Knoop and Welde [197 3] suggested other factors to improve reliability that are listed below: (1) Calibration of performance measurement equipment should be conducted on a continuing basis. (2) Software processes involved in data collection, reduction, conversion, analysis, and plotting should be validated and monitored to avoid data loss. Accurate records of flight conditions and mission requirements should be maintained to facilitate measurement interpretation. (3) 7 . Validity of Measurement Validity is the degree to which the measurement or evaluation process correctly measures the variable or property intended to be measured [Knoop and Welde, 197 3] . In regards to the evaluation process, validity has two aspects; (1) relevance or closeness of agreement between what the test measures and the function that it is used to measure, and (2) reliability or the accuracy and consistency with which the test measures whatever it does measure in the group with 75 which it is used [Cureton, 1951] . In the training environment, a performance test is a stimulus situation constructed to evoke the particular kinds of operator behavior to be measured or assessed. The validity of a performance test is established by demonstrating that the test results reflect differences in skill levels of the performance being assessed [Glaser and Klaus, 1966]. a. Types of Validity Four types of validity which have applicability to performance measurement in general have been described by Smode, et al. [1962], iMcCoy [1963], and Chiles [1977] as follows: (1) Predictive validity refers to the correlational agreement between obtained measures and future status on some task or dimension external to the measurement and requires statistical operations for evaluation. (2) Concurrent validity refers to the correlational agreement between obtained measures and the present status of the units being measured on some task or dimension external to the measurement and also requires statistical computation. (3) Content validity is based on expert opinion and is evaluated by qualified people determining if the measures to be taken truly sample the types of performance or subject matter about which conclusions will be drawn. (4) Construct validity is a logical process where the emphasis is on trait, quality, or ability presumed to underlie the measures being taken. While the measures themselves do not reflect directly the performance to be evaluated, they are accepted to be valid on the basis of logical considerations and related empirical evidence. 76 . b. Validity of Measurement in the Simulator Without empirical or judgmental evidence, the use of full-scale state-of-the-art flight simulation provides maximum face validity, where the performance evaluation situation in the simulator appears to duplicate the actual task of flying or navigating an aircraft 1977] Bowen, et al. . [1966] , [Alluisi, 1967; Chiles, in an experiment using twenty A-4 pilots to study and assess pilot proficiency in an Oper- ational Flight Trainer (OFT; device 2F62) , found that: For measures taJcen in the OFT to be valid, the task set to the pilot should be multiple in nature and have a considerable difficulty level equivalent to the more difficult levels found in actual flight. In this manner, the pilot is more likely to display his skills in the same pattern of priority (i.e., time-sharing, attention-shifting, standard-setting, etc.) as he does in actual flight. This conclusion is also supported by Kelley and Wargo [1968] and in terms of the relevance component of validity as previously discussed by Cureton [1951] c. Improving Validity of Measurement A high degree of validity is essential to the effectiveness of any measurement system. Improving validity can be achieved by increasing either or both relevance and reliability. Relevance may be increased by reproducing the simulation situation to closely resemble that of the actual aircraft itself, or by simulating the task or mission being performed more closely to the actual task or mission environment. Reliability improvements were discussed in the previous section. 77 . d. . Relationship of Validity and Reliability Validity has been described as having aspects of relevance and reliability. Reliability is the consistency or self-correlation of a measurement while validity is its correlation with some independent standard or reference from that which is measured [Kelley and Wargo, 1968] . A given performance measurement can be highly reliable yet not have validity [Smode, et al. , 1962; Kelley and Wargo, 1968]. ever, an unreliable test cannot have practical validity, Howi.e., a measurement that does not even correlate well with itself will not correlate well with other measurements [Kelley and Wargo, 1968; Steyn, 1969]. In performance measurement, high validity must be combined with high reliability; this combination means that a highly skilled operator consistently must achieve a higher performance evaluation result than a less skilled operator [Smode, et al. , 1962; Kelly, et al . , 1979]. If the performance evaluation occurs during the actual task instead of a simulated task, the question of validity reduces simply to the question of reliability, as perfect relevance will have been achieved [Cureton, 1951] Because of the unique relationship of validity and reliability, it is generally easier and more efficient to improve the reliability of a measure than to raise its validity, On the other hand, ing, if the validity of a measure appears promis- improving reliability is preferred to using a reliable measure which has lower validity [Glaser and Klaus, 1966] 78 . , Selection of Initial Measures 8. After the identification and selection of a desired mission, scenario, flight segment, and human operator with the behavior of interest, performance associated with these requirements can be specified and defined. The initial selection of appropriate measures has been a major problem, as evidenced by the lack of concordance in recent aircrew performance measurement research [Mixon and Moroney, 1981] Unless aircrew performance measurement empirical results are collected and standardized, an analytical approach must be taken when initially selecting which measures best describe the performance being examined. Some optimum balance exists between the "measure everything that moves" philosophy and the measurement of a few measures with apparent face validity. The initial selection of any measures, however, should be guided by both the purpose of the measurement and the man- machine system as well as the facility for collecting and processing the data. The following criteria for initial measure selection are provided by Meister and Rabideau [1965] Parsons [1972] [1977] , Greening [1975] , and Vreuls and Wooldridge : (1) Relevance - the measures should be pertinent to the purpose of measurement. (2) Quantifiable - measures should be in the form of numerals on a ratio scale for statistical analysis, except where only subjective collection is feasible. (3) Accessibility - measures should not only be observable but easily collectable, preferably by electronic means. 79 . Operational utility - a measure that has relevance and accessibility in both the aircraft and simulator environments (4) Efficiency - a measure with utility at minimum cost of collection and transformation into usable infor- (5) mation. Content validity - a positive correspondence between the performance measure and what is known about the underlying behavior. (6) (7) Reliability - collection of more measures or samples of a measure than planned would offset the likelihood of electronic data collection equipment failures. (8) Dependence - the availability of human or automatic data observers and collectors limit what measures are feasible to collect. (9) Objectivity - where possible, automatic data observation and recording is preferred to human observers and recorders. (10) Usable - measures collected should be usable for either evaluation information or supportive to evaluation results. (11) Acceptable - measures that are used by instructors in the operational environment must be consistent with their expert judgement. (12) Collection criteria - data must be accurate and precise to what is known about the underlying behavior. Once the performance measures of interest are identified and selected, they should be defined, as a minimum, in terms of the descriptive structure as previously outlined by Vreuls and Cotton [1980] . Some consolation in not selecting the appropriate measures for performance is offered by Knoop [1968] , "Determining exact criteria [standards] and performance measures for virtually any flight task or mission is an accomplishment yet to be made." 80 . CRITERIA CONSIDERATIONS B. 1. Definition and Purpose of Criteria Criteria are standards, rules, or tests by which measures of system behavior are evaluated in terms of success or failure, or to some degree of success or failure. The purpose of human performance criteria is to provide standards or baselines for evaluating the success or failure, goodness or badness, or usefulness of human behavior [Knoop, 1968; Vreuls and Cotton, 1980; Cureton, 1951; Steyn, 1969; Davis and Behan, 1966; Shipley, 1976; Buckhout and Cotterman, 1963]. The criterion is a measuring device which is not generally or readily available, but a device which should be constructed from the beginning for each particular situation [Steyn, 19 69; Christensen and Mills, 1967]. Criteria should not only define the unique manner in which the operator should perform a task, but should define the performance objectives of the entire man-machine system [Demaree and Matheny, 1965; Connelly, et al., 1974]. 2. Types of Criteria The classification of criteria can be accomplished from a measurement standpoint; begin with the smallest known entity and end with the "ultimate" quantity that may exist. Several types identified are listed below: (1) Parametric referent or standard of performance which is sought to be met by the operator or system. Example: maintain 500 feet of altitude [Demaree and Matheny, 19 65; Shipley, 197 6] 81 . . Parametric limit about the parametric standard within which the operator or system is required, Example: maintain plus or or seeks, to remain. minus 100 feet while at 500 feet altitude [Demaree and Matheny, 1965; Shipley, 1976]. (2) System component criteria which distinguishes the relationship between system components and system Example: "least effort" measured from output. the pilot in relation to maintaining altitude [Krendel and Bloom, 19 63; Uhlaner and Drucker, (3) 1964] Test criterion used to evaluate overall human ability, usually expressed as a single overall subjective judgement of inExample: measure. structor for a student as to "pass" or "fail" (4) [Marks, 1961] Ultimate criteria are multidimensional in nature and represent the complete desired end result of a system. This type of criterion is impossible to quantify due to the multidimensional nature of the system's purpose, and hence, is a theoretical entity Example: Any aircraft's that must be approximated. mission [Cureton, 1951; Smode, et al., 1962; Uhlaner and Drucker, 1964; Steyn, 1969; Shannon, 1972]. (5) It may be noted that all five types of criteria can be quantified or approximated in some manner, with decreasing accuracy as the ultimate criteria level is reached. Obtain- ing direct measures of the ultimate criteria for a complex system is seldom feasible. This is particularly true in military systems where such criteria would be expressed in terms of combat effectiveness or effectiveness in preventing a potential aggressor from starting a conflict [Smode, et al., 1962] . Therefore, it becomes apparent that we must select intermediate criteria (types one through four above) evaluating skilled operator behavior. 82 in . . 3. . Characteristics of Good Criteria Using actual criteria as approximations of the ultimate criteria can be accomplished by several methods that will be discussed in a later section. Although there is no certain method that will lead to the specification of good criteria, there are some considerations that can be taken into account which are discussed below: (1) A good criterion is both reliable and relevant [Smode, et al 19 62; Krendel and Bloom, 1963; Cureton, 1951; Grodsky, 1967; Steyn, 1969]. . , (2) Criteria must be comprehensive in that the utility of the individual being evaluated is unambiguously reflected [Steyn, 1969] (3) Criteria should possess selectivity and have ready applicability [Krendel and Bloom, 1963] 4 Other Criteria Characteristics Steyn [1969], in a review of criterion studies, noted that performance measures under simulated conditions can at best serve as substitute criteria. This observation reflects the engineering and mathematical model of reality represented by the simulator that can, at best, approximate an aircraft and its systems. Shipley and Gerlach [1974] measured pilot performance in a flight simulator CT4-G) and found that differences in pilot performance outcomes varied as a function of the difference in criterion limits that were established, with the relationship between criterion limits and tracking performance found to be a nonlinear one. The multidimension- ality of a criterion was also noted by Steyn [1969] 83 , Cureton [1951] and Connelly, et al. , [1974] These latter two studies . observed that multiple criteria must exist for a single task since different operator action patterns having no single feature in common could conceivably obtain the same desired system output. Establishment of Criteria 5. Criteria may either be derived from regulatory requirements, system operating limits, knowledge of common practice, or empirical studies [Vreuls and Cotton, 198 0] . When criteria are established analytically, some caution must be taken. Campbell, et al. [1976], in designing the A-6E TRAJyi training program using ISD methodology, observed that: A standard or criterion of performance for that terminal behavior must also be established at a level comparable to the earlier established operational standards. These latter criteria, however, while reflecting an acceptable level of behavior, imply a repeatability, that whenever they are performed, that some acceptable level will be attained. . . . . . . Criteria derived from objective, empirical techniques are preferable to analytical methods [Steyn, 1969] . Regression analysis, discriminant function analysis, multivariable regression, and norm or group referencing are but a few of the empirical approaches to establishing criteria [Connelly, et al., 1974; Danneskiold, 1955; Dawes, 1979]. No matter which method is used, criteria must be defined and are necessary for the evaluation process. 84 . 6. Sources of Criterion Error As previously mentioned, a good criterion is one that Reliability, as previously is both reliable and relevant. defined, implies that what constitutes successful performance will be resistant to the effects of chance factors. Relevancy refers to the validity of the actual or approximated criterion to the ultimate criterion. By definition, the ultimate cri- Sources of criterion error terion is completely relevant. can then be identified in terms of reliability and relevance. Smode, et al. [1962], lists some significant sources of criterion error below: (1) Low reliability, as previously mentioned. (2) Irrelevancy or the lack of relation of the actual criterion with respect to the ultimate or "ideal" criterion. (3) Contamination of the criterion by the presence of factors or ingredients in the actual criterion which do not in fact comprise the ultimate criterion. (4) Distortion in the criterion caused by errors arising from assigning incorrect weights to the separate factors that comprise the actual criterion (combining criteria is discussed below) 7. Measures of Effectiveness Aircraft missions are all multidimensional in nature. This means that every mission can be divided into usually one overall goal or purpose (i.e., destroy the target, deliver the supplies, rescue the survivors, etc.), with several subgoals (safety, minimize susceptibility, timeliness, etc.). Since missions are multidimensional, the operator effort in 85 the form of mental and physical action (performance) becomes multidimensional. The multidimensional nature of skilled aircrew performance, in turn, requires that several criteria, all of which are relevant for a particular activity, be defined Each of these criteria must and used [Smode, et al., 1962]. be operationally defined, theoretically quantifiable, and collectively give a reasonable portrayal of operator and system performance. Typically, one may wish to bring these component criteria together in an overall effectiveness measure a single - "measure of effectiveness" for the system being investigated. The process of combining criteria into a single composite measure of effectiveness is one of the most difficult tasks to undertake in any field of research, and has been the focus of continuous investigation in the science of Operations Re- search for decades [Hitch, 1953; Morris, 1963; Steyn, 1969; Lindsay, 1979; Dawes, 1979]. A Measure of Effectiveness (MOE) is a quantifiable measure used to compare the effectiveness of the alternatives in achieving the objective, and must measure to what degree the actual objective or mission is achieved mittee, Naval Science Department, 1968] . [Operations Com- For the particular situation of measurement and evaluation of an aircrew member's performance in an aircraft, criteria can be thought of as "alternatives," each of which has an individual effectiveness for each performance component of an aircrew member, who is accomplishing an overall objective 86 - the mission. MOE ' s have been applied in economics, management, and military problems - just about any area where a decision based on information from system performance has to be made. Combining criteria into an MOE can be accomplished either analytically or statistically. Most methods concentrate on assigning weights that are either determined on the basis of "expert" opinion or statistical treatment [Smode, et al. , 1962; Steyn, 1969]. In a review consisting of numerous criterion weighting studies, Steyn [1969] concluded that "it would appear that the most acceptable approach [to weighting criterion variables] would be to identify the job dimensions clearly and unambiguously and to use these pure dimensions as criteria to be predicted independently." There is no established procedure for combining cri- Lindsay [1979] and Smode, teria into a single overall MOE. et al. [1962] offer some suggestions to approach the problem: (1) Look at the big picture. Examine what is to be done with the results of the aggregation. Determine how the numbers will be used, and in what decisions. (Usually one finds that this has not been thought out in advance.) It may be that all that is really needed is the identification of satisfactory systems. (2) If possible, aggregate subjectively. Give the subcriteria values to the decision-makers or their advisers and let them subjectively determine how effective the systems are. (3) Recognize that one is defining, not approximating. The development of a formal scoring system should be done with the awareness that a definition of system effectiveness is being made. The procedure developed should include reference points, diminishing marginal returns, and avoid substitutability except where appropriate. 87 (4) Sub-criteria should be weighted in accordance with their relevance to the ultimate criterion. (5) Sub-criteria which repeat or overlap factors in other sub-criteria should receive a low weight. (6) Other things being equal, the more reliable subcriteria should be given greater weight. The unique situation of an aircrew flying an aircraft for a specific mission and the necessary determination of subcriteria for evaluating the overall accomplishment of that mission requires further research of an analytical and empirical nature. The relationship among altitude, airspeed, operator activity, and the hundreds of other system variables that comprise the total system must be compared to mission success in quantifiable terms. Since "mission success" is multidimensional and may not be totally measurable and quantifiable, some analytical approaches toward combining criteria into an overall MOE appear to be feasible, notwithstanding the possible use of empirical methods to describe some aspects of the process. 8 . Selection of Criteria Criteria have been defined, their purpose established, some types identified, and some characteristics discussed. Since a large number of criteria may exist to evaluate a particular system, some selection in the way of "trade-offs" may be necessary [Davis and Behan, 1976] . More than one criterion may describe the same dimension of performance whereas another carefully selected criterion may accurately describe more than 88 , one performance dimension. Reducing the number of criteria that are relevant, reliable, and practical into a feasible and usable set that can accurately and consistently evaluate the performance of an aircrew and the accomplishment of their mission is extremely difficult at present and will probably remain as an unsolved future problem in the Human Factors field unless specific research is undertaken to attack it. In the meantime, teria some general guidelines for selecting cri- as discussed by Hitch [1953] and Smode, et al . [1962] are listed below: (1) Selection of any criteria should always be consistent with the highest level or type of criterion associated with the system mission. (2) Specify the activity in which it is desired to determine successful and skillful performance. (3) Consider the activity in terms of the purpose or goals, the types of behaviors and skills that seem to be involved, the relative importance of the various skills involved, and the standards of performance which are expected. (4) Identify the elements that contribute to succesful performance and weight these elements in terms of their relative importance. (5) Develop a combined measure of successful performance composed of sub-criteria that measure each element of success and are weighted in accordance with the relative importance of each. The definition, computation, combining and selection of criteria is perhaps the most difficult problem encountered by researchers investigating complex man-machine systems [Osborn, 1973; Krendel and Bloom, 1963]. The importance of criteria is once more emphasized, as stated by McCoy [1963] 89 "You must define the criterion precisely and accurately before interpreting any measures used in investigating a system." Christensen and Mills [1967], in quoting earlier work done by H.R. Leuba, stated: There are many ludicrous errors in quantification as it is practiced today, but none quite as foolish as It is awkward trying to quantify without a criterion. thing when a criterion wrong quantify the enough to most unprofessional exists, but it is a sham of the sort to quantify in the absence of a criterion. C. PERFORMANCE MEASUREMENT CONSIDERATIONS 1. Subjective Versus Objective Measures As previously discussed in Chapter I, subjective and objective measures are not dichotomous, but rather represent a continuum of performance measurement. At one extreme of the continuum, a human observer mentally records actual performance during a specified mission, and uses his perceptions to form a judgement or degree of success rating as to how skillful the operator was in achieving the system objectives. This extreme is the subjective method of measurement and evaluation. At the other extreme of the performance measurement continuum, automatic digital computers sense, record, transform, analyze, and compare actual man and system performance to statistically established criteria and form a complete set of performance data or information to be used by the decision-maker (instructor or training officer) the skills and abilities of an operator. is the objective method of in evaluating This other extreme measurement and evaluation. 90 . Each method of performance measurement has advantages and disadvantages, as were discussed previously, and will not be repeated here. Objective measures and measurement have become more feasible and less costly to implement for aircrew performance than in previous decades, and the method has established itself as a very powerful and useful model for describing actual human behavior [Mixon and Moroney, 1981] 2. Combining Measures What has been discussed previously with respect to combining criteria also applies to combining performance measures into a single overall index of skill level or proficiency. As Smode, et al. [1962] indicated: (1) Measures should be weighted in accordance with their relevance to the criterion. (2) Measures which repeat or overlap factors included in another measure should receive a low weight. (3) Other things being equal, the more reliable measures should be given greater weight. In combining performance measures, it is often possible to determine quantitatively the interrelationships among the performance measures and the relationship between each measure and the actual or immediate criterion [Smode, et al., 1962]. A single overall measure or score composed of numerous performance measures along different dimensions of system behavior is highly desirable in any performance measurement and evaluation system, due to the use of the total score in determining overall performance when compared to a criterion. 91 A single . score or estimate of total performance, when compared to a criterion or MOE, provides the necessary information for evaluation that determines goodness or badness, success or failure, and usefulness of human performance [Buckhout and Cotterman, 1963] Combining performance measures, like criteria, can be performed by either analytical or empirical methods which commonly assign weights to each measure which are then mathematically combined into a single proficiency score. Analytical methods employ the judgement of experts for situations usually involving complex man-machine systems where definitive and quantifiable measures of output are not available [Glaser and 1966; Marks, Klaus, 1961]. Empirical methods of combining aircraft system performance measures with relative weightings into a single score were reviewed by Vreuls and Obermayer [1971] . Among some of the methods from that study for developing multidimensional algorithms were: factor analysis, multiple discriminate analysis, linear-weighted algorithm, nonlinear (threshold) model, energy maneuverability model, time demand, recursion models, and empirical curve fit. The interested reader is referred to that study for more detail on each model and the circumstances in which it was employed. In summary, separate performance measures of different behavioral dimensions are combined by various analytical and empirical methods into a single overall or composite score with the idea that when the score is high, as compared to a 92 predetermined criterion or MOE, it indicates "good" or "successful" performance, and when low, indicates "poor" or "unsuccessful" performance [Cureton, 1951] . Difficulties in how to combine the measures into a single overall score leads to preservation of the behavior dimensions and a "vector" of measures. 3. Overall Versus Diagnostic Measures Overall measures of skilled performance, as previously discussed, along with total system output measures (e.g., bomb-drop accuracy, number of targets hit, fuel consumed) are beneficial in assessing total system performance but are seriously lacking in diagnostic information of potential value to the trainee [Buckhout and Cotterman, Wargo, 1968; Bergman and Siegel, 1972] . 1963; Kelley and Overall scores tell nothing about the operator's performance on various specific tasks which are involved in flying an aircraft on a mission, but are highly useful for performance evaluation. Diagnostic measures are the same measures that result from performance measurement before any combining operations take place. These measures identify certain aspects or elements of a task or performance in specific skill areas and provide useful information on strengths and weaknesses in individual skills [Smode, et al., 1962]. Since they are concerned with smaller and more precisely defined units of behavior, they are easier to measure by objective methods. 93 It thus appears that overall and diagnostic measures are contradictory but both essential. For the training environment/ where a student is learning skills necessary to perform a task, both measures are valuable for what information they provide, as discussed above. Using the two together in a complementary fashion was perhaps best stated by Smode, al. [1962] , et "A prime value of an overall measure is the support it provides in evaluation since diagnostic measures alone are difficult to interpret without some terminal output measure of performance." Kelley and Wargo [1968] recommended that performance be measured in each dimension, evaluated separately by comparison to specific and predefined criteria, and then combined into an overall total score, so the trainee can receive feedback relating to his relative performance on the various dimensions of his task, as well as on his overall performance. More recently, Vreuls and Wooldridge. [1977] described multivariate statistical modeling techniques that are powerful enough to provide measures for diagnosis, and yet also provide single measures that could be combined into an overall score. The qualities of overall and diagnostic measures have been described and their relationship has been discussed. Within the training environment, using one without the other appears to cause a decrease in the quality of information available to the individuals who most need and instructor. it: both student Therefore, for the design of a system to 94 measure student B/N performance during a radar navigation mission, it appears advantageous to employ the use of both overall and diagnostic measures in a mutually beneficial manner that will provide the maximum amount of accurate information for the purposes of training situations. 4. Individual Versus Crew Performance One of the assumptions of this thesis is that the variability of the total contribution of the pilot in the conduct and successful accomplishment of the radar navigation mission is small enough to essentially be ignored when measuring the performance of the B/N during the same mission. This assumption was based on the major role played by the B/N during radar navigation, the unique design of the A-6E CAINS navigation system, and the radar navigation mission itself. Although it is recognized that any successful accomplishment of a mission depends to some degree on the crew interactions and coordination, the actual measurement of the interaction and coordination was beyond the current scope of study, and will be left for future investigation and research. The unique problem of measuring crew coordinated performance has been the focus of much research, but the question of what "crew coordination" is remains unanswered Moroney, 1981]. Smode, et al . [1962] [Mixon and provided a detailed discussion on the problems and approaches taken in measuring aircrew coordination, and concluded that measured interaction and communication were good for differentiating "good" and 95 . "bad" crews. Good crews reduced individual interaction and communication to a minimum so that more time was available to devote effort to performing the individual tasks associated This conclusion does not with accomplishing the mission. acknowledge the inherent difficulties involved in objectively measuring individual interaction and communications. Until further research uncovers objective, valid, reliable, and practical methods of measuring crew coordination, this area is perhaps a measurement function best delegated to a human observer (instructor) 5. Measures and Training The importance of overall and diagnostic measures in the training environment has been previously discussed. Meas- ures for the evaluation of performance are related in some degree to the stages of training. Early in training, when skilled behavior is made up largely of familiarization with the task and basic knowledge of procedures, measurement may consist of more familiarization and procedure-related measures Late in training, when skilled performance has become more or less automatic, measurement becomes more difficult due to the highly cognitive and covert nature of the skilled behavior [Fitts, 1965; Glaser and Klaus, 1966] . In this case, measurement becomes more indirect than direct. Designing any measurement system within the training environment requires a detailed understanding of the training process and its relationship to performance measures, in 96 , addition to an explicit understanding of the basic nature of the skills involved in performing the task. The latter subject will be discussed in Chapter V. D. PERFORMANCE EVALUATION CONSIDERATIONS Although the purpose of this thesis is to design a system to improve current performance measurement techniques for the FRS B/N student/ some mention must be made of performance evaluation since performance measurement exists as information With- necessary to evaluate individual and system performance. out evaluation, little reason exists for the measurement, recording, and storage of performance measures. This section will briefly outline current evaluation methods and the characteristics of evaluation itself. 1. Definition and Purpose Being consistent with the previous discussions on performance measurement and criteria, performance evaluation is simply the process of identifying and defining performance criteria and then comparing the criteria to performance measures produced by performance measurement. All performance evaluation requires some comparison between a standard and an estimate of what the standard truly represents [Angell, et al 1964; Demaree and Matheny, 1965] . . The purpose of performance evaluation in the training environment is usually multidimensional in nature but all evaluation occurs for the purpose of accurate decision-making by the instructor regarding student 97 : performance and by the training officer for effective training control. On the instructor level of evaluation, faulty dec is ion- making due to any performance evaluation involves two possible errors: TABLE V: Type I and Type II, as found in Table V. PERFORMANCE EVALUATION DECISION MODEL FOR THE INSTRUCTOR REALITY UNSKILLED DECISION: Student has not acquired skill to perform task Correct decision Student has acquired skill to perform task Type II The tangible effects of a Type I SKILLED Type (3) I (a) Correct decision error are possible increased costs due to overtraining, an inefficient training flow of students, and a demotivated student. On the other hand, a Type II error may result in increased costs due to an aircraft accident and the loss of human life. This example illustrates the important role that performance measurement and subsequent evaluation plays in providing accurate information necessary for correct decision-making by the instructor. 98 . Types of Performance Evaluation 2. Based on the purpose of the evaluation, evaluation may be divided into two general types; ment. According to Marks [1961] , aptitude and achieve- if the purpose is to predict the capacity of a trainee to absorb training and perform a task, the evaluation is called an aptitude test. If the purpose is to tell how well the trainee has absorbed training or can perform the task, the evaluation is called an achievement When considering achievement measures, it is possible measure. to distinguish three basic kinds: (1) Proficiency tests require the individual to answer questions about his job or about some content knowledge area related to his job. (2) Performance tests involve controlled observation of an individual actually performing his job. (3) Rating methods use the opinion of someone who has actually seen the man's performance on the job. For details on the characteristics, advantages, and disadvantages of each kind of achievement measure, the interested reader is referred to Marks [1961] A model is anything that represents reality. Two performance evaluation models that are utilized in achievement measures are norm-referenced testing and criterion- referenced testing. a. Norm-Referenced Testing Norm-referenced testing involves the use of normreferenced measures in evaluating performance. Norm-referenced measures compare the performance of an individual with the 99 . . performance of other individuals having similar backgrounds and experience [Glaser and Klaus, 1966; Knoop and Welde, 1973; Danneskiold, 1955]. The stability of a norm-referenced measure is highly dependent upon sample size. Too small a sample can yield measures of central tendency and variability that poorly approximate actual population values 1966; Danneskiold, b. [Glaser and Klaus, 1955]. Criterion-Referenced Testing Criterion-referenced testing uses criterionreferenced measures for making an evaluation of performance. These measures involve a comparison between system capabilities and individual performance [Glaser and Klaus, 1966] Such measures indicate whether an individual has reached a given performance standard [Knoop and Welde, 1973]. The standard for criterion-referenced measures may be determined either by analysis, subjective judgements by a panel of experts, or numerous successful performances as sampled from a large population [Knoop and Welde, 197 3] c. Criterion- Versus Norm- Referenced Testing A recent article by Swezey [1973] reviewed and described the relative advantages and disadvantages of criterion-referenced and norm-referenced testing, from which the conclusions are cited below: Content validated criterion-referenced tests, which are derived from appropriate job, task, or training analyses, often provide the best available measure of performance; particularly in objectives-oriented situations. It is often the case that no better criterion exists upon which to validate the instrument. 100 other researchers have supported this conclusion, especially in the field of aircrew training performance measurement, and it is perhaps a more feasible alternative to the more traditional and less efficient method of norm-referenced testing [Knoop and Welde, 1973; Waag and Eddowes, 1978; Uhlaner and Drucker, 3 . 1975; McDowell, 1980]. Accuracy of Evaluation The accuracy of evaluation is dependent upon the accuracy of measurement, the accuracy and relevance of the criteria, and the evaluation conditions. Since the accuracy of measurement and criteria have already been addressed, this section will be limited to evaluation conditions. During any evaluation, several sources of contamination, or bias, as discussed by Danneskiold and Klaus [1966] , [1955] and Glaser may affect the performance evaluation of individuals, and are listed below: (1) In performance testing, one individual may naturally perform better than another during the examination situation, even though both may actually possess the same skill level. (2) The sequence and construction of the simulated mission test may cause some individuals to respond in a way that is dependent only on the test sequence and construction. (3) Judgemental errors occur whenever individuals are used to observe performance, due to prejudices and st.ereotypes formed by the observer. (4) Evaluating condenses performance dimensions into a compact and meaningful unit, where in the process some information is lost. (5) Observed performance is only a sample of the total skills and knowledge of the individual. 101 The accuracy of evaluation may be increased by improving either measurement accuracy, the accuracy and relevance of criteria, or the evaluation conditions. For evaluation conditions, the common method of eliminating some of the five bias factors mentioned above is to increase objectivity in measurement and to standardize the test conditions [Glaser and Klaus, 1966] . 4. Evaluating Individual and Group Differences The measurement of differences in individual performance is highly desirable in a training situation. As training progresses, the performance of individual trainees gradually approaches the desired minimum skill level required for system operation. The accurate evaluation of what skill level the individual actually possesses during the training process is necessary for efficient training control and for efficient instruction [Glaser and Klaus, 1966]. Several methods have been developed to identify which performance measures can best discriminate among individuals at various ability levels; these will be discussed in Chapter VII due to their applicability in designing the measurement system at hand [Parker, 1967; Buckhout and Cotterman, 1963; Thorndike, 1951]. Measuring group differences, as opposed to individual differences, is more suitable for the purposes of treatment evaluation, such as the training method, length of instruction, and design of displays and controls, and will not be addressed in detail here. Interested readers may consult Glaser and 102 . . . . Klaus [1966] or Moore and Meshier [1979] for further discussions of measuring methods for group differences. 5. Characteristics of Evaluation Some characteristics and considerations that contribute to improving the evaluation process are listed below: (1) Repeatability of a measure implies that a specified score achieved today represents the same level of performance as it did at a previous time (temporal invariance) [McDowell, 197 8] (2) Sensitivity of a measure occurs when a measure reliably changes whenever the operator's performance changes [Grodsky, 1967; Kelley and Wargo, 1968; Knoop and Welde, 1973]. (3) Comprehensiveness of the measures employed in covering as wide a range of flying sJcills as possible [Ericksen, 1952] (4) Interpretability of measures and evaluation results [Demaree and Matheny, 19 65; Waag and Eddowes, 197 5; McDowell, 1978]. (5) Immediately available measures and scores to provide the student with knowledge of results [Buckhout and Cotterman, 1963; Demaree and Matheny, 1965; Welford, 1971; Waag and Eddowes, 1975; McDowell, 1978; Kelly, 1979] (6) Economical considerations require that evaluation be constrained by cost and availability of personnel, yet adequate at a minimum level for the purpose at hand [Marks, 1961; Demaree and Matheny, 1965] (7) Standardization of test conditions and environments enables performance to more accurately reflect true operator skill [Ericksen, 1952; Marks, 1961; Smode, at al., 1962; Demaree and Matheny, 1965]. This list of desirable characteristics of evaluation is not all-inclusive but does provide some foundation for examining existing evaluation systems for those properties that are in consonance with system and evaluation goals. 103 V. A. THE NATURE OF THE BOMBARDIER/NAVIGATOR TASK INTRODUCTION Navigating an A-6E TRAi!4 aircraft during a low altitude, non-visual air interdiction mission is perhaps one of the most demanding and complex tasks expected of navigators today. The aircraft must avoid rough or mountainous terrain while traveling narrow corridors between geographical turn points, which must be crossed with pinpoint accuracy at predesignated times. Literally hundreds of individual steps or procedures are involved in navigating the aircraft, each of which contribute in some dimension to attaining the mission objective. 2, adapted from Obermayer and Vreuls [1974] system interactions in the A-6E aircraft. , Figure shows the crew- The pilot controls the aircraft and manages aircraft flight systems while receiving visual and auditory navigational information from the Vertical Display Indicator (VDI) and B/N, respectively. can be noted in Figure 2, As it the B/N manages the navigational equipment, processes large amounts of concurrent information, and makes critical decisions regarding the navigational accuracy of the aircraft. At any one time, the B/N may be executing tasks that are dichotomous, sequential, continuous, monitorial, computational, or decisional in nature. serving as the systems manager. 104 At all times he is TK ± Eh O H M <r U O ? 0) IK (U 4-» ±. s o H Pi H < Q CJ H > <; s ^^HM ^ >1 CO 1 5 (U M u w VT) 1 < a: ^^ 2 w w El a. < s o u U D ^ •H /^ 7R ^ -Vw > T> Q 2 > ^ 105 One of the more difficult subtasks is radar scope interpretation. This activity involves the recognition of the relationship between specific symbols or patterns of symbols on a flight chart with the specific returns or patterns of returns on the radar scope [Beverly, 1952] . The success of this identification subtask depends largely upon the quantity and quality of a priori information about the target that was available to the radar navigator [Williams, et al., I960]. This subtask may be performed while the B/N is monitoring the Inertial Navigation System (INS) , observing computer-generated navigational information displays, and informing the pilot about current equipment status. It is an axiom among student B/Ns that "if you are sitting there perceiving that everything has been done, you are getting behind." This section will define and describe the nature of the B/N's tasks in terms of: (1) aircrew-aircraft system, and the physical variables of the (2) the complex skills and abilities of a perceptual, psychomotor, and cognitive nature. The importance of operationally describing and systematically classifying the B/N's tasks from a behavioral point of view is that such an analysis may point to areas where measurement of performance is both desirable and feasible, and may indicate the relationships of individual tasks to overall mission success [Smode, et al., 1962; Vreuls, et al., 1974]. The tool used to define and describe the tasks of the B/N will be a task analysis. 106 . B. . TASK CONSIDERATIONS Task Definition 1. A task is one or more activities performed by a single human operator to accomplish a specified objective [Connelly, et al., 1974]. In the aviation training environment, a navigation task is the successful action of the navigator in response to visual, aural, vestibular, and tactile information concerning the actual and desired values of ameter (or a particular par- more than one parameter) associated with navigating the aircraft, usually after completing a lesson or series of lessons [Demaree and Matheny, 1965; Anderson and Faust, 197 4] 2 Classification of Tasks There are numerous task classification methods, all of which depend on the purpose of describing the tasks and the nature of the tasks themselves. fied Smode, et al. [1962] classi- behavior for performance measurement purposes with the idea of accommodating both diagnostic measures relating to elemental tasks as well as the more global measurements relating to overall system performance. These general behavior classes are listed and defined as follows: a. Level I - Elemental Tasks The simplest level of analysis, referring to any homogeneous series of work sequences conducted at one time, or single actions taken toward accomplishing a desired objective. These tasks range from short duration discrete homogeneous acts to longer sequences of routing activity. 107 b. Level II Complex Tasks - The composite of activities which involve identifiable sequences of homogeneous activity or recurring single Each complex task actions and sub-routines in performance. is made up of tasks from Level I, involving either the simultaneous and/or sequential integration of combinations of elemental tasks, or the repetition of a single Level I activity over time. c. Level III - Mission Segments The segments or phases of performance that are identified in full mission activity. Essentially, a segment is composed of a group of complex tasks (Level II) which are integrated in the performance at this level of description. d. Level IV - Overall Missions The major types of missions anticipated for advanced flight vehicles. Each mission is composed of a group of segments of activity which are integrated in the performance at this level of description. Beginning with overall missions and ending with elemental tasks, these progressive refinements in task specificity allow performance measurement decisions to be made at progressively more detailed levels. 3. Task Relation to Performance and System Purpose For measurement purposes, it is neither practical nor desirable to measure all possible task conditions which might occur in accomplishing a mission objective [Smode, et al., 108 : 1962; Vreuls, et al., 1974J . To be practical, an attempt should be made to simplify the analysis of tasks and remove irrelevant measurement [Vreuls, et al., 1974]. Since measurement is only possible on the basis of specific, observable events, a great deal of investigation and analysis must be accomplished to describe tasks that are representative of accomplishing the mission purpose while at the same time are measurable [Glaser and Klaus, 1966] . Glaser and Klaus [1966] identified two kinds of observable performance that are useful for performance measurement: the behavioral repertory of the operator in the form of verbal and motor actions , and the operator's effects on overall system performance or output. From the measurement of either of these two observed performances, some inference can be made about the operator's level of skill in performing operational and describable tasks. The intimate relationship between the task of the operator and performance measurement of the operator for the purpose of estimating his level of skill was best stated by Smode, et al . [1962] The behaviors and tasks which are observed and measured necessarily will be a sampling from those which comprise the complete system activity, for it is neither feasible nor necessary to measure everything in order to evaluate proficiency. What one evaluates depends on purpose. Determining those properties of behavior significant to the purpose aids in defining the areas of human behavior for assessment. In the interest of maximizing validity of measurement, this sampling should be guided by the criticality of the tasks and operational segments to mission or system success. As a rule, those tasks should be selected for measurement on which good performance results in mission success and on v/hich poor performance means failure. 109 As discussed previously, identifying the purpose of the system is important for defining performance standards and for accurate measurement of behavior. Likewise, the purpose of the system helps define those tasks which should be measured, and is essential for discovering what the relationship is between mission tasks performed by the operator and the probability of mission success [Smode, et al. , 1962; Buckhout and Cotterman, 1963; Cotterman and Wood, 1967]. Actions that are critical to performance in that they differ- entiate between success and failure in performance can only be identified properly in terms of the ultimate purpose or goal of the man-machine system [Smode, et al., 1962]. C. RADAR NAVIGATION TASK ANALYSIS 1 . Background A task analysis is a time-oriented description of man-machine interactions brought about by an operator in accomplishing a unit of work with an item of the machine, and shows the sequential and simultaneous manual and intellectual activities of the man operating, maintaining, or controlling equipment, rather than a sequential operation of the equipment [Department of Defense, iMIL-H-4 68 55B, 1979] . Miller [1953] presented a more usable definition of a task analysis: the gathering and organization of the psychological aspects of the indication to be observed (stimulus and channel) , the action required (response behavior, including decision-making) 110 , the skills and knowledge required for task performance, and probable characteristic human errors and equipment malfunctions. A task analysis is conducted mainly for the design of new systems or for improvements to existing systems and provides basic building blocks for the rest of human engineering analysis [Van Cott and Kinkade, 197 2] . The purpose of the task analysis presented here is to improve current performance measurement of the B/N in the A-6E WST, and will be discussed more fully in respect to this purpose in Chapter VII. There are several methods of conducting a task analysis which are classified as either empirical, analytical, or some combination of both. The empirical methods rely on industrial engineering techniques such as time and motion study [Mundel, 1978] while the analytical techniques involve the use of expert opinions through interviews or questionnaires. Van Cott and Kinkade [197 2] advocated seeking information from a wide variety of sources and employing more than one technique in order to adequately describe what an operator actually does in a system. "A completely developed task analysis will present a detailed description of the component behavioral skills that the accomplishment of the task entails, the relationships among those components, and the function of each component in the total task [Anderson and Faust, 1974] ." Since a task analysis involves breaking down a task into behavioral 111 components for the purpose of performance measurement, the question of when to stop subdividing the task is most impor- Anderson and Faust [197 4] proposed that enough task tant. analysis detail is reached when the intact or component skill is part of the student's entering behavior. The use of a task analysis for the purpose of performance measurement assumes that behavior can be analyzed in terms of basic components that are conceptually identified in a way that is convenient and agreeable to people and that specific measurement techniques appropriate for the various behavioral components exist [Smode, et al., 1962]. This assumption becomes less theory and more factual in light of research conducted in the helicopter community. [1965] , Locke, et al. in a study of over 500 primary helicopter students using the OH-23D helicopter, reported that nearly all complex man-machine maneuvers can be broken down into independent component parts with associated component abilities. A more recent study by Rankin and iMcDaniel [1980] assessed helicopter flight task proficiency using a Computer Aided Training Evalu- ation and Scheduling (GATES) system, where flight maneuvers were divided into tasks that were used for performance measurement and evaluation, and were then utilized to determine overall aviator proficiency. Just as no two task analyses are ever the same, there may be multiple sets of operator behavior possible to accomplish the tasks as described in one task analysis [Fleishman, 112 . 1967; Vreuls and Wooldridge, This major limitation 1977]. of using a task analysis to measure performance of an operator reaffirms the idea of measuring all observable system outputs and establishing the relationship among operator actions, system outputs, and mission success or failure. By empirically validating a task analysis in the operational environment and establishing the above mentioned relationships, any limitations imposed by differences in operator strategy on the measurement system may be circumvented. 2 . Previous A-6 Task Analyses a. Naval Flight Officer Function Analysis In 1972, the Chief of Naval Operations requested the Naval Aerospace Medical Research Laboratory (NAMRL) to conduct a series of investigations analyzing the operational functions of the Naval Flight Officer (NFO) for the purposes of revising NFO training programs and to aid in determining future training equipment requirements and characteristics. Addressing NFOs of P-3B/C, RA-5C, A-6A, EA-6B, E-2C, and F-4B/J aircraft, the investigations determined the roles, duties and tasks performed by the NFO in a given aircraft, the percent of NFOs performing a given task/duty, the time and effort spent on various roles, duties and tasks, and finally, the task criticality The study of interest for the purposes of this thesis involves the analysis of the B/N operational functions in the A-6A [Doll, et al., 1972]. 113 The procedure used for the . function analysis was based on a method of job analysis developed by the USAF Personnel Research Laboratory at LackThe principal method of analyzing land Air Force Base, Texas. functions was an inventory of activities approach that combined features of the checklist, open-ended questionnaire, and interview methods The results of analyzing the A-6A B/N tasks was based on 84 surveys completed by operational B/Ns. major operational roles identified Of six (communication, navigation, tactics, sensors, armament, and system data processing), more time and effort was spent (28 percent) in flight by the B/N performing the navigation role than any other single role. Within the navigation role, five duties were identified: navigate using Inertial Doppler systems, using ADF/UHF-ADF, and (5) (4) using radar. were listed. (2) (1) using TACAN, (3) using visual references/Dead Reckoning, Over 98 tasks within those five duties Amount of time and effort as well as the criticality of each task was recorded and a rank order listing of all tasks for these two categories was presented. In developing a task analysis for the B/N during radar navigation (presented later in this section) , this A-6A function analysis for the B/N shows the importance of navigation in terms of time and effort spent by B/Ns in the operational environment. A measurement system that accurately describes B/N performance during radar navigation would be extremely useful from this standpoint. 114 The time and effort. and criticality rankings of this source were also useful for those tasks that corresponded with the same task or subtask in the current effort, and in developing a performance measurement system that encompassed critical tasks in terms of their contribution to overall mission success, b. Grumman A-6E Training Program TRAT-I Grumman Aerospace Corporation completed a study on the application of ISD methodology to the design of a training program for A-6E TRAM FRS pilots and B/Ns in mid-1976. Comprised of over seven volumes, the study included a task analysis, development of SBOs, media analysis, and formulation of lesson specifications 1975; Campbell, et al., et al. , 197 5; Campbell, [Campbell, 1975; Campbell and Sohl, Hanish and Feddern, 1975; Graham, 1975; et al. 1977] , . The task analysis phase of the ISD process was performed jointly by a team consisting of Navy Subject Matter Experts (SMEs) and Grumman training psychologists, educational specialists, and flight test personnel. Tasks were to be identified based on performance in the operational environment and described in sufficient depth to permit an identification of the underlying skills and knowledge required by the crewmen to perform the task. A hierarchical approach for describing the pilot and B/N behaviors during a mission resulted in three levels of description: major mission events, the tasks which comprise the events, and the steps which describe the incremental actions an aircrewman must take to complete a task. 115 The first result of the task analysis effort was a comprehensive task listing comprised of over 4 00 nominal pilot tasks, each with an average of approximately 10 steps; 7 airframe emergency sequences involving an average of 7-10 steps each/ 35 system malfunctions, and more than 200 nominal B/N tasks with an average of 10 steps each. The listings represented tasks for which training needed to be conducted at the FRS level. A Task Analysis Record (TAR) form was utilized for each task to ascertain the following: (1) crewman where training was given, (3) skills and performing task, (2) knowledge required by task, is performed, (5) system involved, difficulty, (9) (4) conditions under which task cues involved in performance, (7) degree of difficulty, task criticality, (10) (8) (6) aircraft factors in task factors in performance measurement, and (11) other special factors which impacted on training. Because the TAR was used for the purpose of instructional sequencing and blocking downstream in the ISD process and as an aid in selecting appropriate instructional strategies, it was not published as part of the study. The actual task analysis appears in the form of an ISD record developed from the TAR and SBOs. Objectives were classified on the basis of eight major taxonomic categories: (4) and (1) knowledge, application, (8) (5) (2) comprehension, analysis, complex performance. (6) (3) synthesis, discrimination, (7) evaluation, This taxonomy was retained for the current thesis task analysis effort and will be defined 116 in Table VII. The ISD record then contained the SBO, task identification data, condition/constraints, performance standard, taxonomic data, a criterion test statement, and test type and format. The Grumman task analysis effort becomes useful to the current effort of developing a task analysis for the B/N during the radar navigation maneuver, and using that task analysis for the purpose of performance measurement. In this respect, the Grumman study was used as a guiding outline in developing the current task analysis. Prophet [1978] reviewed past ISD efforts in Navy fleet aviation training program development that included the Grumman A-6E ISD program, and made the following comments in reference to measurement and evaluation for that program: (1) Methodologies being followed did not necessarily require a systematic treatment of measurement and evaluation. (2) No discussion of the mechanics of measurement for standards found in SBOs is given. (3) While a clear recognition of when and where measurement will take place is addressed, no information is given concerning how. (4) The problems of flight versus non-flight measurement were not discussed. Although some criticism may be found in the lack of measurement mechanics from the Grumman task analysis effort, it is not a surprising revelation given the purpose of their task analysis. Their effort is still deserving in the light of the complexity of the aircrew and A-6E combination, and this author 117 used their unclassified task analysis material in defining and describing exactly what a B/N does during a radar navigation maneuver. c. Perceptronics Incorporated Decision Task Analysis In early 1980, a study designed to identify significant aircrew decisions in Navy attack aircraft was performed by Perceptronics, Inc. for the Naval Weapons Center, China Lake, California. The study selected two mission scenarios that were representative of A-6E and A-7E aircraft: close air support and fixed target attack [Saleh, et al., 1980]. A mission analysis followed by an Air crew/ Avionics Functions Analysis was performed on each scenario. Finally, a decision identification analysis was performed which resulted in a listing of significant decisions in each mission for each aircrew. The study results provided information on decision type, difficulty, and criticality. Limited use was made of this decision identification analysis due to the scenarios developed and the purpose of the task analysis: decision-making, and some dependence of that task analysis upon the previous efforts by Grumman. Nevertheless, a few decisional tasks were reviewed for use in the current effort. 3. Current Task Analysis for Performance Measurement Using the research provided by the Naval Flight Officer function analysis, the Grumman A-6E TRAM training program task analysis, and the Perceptronics, Inc. decision task analysis, 118 " as discussed previously, performed with a task analysis v;as the purpose of measuring B/N performance during radar navigation in the A-6E WST. The results of that effort, in the form of a task listing (Appendix A) C) , , and a Mission Time Line Analysis a task analysis (Appendix (MTLA; Appendix D) , are each presented separately below. Task Listing a. As shown in Appendix A, the radar navigation maneuver was divided into three segments: (2) (1) after takeoff checks, navigation to the initial point (IP), and to the turn point (TP) . (3) navigation The navigation to TP segment (3) was the portion of the A-6E CAINS flight within the scope of this thesis, and was the segment of interest to be later expanded upon in the form of a task analysis and MTLA that will be discussed later in this section. The following definitions will explain the significance of the symbology within the navigation to TP segment of Appendix C: (1) Tn - Task number, where the number is represented by "n. (2) Sn - Subtask number. (3) (a) - Subtask element, where the element is represented by the lower case letter "a," or other letters. The nomenclature for actual switches, keys, controls, or buttons in the A-6E CAINS cockpit is underlined throughout the task listing (e.g., Rcvr control). 119 Discrete or continuous , . settings for each switch, key, control, or button is to the far right of the task, subtask, or subtask element, and is separated by a line of periods. The choice of language in the form of action verbs for which behaviors are described was a difficult process, due to the lack of standardization in both the science of analyzing tasks and in aircrew performance measurement research. The necessity of employing action verbs that described simple and easily observable activities and were easily identified in terms of performance measurement was paramount to the current effort. A hybrid taxonomy, using 31 action verbs as shown in Table VI, was developed from earlier work by Angell, et al. [1964] that was, in a sense, later validated by Chris- tensen and Mills [1967] in an analysis of locating representative data on human activities in complex operational systems. Using a later study by Oiler [19 63] in the form of a human factors data thesaurus as applied to task data, the original 50 action verbs used by Angell, a total of 31 et al. [1964] was reduced to action verbs by eliminating redundant synonyms and by using the recommended acceptable action verbs and nouns. Except for the reduction of action verbs (specific behaviors) the remainder of the original taxonomy was preserved. For the convenience of the reader, the 31 action verbs or specific behaviors utilized in the current task analysis are presented as a glossary (Appendix B) 120 TABLE VI: CLASSIFICATION OF BOMBARDIER/ NAVIGATOR BEHAVIORS SPECIFIC BEHAVIORS ACTIVITIES PROCESSES Perceptual Searching for information Receiving information Identifying objects, actions, or events Checks Monitors Observes Reads Information processing Initiates Records Uses Mediational Problem solving and decision-making Communication Communicating Alerts Informs Instructs Simple/discrete Activates Depresses Places Pushes Throws Sets Complex/continuous Adjusts Inserts Positions Rotates Tunes Motor Source: Checkouts Compares Continues Delays Determines Evaluates Performs Repeats Selects Troubleshoots Angell, et al. [1964] 121 (adapted) b. Task Analysis A task analysis for the specific purpose of measuring B/N performance during radar navigation was performed and is presented as Appendix C. As previously discussed, only segment three, navigation to TP, was examined during the task analysis to limit the scope of this study. Since the concepts of segment and tasks have already been addressed, the seven columns of the A-6E TRAI"! radar navigation task analysis form in Appendix C will now be explained in detail, provided by Van Cott and Kinkade [197 2] [1974], Pickrel and McDonald Rosenmayer and Asiala [197 6] , using guidance Anderson and Faust [1964], Smode, et al. [1962], and : - a component activity of a task. Within a task, collectively all subtasks comprise the task. Subtasks are represented by the letter "S" followed immediately by a numeral. Subtask elements are represented by a small letter in parentheses. (1) Subtask (2) Feedback (3) Action Stimulus - the event or cue that instigates performance of the subtask. This stimulus may be an out-of-tolerance display indication, a requirement of periodic inspection, a command, the indication of adequacy of response or action. Listed as VISUAL, TACTILE, AUDITORY, or VESTIBULAR and located in the subtask column for convenience only. - a failure, etc. (4) Time (5) Criticality - the relationship between mission success and the below-minimum performance or required excessive performance time of a particular subtask or subtask element. "High" (H) indicates poor subtask performance may lead to mission failure or an accident. the estimated time in seconds to perform the subtask or task element calculated from initiation to completion. - 122 . . , "Medium" (M) indicates the possibility of "Low" (L) indegraded mission capability. dicates that poor performance may have little effect on mission success. (6) Potential Error - errors are classified as either performing failure to perform the task (OMIT) the task inappropriately in time or accuracy (C0MI4IT) or performing sequential task steps in the incorrect order (SEQUENTIAL) , , (7) Skills Required - the taxonomy of training objectives used for the Grumman task analysis was retained and is presented in Table VII [Campbell, et al., This concept will be discussed in more 1977] detail later in this section. . (8) Performance Measure Metrics - a candidate metric which may best describe the successful performance of the task or a genuine display of the required skills. The types of metrics suggested were classified as TIME (time in seconds from start to finish of task) T-S (time-sharing or proportion of time that particular task is performed in relation to other tasks being performed in the same time period) R-T (reaction time in seconds from the onset of an action stimulus to task initiation) ACC (accuracy of task performance) FREQ (number of task occurrences) DEC (decisions made as a correct or incorrect choice depending on the particular situation and mission requirements) QUAL (quality of a task, especially in regards to radar scope tuning quality) and SUBJ (subjective observation or comprehension of task execution success by an instructor) , , , / , , Due to the lack of operational data, the task analysis was derived analytically with close attention being paid to consistency with previous A-6 task analysis efforts. The validation of any task analysis can only occur when it is subjected to the operational environment for repeated empirical analysis. Unfortunately, time, cost and system availability constraints precluded the execution of this important phase. 123 TABLE VII: TAXONOMY OF SKILLS REQUIRED Knowledge Technological - Learning "how to" perforin a single Learning switch and control configuring procedure. "how to" read meters, digital displays, scopes, lighting displays, etc. In general, learning "how to." Formal - Learning the meaning of special symbols, acronyms, words, nomenclature, etc. Descriptive - To describe "what is" and "what was": facts, data, special information about systems, subsystems, equipment, weapons, tactics, missions, etc. Concepts and Principles - Fundamental truths, ideas, opinions and thoughts formed from generalizations of particulars. Comprehension Understanding the meaning of meter readings, scope, digital and lighting displays. Understanding the switch and control configuring procedure, i.e., the reason for a specified sequence, the reason for a switch or control position, the reason for a verification, etc. Grasping the meaning of concepts and principles, i.e., understanding the basic principles of infrared and radar detection. Understanding the meaning of facts, data, specific information, etc. Discrimination Distinguishing among different external stimuli and making appropriate responses to them, e.g., scanning gages for out-of-tolerance trends. Also includes the recognition of the essential similarity among a class of objects or events, e.g., classifying aircraft types or radar* return images. Source: Campbell, et al. [1977]. 124 . TABLE VII CContinued) Application Simple Procedure - A demonstration of a simple learned procedure in the cockpit or simulator requiring not more than simply repeating required switch and control configuring and simple visual verification (i.e., advisory light status) Complex Procedure - A demonstration of a learned procedure in a cockpit or simulator that requires differentiating or distinguishing between readings on meters, digital displays, and images on video and radar displays and interpreting and applying the meaning of the readings and images. General - Using learned materials in new and concrete situations (e.g., using rules, methods, concepts, principles, procedures, etc.). Analysis A demonstration material (i.e., ents so that it safety, mission of a learned process of breaking down data, other information) into its componmay be evaluated with respect to crew's success, A/C maintenance, etc. Synthesis A demonstration of learned process, i.e., putting tactical elements together (e.g., weapons, targets, available systems, A/C capability, etc.) to formulate a mission. Evaluation A demonstration of a learned process of assessing or judging a system or situation, based on criteria (i.e., data, rules, available equipment, conditions, etc.) and then reaching a conclusion based on this assessment. Complex Performance A demonstration that requires psychomotor skills and/or critical thinking skills usually requiring practice. 125 As it stands, a reasonable assumption of the existence of some face validity in the current task analysis can be made in the light of the author's operational experience as a B/N in the A-6E CAINS aircraft (over 600 hours) and the dependence of the task analysis upon previous task analysis efforts, even though none of the previous efforts were formally validated by empirical methods. The current task analysis was also informally reviewed by other A-6E B/Ns before finalization of the effort. The purpose of the task analysis was to improve current performance measurement of the 3/N during radar navigation in the WST by providing performance measure metrics (right-hand column of Appendix C) that are possible candidates for describing successful task performance or B/N skill acquisition. Several hundred metrics are available from which a candidate set can be chosen based on the initial measure selection criteria as previously discussed in Chapter IV. From the "performance measure metrics" column of Appendix C, several potential candidate measures were identified and will be combined with potential measures from Table II in Chapter IV and presented as part of the final candidate measure set listed in Table XI c. (Chapter VII). Mission Time Line Analysis An MTLA is a graphical analysis which relates the sequence of tasks to be performed by the operator to a real time basis [Matheny, et al., 1970]. 126 The purpose of an MTLA as used in the current study is to identify those performance measurement points within a man-machine system where standards of accuracy and time may be applied in the evaluation process. Essentially a bar chart, an MTLA for the navigation- to-TP segment of the radar navigation maneuver is presented as Appendix D. The time of execution for each subtask was extracted from estimated completion times on the task analysis record form (Appendix C) . Time was estimated with the assumption that sensing conditions were good and the B/N was highly skilled. Darkly shaded time lines represent tasks that demand full mental attention whereas shaded time bars represent "monitoring" tasks or "troubleshooting" tasks that may not have to be executed. The MTLA is a performance measurement source for both the identification of critical subtasks and the use of time to perform as a measure of skilled behavior. Thus, the MTLA was utilized to identify candidate performance measures as found in the "performance measure metric" column of the task analysis record form (Appendix C) that were later used for the final candidate measure set as will be described in Chapter VII 4. (Table XI). B/N Skills and Knowledge This section will relate current skill acquisition principles to performance measurement, and present a skill acquisition model of the relationship between skill acquisition, the task, performance measurement, and performance evaluation. A great deal of discussion about the concept of 127 : : skill, how skill is attained, and how skill acquisition is measured can be found in the literature from such diverse areas as private industry, control theory, information processing, and education [Bilodeau, 1966 and 1969; Jones, 1970; Welford, 1971; Hulin and Alvares, 1971 and 1971; Singleton, 1971; Leshowitz, et al., 1974; Shipley, 1976; Welford, 1976]. Despite the global interest in skill, this discussion will be limited to aircrew skill acquisition and the measurement of that skill. a. Definition of Skill Skill may be defined as the ability to perform given tasks successfully or competently in relation to specified standards [Cureton, Prophet, 1978]. 1951; Senders, 197 4; Smit, 197 6; A more precise definition of skill is offered by Connelly, et al. [1974] The ability to use knowledge to perform manual operations in the achievement of a specific task objective in a manner which provides for the elimination of irrelevant action and erroneous response. This conceptualization exists only in conjunction with an individual task and is reflected in the quality with which this task is performed. Most definitions of skill rely on the fundamental concept that the use of capacities efficiently and effectively as the result of experience and practice would generally characterize skill [Welford, 1976]. Indeed, the concept of skill cannot be well defined due to the diversity of its nature and remains more or less an amorphous quantity, best described by its characteristics [Singleton, 1971] 128 , 1. It is continuous, there is always an extensive overlap Even in principle, it cannot be and interaction. analyzed by separation into discrete units along either space or time axes. 2. It involves all the stages of information processing identifiable in the organism, basically inputs, processing and outputs. 3. It is learned and therefore highly variable within and between individuals. 4. There is a purpose, objective, or goal providing meaning to the activity. b. Skill Acquisition The development of skill, as previously discussed, is due mainly to the effects of practice and experience on the use of basic capacities. Therefore, the acquisition of skill appears to result from learning and seems to improve the efficiency and effectiveness of underlying basic capacities [Welford, 1976]. Singleton [1971] advanced the idea that skill develops by selectivity and by the integration of activities. For most skill development theories, it is generally agreed that as learning a new task takes place, operators learn a basic strategy in performing the task, that in effect becomes an increasingly skilled template with the qualities of organizational and efficiency of operation 1980] . [Engler, et al. Depending on the task, the level of skill required to perform the task is universally measured with the property of variability. Bowen, et al. [1966] found considerable variability as measured by a lack of consistency for all skill levels of pilots performing tasks in an OFT, even pilots with 129 substantial flight experience. operator begins to learn a It thus appears that as an new task, his control strategy in performing the task is highly inefficient, resulting in a large variability of actions. As skill development progresses, his control strategy becomes highly efficient and effective, resulting in what should be smaller variability of actions. Three phases of skill development have been hypothesized in earlier research by Fitts [1962] and discussed in terms of aircrew skill acquisition by Smode, et al. and Prophet [1976] . [1962] The stages of skill acquisition are discussed below. (1) Early Skill Development. In this phase the student seeks to develop a cognitive structure of the task in the form of discriminating the task purpose, ascertaining standards of task performance, and interpreting performance information feedback. Actions tend to be slow and deliberate, and depend a great deal on concentrated attention and effort in performing the task. (2) Intermediate Skill Development. After learning the task purpose and experiencing some practice at the task, the student begins to organize his control strategy by becoming more efficient in responding to signals displayed to him. Perceptual and response patterns become fixed with less reliance placed on verbal mediation of response integration by the student. 130 . (3) Advanced Skill Development. This phase represents the higher level of skill acquisition, where performance becomes more resistant performed concurrently. to stress and activities are The rate to acquire this stage through practice is different for each individual, as practice on any complex task generally spreads individuals out into stable but different skill levels [Jones, 1970]. Navi- gating an A-6E CAINS aircraft falls into this category of complex tasks. This stage is characterized by the individual performing in an automated manner requiring little conscious awareness and little allocation of mental effort [Norman, [1976] c. Measurement of Skill Acquisition Figure 3 is a model developed by the author to illustrate the relationship among B/N skill acquisition, the radar navigation task, and performance measurement and evaluation. An understanding of the model depends heavily upon concepts defined and discussed in Chapter IV and in the early part of this chapter. This model will be used for the current discussion of skill acquisition measurement. The actual measurement of skill acquisition \ ' through its various stages has received little practical atteni tion and research, most likely due to the complexity of the subject and the difficulty involved in accurately assessing human performance as system complexity increases [Glaser and Klaus, 1966; Vreuls and Wooldridge, 131 1977; Kelley and Wargo, -\ U o u o 4J lu 2 O M D •J < > NX > O • w O 0) C M 'H 3 13 -§ o 01 IT3 OJ OS, •W > CO 1-3 ""^ ^ C a n a o 0) U C Ul (U M 6 3 rd 2 O M H < fl M-l •h d) s \ V" / / / \\ \/ M. • X ^ y^ U M X X ' Q CO / • b3 rt; \ •H N < > o 2 U H +J \\ // \V \ // \ \ / ^ / •H / V / '^^. N _ v» • • ^ • ^ • • • ^ • ^< . • '' •^ A^ r ^ X X N N .Â w U3 0) o la C-H M 10 «J w U IT] tJ O-H •H •H 4J -U la 4J •W 0) MS V« 0) s (U Met Crite Met/N •H U U u 3 2 ^ 04 L 00 C <t3 •H 6-» 132 J •H fa 1963; Senders, 1974] . As shown in Figure 3, through the process of learning and practice of the task over several trials, the student (represented by the oval shapes in the center column) should progress from the early or unskilled state through the intermediate stage and into the skilled or "proficient" stage where he is "trained." Over the course of one task trial, the most that can be accomplished is to obtain objective performance measures and combined measures, and to obtain a subjective opinion of the skill level from the one individual well qualified and skilled in performing the task: an instructor. a Once this "indirect" measurement takes place, comparison is made between the objective and subjective measures and the predefined performance criteria or MOEs. is from this It comparison that the student's skill level is finally evaluated, with severe limitations imposed due to the measurement over one trial. Skill acquisition through the three stages of development occurs not only at different rates, but over the course of several trials. This fact would lend support to a measurement system that indirectly measured skill development over one trial and used historical records of performance to measure skill development over several task trials. The model shows highly likely, likely, and very unlikely evaluation results by assuming that both criteria and measures are ^ valid, reliable, and specific to the purpose of measuring skill i acquisition. 133 . Early conceptions of measuring aircrew skill were discussed by Smode, et al. [1962] and Angell, et al. [1964] Both structured tasks or skills into a hierarchical model and theorized what types of measures (e.g., time, accuracy, frequency, etc.) would be appropriate for a particular level of task or skill. The former study also discussed measurement at the three stages of skill acquisition: Measurement at (1) the first stage should be concerned with knowledge and task familiarity as well as distinctions between task relevant and task irrelevant information and cues, and a differentiation between in- tolerance and out-of-tolerance conditions; (2) the intermediate stage has measurement concerned with procedure learning, the identification of action stimuli, and the performance of manipulative activities; and (3) measurement of highly developed procedural, perceptual-discriminative motor and concept-using skills and the integration of these combinations into more complex units of performance is of concern. An experiment by Ryack and Krendel [1963] based on research by Krendel and Bloom [1963] measured highly skilled pilots performing a tracking task using a laboratory apparatus. The measurement was based on a theory that a highly skilled pilot displays consistency of system performance, is highly adaptable to changing dynamic requirements, and performs the task with least effort. The transfer of the measurement of these three conceptualizations of high pilot skill from the laboratory to an actual aircraft was not demonstrated. 134 . Later conceptions of skill acquisition measurement were advanced by Welford [1971 and 197 6] who proposed that as practice on a task increased, the speed of performance on that task as measured by time would fall exponentially. Haygood and Leshowitz [197 4] proposed using an information processing model to measure flying skill acquisition. [1979] Bittner evaluated three methods for assessing "differential stability": (1) graphical analysis, (2) early versus late correlational Analysis of Variance (ANOVA) Test of Correlational Equality. , and (3) Lawley That study recommended graphical analysis as a method of first choice. Three recent experiments regarding measurement of aircrew skill acquisition are noteworthy. [1974] Vreuls, et al used multiple discriminant and canonical correlation analyses to discriminate between different levels of skill using four pilots in an F-4E configured simulator. Using six pilots in a UH-IB (helicopter) simulator, Murphy [197 6] investigated individual differences in pilot performance by measuring both man-machine system outputs and pilot control outputs during an instruinent approach and landing. This study concluded that performance differences may be attributed to crewmember differences in cognitive styles, information processing abilities, or experience. Pierce, et al. [1979] concentrated on procedures to assess cognitive skills through the use of behavioral data on eight pilots performing F-4 aircraft pop-up maneuvers. The primary measurement instrument 135 : was an instructor using subjective ratings that were validated by comparison to actual bomb scores. From this previous research and the discussion of the skill acquisition measurement model, it becomes readily apparent that measurement of B/N skill acquisition in the A-6E WST during radar navigation will require both an analytical foundation, as described in this thesis, and empirical validation that would result from implementation of the proposed measurement system. This section is concluded with six recommendations for the procedure of skill appraisal, as discussed by Singleton [1971] (1) Discuss the skilled activity almost ad nauseam with the individuals who practice it and with those to whom and for whom they are responsible. It is not enough to pop in at intervals, the investigator must spend whole shifts and weeks with the practitioners to absorb the operational climate. (2) Try to make this verbal communication more precise by using protocol techniques, critical incident techniques, good/poor contrast techniques, and so on. (3) Observe the development of the skill in trainees and by analysis of what goes on in the formal and informal training procedures and in professional assessment. Make due allowance for history, tradition, technological change, and so on. (4) Structure the activity. Identify the dimensions of the percepts, the decision making, the strategies of action and the overt activities, and try to provide scales of measurement along each dimension. (5) Check as many conclusions as possible by direct observation, performance measurement, and by experiment . (6) Implement the conclusions and provide techniques for assessing the limitations and successes of the innovations. 136 A-6E WST-PERFORiMANCE MEASUREMENT SYSTEM VI. The A-6E WST, Device 2F114, is designed to provide full mission capability for pilot transition training, B/N transition training, integrated crew training, and maintenance of flight and weapon system proficiency in the A-6E Intruder aircraft. The WST will be used to train Navy /Marine flight crew members in all A-6E procedures - ground handling, normal and emergency flight modes, communications, navigation and cross-country missions, tactics, and crew coordination [Read, 1974]. Inherent in these design and training requirements is the necessity for the measurement of aircrew performance and the subsequent measurement of improved performance after training has occurred; a necessary goal for any training simulator [Knoop, 1968]. This section will discuss the general characteristics of the WST together with current performance measurement capabilities, generic performance measurement systems (PMS) for simulators, and current performance measurement and evaluation practices for student B/Ns in the WST. A. WST CHARACTERISTICS 1. General Description The trainer system consists of the following elements Trainee Station, Instructor Station, Simulation Area, and Mechanical Devices Room (see Figure 137 4) . The Trainee Station Instructor Station Trainee Station A»e« AWSAPOW SWrSM" TRAIfMBR OVVICS aft^t^> Mechanical Devices Room Figure 4. Simulation Area Device 2F114, A-6E WST, 138 is an exact replica of the A-6E CAINS cockpit and is mounted on a six degree-of-f reedom motion base to give realistic motion cues. Sound cues, environmental controls, and controls with natural "feel" increase the similarity between and V7ST Normal and emergency flight configurations aircraft cockpits. are simulated, together with all modes of weapon system operation. The Instructor Station area consists of a wrap-around console that can accommodate two principal instructors and two assistant instructors. Controls and displays are utilized by the instructors to: set up and control the training problem, (2) (1) introduce malfunctions and failures, (3) trainee actions and responses to malfunctions, and trainee performance. monitor (4) evaluate Four interactive CRT displays for alpha- numeric and graphic presentations, together with repeater displays of the VDI, direct view radar indication (DVRI) electronic countermeasures (ECM) , and found in the aircraft are available to the instructors for use in training. The Simulation Area contains the computers necessary for simulation of the A-6E CAINS aircraft and its missions. Four real-time minicomputers are utilized, two for flight, one for tactics, and one for the Digital Radar Land Mass Simulation (DRLMS) . iMagnetic tape units, teletypewriter printers, digital conversion equipment, and the DRLMS are also contained in this area. The DRLMS is designed to sim- ulate landmass radar return for the AN/APQ-15 6 radar, which 139 is a major component of the A- 6 navigation/weapon system and is used by the B/N for the tasks of radar scope interpretation and target location. The Mechanical Devices Room contains hydraulic and power equipment for positioning of the Trainee Station motion Also provided from this area is compressed air needed system. for g-suit and environmental control requirements. 2. Performance Evaluation System A comprehensive list of system features and characteristics is beyond the scope of the present effort, but can be found in the Grumman Aerospace Corporation Final Configuration or Criteria Reports for the A-6E WST and 1977; Rinsky, 1977]. [Blum, et al., 1977, 1977, Those features which are of interest to performance evaluation in the WST are shown in Figure 5 and discussed below: Program Mission Modes are provided for up to ten a. missions. In this mode, the computer system automatically generates and sequences mission profiles. During this mode, the instructor station monitor displays a listing of the mission leg number, maneuver to be performed, mission leg end, parameters to be monitored, and remarks in order of occurrence. Programmed missions are activated by controls from the instructor station. b. Program Mission Mode Critiquing is provided by instructor selection of up to six parameters for monitoring on each leg of the program mission. 140 The system computes the — 't t »« ^ z u -] iJ u a: u u 62 q (^ W « 4) 0* CS cu 0. 1 S n u M C • b ot i« "^ u 19 4) a. V5 IQ fl H CO a 4) <-< ^ ig u « 'U 1) a u b 3 i:^ m 3 -U ^ J/5 > a UI 'J ^ U3 > C -o M 1 •-• 1 4J (/I OI u> 1 -W -W 0-1 u Z u 1 fl 1 04 1 b •^ z o hi £-• < 3 •n -1 C 3 U C (A H) o X «J C3 D :> a« to TS TJ z U a. 4) ca '^ a Q CJ U i wi CJ 4> 4) w CO jrf CJ -W X w U ca fl 4) -u CO i4 u s -^ V4 s CO T SlO 4J < CU c ^ 2S >i c o •H U 1 4J CTI (tJ U 3 ifl rH > i o c fO 6 o 04 (M U « to J -u (0 5< 4) 4> 4) cJ a fl w Ot X u 01 cn 4) CO Oi C •*4 k • , 4) J 1- S «3 u D U 3 ^ k -M U •w E" \> u *i c-1 CO a< f_J] s-t > u a u 1 — TJ >J 04 <£ rt3 :T 1 / t» CO > — CO 1 a^ 1 1 U u 6H q 0) ., '^t'^ _ 2S 2 u 1 L tt T3 CO ifl H < MS'U4)U<«4iqU4] ^ 1 ,^ 4) U "o -n ja _ \j i CO <0 ^t T?t i X St w / 4J CO 4) .-4 f^ s /\ I CO <«>• yi a n 1) ••4 a. yi fl S u H Q > 4» e u 4) r4 >* fl JJ 'U > / . 4> 14 <9 4J to _ vw 4> ^ 05 > / ^ . 4) 9-> CO •»< 1-} 4J cn Cd ~ /> »m ^ ^m. o *t c a -— 3 3^-OJ ,«-• UJ jj en 1) 13 'J — fl j^ o n-noi'^-iJi-uaoa)'-! *j r-o— <a3-t> 3 <o ^ '^ U ^ U ^' aj '^ (X a 1^ CJ S < < a. yjiiiû-Ltu ^C-^Ct JJ fl D U TÎTtCJ U-U4J --» •-• C C U OJ -1 -• •-• <<=:<<<<<>c5a. iTl a>.-. CO 4) ^^ 2; > 0. s ^^ c Q u U c ^^ V -U •-< —» Jl C cu 4) y — « (13 <a U U Cu c / V ^ ^ 'O 3 CJ OS < in -J a. 4J <a 4; -» Q. Cu CO 141 c fl 0) -H 4J .-4 u 4) M 3 > fl ai 4J -^ •-• -u as U -M C C •H CO 00 -W —1 «i -H -^ UI ca T> tj c .^^ •-4 -u •-» ^-a J >-» •^-* !Ti— 4J -a 1-4 «- y i 3 ^ U H in ra z ij T3 c s < 3 X. r. jz -a.ôi^pHa-wM'j ffJ-J ^ '-I -. iJ 0) CS CJCJ \ \ ^ ^ -H 4) lyigiMiutg^ ,g ^ CTi-< StJO^-OOIOO'-^'fl'^CS ^ Oa3C3(y (UUO jj.Mija.5)ur-^-w S m CJ / \ VJ 4) oj ^ •^ - >. \3< j^ — T3 — -t ^ U --— e- M z 3 I / c S a 3; 4J S > 4> > O •M 'Ji a. /k (0 3 a. 0. — o< ^ 2 a s . 4) 4) -• -> ^ jJ rg Q a« S Q (0 .^ SZ. 1) 4J CJ tl C> 4J •-• S '-I "4 cd - i :-> c :: 'fl £-< &• Ci.I 1 s J ^ 3 > = C: 00u w ":» "" -a 4) u a 0) •-. 4J c > N < < *4 — CO CO M z a = -M 9-« •H difference between the parameter and a tolerance value for When the toler- the parameter preselected by the instructor. ance is exceeded for a selected parameter, the exceeded amount is displayed to the instructor and a printout record is provided at rates of every 30 seconds, 1 5 minute, and seconds, 2 10 seconds, 15 seconds, minutes on a printer/plotter. Available parameters are shown in the left-hand column of Figure 5. c. Procedure Monitor Display is a mode in which all steps of up to any two procedures, normal or emergency, appear automatically on the instructor's display system when called up by the instructor. The text for the procedures and malfunctions is a listing of the steps to be performed by the trainee and an indication of the elapsed time required by the trainee to complete the procedure. This mode is also available during the Programmed Mission Mode. d. Paramêter Recording is available on all modes of WST operation. A minimum of six parameters may be simultaneously recorded on a continuous basis as a function of time, and compared to preselected tolerance values as discussed in (b) above. The parameter record mathematical model program is available in the flight simulation computer. e. CRITIQUE Mode calculates miss distance and clock code of trainee delivered weapons on an instructor-designated target by the use of a Scoring Math Model. Bomb circular probable error (CEP) is calculated using available functions 142 and parameters at time of release. Missile releases are scored as "hit" or "miss" based on computed comparisons to a respective missile envelope. A CRITIQUE display for a permanent record of the results is available. f. Event Recording is provided where up to thirty events selected by the instructor are monitored by the Per- formance Evaluation System. After the event is selected for recording by the instructor, a printout is initiated which contains a statement of the event, other parameter values, and the time of occurrence. Table VIII is a listing of available events for recording along with other recorded parameters. g. Audio voice recording with a time mark and rapid recall function permits the instructor to access desired portions of trainee headset radio during and after a training All pertinent communications can be recorded for up mission. to 2.5 hours. h. Navigational computations, display drive signals and positional readouts were designed to be within 0.1 nautical mile of true position based on ground speed and true course. i. A Versatec electrostatic Printer Plotter unit is furnished at the instructor console area and has the capability of simultaneously printing and plotting parameter recordings. Program iMission parameter recordings, and event recordings during Free Flight, Program Mission, and Demonstration Man- euver/Trainee mode training missions. 143 This unit can also TABLE VIII. A-6E WST EVENT RECORDING SELECTION OTHER EVENT PARAMETERS EVENT 1. 2. 3. 4. 5. 6. 7. 8. 9. 10. 11. 12. 13. 14. 15. 16. 17. 18. 19. 20. 21. 22. 23. 24. 25. 26. 27. 28. 29. 30. Airborne Gear-up and locked Flaps/Slats-up Stability Augmentation Engaged Search Radar Switch Stby/on Doppler Radar Switch Stby/on Chaff emitted Isolation Valve Switch FH/Land Fuel Dump (wing or FUS) -on/secured Tank Pressure Switch-on/off Present Position Correct Button-Depr essed Computer - on/off Computer Error Lite Lit Master Caution Lite Lit ALQ-126 Switch - Rec/Repeat Reselect Lite Flashing/Steady Master Arm - on/off Attack; Step In to/Out of Bomb Release Commit Trigger - Depressed AZ Range Switch - on/off Velocity Correct Switch - Memory/Off Save Track-While-Scan; on Computer - Out of Attack Throttle (s) below 75 percent Gear Handle Down Flap/Slat Lever -30/40 degrees Touchdown Ram Air Turbine - in/out Designate - on/off Source: Blum, et al. [1977] 144 IAS IAS ALT - — - IAS IAS IAS, AOA — Slant Range print out any display type designated by the instructor with a maximum of 20 printouts possible during any 2.5-hour mission. j . The tactics computer exercises master control of the system and includes functional control of the attack navigation system, system displays, weapons release system, inflight refueling system, ECM, threats, magnetic variations. Programmed Missions, dynamic replay, malfunctions, instructor display system, malfunction control, displays, instructor flight control, demonstration maneuvers, CRITIQUE mode, and others. This computer was installed with future hardware and software growth for input-output, memory core, and computation as a design specification. The perfonniance measurement capability of the A-6E WST appears to have an impressive objective measurement capability. The hardware and software computer system was designed with objective performance measurement in mind, although no definite model or technique was provided by the designers for evaluating B/N skill acquisition during a radar navigation mission. The foundation has been laid for objective measurement; all that remains is building a sound performance measurement structure based on principles and models that have been examined and evaluated thus far. B. PERFORMANCE MEASUREMENT SYSTEMS Measuring aircrew performance can be viewed as within itself. a system Every system consists of an assemblage or 145 combination of objects or parts forming a complex or unitary whole with definable characteristics and a common purpose. The purpose of the performance measurement system (P24S) examined here will be to provide FRS instructors and training managers with valid, reliable, and objective information needed to guide decisions about trainee skill acquisition. for the similator has definable components, The PMS functions, inputs, outputs, communication links, and procedures that all interact to form a system that may or may not be efficiently designed or implemented. Criteria for PMS selection may outweigh some desirable system characteristics as well as the optimal allocation of functions to a particular component. has been analyzed, After a PMS functions allocated, and system criteria selected, implementation of the PMS within the operational environment may impose further constraints that cause redesign of the system. The interactions of PMS analysis, functional allocation, criteria, and implementation are discussed below. 1. Systems Analysis Since the purpose of the PMS being discussed is to provide information about student skill level to training personnel for accurate training control decision-making, this system can be viewed from an information-processing approach. Information in the form of data is sensed and collected, recorded and processed, and presented in an output form that is useful for performance evaluation purposes. These functions of the system are interdependent and may be served by the 146 . same component. Major components of the PMS are instructors Discussion of and computers with data storage capability. the PMS analysis follows. Data Sensing and Acquisition a. Performance must be observed to be sensed and Performance measurement considerations in data collected. collection include but are not limited to: pose, (5) (2) flight regime, skills required, ures, (7) measures, (6) (3) maneuver performed, (8) mission results, procedural control, (12) operator motivation, and mission pur(4) tasks, operator physiological output meas- aircraft measures, (9) (1) aircrew-aircraft system output (10) flight management, aircraft systems management, (14) (11) (13) historical data. Sensing and collecting performance information in a simulator can be accomplished by: (1) mechanical and electronic devices including digital computers and human observation [Angell, et al. , 1964]. is usually referred to as "automated" (2) direct The first category measurement devices, and may include video/photo recorders, audio/digital recorders, timers and counters, graphic recorders, and plotters [Smode, et al., 1962; Angell, Hagin, et al., 1977]. et al., 1964; Obermayer , et al., 1974; Direct human observation may or may not be standardized by preplanned performance checklists or instructions Video/photo recorders provide permanent records of performance and are suitable for instructors to use in 147 observing performance more objectively because of playback features. Audio/digital recorders involve recording of communications and direct measurement conversion to digital form by a computer for selected observable parameters. Audio recordings may be utilized as a more objective measurement for instructor use but currently have data conversion limitations. Digital recording of discrete and continuous measures from all levels of aircrew-aircraft system performance has been demonstrated in both simulators and in actual aircraft flight over the past twenty years 1978; Mixon and Moroney, 1981]. [Wierwille and Williges, Timers and counters are suitable as auxiliary components to digital computer measurement for both time and frequency performance measures al., 1964]. [Angell, et Graphic recorders are electrom.echanical in operation and provide continuous records of event states and magnitudes along a time continuum. Graphic recorders are usually either classified as event (discrete performance) or continuous (magnitude of continuous variable) [Angell, et al., 1964] Plotters display information in Cartesian or rectangular coordinates, and are useful for both performance data collection and output. Charles [1978J thoroughly studied the role of the instructor in a simulator and determined one function of the instructor was to monitor performance in the form of student procedures, techniques, skill level, and simulator performance, Indeed, the direct observation of student performance may 148 sometimes be the sole source of valuable performance measurement, especially when unexpected events occur during a simulator mission [Smode, et al., 1962]. As previously mentioned, video recording with playback capability improves the human observation data collection method. Usually, performance measurements resulting from this technique must be converted for digital computer use in the data playback and processing stage. b. Data Processing and Analysis Once performance data are sensed and collected by mechanical or human means, some conversion is usually required to make the raw data more useful for the system purpose, i.e., tion. to provide information for accurate performance evalua- Usually all data are converted to a digital format appropriate to the general purpose computer. It is in this stage where computers and peripheral equipment such as input/ output devices, memory core units, and magnetic tape drives are extremely accurate, efficient and cost-effective as compared to human processing of data, although some data types may not be convertible to a digital format and must be carried to the system output stage in raw form. In this stage, usually video recordings are reviewed by the instructor to increase the objectivity of his direct human observation of performance. [1974] For the interested reader, Obermayer and Vreuls present a more detailed account of data playback and processing components and interactions. 149 c. Data Presentation After data analysis, the data will be available as output measures for the evaluation process. The output format may be numerical, graphical, audio, visual, or some other form. Since the evaluation process involves the comparison of performance data to standards or criteria, some of the performance data may be utilized as criteria for subsequent evaluation use. Most likely, the data output will be typical measures of time, accuracy, and frequency for various task levels. Some measures may be combined in the processing stage and used as output data for comparisons to established MOEs. 2 . Allocation of Functions One result of the systems analysis of the simulator PMS was to identify functions that have to be performed. Given there may be an option as to whether any particular function should be allocated to the human or a machine, some knowledge of the relative capabilities of humans and machines would be useful for determining the allocation of functions. Some relative capabilities among mechanical devices and h'oman observers were discussed in the previous section but more detail is necessary. Using the results of iMcCormick [1976] Buckhout and Cotterman [1963], Obermayer Angell, et al. [1964], and Coburn , [1973], et al. [1974], the capabilities of humans and machines for the purpose of performance measurement in the simulator are presented in Table IX. 150 , . TABLE IX: HUI'4AN . . AND MACHINE CAPABILITIES AND LIxMITATIONS HUMAN CAPABILITIES Detect stimuli against background of high noise (CRT) Recognize patterns of complex stimuli (DVRI) Sense and respond to unexpected events. Store large amounts of diverse information for long periods. Retrieve information from storage (with low reliability) Draw upon experience in making decisions. Reason inductively, generalizing from observations. Apply principles to solutions of varied problems. Make subjective estimates and judgements. Develop entirely new solutions. Select only most important events for sensing inputs. Acquire and record information incidental to primary mission. High tolerance for ambiguity, uncertainty, and vagueness. Highly flexible in terms of task performance. Performance degrades gradually and gracefully. Override own actions should need arise. Uses machines in spite of design failures or for a different task. Modify performance as a function of experience. MACHINE CAPABILITIES Sense stimuli beyond man's range of sensitivity. Apply deductive reasoning when classes are specified. Monitor for prespecified frequent and infrequent events. Store coded information quickly and in quantity. Retrieve coded information quickly and accurately. 151 TABLE IX (Continued) MACHINE CAPABILITIES (Continued) Process quantitative information. Make rapid, consistent, and repetitive responses. Perform repetitive and concurrent activities reliably. Maintain performance over time. Count or measure physical quantities. Transfer function is known. Data coding, amplification, and transformation tasks. Large channel capacity. Not influenced by social and physiological factors. HUMAN LIMITATIONS Sense stimuli within a limited range. Poor monitoring capability for activities. Mathematical computations are poor. Cannot retrieve large amounts of information rapidly and reliably. Cannot reliably perform repetitive acts. Cannot respond rapidly and consistently to stimuli. Cannot perform work continuously over long periods. Requires time to train for measurement and evaluation. Expectation set leads to "see what he expects to see." Requires review time for decisions based on memory. Does not always follow an optimum strategy. Short-term memory for factual material. Not suited for data coding, amplification, or transformation. Performance degraded by fatigue, boredom and anxiety. 152 . TABLE IX (Continued; HUMAN LIMITATIONS (Continued) Cannot perform simultaneous tasks for long periods. Channel capacity limited. Dependent upon social environment. MACHINE LIMITATIONS Cannot adapt to unexpected, unprograramed events. Cannot learn or modify behavior based on experience. Cannot "reason" or exercise judgement. Uncoded information useless. Inflexible. Requires stringent environmental control (computers) Cannot predict events in unusual situations. Performance degraded by wearing out or lack of calibration Limited perceptual constancy and are expensive. Non- portable. Long-term memory capability is expensive. Generally fail all at once. Little capacity for inductive reasoning or generalization. 153 . When fully exploited with no other limitations imposed, these capabilities and limitations of the human and machine define what might be described as an "optimal" performance measurement system from the engineering standpoint. As Knoop and Welde [197 3] observed, a performance measurement system should "capitalize on the advantages of an automated, objective system and yet retain some of the unique capabilities afforded by the human evaluator." For each task which is to be measured and evaluated, a decision must be made as to whether it would be more efficient for the man or the machine to measure or evaluate performance on that task [Buckhout and Cotterman [1963] 3 . System Criteria In addition to examining human and machine capabilities and limitations for the functions and components of a performance measurement system, other factors with potential impact on system design and implementation must be identified, analyzed, and weighed for importance. Choosing a system solely by human-machine advantages does not take into account other apparently extrinsic influences that may turn out to be deciding factors. The following listing of system criteria for performance measurement systems was gleaned from research by Obermayer and Vreuls [1974], Buckhout' and Cotterman [1963], Demaree and Matheny [1965], Farrell [1974], and Carter [1977]: a. Conflicts of system purpose may exist. is required to provide objective, 154 The PMS reliable, and valid information for decision-making purposes and to also identify changes in student skill level. A component may be the most objective choice available for the first goal but inadequate for the second. An alternative component or function may be identified to do both satisfactorily. b. Data should be provided in a useful form for evaluation purposes. c. Data collection, processing, and presentation must be timely enough to enhance the training process in the form of knowledge of results. d. Costs to modify or supplement equipment and software must be weighed against the utility of the information derived. e. Data distortion must be controlled for accurate and reliable results. f. Minimum interference with the training process should occur with the measurement system having an inconspicuous role requiring little or no attention from the student or instructor. g. Social impacts of any system may have adverse effects on morale or personnel involvement. If an instructor perceives that automated performance measurement is a replacement to his traditional role as an evaluator, the effectiveness of the measurement system will be greatly reduced. h. Economic and political constraints may affect system design. The ideal measurement device may not be 155 recommended for procurement by higher authority, while some selection of components is based on available equipment at time of procurement. i. Other factors such as size, weight, safety, ease of use and reliability should also be considered. Obviously, system criteria should be used in the sense that selecting components and functions for a performance measurement system would maximize those criteria that are advantageous and minimize those aspects that are not optimum for the system purpose. These criteria must be taken into account during allocation of functions for the performance measurement system, and must be weighed at least qualitatively if not in a quantitative sense for overall contribution to the final system configuration. 4 . System Implementation Once the objectives of the performance measurement system are identified and the allocation of functions and system criteria are applied, the system model then requires a deliberate implementation procedure if it is to produce meaningful results and have utility to the end user. et al. [1975] Waag, identified four phases of development for implementation of the measurement system in the simulator: (1) Definition of criterion objective in terms of a candidate set of simulator parameters. (2) Evaluation of the proposed set of measures for the purpose of validation and simplification. 156 Specification of criterion performance by requiring experienced instructor aircrew to fly the maneuver in question. (3) Collection of normative data using students as they progress through the training program. (4) When using automatic or human measurement components within the performance measurement system, other implementation considerations should apply. a. These are discussed below, Automatic Measurement Considerations In simulator environments, the organization of the software will be the key to successful implementation of flight training measurement systems [Vreuls and Obermayer, 1971] . Extensive research into programming techniques for the automatic monitoring of human performance in the flight regime has been accomplished [Knoop, 1966 and 1968; Vreuls and Obermayer, 1974; Vreuls, et al., 1973, 1974, and 1975]. Knoop [1968] examined some of the prerequisites for automatically monitoring and evaluating human performance: (1) Knowledge is required of which performance variables are important in evaluating an operator's proficiency, (2) Knowledge is required of how these variables should be related for optimal performance. (3) A digital computer program is required which compares actual relationships among these variables during performance with those required for optimal performance to evaluate operator proficiency. These prerequisites point out the need for careful front-end analysis of the system in terms of performance measures and criteria, and the complex problem involved of programming this analysis for automatic measurement and evaluation. 157 b. Human Measurement Considerations Smode [1962] provides some rules for enhancing the validity and reliability of resulting measurement where human observers are employed in data collection; (1) Provide standardized checklists that specify what to observe, when, and how often observations are recorded. (2) Train observers for the measurement process to insure full understanding of how to make the observations. (3) Provide data collection sheets that conveniently indicate what is to be observed and the sequence of observation. (4) Avoid overloading the observer with too much simultaneous observation and recording. (5) Data collection forms should have notation or symbology for recording observations, when feasible, and should result in a permanent record that can be easily transformed into a form for rapid analysis. These guidelines still appear sensible today, with perhaps some additional information about the relationship between automatic measurement and the human observer being established and provided within the system. C. CURRENT PERFORMANCE MEASURExMENT IN THE WST Navy B/N replacement training is conducted by both the East Coast FRS, Attack Squadron Forty-Two (VA-42), and the West Coast FRS, Attack Squadron One Twenty Eight (VA-128). Each FRS has developed and maintains its own training program; however, these programs are similar in nature and utilize virtually the same performance measurement and evaluation techniques. Each training program is divided into specific 158 phases designed to develop certain skills such as navigation, system operation, or attack procedures, and each uses the mediums of classroom lecture, simulator, or actual aircraft flight. A building-block approach to developing progressive knowledge of skills is utilized, including training missions in the WST. This study will focus on a Category One (CAT I) B/N student, where entry skills and knowledge for measurement in the WST are minimal. Determination of the skill level of CAT I B/Ns performing simulated missions in the A-6E WST continues to be based on subjective judgements made by instructor B/Ns and pilots. A syllabus of progressively more difficult flights in the WST is part of the training curriculum. During or shortly after each flight, the instructors "grade" the student on tasks performed employing mostly personal criteria, based on experience and normative comparisons of the student's performance with other student performances. Table X is a compilation of tasks taken from B/N flight evaluation sheets for the VA-4 2 simulator curriculum for which B/N performance is graded on a scale using four categories: average, (3) average, and (4) (1) unsatisfactory, above average. (2) below A typical B/N flight evaluation sheet for the WST is shown in Figure 6. The four ratings listed above are then converted to a 4.0 scale for numerical analysis and student rankings. During a personal visit by the author to the VA-42 A-6E WST in June 198 0, subjective performance measurement 159 TABLE X: CURRENT B/N TASKS GRADED IN WST CURRICULUM Aggressiveness Aircraft familiarity Aircraft system knowledge - NATOPS Aircraft/system operations Aircraft/system turnup Aircraft/system utilization ALE-3 9 awareness ALQ-12 6 awareness ALR-4 5/50 awareness AMTI Approach Approach (TACAN, GCA, ASR) Attack procedures/ACU knowledge Attitude Basic air work BINGO procedures Communications Computer operation Crew concept Degraded aircraft/system operation Degraded aircraft/system utilization Degraded system CEP Degraded system utilization Departure Departure procedures ECM equipment knowledge ECM tactics Emergency procedures 160 TABLE X (Continued) Flight briefing Fuel management Full system utilization General attack CEP Glideslope control Headwork HI Loft attack CEP Impact accuracy Knowledge of the cockpit LABS attack CEP Landing transition Line-up corrections Low level navigation Marshall pattern Mining procedures NATOPS Navigation procedures NORDO procedures Normal procedures Planning/preparation Point checks Post landing/shutdown procedures Post start Prestart Radar interpretation Radar operation S-1 pattern S-3 pattern 161 TABLE X (Continued) Shipboard procedures SRTC utilization Stall series Start Start/point checks Straight path attack CEP System shutdown System turnup Takeoff /departure Target procedures Targets (geographical turn point listed) Turn point acquisition UHF communications Use of checklists Use of the clock 162 ; . B/H FLIGHT EVALUATION SHEET >>> <<< BCWOl REPLACEMENT BUNO:. DISPOSITION CODE:_ INSTRUCTOR FLIGHT TIME: NIGHT TOTAL_ T/0 BRIEF TIME DEPARTURE PROCEDURES 2. 7. MARSHALL PATTERN APPROACH GLIDESLOPE CONTROL LINE-UP CORRECTIONS COMMUNICATIONS NORDO PROCEDURES 8. BINGO PROCEDURES 9. EMERGENCY PROCEDURES 5. 6. 10. HEADWORK 11. CREW CONCEPT . . . . • • • • • : ; : • • « * * • • • • • a • • • « . " m * • • * * * • * • • * • • '• • • • i Figure • * * * • • • i ; • • : I ', DATE: 6. Typical B/N Flight Evaluation Sheet for WST 163 2 I COMMENTS SIGNATURE • • * • • T OTAL AA 3A UN 1. 4. DEBRIEF Tir4E ITEM 3. INST_ and evaluation was exclusively being conducted for training missions of students in the V7ST. The use of subjective performance measurement and evaluation was due mainly to the newness of the simulator and the traditional and acceptable role that subjective methods have played for the last three- quarters of a century across all aviation communities. Oper- ational FRS personnel rarely have the time to carefully analyze and employ new performance measurement models and techniques or utilize new systems that are incorporated into a newly-delivered simulator. One purpose of this thesis is to eliminate the gap of carefully analyzing and evaluating performance measurement models for operational use. Due to the exclusive use of subjective performance methods, standards of performance in the A- 6 FRS are established analytically based on perceptions by the instructors on what constitutes "proficient" or "skilled" performance. and radar target identification (RTI) Bombing criteria are used internally, with RTI grading based on target difficulty and the replacement B/N's level of exposure and experience. Essentially, RTI criteria use a trinomial division of "hit the target," "in ball park," and "out of ball park" that has been defined well enough to be converted to a numerical grade In addition, the replacement B/N must identify 75 percent of the assigned targets on a specific radar navigation "check flight" as a criterion for radar navigation skill. 164 The proposed model for measuring 3/N performance during radar navigation in the WST, to be discussed in Chapter VII, incorporates the best qualities of both subjective and objective measurement, and uses the results to provide accurate and valid information for making decisions about student progress within the training process. Under the proposed model, criteria for successful performance will be established empirically for operational use by either the A- 6 FRS or the A- 6 community as a whole. 165 VII. RESULTS AND DISCUSSION The purpose for designing a performance measurement system for the B/N during radar navigation in the A-6E WST was to provide objective, reliable, valid, and timtely performance information for accurate decision-making to the training mission instructor and FRS training manager. This section presents the performance measurement system model developed from the previous analysis of related aircrew performance literature, generic performance measurement systems concepts, the B/N radar task analysis and MTLA, and the A-6E crew-system network model. The model developed is specific from the standpoint of identifying what to measure, when to measure, scaling, sampling frequency, criteria establishment, applicable transformations, observation method, current availability in the A-6E WST, and the accessibility of the measure if not currently available. The proposed model embodies: (1) the establishment of standards of performance for the candidate measure set by utilizing fleet-experienced and motivated A-6E aircrews performing well defined radar navigation maneuvers and segments, (2) techniques for reducing the candidate measures to a small and efficient set by statistical analysis, (3) evaluation methods which use the results from established performance standards and performance measurement of student B/Ns for decision analysis bv the FRS instructor and training manager, and 166 (4) some . , performance measurement informational displays which present diagnostic and overall evaluation results in a usable and efficient format. A. CANDIDATE MEASURES FOR SKILL ACQUISITION Using the candidate performance measure metrics derived from the 3/N task analysis research (Table II) , a (Appendix C) and previous aircrew composite list of candidate measures for B/N skill acquisition is presented as Table XI. Informa- tion is provided for each measure in terms of the method of measurement, measure segment, scaling, sampling rate, criteria establishment, transformations, availability in the A-6E WST, and accessibility in the A-6E WST (if not available) . Each of these terms is defined below. 1. Method of Measurement Either electronically or both. (E) , instructor observation (0) The primary basis for the determination of the best method was both measure selection criteria (Chapter IV) and human and machine capabilities and limitations (Table IX) 2. Measure Segment This is the period of time or segment of flight in which the measure should be observed. "ENTIRE LEG" defines the radar navigation segment from TP to TP, "MISSION" defines the segment from takeoff to landing, and "TP" defines within one minute of approaching the TP. 167 » X ^ ^ o 2 en en •H M OJ rH +J u-i fU O •H M C C -P 3 U S (U g U-) U w c: ^ 4J K '- G cn fal DU ^M 0) Q) •H r!3 Cm o IT) T3 en \ rH 1 o 2 w "J-i 0) o C •H en en (0 •H rH \ O en en -H fa •H s '- s w D U — H CO V S o G j^ 73 fa*" 2 2 H 2 ^^ \ X &I W •H -H Pi r) "V o o 2 « fa Oi o H Eh 05 X o CU O \. ff; rH CU rH < C •H C -P -P C (C 1 \ •HOOT) 4J S-4 a 5-1 -H cu -p nj fa X u G 73 •r^ (^ U G MO fa a: D ^ G 3 1 (U g g -H -P -P fd a) a r-H w M •H -P -- (U 0) CO v^ g C7^ a (U H \ H 1 o G 04 2 fa a nj en to !^ D U -H "-' H a-P 73 •H -H 73 G +J (U rH -P a-H c w w C 3 1 g g 73 U x ^ 12 en -P M e +J G C 0) en en U H .H (d 4J M Cr> CU-P en (13 0) -H (D -^ 0) -H g G -H 3 4J 1 g (1) w en CD o o o 5h rH rH u OJ en \ H •H -P G W u-j 1 X ^ 2 H 2 X ^ ^ o 2 u 01 2 a* s fa O -P fa X 2 ^ 2 H 2 •z o H e- E-t 2 O 2 fa fa fa Q z S O 3 :d < < o W 2 S cij a: CO CO £CJ CJ CJ CO r^ 168 < CJ oq fa E^ Qi Di DJ ^4 z M J Q. \~* m fa J J J a hca 2 < CO CO fa J oi h^ fa 2i u u < < £-• u :imes sent DATA r ^ 0) 4-1 ^ fC o a c ^ ^ S -H 0) ^ g a 2 0) re (u T3 13 +J O Q Q ^ +j -P C 5 w -H W a. nj (u V^ W O 04 w ^ o o 0) in w •P G cr; W s a. CO = \ rH 1 o ^ &. 2 W 0) rH U T! T3 <D 0) -P s X V a w « d" C7> a !-i u (U w fa 4J .T! i: M •H cn +J -H fd 4-4 o C7» 0) M j:: +J -H -H no 4J +J rd CO >i ^ o &a o in OJ en \ rH !-l 0) cn J-i 3 •H S^ Q) Q) Cu -P C e e +j s 3 -H C 3 4J (U O 2 o o w 1 o cu s w 2 X V 2X h-^ cn < a 2 M-i rd ^ +J OJ X da mput ion ent rH g a w C M W O u +j -H a ^ 0) S -H CU -P i-l •H +J 0) rH \ 1 rH o a, S a w c c o" X 2 *» r-T H 2 c -P -P Q) rH cu -H '0 +j S-1 i^ S Q) IT3 en = o ^ D^ r 0) (0 4J 0) -P p g ^«W fT3 g O 2 W ^s W W •H rn a -P -P r^ U (d OJ en 4J O Di >iQ Eh = G -H C C W 3 Eh -H -H S g o ^ U "-'^ w E^ cn = w <U o o o ^ rH -H ffi 0) OÔl ^ E^ * W 2 OJ '^ 0) en \ rH •H •P G W CL, s a 1 o u Eh Ou Eh X U u o < 2 -P c < X 2 ^ ^ in o Q O E EH w s z U s S 2 z 3 D u CO w s < <: o cu W CJ s: X m 169 w ^ <: u en O 2 M < M hJ CIh CJ S§ C.T CX a; CJ E^ M Qi U 2 O M E^ ^ 2 2 o fa en 2 ^ C-i >> fa .J Eh J m 2 < fa fa Di M Qi < D > CJ < fa fa aa M en en fa u 'J < » t IM u -P T3 a •H 4J 5-1 a a en OJ rH nj « rH w t7> CO) a. (u M -H a. E-t -H CD M rH -p Oi D^ 0) -H G e G a E-t X ^ 2j Cn (T3 C 1 o \ .H W CO Eh W n3 rH OJ ^ }-( fd .H 4J E-t -H CU c -> W (U (T3 a. +j iH T( G S-l (T3 w Eh G O a W ^3 rH (T3 Oi < w 0) c cc; -U cn (t3 iH s H s w s u OJ CO) e c •H p U (U ^ ^ rH en X cu S w 0) ^ H 4J a G g G W --U U OJ cn o o o O (U tn \ rH '-\ 2 cu s w 1 o 5x >» X ^^ 4-1^ S -^ en = wq W= Q « >iO W U S W >iS X5 e/3 cn J < 0) < o G 0) Eh Eh Eh 3 < H W -H VJ &4 en Qo "^ = rH Si rH D O O S -M <y o o Q 3 ZQ>rt:--'-Pt.Q -P ^3 OJ 73 r^ o o O Q) in QJ w n3 •H M-l 0) (Ti -r^ en \ >-J +J ':i G 5-1 W 03 1 iH o 0^ s w 2 -O W H 2 EH « -O Ol cu W O « oi X fe a. 4-) u td -P 0) G d G Oi-H en S -P G-H O 4J (d Q M-J ^ -p a U ^ g a rH U u <: -P en ^— >i 0) 4J 3 rd 'Z >A en G G ^ O 0. -r^ --^ (D w Q) X! 2 O H cn en V4 •H rH \ o cn \ rH 4J o Eh oi 0) G cu 2 a X Qi ci Cm a. W "^ -O O, O o o 4-1 C 2 O M U Eh Dl. X! CQ < Eh o Q O ~ H U3 S 170 E- 2 u s W CJ Qi a: D < W S e/2 CJ )Cj* 3 < w S e/3 o < M z M Eh z u S O w .J e/1 'Si '^ H Cd hJ >-5 Eh OQ 2 < as J CQ 1— en en CJ u U < < 2i CD ^ G OJ •* H Q) -P -H w s M -P -M Eh > tT3 cn D^ TJ -H 3 -P rH W *+-! CQ C 05 -H fd +J (T3 O, G (D 0^ TJ E^ H H Di > Q Q) Oi 6 Q; > ^S fi dJ -r^ W a, Eh 4-1 O CT> W G G W C7>-H XI G -H t3 g M W 3 3 03 OJ +J G ftJ ^3 a M •H OJ M a o o U o in ^4 < Z O M Eh UO <: w W tf ce; TJ Eh 2 u CO 'H 04 OJ 2 03 \ O cu W rH 0) 0) • -H M-( 4-) rH rH tT> w < ( 03 \ tH 1 o X li. w s cu S w X *» a a ci t4 W ffi Pi V) < s en ^ fi3 V^ 03 W W (T3 cno^ G d Q) i2 G H •* cr> w G G (U -H T3 D T3 fC S-( n3 0) x) Eh 0) K T3 ^ •H CU E-t (T3 o 2 cn 0^ Eh o u o VD 00 03 \ rH 1 ^K Oi s a o >i M rH -^ ^3 3 rH i3 G O d 3 O CU G <C Xi 4J d O H D^CO G X -H Êh » (ti — CT" U a dJ w CI. Eh TJ litJ O ^ 4J ro 03 \ rH 0^ s w 1 1 O X ô. w s H Eh c Da S Z o M u Eh Eh z Dm O S W Q Oi <J* ClI < ^ 0. 0} C/3 u X u u u u z u o < M £-5 **— +J OJ a: X ^ < rH 13 W s O ::d r: en Eh CJ g| a W < < 3 M W M s s S 03 171 O z M CJ J U < >H hJ E- CQ u J 03 1— 73 CO W U u < t 3 o H Steer- G uble- g •H G g M -H o; -u E-" •H O W CO S W en >i c s: en -H tn 0) u M 0^ ^^ W C < u s S M cn -HO +J -H G 4J Cn +J -H s a; M-J — -4J OJ U (U O g W i cn \ rH 0) W ^ o. \ o 1 (D u U i-i g ^ z H 2 0) cn rH O M-i cn 04 Qi Eh -P CO 0) —M (U cn tH o Q) «k H M o g G O -H G M tT> cn -H X Clj G (U >i s \ o CM o « W O Oi rH g CO cn •H \ o / -H W Eh Oi 05 cn .H GO W \ o •H -P -u iH « E-t o W o -P o o o c r^ U •H w U o 0) rn Ji cn \ o^ 2 V H 2 H X <J Eh 2 s w <H X I o <-i 1 03 o -P td > •H te 5^ U M <C }-l W Ai o G t3 g -H CO Eh W 1^4 Eh -- 03 Oi Q) 0} 03 0} rH -H C4 -P Eh U o CU Cn -P -- O o o Q) CN cn >i OJ rM i-q cn cn \ rH 0^ 2 w 1 G 1 o Di c £h Eu OQ [ii X Cli z o M u X u 2 W M 2 Eh ^Z Z p. H Eh O ^ H '^ a; Eh -O U O 04 w < o w « « O Q o c Eh CJ s 172 Z U S 2 s < W s '«n £h rf U Eh Oi z D W CO S <: o u u S 'J^ M ^ J ez W w J a < hJ oi M Qi < D > U < CJ J 03 )— 'J5 eo a D U < 1 1 W 2 w 0^ Eh cri en QJ -P i ^ Eh Pi E-t 0-P(T3fdDfHi< = S (ua3M-p&4= o w J o g e S-l i2 ^-1 2 i-H (U O Jh -H -H ti rH Q) en \ o \ rH -P G 0) •HOo>coocn E-io<:^H-p-p W en a G cu tH 1 G O 0) en w D^ o T3 Oi -H g en Eh -H en Eh fC — G G "4-1 S 0^ (U Cn-H (0 rH rO 03 (U en a en 4J •H nj 1 J-i Oi a g B U M G -H Eh W Eh CU V W CO 2 2 o (X (U en a. Eh w 0^ Eh u o o Cl^ cn 2 \ LO + 0) w iH 2 M X 2 ^ *» X *^ U ^ u < 2 1 i+H < c 2 H u Eh 2 U o s u W Q Oi Qi O D E en CO Eh < <: U u 2 2 2 Eh ClH u K PU CT> (T3 OJ Oi 'oj 0) tT3 CQ X cr; = TJ X U X u <; < 2 nj = M 2 ^ rH cn cn -H cojUrHooo:!--3 2" a, o < H 1 U X < 2 1 1 1 X •ik o (i^ a. Eh -H 4J 2 H 2 1 X) •H CU < w S 2 W -i4 £^ Eh S-l s \ iH a. -P !Ti &. en en <n Q) o ::3 tjj 173 ej 2 »— E- Z W 2 o u M J H Sri Oi < 3 > u < M J 03 t— '-0 CO u u u < a< 5 fi OJ ^ U ^ ** (u •H rH -H CO <D 5 E 0) en 4J — cn •H rH ^ tT3 -p w ^^ xJ -^ 03 G ^ J^ Pi C W 4J 4-> 0) M a ^3 4J a3 c 3 Eh en U U tT3 CTi Q) Q) cn .H (tJ Eh en rH w s o rH w 5h o o o •H iH o c cn a, X w rH JJ w cu \ 1 1 o CQ 5x X -s w " ^ H O Eh S '^ >i 0) +J c c ^ s ^ -H G 3 S 4J rH t3 Q W >i O a C •H o oi (U td (^ <+-( i-i fO >-i M ^ 0^ a 3 fo g U T3 T3 O 03 a (T3 w u <: Qi Its -P 2 O H en 0) iH o ^ rH (D ^ j-i o •H 03 -H +J ^ 3 u a CT> OJ rH \ o \ rH Pi w cu O C Eh Pi o Cu o X Pi cu W a; 1 en < ^ u (U 0) 0^ 4J Eh g >iT3 QJ «4H s^ n3 -H oi 4J 4J u -H -H fi -U W M 3 U 0) en 03 0) -p rH c: •H 'H Q j:: cr 4J ^3 +J -p c 0) 0) T! -H i^ en -i^ M 0) C C (^ M o en 3 a )H o CTi U -H x: rH -H +J (T3 S-i tT3 •H T! Eh i-i »k w •H .H +J c -r^ "^ u o o o 0) w •H Q) •• w (U cn 1 1 s o cu cn \ Oi s w rH o 2 O M CD rH u en er" >i o 4J to -H rH T3 03 03 a i-l Pi 0) M •H 4J r-i QJ \ o \ Pi rH w rH o C 0^ W 3 X *». CT" a 2 § U u o < 2 CH Pi o cu o X Pi cu -P C 2 o M u Eh E-^ <j* z u o s Ct4 u qS ag o o D u CQ S a: en en CJ3 Eh Ci3 Cd CJ CJ en s: < < S s 174 DJ J <: u en < M hJ cu DJ ^3 en ci Pi DJ Eh t-t Di u 2 2 o en 2 s E- >• CJ CJ kJ CQ J J M 2 < en CO pq k4 Qi M CJ CX u u < < V r— •- ( C Q) SO M —W +J (T3 --I Xi M " CHrH ctjiGw^-iaj-Ho o o-H(u<uo-i->»^c: •H > e (0 3 s-i CcjUWfaO^WH (U rH w ^ >-^ o cu w •H o o o \ •P rH r-i U w G (T3 W 0) a. s w 1 1 o cnu 4^ (D -H 03 -H -P T3 T3 U2 -P G Eh Oh (13 a D 03 ff; ^ W X 2 H < Eh 2 (U -H J +J 3 Vh aou e X u uw ^ •. o o M -H o 2 O (D W \ H CD M S £ W D -H 2 -H OJ -r^ ^ w fl H 2 D- U X ^ ^ C O W W -H -p x: 0) Q) ^ -ptua^-pcuMc^'O S -H rH (U S 3-HC0O5-ICOU OOCrnJCUS-iS-P +j -— tj" -l-l 5-1 4J «. w 2 ^ MS-) l+-( -r-^ E-t \ o a U (T3 1 o Oh 2 W V 2 H 2 W O X 2 CM <: 3 X 2 5-1 ^ y-i M (U X! 5-1 0) Q) 13 3 rH o o o 0) rH U W e e D -H c-t C -P -P !-< a w (/) P G W oj S-l 03 0) \ 1 Oh 2 W -H o Q) QJ rH Qj cn G X ^ 2 rH tn s-j -HO 5 w e oj M M -H rH d 5h •H 0) Oj Eh 03 Cn a) w P G W U OJ rH \ 1 o 04 2 W X ^ 2 H 2 C 2 O H u Eh Ct4 O Q O < ^ O 2 <-{ "3 an X 2 2 M 2 •» C7> a en OJ M •H QJ 2 H 0) r-l •» O 2 a: &H w s 175 2 U S 2 §1 D D W en cn 2 < <: o u W CJ s S ui Eh DJ o u -3 H CU hJ E-i ca J 2 < W k4 M Qi < 3 > U < c:; J CO ^H 73 cn y u u < « ( T3 CD a) iH a, T) w c 3 u U O O •H <o Q) o cu en •\ cu S w rH 4J 1 1 o H S X ^ u x < 2 }H u u -p w 4H cn w •H 4i4 o o m Cm Eh o p^ en \ rH s w 1 1 (TJ o 5-1 S X >• w 2 Cl3 U 1 W iT> 4J C c; CO) •H ^ OJ 5 U ^ fd O S^ Eh CQ H C (d G >-i CP -P '-^ a I4H Oh Eh .H H -H (T3 C/3 -P 3 i-l tj -P --U M •H rn W -P c w d a en 3 e C C 6 +J U (K o -H S-4 S 0) !-l M-l 0) -H 0) rH — -H -H 4J -p -p -H -H W U tT3 >, U M +J 4-» 0) -H -H OJ >H -P g U3 -H -H a M U rH (u M-i cc;4J> ^ K X •» w 2 1 1 ^ w 2 ^ C G W W X w o ,— W +J +J 4J •H \ Oh ^ \ o •H Eh Q) cn <: +J 0) cn ui o o QJ CP CJ 0) Q) rH (J) O •r^ o o o -P rH o M ^ w G c Oh:>OUM W T3 ^ Q) cn \ 0^ s w rH 2 H 2 X « w 2 X Eh t 1 1 O c 2 O M u £h &^ Cm O Q O K H DJ X 176 z U S W Oi D . CJ hJ CJ hJ CO E-, h-l J CQ Z < cn CO CU J Ci w CJ ex u u < < ( w" 2 Ul 51 cu Eh 0) TJ C 4J 0) 0) •H 4.) 3 -H < w 0) -p rH U-l (U i4 S-i 5 O •H U3 -P 4-1 1 1 c 0) O 0) w \ Oh 2 a rH o w CQ ^ to <y (D (T3 M-l w 3 W TJ w Oh « +J -P C (u c td -H -H 4-> Iji M 4J nj fO S > 0) CO < > '-^ 0) >i rH rrj rH a u < -H S Q — w en M O O •H rH (U -p o Q) W \ 2 W rH 1 1 c 0* O w a 2 X *k u u < _ w cn ca c (0 a e U H 1 <d CI Jj Q) •H rH C 3 ^-^ -H 4J Q) o rH (D -73 ^ o; M-i OJ (U 0) M w M o •H c cd u z w o s w CJ Q 3i oi O 3 s a: W Ch < < M DJ W s r: 2 irf Ct, w c/5 < X ^ < C X a 2 u u 1 1 w S S Q W 2 rH 4J M M en 177 O 2 £-. z u z o Cd W u < >* vj a a oa M cn cn MM o u < 1 1 W 2 '-^ 2 +J CT>\rH (U g c U •H -H -H S M E-. ^ 4J (T3 6 -P (C C U-i 4J (U 5-1 -P (u 3 a Eh CO CU CU > cn 0) >H H £h H U ^ U w fO u kJ « P 1 M M w <S -H OJ QJ rH CO «. \ •H C > u -- (U CO U -P C >iW -p u Q) ce; (tj en w <-i Oi 2 W rH CU EH fO c 0) ^ S a oa H M-l -H 5^ -P CO M c a -H S-i 1 1 X 2 2 H W CH Pi 2 •w tjl (U Cl. rH S \ s 0) Tl -H SU M LT) -H CO CO OJ X ^ <J -H -P X < X 2 H 2 I 1 1 u-i ( a rH +J X -Oi s M a S CU 02; 3 U2 < g" OJ 0) e CO "Z ^ •H U Eh w rH &, 03 cu Eh OJ CO rH Eh 4J \ cu s w r-i a 1 Eh C j-i •H CO ^ D U ^ -H 5 U a (U Q) CO -P c nq Eh rH 1 1 \ rH a, 2 w w c Eh 2 w s w Q Qi sg D :3 w = CO CO S Eh < < CJ w u W DJ s p: S cn E-t X V 2 X H <; 2 2 Eh Dl, < a 2 2 M u X w 2 X S 2 «. U H X ^ 2 Q) CO rH -H 0) 4J x: -P -H -P -H g --H S-i -H Oi w 2 178 w — en S Qi U y 2 Du cn 2 Dj £- ui M J J CQ 1— t- CQ 2 < M J 00 a; w Ci u u < _^ >-' 1-; '/I "-^ w c P4 -H Eh x: +J "W -H V^ a S ^•H U Q) S-l Ti <D M U P c o ai O C e =s -H S IJ4 c (T3 Xi M U OJ -P AJ :3 ^ >i 4-1 Ln fH cn •H W W cn < •H s en Q) rH \ 1 o 04 2 w ^ c O 'C >i W (U c O -H 5 X ^ IS -P 14H (tJ n3 -P ^ D rH » a. Eh CO \ Eh l-l <: -p lOi o o -M M-l td 0) X H « 2 (^ &^ 1 td o z o -Oi rH -H cu U Q X <^ 2 2 O ^ H W Eh 2 « ^ ttJ -H Oi -' 4J <: Eh a. s w rH 1 W 2 X «» Z cn H 2 2 2 o a: 3 en < w 2 -P x: oj QJ n3 C^T3 3 (13 OJ -P G -P td S tt3 g D G &:. to w 0^ Eh M-l rH +J :d tn < CO a *» (T3 W U M •H w -H — S C^ ja rH d* rH \ « o rH -QJ CT" 3 0) Cti s w rH r-l + 1 J IW C -P •H (U -H W g M +j M a t3 01 g M -H Ph Eh rH o > 2 \ u a > a. < rH tJI cu w dJ U •H ^-1 +J s-l C 1 o rH \ Oi s H *. o 2 X rH » 2 M 2 w C z o H u £- E^ z Cu CJ O S CJ Q Oi O - 75 Eh < CJ u s r: ::> CQ X s o -cu z o M oi 2 cu X i 0) c c a X < 2 o z 2 O M 179 (rf o 2 M CJ Eh Oi Z D a 2 < C U CJ S a:) c/] CJ J il J J Eh a z < W .J -H Oi Oi < 3 > u < a3 J CQ »H cn cn CjJ u u < ( M en u § S-i (D Eh rH T3 0) T3 CD S-I o o \ -H en u w C M D u u u •H 4-1 c o w fa en ^ a* S *k U < 1 fa (U '0 •H -P V^ rO cu a. rH o o O 0) in en -H V-l fa 4-> a x: j3 (U 4-) en g g -H -H 3 -H M rH dj 2 03 Eh U ^ en 4-> C =5 O 2 £ W <+-i X 2 fa rH + fa !h •H 4J c 1 0) \ cu 2 fa rH X «k O S >^ oi <c fa o 2 fa fa 2 H 2 ^ en s «^2 CO C O S-l +J 0) x: -P g -H -H Eh rH -H oj •H cu nj CU i-) a o +J •H M U w a -H ^ a. £h +j u-( \ &4 2 W rH fa X - 2 X Eh 2 1 1 o O tn M u 3 M G o QJ n3 U r-\ •H en fa -u <; en •H en 3 ^ Q Ô V c -H 4-1 ^ O o 5 ^ a: X ^ 2 OJ -H -H M en >i u-l c; 0. 0) M -^ C 3 a rH c fa •H fa en -t-l H c: (^ •H M cr> en o o o OJ w T3 +j •^ Eh s -H a. H Cn OJ 1 O rH \. fa 0^ 2 w rH 2 * X 2 H 2 T3 c o 2 O M u C-l Eh 2 fa w O X Q Di o a X 75 Eh H DJ hJ vJ Eh 03 J J3 1— 2 < en en fa -S (X M fa (X u u < r' Da D CO < 2 ^ G 5 O-H i 4J c (TJ a. •H i^ 4-) x: en •H -H M rH 0) P u p4 5-1 4J M-l -H ^ CO -p +j a o u c Ph >i (TJ M 3 < rd 0) Oi fO rH (T3 -P ft3 OJ x: T! 01 3 Q O -' CT^ •H rH ^ to 1^ >i -P 2 M-l -^ ^ Vh -H 4J O en o 4J 0) w (T3 !-^ •H C 3 O O n 04 2 Oh H <; X -P c w I o <: -P fa CQ c o u fa o Q O CQ &^ fa fa fa fa CO 2 S 2 181 2 M fa < kJ fa a. u CO a Qi fa M E-. r-t ^:i X <c D > u < u u < CO fa , . 3. Scale Scale shows the numerical limits and units of the measure, e.g., seconds, feet, miles, or if units are unassigned, the values that the measure is assumed to take on over the "0/1" defines a dichotomous "not occurred/ measure segment. occurred" situation. (e.g., 4. Subjective scales are listed separately "effectiveness of communication"). Sampling Rate The sampling rate is the rate of measurement defined by time or by measure segment. Determination was based on research by Vreuls and Cotton [1980] 5. Criteria The recommended method of establishing performance criteria for the performance measure is defined as "EMP" for empirical, "OPER" for operational by fleet aircrew) 6. , (subjective determination or both. Transformation A recommended mathematical or statistical process for the measure is provided based on the literature review by Mixon and Moroney [198 0] . A caution is provided that the determination of the measure's distribution would be in order before applying any transformations, as previously discussed in Chapter IV. (frequency) , ACC Transformations are listed as (accuracy) , ME (of successes to total number) RMS (root mean squared error) , 1S2 , (mean) , MO (mode) MIN (minimum) or N/A TIxME, , , FREQ PROPORTION MAX (maximum) (not applicable; if . measured for computational purposes only) , Other transformations may be possible; see Table IV. 7 Currently Available The measure exists within the A-6E WST PMS in Table VIII or Figure 5. If blank, , as listed the measure is not available. 8 Accessible If not currently available in the A-6E WST PMS , a determination was made as to the feasibility of incorporating the measure into the existing WST PMS with a minor software If blank, major changes may be required in the WST change. PMS to facilitate the accessibility of the measure. It is recognized that the resultant candidate measure set contains redundant and perhaps overlapping measures but analytical derivation is necessary before empirical analysis is possible. Only "objective" aspects of performance were listed; the determination of subjective components (i.e., motivation) and their measurement is a subject for further research. B. ESTABLISHMENT OF PERFORMANCE STANDARDS 1. Radar Navigation Maneuver The radar navigation maneuver consists of multiple "legs" of varying distances between predesignated turn points or targets that are selected using radar significant criteria. Each turn point (TP) must be reached within a criterion distance at a predetermined time, airspeed, heading, and altitude 183 for the success of the air interdiction mission. are more difficult than others to detect, Some TPs identify and successfully fly over using the A-6E CAINS navigation system. Operationally, a TP is "crossed" or "reached" when the actual position of the TP passes more than ninety degrees abeam the actual aircraft position. This operational definition is independent of the DVRI moving bug cue used by the B/N, as based on cursor intersection on the perceived TP radar return. This operational definition of TP "passage" can easily be converted mathematically using Boolean functions for use in the A-6E WST performance measurement system. Appropriate radar navigation routes can be planned by either FRS or Medium Attack Wing (I4ATWING) personnel and programmed into the WST. This task is simplified by the current capability of the simulator to facilitate preplanned mission routes for training purposes. 2. Simulator "Intruder Derby" A competitive exercise is currently conducted on an annual basis using actual A-6E TRA^4 or CAINS aircraft that perform radar navigation maneuvers in both East and West Coast A-6 communities. A similar competition could be applied in the A-6E WST using the programmed radar navigation routes as previously discussed. Each route is then flown by fleet- experienced aircrew on a competitive basis under the cognizance of the appropriate r4ATWING command with performance measured by the proposed model. Using fleet aircrew that are carefully 184 selected by individual squadron Commanding Officers almost guarantees motivated and skilled performance during the radar navigation routes due to the intrinsic importance placed upon the competition results as an aid in determining that A-6 squadron which is the most "excellent" in each MATWING community. The performance results of the simulator "Intruder Derby" would be most useful for establishing standards of performance for each radar navigation route. Establishing standards in this manner for comparisons of performance to other groups is both feasible and operationally acceptable. The use of "ideal" flight path performance criteria lacks this acceptance criteria among operational FRS and fleet aviators, as "ideal" performance may not be achieved by even the most highly skilled aviator in a consistent manner. The fleet- established standards of performance would be carefully analyzed and performance limits set by operational personnel for those performance dimensions which are within or approaching the performance standards of the fleet. C. MEASURE SELECTION TECHNIQUES Various empirical methods and models have been formulated and applied toward reducing a list of candidate measures to a small, efficient set with the characteristics of reliability, validity, objectivity and timeliness. One technique, employed successfully by the Air Force for air combat maneuvering (ACM) 185 performance measurement, used univariate and multivariate analysis techniques to find the smallest comprehensive set of measures which discriminated skill differences in "novice" and "expert" ACM pilots in one-versus-one free engagements [Kelly, et al., 1979]. Using multivariate analysis, correlational analysis, regression analysis, and ridge adjusted discriminant analysis, an original set of twenty-seven candidate performance measures were reduced to a final set of sixteen measures that were: (1) Sensitive to differences in pilot ACM skill level. (2) Diagnostic of performance proficiencies and deficiencies. (3) Usable by instructor pilots and compatible with their judgements. (4) Capable of providing results immediately after the end of the engagement. (5) Compatible with current projected training and measurement hardware. These statistical analysis techniques appear to be appropriate for application to the measurement of B/N performance during radar navigation in the A-6E WST. Computer programs have been developed and are available at minimum cost for possible software alterations to the current performance system in the WST. D. EVALUATION METHODS Once a small and efficient set of performance measures has been derived using suitable statistical techniques, measurement and evaluation of student performance may then occur. 186 . The proposed performance measurement model uses the results of a student's performance com.pared to fleet performance as usable information for several decision levels. The task of operating the radar by the student on each leg of the radar navigation maneuver is measured and evaluated in terms of the underlying skill level of the student, leading to a decision of "proficient" or "not proficient" for that task. Decisions on quality of performance must also be made for global indices of navigational skill, e.g., managem.ent. fuel management or time-on-TP Students just learning the task are expected to be "not proficient" when compared to fleet performance on the same mission whereas students near the end of scheduled training are expected to meet fleet standards. An evaluation technique proposed by Rankin and McDaniel [1980] that uses a sequential method of making statistical decisions has the capability of utilizing both objective and subjective performance results for more accurate and potentially less costly training evaluation. This decision model focuses on proportions of "proficient" trials, where "proficient" is determined by the instructor using either subjective evaluation or objective standards established prior to performance. The model sequentially samples performance during the training of a particular task or maneuver and uses the historically sampled performance results to eventually terminate training for that particular task or maneuver. 187 Figure 7 illustrates the proposed sequential sampling decision model using the task of navigating the A-6E aircraft to a radar navigation TP as an example. For this particular radar navigation route, eight TPs must each be navigated to within an empirically established criterion radial distance by the student. The instructor must use the performance measurement results to evaluate the student's actual performance on each TP and assign a "proficient" ficient" a trial. (1) (P) or "not pro- score for each TP, where each TP is considered The figure shows "proficient" (P) trials plotted against total trials and indicates in this example that three TPs were successively and accurately navigated, followed by a missed fourth TP and ending with the remaining four TPs successfully navigated. The regions of "proficient," "undeter- mined," and "not proficient" are derived statistically; more detail on their actual calculation E. is presented in Appendix As can be shown in the figure, the student has "mastered" the important task of navigating to a TP on his eighth trial. This information can then be used by the training manager in deciding whether the student has actually mastered the task and needs to progress to more difficult tasks or the student has not mastered the task in a previously established, statistically-based number of training trials and needs remedial training for that task. For this example, the training manager could safely determine that the student is ready for training on tasks other than accurately navigating to a TP. 130 ^"^ 00 a; CO 03 rH .•^ Eh •H (ti !^ •H U Eh (U 0) U C 0) Q) C CT •H Q) fU 03 &4 0^ Eh O .f^ •H • cu • Oi 4J i-H C cu Ot a. OJ •r4 4J <0 CN •H > U M •H &H M-l 1^ Eh 5-1 O t en a\ U O M-l iH •H -' Eh U 04 189 This sequential sampling decision model has been previously used in educational and training settings. [1969] Ferguson used the sequential test to determine whether individual students should be advanced or given remedial assistance after they completed instructional learning modules, and Kalisch [1980] employed the model for an Air Force Weapons Mechanics Training Course (63ABR46320) conducted at Lowry Air Force Base, Colorado. Both applications resulted in greater test efficiency than for tests composed of a fixed number of items and substantially reduced testing time. As discussed in Chapter IV (Table V) , the resulting costs associated with training manager decisional errors predicates that statistical and systematic methods be employed to measure and evaluate student performance. The sequential sampling plan accomplishes this by fixing the error rates as previously discussed in Table V) (Types I and II and allowing the number of trials to vary according to the performance demonstrated by the student. This evaluation decision model is currently being integrated into the training program of a helicopter FRS that uses only subjective determinations of "proficient" instructors [RanJîn and McDaniel, 1980] E. . INFORMATION DISPLAYS This section discusses some displays proposed for use by the instructor at the A-6E WST console for the purpose of performance evaluation. Several classes of performance measures 190 Figure are represented. 8 shows a time activity record for the B/N's interactive control of the A-6E CAINS navigation system during one leg of a radar navigation maneuver. This type of display tells what, when, and for how long a particular From this, one may infer what equipment was being operated. particular task was being accomplished at during a radar navigation leg. performance may be used as a a particular time Time activity records of fleet performance standard by simply preparing a transparent overlay to show means and ranges of activity by the fleet performing the same radar navigation leg. This comparison provides diagnostic information for the individual student in regards to efficient or appropriate operation of the complex navigation system of the A-5E. Specific tasks may be measured directly and displayed as shown in Figures 9 and 10. The tasks of time and fuel management are measured in the simulator by comparing planned time and fuel values with actual time and fuel values at each turn point, based on the premise of facilitating the input of the student-planned values by the instructor into the simulator computer prior to the simulated mission. Trends may show decisional errors not otherwise detectable by an instructor. Again, fleet performance standards can be compared by using simple overlays to an individual student's performance, as well as the performance of other students, to facilitate evaluation. 191 CTi 1 G a -H e M 5 u •H ra t^ j:: iH CN O in iH ro <-{ o LD rH • • • • • • r- -g* en o« u 1 1 rH • 1 1 1 1 (T>i • co- • i • [^- i •H - 1 ' >1 1 • • (0 pH C4 I Eh i 1 Cfl C7> •H 1 Q (U t-q "^ - 'T3 ' u n- i 1 i 1 1 CN p>1 - •H > CD •H r-{ « -P 1 1 t 1 i rH G OJ •H C a -P a. 03 U J >1 C O •H cn > H 4-> •H S O < O o u -P -P f a 3 o C CO C i •H rO +J rH (TJ a G i-1 +J 0) •H U U 4J •H 3 O •H T3 G 3 S Q Q U O 5 ttj iH OJ ;>i Eh rH •H tt3 D M 73 iA rH C •H ^ ^-1 -p •H 1 rH +J O < • • OJ w C >i ci. c 5 (D c-) 192 rH (U > g CO 36 30 Fast 24 18 12 Planned minus actual time 6 (sec) 6 -6 -12 -18 -24 Slow -30 -36 Figure 9. Time-on-TP Management Display, 193 8 Leg 3000 t 2500 Over 2000 1500 Estimated minus actual fuel 1000 . 500 . (lbs) r -500 • -1000 • -1500 - 12 I r 3 t 4 r 5 -2000 Under -2500 - -3000 . Figure 10. Fuel Management Display 194 r i 6 7 .eg . . Figure 11 shows the usefulness of displaying the results of an overall measure of performance: acy. radar navigation accur- The solid boundary lines between and surrounding each TP are performance standards established by fleet A-6E CAINS aircrew; in this situation a 90 percent confidence interval Student has been constructed about the mean flight path. navigational accuracy over the entire mission may be evaluated from this display as well as diagnostic navigational informaThis figure illustrates the perfor- tion for each leg or TP. mance of a student who has met fleet-established criterion limits for all leg and TP navigation portions of the route except TP number two Summarized performance of the entire mission is depicted by Table XII. Each turn point (TP) is evaluated for B/N equipment time-sharing activity on the previous leg, time-onTP management, fuel management, heading accuracy to planned run-in heading) as a "P" , (as related navigational accuracy (reported if within criterion limits or reported in miles from TP if not within limits) , minim.um altitude for the previous leg, and indicated airspeed (IAS) at the TP This brief discussion on information displays for B/N perfonnance is not exhaustive and does not reflect what may be the most efficient and reliable measures for determining skill acquisition. Only statistical methods will produce those measures that should be displayed and used. presented here are for illustrative purposes only. 195 The examples >1 71 •H Q >i O (tj M 3 O O < (T3 •H > TJ 3 •H 196 en < M -P -H < m LD LD en in ro o o rH <N in CN > O VO m in ^ m s • CJi ro o r^ H o O o <T» rH CN CN in (N a. a. &. a. O >^ ro O ^ in in ro o <Ti in in ro ro in o (N rH ^n =^2 ^ 00 1 m o E-t iH rH 04 CO 00 ro rM 1 1 o H in CM CN CN CN ro ro in in rvi CN 1 1 1 1 1 1 1 1 rg in VO CN ^ o CN C» in CN CO • r^ H 1 Eh H H H H H H 1 H 1 * U3 >1 ID rH CN H r^ rH in rH o H rH o O H H • • • • • • • • • ro CN +J •H > in •H o \D in (N rH OJ Pu • CN CN o. (7\ • a: B C o ^ rH T3 •H '^ ro r-i Oh <n Cn O O • • in (N CN) r-- ro • • in o • o H O CN CN ro CN O H O • • • • • • • • H o •^ O O O 4-> < T3< H H H o H O o H o o o o CO o <o r-\ ' • • • • t • • • • ^ ro o cn C •H M (^ 03 ro H in CN rH H H O O o o o • • • • • • r^ o H H o X r^ • • • • CM o x: w o (y\ >^ CN in (Ti ro rH ^ o ro CN t • • * • • • • • • ^ rH CN rCN CN CN ^ H r~ un CN H (X) ro H ro CN ^ o • • « • t 1 s o O •H Eh M H H CN o O O • H H Q) G x: < 04 0. Eh Q) -h )-( 03 H (TJ C^ Eh 0^ Eh K &• (t3 rd H Oi Di i2 >H 04 04 0) 4-> H H Q) 5 QJ +J H •H 03 o -P U e > c •H •H (X( •H ITJ (T3 rH U CN > ro x •^ 3. r-i cu Ci Q in 197 en H M TJ •H UH x: rH g 2 CO -u e 02 V£> e 'h cn K > CQ U2 r^ D en TJ Oi-U fO J O M ^ H a O C w (U U CuC 4J (T3 >,Jh >rrM (U +J Sh C -U ^ O 03 H 4J 03 G M >.a.u u 1) -P cn-H >i T3 03 03 -H-H C 3 O'd 3 5 Q4O E g H QJQ H CN ro •^ in U3 OJ Eh 5h 03 Cl4 00 X O 4J gHD Ul U>QU F. IMPLEMENTING THE MODEL A model for measuring B/N performance during radar navigation in the A-6E WST has been designed and proposed for use by the East and West Coast A- 6 Fleet Replacement Squadrons. The final model design was predicated on: the model at minimum cost, (2) (1) implementing utilizing existing computer algorithms and software that have been validated, and (3) requiring no additional personnel to operate the model after implementation. Some software changes are necessary, but they appear to be minor in light of the 2F114 design specifications for currently accessible programs. Some translation may be necessary due to different computer languages but these are feasible alternatives given the implications for reducing training costs and increasing the effectiveness of both the instructor and the simulator. Additionally, a computer-managed system will be necessary for implementing the sequential sampling decision model. Currently available desk-top computers could accomplish this function assuming the simulator's computer capacity was fully utilized after the measurement portion of the model was installed. Objective performance measurement provides useful information necessary for training evaluation and control. Per- formance measurement models that incorporate this powerful technique can increase simulator and instructor effectiveness, reduce training costs, and may contribute toward reducing accidents attributed to "unskilled" aviators. 193 Implementation of these systems appears to be cost-effective in view of the potential savings from their effective utilization. 199 VIII. SUxMMARY The purpose of this thesis was to model a performance measurement system for the Bombardier/Navigator (B/N) Fleet Replacement Squadron (FRS) student during the radar navigation maneuver in the A-6E V7eapon System Trainer (WST, device 2F114) that would best determine student skill acquisition and would incorporate the advantages of both objective and subjective aircrew performance measurement methods. This chapter is provided as a compendium due to the extensive material covered and assumes reader unf amiliarity of previous chapters. A. STATEMENT OF PROBLEM Traditional and current FRS student performance measurement and assessment in the A-6E WST by an instructor is mostly subjective in nature with disadvantages of low reliability, lack of established performance standards, and human perceptual measurement inadequacies. The recently delivered A-6E WST has the capability to objectively measure student performance but is not being utilized in this fashion due to the lack of an operational performance measurement system that incorporates the characteristics of objective performance measurement and still retains the valuable judgement and experience of the instructor as a measuring and evaluating system component. Objectivity in performance measurement is a highly desirable 200 component of the performance measurement and evaluation process that enables the establishment of performance standards, increases instructor and simulator effectiveness, and fulfills the requirements of Department of Defense policy. B. APPROACH TO PROBLEM Designing a model to measure B/N performance during radar navigation in the A-6E WST necessarily assumed: (1) the A-6E WST realistically duplicated the A-6E aircraft in both engineering and mission aspects, (2) little variability in overall A-6E crew-system performance is attributable to the pilot, (3) that results from pilot performance measurement literature were applicable to the B/N, (4) that a mathematical relationship existed between some aspects of B/N behavior and performance measurement and evaluation, and (5) competitively selected, motivated and experienced A-6E fleet aircrew exhibit advanced skill or "proficiency" characterized by minimum effort and consistent responses ordinarily found in actual aircraft flight. The methodology used in formulating a model to measure 3/N performance was based on an extensive literature review of aircrew performance measurement from 1962-198 tical task analysis of the B/N's duties. and an analy- After selection of the Air Interdiction scenario and turn point-to-turn point radar navigation flight segment, the review concentrated on aircrew performance measurement research which emphasized navigation, training, and skill acquisition. 201 A brief review was presented of the concepts of performance measurement and evaluation including measure types, reliability, validity, measure selection criteria, performance standards, aviation measures of effectiveness and types of evaluation within the framework of aircrew training. A model was then formulated to illustrate the relationship among student B/N skill acquisition, the radar navigation task, and performance measurement and evaluation. Candidate measures for navigation training and the radar navigation flight segment were identified from an original listing of 182 performance measures from previous aircrew performance measurement research. The task analysis was performed to identify skills and knowledge required of the B/N and to identify candidate performance measures for both B/N skill acquisition and the radar navigation segment. A Mission Time Line Analysis (MTLA) was conducted to identify B/N tasks critical to performance. A model was then formulated to illustrate A-6E crew-system interaction and the complexity involved in measuring B/N performance. Generic aircrew performance measurement system concepts were reviewed for the training environment. Current performance measurement and evaluation practices of the A-6 FRS for the B/N student in the WST were reviewed as well as the current objective performance capabilities of the WST. A final list of candidate measures was presented that had met selection criteria of face validity, ease of use, instructor and student acceptance, and appropriateness to training. 202 C. FORMULATION OF THE MODEL The purpose of measuring B/N performance during radar navigation in the WST was to provide objective, reliable, and valid information for accurate decision-making about B/N skill acquisition to the FRS instructor and training manager. model developed candidate measures (Table XI) The that determined what to measure, observation method, when to measure, scaling, sampling frequency, criteria establishment, applicable transformations, current availability in the A-6E WST, and accessibility of the measure if not currently available. After operationally defining the radar navigation segment, a proposal was made to conduct an annual competitive exercise in the A-6E WST under the cognizance of the appropriate Medium Attack Wing (MATWING) command utilizing A-6E fleet squadron aircrew. A-6E fleet squadron personnel would fly preprogrammed radar navigation routes while their performance was measured using the candidate measures previously developed. The results from the proposed competitive exercise were cited as being useful and operationally acceptable for establishing standards of performance for each radar navigation route since the selected aircrew would be highly motivated and fleet-experienced. Statistical techniques for reducing the initial candidate measures for B/N radar navigation performance were reviewed and evaluated with respect to skill acquisition, and a multivariate discriminant analysis model was selected as applicable due to the model's utility and previous practical development and applications. 203 The final part of the performance measurement model proposed an evaluation application which used dichotomous results of student perf oirraance compared to fleet performance as information for several decision levels. Performance results of the student for the majority of the statistically reduced performance measures can be dichotomized by the instructor as either "proficient" (unskilled) (skilled) or "not proficient" based on the empirically established objective performance standards or operationally-defined subjective performance standards. and iMcDaniel [198 0] An evaluation model developed by Rankin that used a sequential method of making statistical decisions incorporating the dichotomized results of student performance was adapted and modified for determining successful completion of task training for the B/N based on both objective and subjective performance measurement and evaluation. The sequential sampling model was selected due to its inherent power to fix decisional error rates, previous practical developments, and potential for reducing training costs. Sev- eral informational displays specific to the B/N radar navigation segment and based on hypothetical model results were presented. Model implementation was discussed with regards to personnel, costs, A-6E WST software changes, and effective training control. D. liMPACT TO THE FLEET The performance measurement model as outlined in this thesis is specific to measuring A-6E 3/N performance during radar 204 navigation but has generic qualities applicable to aircrew members of any aircraft. The advantages of objective measurement in the form of reduced paperwork, permanent performance records, established performance standards, diagnostic and timely information, and high reliability are fulfilled. At the same time, subjective measurements for those aircrew behavioral aspects that currently defy objective measurement are made using the experienced simulator mission instructor, who also remains as the final decision-maker on whether or not a student has demonstrated task performance that reflects an acquired skill for that task. The application of the model to individual aircrew readiness, fleet squadron unit readiness, and selection of individual aircrew teams for multi-crew aircraft appears to be feasible and operationally acceptable. The model has potential utility in tactics development, accident prevention, predictive performance, and proficiency training of reserve aviators. Almost certainly some reduction in aircrew training costs and an increase in instructor, simulator, and training program effectiveness would be realized. 205 , , BIBLIOGRAPHY "Methodology in the Use of Synthetic Tasks to Alluisi, E.A. Complex Performance," Human Factors v. 9(4) Assess p. 375384, August 1967. , , : Educational Psychology; The Anderson, R.C. and Faust, G.W. Science of Instruction and Learning Dodd, Mead & Company, , , 1974. Shearer, J.W. and Berliner, D.C., Study of Angell, D. Training Performance Evaluation Techniques U.S. Naval Training Device Center, Port Washington, NY, NAVTRADEVCEN 14 4 9-1, October 1964, AD 609605. , , , Bergman, B., and Siegel, A., Training Evaluation and Student Achievement Measurement; A Review of the Literature Air Force Human Resources Laboratory, Lowry AFB, CO 8 230, AFHRL-TR-3, January 1972, AD 747040. , Beverly, R.F., A Measure of Radar Scope Interpretation Ability RSI Test #1 Human Resources Research Center, Lackland AFB, San Antonio, TX, Research Note AO 52-9, December 1952, AD 7956. , ; Billings, C.E., "Studies of Pilot Performance: I. Theoretical Considerations," Aerospace Medicine v. 39; p. 17-19, January , 1968. Billings, C.E., Eggspuehler, J.J., Gerke, R.J., and Chase, R.C, "Studies of Pilot Performance: II. Evaluation of Performance During Low Altitude Flight in Helicopters," Aerospace Medicine V. 39: p. 19-31, January 1968. , Bilodeau, E.A. , Acquisition of Skill Bilodeau, E.A. (Editor) Academic Press, 1969. , , Academic Press, 1966. Principles of Skill Acquisition , Bittner, A.C. Jr., Statistical Tests for Differential Stability Proceedings of the 23rd Annual Meeting of the Human Factors Society, p. 541-545, October 1979. Blum, A., Melynis J., Flader, D. Beardsley, VI., Ramberg, T. Criteria Report for A-6E Weapon System Trainer Device 2F114, Final, Vol. I Grumman Aerospace Corporation, TR-T01-CD-A005-04 January 19 77. , , 206 , , , , , Kapp, N. Blum, A., Lovrecich, J., Force, J., Cohen, L. Configuration Report for A-6E Weapon System Trainer Device 2F114, Final, Vol. 1 Gr'omraan Aerospace Corporation, TR-TOlCD-AOOl-06, May 1977. , , Happ, N. Blum, A., Lovrecich, J., Force, J., Cohen, L. Device Trainer Weapon System Configuration Report for A-6E 2F114, Final, Vol. II Grumman Aerospace Corporation, TR-TOl-CD-AOOl-06, May 1977. , , and Robins, J., Study, Promisel, D. Bishop, E.W. Bowen, H.M. Assessment of Pilot Proficiency U.S. Naval Training Device Center, Port Washington, NY, NAVTRADEVCEN 1614-1, August 1966, , , , , AD 637659. Brictson, C.A., Human Factors Research on Carrier Landing Office of Naval System Performance Final Report (1966-1971) Research, July 1971, AD 733703. , , Buckhout, R. A Bibliography on Aircrew Proficiency Measurement Aerospace Medical Division, Wright-Patterson AFB, OH, MRL-TDR-62-4 9, May 1962, AD 283545. , , Buckhout, R. and Cotterman, T.E., Considerations in the Design of Automatic Proficiency Measurement Equipment in Simulators , , 6570th Aerospace Medical Research Laboratories, Wright-Patterson AFB, OH, AIvIRL Memorandum P-40, June 1963, AD 417 425. Campbell, S. Task Listing, A-6E (TRAM) Aircraft, Pilot Procedures, Vol. I Grumman Aerospace Corporation, Report ISD (SAT) 0003-1, September 1975. , , Campbell, S., and Sohl, D. Task Listing, A-6E (TRAM) Aircraft Airframe Emergencies and System Malfunctions, Vol. Ill Grumman Aerospace Corporation, Report ISD (SAT) 0003-3A, October 1975. , , , Campbell, S., Graham, G. and Hersh, S., Specific Behavioral Objectives and Criterion Test Statements, Pilot Procedures A-6E (TRAM) Aircraft, Vol. I Grumman Aerospace Corporation, Report ISD (SAT) 0003-4, December 1975. , , Campbell, S.C., Feddern, J., Graham, G. and Morganlander M. Design of A-6E TRAJ4 Training Program Using ISD Methodology. Phase I Final Report Naval Training Equipment Center, Orlando, FL 32313, NAVTRAEQUIPCEN 75-C-0099-1, May 1976. Also titled as A-6E Systems Approach to Training. Phase I Final Report February 1977. , , , , Carter, V.E., Development of Automated Performance Measures for Introductory Air Combat Maneuvers Proceedings of the 21st Annual Meeting of the Human Factors Society, p. 436-439, October , 1977. 207 Chapanis, A., Theory and Methods for Analyzing Errors in Man-Machine Systems from Sinaiko, H.W. (Editor), Selected Papers on Human Factors in the Design and Use of Control Systems Dover Publications, Inc., p. 42-66, 1961. , , Instructor Pilot's Role in Simulator Training Charles, J. P. _ III) Nava: (aval Training Equipment Center, Orlando, FL 3 2813, Phase NAVTRAEQUIPCEN 76-C-0034-2, June 1978, AD A061732. , , Objective Methods for Developing Indices of Pilot Chiles, W. Workload, Federal Aviation Administration, Oklahoma City, OK 73125, FAA-AM-77-15, July 1977, AD A044556. , and Mills, R.G., "What Does the Operator Christiansen, J.M. Do in Complex Systems," Human Factors v. 9(4): p. 329-340, August 197 0. , , Human Engineering Guide to Ship System Development Coburn, R. Naval Electronics Laboratory Center, San Diego, CA 92152, NLEC/TD 27 8, October 197 3. , , Comptroller General of the United States, Greater Use of Flight Simulators in Military Pilot Training Can Lower Costs and Increase Pilot Proficiency General Accounting Office, Washington, D.C. 20548, B-157905, 9 August 1973. , Connelly, E.M. Bourne, F.J., and Loental, D.G., Computer- Aided Techniques for Providing Operator Performance Measures Air Force Human Resources Laboratory, Wright-Patterson AFB, OH 45433, AFHRL-TR-74-87, December 1974, AD A014330. See also Proceedings of the 18th Annual Meeting of the Human Factors Society, Huntsville, AL, p. 359-367, October 1974. , , Connelly, E.M. Bourne, F.J., Loental, D.G., Migliaccio, J.S., Burchick, D.A. and Knoop, P.A. Candidate T-37 Pilot Performance Measures for Five Contact Maneuvers Air Force Human Resources Laboratory, Wright-Patterson AFB, OH 45433, AFHRLTR-74-88, December 1974, AD A014331. , , , , Cotterman, T.E., and Wood, M.E., Retention of Simulated Lunar Landing Mission Skills: A Test of Pilot Reliability Aerospace Medical Research Laboratories, Wright-Patterson AFB, OH 45433, AMRL-TR-66-222, April 1967, AD 817232. , Cureton, E.E. Validity Educational Measurement , , , from Lindquist, E.F. (Editor), George Banta Publishing Company, 1951. Daniel, R.S., and Eason, R.G. Error in Aiming Point Identification on a Radar Bombing Mission Air Force Personnel and Training Research Center, Mather AFB, CA, Project No. 7711, April 1954, AD 36050. , , 208 , Danneskiold, R.D., Objective Scoring Procedure for Operational Flight Trainer Performance U.S. Naval Special Devices Center, Port Washington, NY, SPECDEVCEN 999-2-4, February 1955, AD , 110925. and Behan, R.A., Evaluating System Performance Davis, R.H. from Gagne, R.M. (Editor) Psychoin Simulated Environments logical Principles in System Development Holt, Rinehart, & Winston, Inc., 1966. , , , , "The Robust Beauty of Improper Linear Models in Dawes, R.M. Decision Making," American Psychologist v. 34(7): p. 571-582, July 1979. , , Deberg, O.K., A Scoring Algorithm for Assessing Air-to-Air Combat Performance. Phase I: A Factor Analysis Air Command and Staff College, Maxwell AFB, AL 36112, Report No. 0590-77, May 1977, AD B019569. , Demaree, R.G., Norman, D.A., and Matheny, W.G., An Experimental Program for Relating Transfer of Training to Pilot Performance and Degree of Simulation U.S. Naval Training Device Center, Port Washington, NY, NAVTRADEVCEN 138 8-1, June 1965, AD 471806. , Department of Defense Military Specification MIL-H-46855B, Human Engineering Requirements for Military Systems, Equipment and Facilities January 1979. , Dixon, W.J., and Massey, F.J. Jr., Introduction to Statistical Analysis 3rd ed., McGraw-Hill, 1969. , R.E., Lane, N.E., Shannon, R.H., and Clisham, W.F. Jr., Naval Flight Officer Function Analysis Volume V: A- 6 Bombardier/ Navigator (B/N) Naval Aerospace Medical Research Laboratory, Pensacola, FL, December 1972. Doll, , Engler, H.F., Davenport, E.L., Green, J., and Sears, W.E. Ill, Human Operator Control Strategy Model Air Force Human Resources Laboratory, Wright-Patterson AFB, OH 45433, AFHRL-TR-7 9-60 April 1980, AD A084695. , Ericksen, S.C, A Review of the Literature on Methods of Measuring Pilot Proficiency Human Resources Research Center, Lackland AFB, San Antonio, TX, Research Bulletin 52-25, August 1952, ATI No. 169181. , Farrell, J.P. Measurement Criteria in the Assessment of Helicopter Pilot Performance Proceedings of a Conference on Aircrew Performance in Army Aviation, held at U.S. Army Aviation Center, Fort Rucker, Alabama on November 27-2 9, 197 3, Office of the Chief of Research, Development and Acquisition (Army) and U.S. Army Research Institute for the Behavioral and Social Sciences, Arlington, VA 22209, July 1974, p. 141-148, AD A001539. , , 209 , Farrell, J. P., and Fineberg, M.L., "Specialized Training versus Experience in Helicopter Navigation at Extremely Low Altitudes," Human Factors v. 18(3): p. 305-308, June 1976. , Federman, P.J., and Siegel, A.I., Communications as a Measurable Index of Team Behavior U.S. Naval Training Device Center, Port Washington, NY, NAVTRADEVCEN 1537-1, October 196 5, AD 623135. , The Development, Implementation, and Evaluation Ferguson, R. of A Computer-Assisted Branch Test for A Program of Individually Prescribed Instruction Ph.D. Dissertation, University of Pittsburgh, 1969. , , Fineberg, M.L., Navigation and Flight Proficiency under Nap of the Earth Conditions as a Function of Aviator Trainina and Experience Proceedings of the 18th Annual Meeting of the Human Factors Society, Huntsville, AL, p. 249-254, October 1974. , Fineberg, M.L., Meister, D., and Farrell, J. P., An Assessment of the Navigation Performance of Army Aviators Under Nap-ofthe-Earth Conditions U.S. Army Research Institute for the Behavioral and Social Sciences, Alexandria, VA 22333, Research Report 1195, August 1978, AD A060563. , Fitts, P.M., Psychomotor Skills and Behavior Modification Academic Press, 1965. , Forrest, E.G., Develop An Objective Flight Test for the Certification of A Private Pilot Federal Aviation Administration, Washington, D.C. 20590, FAA-DS-70-17 May 1970, AD 708293. , , Geiselhart, R. Kemmerling, P., Cronburg, J.G., and Thorburn, D.E., A Comparison of Pilot Performance Using a Center Stick VS Sidearm Control Configuration Aeronautical Systems Division, Wright-Patterson AFB, OH, Technical Report ASD-TR-70-39 November 1970, AD 720846. , , Geiselhart, R. Jarboe, J.K., and Kemmerling, P.T. Jr., Investigation of Pilots' Tracking Capability Using a Roll Command Display Aeronautical Systems Division, WrightPatterson AFB, OH, ASD-TR-71-46, December 1971, AD A009590. , , Geiselhart, R. Schiffler, R.J., and Ivey, L.J., A Study of the Task Loading Using a Three Man Crew on a KC-135 Aircraft Aeronautical Systems Division, Wright-Patterson AFB, OH 45433, ASD-TR-76-19, October 1976, AD A044257. , , Gerlach, S., Cues, Feedback and Transfer in Undergraduate Pilot Training Air Force Office of Scientific Research, Boiling AFB, DC 20332, AFOSR-TR-76-0056 August 1975, AD A033219, , , 210 and Klaus, D.J., Proficiency Measurement: Glauser, R. Assessing Human Performance from Gagne, R.M. (Editor) Psychological Principles in System Development Holt, Rinehart & Winston, Inc., 19 66. , , , , F.E., Beideman, L.R., and Levine, S.H., The Application of Biocybernetic Techniques to Enhance Pilot Performance Corner, During Tactical Missions Defense Advanced Research Projects Agency, MDC E204 6, October 197 9. , Hanish, J., and Hersh, S., Specific Behavioral Graham, G. Objectives and Criterion Test Statements, Airframe Emergencies and System Malfunctions, A-6E (TRAM) Aircraft, Vol. Ill Grumman Aerospace Corporation, Report ISD (SAT) 0003-6, December 197 5. , , Greening, C.P., Performance Measurement and Prediction in Air-to-Ground Target Acquisition Naval Weapons Center, China Lake, CA 93555, NWC TP 5747, April 1975. , Greer, G.D. Jr., Smith, D., Hatfield, J.L., Colgan, CM., and Duffy, J.O., PPDR Handbook. Use of Pilot Performance Description Record in Flight Training Quality Control Human Resources Research Office, Fort Rucker, AL, Department of the Army, Washington, DC 20310, December 1963, AD 675337. , Grodsky, M.A. "The Use of Full Scale Mission Simulation for the Assessment of Complex Operator Performance," Human Factors v. 9(4): p. 341-348, August 1967. , Hanish, J. and Feddern, J. Specific Behavioral Objectives and Criterion Test Statements, Bombardier/Navigator Procedures A-6E (TRAM) Aircraft, Vol. II Grumman Aerospace Corporation, Report ISD (SAT) 0003-5, December 1975. , , , , , Haygocd, R.C., Leshowitz B. Parkinson, S.R., and Eddowes, E.E., Visual and Auditory Information Processing Aspects of the Acquisition of Flying Skill Air Force Human Resources Laboratory, Williams AF3, AZ 85224, AFHRL-TR-7 4-7 9 December 197 4, AD A007721. , , , , Hitch, C, "Sub-optimization in Operations Problems," Journal of the Operations Research Society of America v. 1(3): p. 87-99, May 1953. , Hitchcock, L. Jr., The Analysis of Human Performance within the Operational Flight Environment Proceedings, Annual AGARD Meeting of the Aerospace Medical Panel, Toronto, Canada, Advisory Group for Aerospace Research and Development, Paris, France, AGARD Conference Proceedings No. 14, d. 9-14, September 1966, AD 661165. , 211 An Evaluation of Three Possible Hulin, C.L., and Alvares, K.M. in Predicting Pilot Decay Temporal Explanations of the Proficiency Air Force Human Resources Laboratory/ Williams AFB/ AZ 85224/ AFHRL TR-71-5/ February 1971/ AD 731191 , / Three Explanations of Temporal Hulin/ C.L./ and Alvares, K.M. Changes in Ability-Skill Relationships: Literature Review and Theoretical Analysis Air Force Human Resources Laboratory/ Williams AFB/ AZ 85224/ AFHRL TR-71-6/ February 1971/ AD 732612. / / Jensen/ R.S./ Vander Kolk/ R.J./ and Roscoe, S.N., Pilotage Error in Area Navigation: Effects of Symbolic Display Variables, Federal Aviation Administration/ Washington/ DC 20590/ FAA-RD-72-29/ January 1972/ AD 750178. Johnson, H.M. and Boots M.L./ Analysis of Ratings in the Preliminary Phase of the CAA Training Program CAA Division of Research Report No. 21/ 1943. / / / JoneS/ M.D., "Rate and Terminal Processes in Skill Acquisition," Journal of Psychology v 8 3: 1970. . / Jones, R.A. The F-lllD Simulator Can Reduce Cost and Improve Aircrew Evaluation Air War College, Maxwell AFB, AL 36112, Professional Study No. 5959, April 1976/ AD B010452. , , Kalisch/ S.J./ Computerized Instructional Adaptive Testing Model: Formulation and Validation Air Force Human Resources Laboratory/ Brooks AFB, TX 78235/ AFHRL-TR-79-33 February / , 1980. Kelley, C.R./ and Wargo, M.J., Adaptive Techniques for Synthetic Flight Training Systems Naval Training Device Center, Orlando, FL/ NAVTRADEVCEN 68-C-0136-1/ October 1968, AD 678536. , Kelly/ M.J./ Wooldridge/ L. Hennessy/ R.T., VreulS/ D., Barnebey, S.F., Cotton, J.C, and Reed, J.C, Air Combat Maneuvering Performance Measurement Naval Training Equipment Center, Orlando, FL 32813, and Air Force Human Resources Laboratory, Brooks AFB/ TX 78 23 5/ NAVTRAEQUIPCEN IH-315/AFHRCTR-79-3/ September 1979; see also Proceedings of the 23rd Annual Meeting of the Human Factors Society/ p. 324-323, 1979. / , Kemmerling, P.T. Jr./ The Impact of Full-Mission Simulation on the TAC Combat Crew Training System Air War College, Maxwell AFB, AL 3 6112, Professional Study No. 5655, April 1975/ AD B004457. , Klier/ S., and Gage/ H. Motion Factors in Flight Simulation Naval Training Device Center, OrlandO/ FL 32813, NAVTRADEVCEN 68-C-0007-1/ December 1970/ AD 880341. / , 212 , Knoop, P. A., Programming Techniques for the Automatic Monitoring of Human Performance Aerospace Medical Research Laboratory, Wright-Patterson SFB, OK 45433, AMRL-TR-66-16 April 1966. AD 637454. , Knoop, P. A., Development and Evaluation of a Digital Computer Program for Automatic Human Performance Monitoring in Flight Simulator Training Aerospace Medical Research Laboratories, Wright-Patterson AFB, OH 45433, AMRL-TR-67-97 August 1968, , , AD 675542. Knoop, P. A., and Welde, W.L., Automated Pilot Performance Assessment in the T-37 A Feasibility Study Air Force Human Resources Laboratory, Advanced Systems Division, AFHRL/AS, Wright-Patterson AFB, OH 45433, AFHRL-TR-7 2-6 April 1973, AD 766446; see also Human Factors v. 15(6): p. 583-597, December 197 3. : , , , Krendel, E.S., and Bloom, J.W. The Natural Pilot Model for Flight Proficiency Evaluation U.S. Naval Training Device Center, Port Washington, NY, NAVTRADEVCEN 323-1, April 1963, AD 410805. , , Leshowitz, B. Parkinson, S.R., and Waag, W.L., Visual and Auditory Information Processing in Flying Skill Acquisition Air Force Human Resources Laboratory, Williams AFB, AZ 85224, AFHRL-TR-74-103, December 1974, AD A009636. , , Lewis, R.E.F., Navigation of Helicopters in Slow and Very Low Flight. A Comparison of Solo and Dual Pilot Performance Proceedings, Annual AGARD Meeting of the Aerospace Medical Panel, Toronto, Canada, Advisory Group for Aerospace Research and Development, Paris, France, AGARD Conference Proceedings No. 14, September 1966, p. 29-34, AD 661165. , Lindsay, G.F. Statistical Considerations in MX Site Characterization Systems Exploration, Inc., Report No. B3440-79462H53BAXSX37718-H-2590D, November 1979. , , Locke, E.A., Zavala, A., and Fleishman, E.A., "Studies of Helicopter Pilot Performance: II. The Analysis of Task Dimensions," H uman Factors v. 7(3): p. 285-301, June 1965. , Lorge, I., The Fundair.ental Nature of Measurement from Lindquist, E.F. (Editor) Educational Measurement George Banta Publishing Company, 1951. , , , Marks, xM.R., Development of Human Proficiency and Performance Measures for Weapon Systems Testing Aerospace Medical Research Laboratories, Wright-Patterson AFB, OH, ASD Technical Report 61-733, December 1961, AD 272975. , 213 Matheny, W.G., Patterson, G.W. Jr., and Evans, G.I., Human Factors in Field Testing Office of Naval Research, Washington, DC, LS-ASR-70-1, May 1970, AD 716438. , McCormick, E.J., Human Factors in Engineering and Design McGraw-Hill, 4th Ed., 1976. , McCoy, W.K. Jr., "Problems of Validity of Measures Used in Investigating Man-Machine Systems," Human Factors v. 5(4): p. 373-377, August 1963. , The Development and Evaluation of Objective McDowell, E.D. Frequency Domain Based Pilot Performance Measures in ASUPT Air Force Office of Scientific Research, Boilings AFB, DC 20332, AFOSR-TR-78-1239, April 1978, AD A059477. , , McMinn, V.L., January 1981. "A Subjective Leap," Approach , v. 24-25, 26(7): p. Meister, D. and Rabideau, G.F., Human Factors Evaluation in System Development Wiley and Sons, 1965. , , Miller, N.E., Gait, W.E. and Gershenson, C.P., A Large Scale Objective Study of the Effects of Additional Training on Flying Skill from Miller, N.E. (Editor), Psychological Research on Pilot Training AAF Aviation Psychology Program Research Report No. 8, 1947. , , , Mixon, T.R., and Moroney, W.F., Annotated Bibliography of Objective Pilot Performance Naval Training Equipment Center, Orlando, FL 32813, NAVTRAEQUIPCEN IH-330, March 1981. (In press) , . Mood, A.M., Graybill, F.A., and Boes, D.C., Introduction to the Theory of Statistics 3rd ed., McGraw-Hill, 197 4. , Moore, S.B., Meshier, C.W., and Coward, R.E., The Good Stick Index. A Performance Measurement for Air Combat Training First Interservice/Industry Training Equipment Conference, sponsored by Naval Training Equipment Center, Orlando, FL 32813, NAVTRAEQUIPCEN IH-316, November 1979, p. 145-154. . Morris, W.T., Management Science in Action Inc., Homewood, IL, 1963. Mundel, M.E., Motion and Time Study: Prentice-Hall, 1978. , Richard D, Irwin, Improving Productivity , Murphy, R. Individual Differences in Pilot Performance Proceedings of the 6th Congress of the International Ergonomics Association, College Park, MD, July 1976, p. 403-409. , , 214 1 Norman, D.A., Memory and Attention ed., 1976. , Wiley & Sons, Inc., 2nd Aviator Selection 1919-1977 and Griffin, G.R. North, R.A. Naval Aerospace Medical Research Laboratory, Pensacola, FL 32508, NAIvlRL-SR-77-2, October 1977, AD A048105. , , , , and Muckler, F.A., Performance Measurement Obermayer, R.W. in Flight Simulation Studies National Aeronautics and Space Administration, Washington, DC, NASA CR-82, July 1964. , , and Vreuls, D., Combat-Ready Crew Performance Obermayer, R.W. Measurement System: Phase III A Crew Performance Measurement Air Force Human Resources Laboratory, Williams AFB, AZ 85224, AFHRL-TR-7 4-10 8 (IV), December 1974, AD B005520. , , Muckler, F.A. Conway, E.J., and Obermayer, R.W., Vreuls, D. Fitzgerald, J. A., Combat-Ready Crew Performance Measurement System: Final Report Air Force Human Resources Laboratory, Williams AFB, AZ 85224, AFHRL-TR-74-108 (I) December 1974, AD B005517. , , , , Obermayer, R.W., Vreuls, D. Muckler, F.A., and Conway, E.J., Combat- Ready Crew Performance Measurement System: Phase III D Specifications and Implementation Plan Air Force Human Resources Laboratory, Williams AFB, AZ 85224, AFHRL-TR-7 4-108 (VII), December 1974, AD B005522. , , Office of the Secretary of Defense, Department of Defense Review of Tactical Jet Operational Readiness Training Assistant Secretary of Defense (Manpower and Reserve Affairs) Washington, DC 20301, November 1968, AD 845259. , Oiler, R.G. Human Factors Data Thesaurus (An Application to Task Data) Aerospace Medical Research Laboratories, WriahtPatterson AFB, OH, AMRL-TR-67-211, March 1968, AD 670578 , , Operations Committee, Naval Science Department, Naval Operations Analysis U.S. Naval Institute, 1968. , W.C, Developing Performance Tests for Training Evaluation Human Resources Research Organization, Alexandria, VA 22314, HumRRO-PP-3-7 3, October 1971, AD 758436. Osborn, , Parker, J.F. Jr., "The Identification of Performance Dimensions Through Factor Analysis," Human Factors v. 9(4): p. 367-373, August 1967. , Parsons, H.M. Press, 1972. , Man-Machine System Experiments " ~ 215 , Johns Hopkins . ,, , "Quantification of Human Pickrel, E.W., and McDonald, T.A. Performance in Large, Complex Systems," Human Factors v. 6(6): p. 647-662, December 1964. , , Pierce, B.J., De Maio, J., Eddowes E.E., and Yates, D. Airborne Performance Methodology Application and Validation; F-4 Pop-Up Training Evaluation Air Force Human Resources Laboratory, Williams AFB, AZ 85224, AFHRL-TR-7 9-7 June 1979, AD A072611. See also Proceedings of the 23rd Annual Meeting of the Human Factors Society, p. 320-324, November 1979. , , , Performance Measurement in Helicopter Training Prophet, W.W. and Operations Office of Chief of Research and Development April 1972, (Army), Washington, DC 20310, HumRRO-PP-10-72 AD 743157. , , , Long-Term Retention of Flying S ills: A Review Prophet, W.W. of the Literature U.S. Air Force Headquarters (AF/SAA) Washington, DC 2 0332, HumRRO FR- ED (P) -7 6-35 October 197 6, AD A036077. , , , Prophet, W.W. U.S. Navy Fleet Aviation Training Program Development Naval Training Equipment Center, Orlando, FL 32813, NAVTRAEQUIPCEN 77-C-0009-1, xMarch 1978, AD A055788. , , Rankin, W.C., and McDaniel, W.C., Computer Aided Training Evaluation and Scheduling (CATES) System: Assessing Flight Task Proficiency Training and Analysis Group, Orlando, FL 32813, TAEG Report No. 94, December' 1980 , Read, J.H., Specification for Trainer, A-6E Weapon System Device 2F114 Grumman Aerospace Corporation, September 1974. , Ricketson, D.S., Pilot Error As A Cause of Army Helicopter Accidents Proceedings of a Conference on Aircrew Performance Army Aviation held at U.S. Army Aviation Center, Fort Rucker, AL, on November 27-29, 1973, Office of the Chief of Research, Development and Acquisition (Army) and U.S. Army Research Institute for the Behavioral and Social Sciences, Arlington, VA 22209, July 1974, p. 75-80, AD A001539. , m Riis, E., Measurement of Performance in F-8 6K Simulator Proceedings, Annual AGARD Meeting of the Aerospace Medical Panel, Toronto, Canada, Advisory Group for Aerospace Research and Development, Paris, France, AGARD Conference Proceedings No. 14, Assessment of Skill and Performance in Flying, September 1966, p. 49-56, AD 661165. , Rinsky, R. Pearlman, T., Nienaltowski, W. and Marchessault CCriteria Report for A-6E Weapon System Trainer Device 2F114, Final, Vol. II Grumman Aerospace Corporation, TR-T01-CD-A005-04, January 1977. , , , , 216 Roscoe, S.N., Review of Flight Training Technology U.S. Army Research Institute for the Behavioral and Social Sciences, Research Problem Review 76-3, July 1976, AD A076641. , Rosenmayer, C.E., and Asiala, C.F., F-18 Human Engineering Task Analysis Report McDonnell Aircraft Company, St. Louis, MO 63166, MDC A4276, August 1976. , Ryack, B.L., and Krendel, E.S., Experimental Study of the Natural Pilot Flight Proficiency Evaluation Model U.S. Naval Training Device Center, Port V7ashington, NY, NAVTRADEVCEN 323-2, April 1963, AD 414666. , Identification of Significant Aircrew Decisions in Navy Attack Aircraft Naval Weapons Center, China Lake, CA 93 555, NWC TP 6117, January 1980, AD B045034. Saleh, J., Thomas, J.O., and Boylan, R.J., , "Visual Sanders, M.G., Simmons, R.R., and Hofmann, M.A. Workload of the Copilot/Navigator During Terrain Flight," Human Factors v. 21(3): p. 369-383, June 1979. , , Schohan, B. Rawson, H.E., and Soliday, S.M., "Pilot and Observer Performance in Simulated Low Altitude High Speed Flight," Human Factors v. 7(3): p. 257-265, June 1965. , , Semple, C.A. Vreuls, D. Cotton, J.C., Durfee, D.R., Hooks, J.T., and Butler, E.A., Functional Design of an Automated Instructional Support System for Operational Flight Trainers Naval Training Equipment Center, Orlando, FL 32813, NAVTRAEQUIPCEN 76-C-0096-1, January 1979, AD A065573. , , , Senders, J.W., A Study of the Attentional Demand of VFR Helicopter Pilotage Proceedings of a Conference on Aircrew Performance in Army Aviation held at U.S. Army Aviation Center, Fort Rucker, AL on November 27-29, 1973, Office of the Chief of Research, Development and Acquisition (Army) and U.S. Army Research Institute for the Behavioral and Social Sciences, Arlington, VA 22209, July 1974, p. 175-187, AD A001539. , Shannon, R.H., and Waag, W.L., Toward the Development of a Criterion for Fleet Effectiveness in the F-4 Fighter Community Proceedings of the 16th Annual Meeting of the Human Factors Society, Los Angeles, CA, October 1972, p. 335-337. See also NAMRL-1173, December 1972, AD 755184, Shelnutt, J.B., Childs, J.M. Prophet, W.W. and Spears, W.D., Human Factors Problems in General Aviation Federal Aviation Administration, Atlantic City, NJ 08405, FAA-CT-8 0-194 April , , , , 1980. 217 , , Shipley, B.D., Gerlach, V.S., and Brecke, F.H., Measurement Air Force Office of Flight Performance in a Flight Simulator AFOSR-TR-75-0208 DC AFB, 20332, of Scientific Research, Boiling August 1974, AD A004488. , Shipley, B.D., An Automated Measurement Technique for Evaluating Pilot Skill Air Force Office of Scientific Research, Boiling February 1976, AD A033920. AFB, DC 20332, AFOSR-TR-76-1253 , , Siegel, A.I., and Federman, P.J., Increasing ASW Helicopter Effectiveness Through Communications Training Naval Training Device Center, Orlando, FL, NAVTRADEVCEN 66-C-0045-1, October 1968, AD 682498. , Siegel, S., Nonparametric Statistics for the Behavioral Sciences McGraw-Hill, 1956. , and Whitfield, D., Measurement of Singleton, W.T., Fox, J.G. Man At Work Taylor & Francis LTD, 1971. , , Smit, J. Pilot Workload Analysis Based Upon In-Flight Physiological Measurements and Task Analysis Methods National Aerospace Laboratory (NLR) The Netherlands, NLR-MP-7 6001-U, June 1976, AD B026957. , , , Smith, B.A. Development and Inflight Testing of a Multi-Media Course for Instructing Navigation for Night Nap-of-the-Earth Flight Proceedings of the 24th Annual Meeting of the Human Factors Society, Los Angeles, CA, October 1980, p. 568-571. , , Smode, A.F., Gruber, A., and Ely, J.H., The Measurement of Advanced Flight Vehicle Crew Proficiency in Synthetic Ground Environments 657 0th Aerospace Medical Research Laboratories, Wright-Patterson AFB, OH, MRL-TDR-62-2 February 1962, , , AD 273449. Smode, A.F., and Meyer, D.E., Research Data and Information Relevant to Pilot Training. Volume I. General Features of Air Force Pilot Training and Some Research Issues Aerospace Medical Research Laboratories, Wright-Patterson AFB, OH, AMRL-TR-66-99-V0I I, July 1966, AD 801776. , Soliday, S.M., "Navigation in Terrain-Following Flight," Human Factors v. 12(5): p. 425-433, October 1970. , Steyn, D.W., "The Criterion - Stagnation or Development? A Selective Review of Published Research Work on Criterion Problems," Psychologia Africana v. 12(3): p. 193-211, March , 1969. Swezey, R.W. "Aspects of Criterion-Referenced Measurement in Performance Evaluation," Human Factors v. 20(2): p. 169-178, April 1978. , , 218 , Lankford, H.E., Miller, R.M. Swink, J.R., Butler, E.A. and Waag, W.L., Definition of Requirements for Watkins, H. Performance Measurement System for C-5 Aircrew Members Air Force Human Resources Laboratory, Williams AFB, AZ 85224, AFHRL-TR-7 8-54, October 1978, AD A063282. , , , Thorndike, R.L., Reliability from Lindquist, E.F. (Editor), Educational Measurement George Banta Publishing Company, 1951. , , Effects on A-6E Bombardier/Navigator Flight Tindle, J.R. Training with the Introduction of Device 2F114, A-6E Weapon System Trainer Master's Thesis, Naval Postgraduate School, March 1979. , , Uhlaner, J.E., and Drucker, A.J., "Criteria for Human Performance Research," Human Factors v. 6(3): p. 265-278, June 1964. , Uhlaner, J.E., and Drucker, A.J., "Military Research on Performance Criteria: A Change of Emphasis," Human Factors v. 22(2): p. 131-139, April 1980. , Van Cott, H.P., and Kinkade, R.G. (Editors), Human Engineering Guide to Equipment Design Joint Army-Navy-Air Force Steering Committee, 197 2. , Voiers, W.D., A Comparison of the Components of Simulated Radar Bombing Error in Terms of Reliability and Sensitivity to Practice Air Force Personnel & Training Research Center, Lackland AFB, TX, AFPTRC-54-74 April 1954, AD 62566. , , Vreuls, D. and Obermayer, R.W. Study of Crew Performance Measurement for High-Performance Aircraft Weapon System Training: Air-to- Air Intercept Naval Training Device Center, Orlando, FL 32813, NAVTRADEVCEN 70-C-0059-1, February 1971, AD 727739. , , , Vreuls, D., and Obermayer, R.W., Emerging Developments in Flight Training Performance Measurement from U.S. Naval Training Device Center 25th Anniversary Commemorative Technical Journal, Orlando, FL 32813, November 1971, p. 199-210. , Vreuls, D., Obermayer, R.W., Goldstein, I., and Lauber J.K., Measurement of Trainee Performance in a Captive Rotary-Wing Device Naval Training Equipment Center, Orlando, FL 32813, June 1973, AD 764088. , , Vreuls, D., Obermayer, R.W., and Goldstein, I., Trainee performance Measurement Development Using Multivariate Measure Selection Techniques Naval Training Equipment Center, Orlando, FL 32813, NAVTRAEQUIPCEN 73-C-00 66-1, September 1974, AD 787594. See also Proceedings, NTEC/Industry Conference, Orlando, FL, NAVTRAEQUIPCEN IH-240, November 1974, P. 227-236, AD A000970. , 219 , Vreuls, D., Wooldridge, A.L., Obermayer, R.W., Johnson, R.M. and Goldstein, I., Development and Evaluation Norman, D.A. Measures in an Automated Instrument Performance of Trainee Flight Maneuvers Trainer Naval Training Equipment Center, Orlando, FL 32813, NAVTRAEQUIPCEN 74-C-0063-1, October 1975, AD A024517. , , Aircrew Performance and Wooldridge, A.L. Vreuls, D. Measurement Proceedings of the Symposium on Productivity Personnel Assessment in Navy Systems, Naval Enhancement: Personnel Research and Development Center, San Diego, CA, October 1977. , , , Vreuls, D. and Cotton, J.C., Feasibility of Aircrew Performance Measurement Standardization for Research Air Force Office of Scientific Research, Boiling AFB, DC 20332, April , , 1980. Waag, W.L., Eddowes, E.E., Fuller, J.H. Jr., and Fuller, R.R., ASUPT Automated Objective Performance Measurement System Air Force Human Resources Laboratory, Williams AFB, AZ 85224, AFHRL-TR-7 5-3, March 1975, AD A014799. , Welford, A.T., Fundamentals of Skill , Methuen & Co. LTD, 1971. Welford, A.T., Skilled Performance: Perceptual and Motor Skills Scott, Foresman and Company, 197 6. , Williams, A.C. Jr., Simon, C.W., Haugen, R. and Roscoe, S.N., Operator Performance in Strike Reconnaissance Wright Air Development Division, Wright-Patterson AFB, OH, WADD Technical Report 60-521, August 1960, AD 246545. , , 220 APPENDIX A A-6E TRAM RADAR NAVIGATION TASK LISTING The enclosed task listing for the B/N during radar navigation in the A-6E WST was compiled from various sources as discussed in Chapter V. This was the first phase of developing a task analysis for the purpose of measuring performance of the B/N during radar navigation. 221 A-6E TRAM RADAR NAVIGATION TASK LISTING SEGMENT Tl T2 T3 T4 1: AFTER TAKEOFF CHECKS ADJUST RADAR PANEL FOR SCOPE RETURN 51 SET SEARCH RADAR PWR SWITCH 52 SET XxMT ON NORM SWITCH ACTIVATE SYSTEM STEERING TO INITIAL POINT 51 CHECK C0MPTM0D5 SWITCH 52 DEPRESS TGT N ADDRESS KEY HAVING IP LAT/LONG 53 CHECK C0J4PT/MAN SWITCH (IP) STEER COMPT CHECK FOR ACCURATE SYSTExM STEERING TO IP 51 READ SYSTEM BEARING AND RANGE TO IP FROM DVRI BUG AND RANGE DISPLAYS 52 COMPARE SYSTEM BEARING AND RANGE TO IP WITH PRE-PLANNED OR ESTIMATED BEARING AND RANGE TO IP 53 GO TO T5 IF SYSTEM STEERING TO IP IS CORRECT 54 GO TO T4 IF SYSTEM STEERING TO IP IS NOT CORRECT TROUBLESHOOT SYSTEM STEERING IF REQUIRED 51 DETERMINE IP LAT/LONG FROM CHART OR IFR SUPPLEMENT 52 COMPARE SYSTEM IP LAT/LONG WITH ACTUAL IP LAT/LONG (a) THROW DDU DATA SWITCH (b) READ SYSTEM TGT N ADDRESS (IP) FROM LOWER DDU LAT/LONG DISPLAYS 222 ON CALL 53 INSERT CORRECT IP LAT/LONG IF REQUIRED 54 EVALUATE SYSTEM PRESENT POSITION AND ESTIMATED POSITION FROM CHART 55 T5 T7 THROW DDU DATA SWITCH (b) READ SYSTEM PRESENT POSITION LAT/LONG FROM LOWER DDU LAT/LONG DISPLAYS (c) COMPARE SYSTEM PRESENT POSITION WITH ESTIMATED PRESENT POSITION FROM CHART (d) INSERT CORRECT PRESENT POSITION IF REQUIRED PRES POS (1) DEPRESS PRES LOC ADDRESS KEY ON COMPUTER KEYBOARD (2) THROW COMPTMODE SWITCH (3) INSERT CORRECT PRESENT POSITION LAT/LONG (4) THROW COMPTMODE SWITCH (5) CHECK LOWER DDU LAT/LONG DISPLAYS FOR ACCURATE DATA ENTRY ENTER STEER INFORM PILOT OF APPROXIxMATE HEADING TO IP IF REQUIRED AND GO TO T6 INFORM PILOT THAT SYSTEM STEERING IS TO IP SI T6 (a) CHECK THAT PILOT MAINTAINS SAFE FLIGHT AND FOLLOWS SYSTEM STEERING CHECKOUT RADAR FOR STATUS 51 GO TO NEXT EVENT IF RADAR RETURN IS PRESENT 52 GO TO T7 IF RADAR RETURN IS NOT PRESENT TROUBLESHOOT RADAR IF REQUIRED SI CHECK TEST MODE SWITCH (RADAR/DRS TEST) CENTERED 223 SEGMENT Tl 52 CHECK FAULT ISLN SWITCH 53 CHECK RADAR CIRCUIT BREAKER 54 USE PCL CHECKLIST FOR RADAR TURN-UP PROCEDURES 55 ALERT PILOT IF RADAR INOPERATIVE AND ABORT RADAR NAVIGATION FLIGHT 2: CENTERED IN NAVIGATION TO IP TUNE RADAR FOR OPTIMUM PPI DISPLAY 51 ROTATE CONTRAST CONTROL CW 52 ROTATE BRT CONTROL (UNTIL SWEEP PRESENT) CW 53 CHECK VIDEO/DIF CONTROLS 54 ROTATE RCVR CONTROL (UNTIL RETURN IS PRESENT) 55 CHECK DISPLAYS BUTTONS 56 ADJUST PPI RANGE CONTROL (UNTIL IP AT TOP OF SCOPE) 57 ROTATE RNG MKR/AZ MKR CONTROLS (UNTIL CURSORS PRESENT) 58 CHECK SCAN STAB CONTROL 59 ROTATE SCAN ANGLE CONTROL (UNTIL DESIRED SWEEP WIDTH PRESENT) CCW CW PPI CW/CCW CW ADL CW/CCW 510 CHECK SCAN RATE SWITCH 511 CHECK AMTI CONTROL 512 CHECK STC SLOPE/DEPTH CONTROLS 513 THROW ANT PATT SWITCH (CONTINUE FOR 10 SECOND MINIMUM) FAR 514 CHECK RCVR SWITCH AFC 515 CHECK BEACON CONTROL CCW 516 CHECK AZ-RNG TRKG SWITCH OFF 224 FAST CCW CW/CCW S17 T2 T3 51 PLACE LEFT HAND ON SLEW CONTROL STICK 52 DEPRESS RADAR SLEW BUTTON AND PUSH SLEW CONTROL STICK IN DIRECTION OF DESIRED CURSOR INTERSECTION MOVEMENT 53 COMPARE RADAR RETURN IMAGE ON CURSOR INTERSECTION WITH PREPLANNED CHART AND QUICK & DIRTY IP 54 REPEAT SI AND S2 IF REQUIRED 55 PUSH CORRECT POS BUTTON ON LOWER DDU PANEL ACTIVATE DOPPLER RADAR 52 T5 ON POSITION CURSOR INTERSECTION ON IP IF REQUIRED 51 T4 CHECK FREQ AGILITY SWITCH SET DOPPLER CONTROL SWITCH (LAND OR SEA AS APPROPRIATE) MONITOR DOPPLER CONTROL PANEL FOR DOPPLER RADAR STATUS (a) OBSERVE MEMORY LIGHT OUT FOR PROPER OPERATION (b) OBSERVE DRIFT AND GND SPEED DISPLAYS FOR DRIFT ANGLE AND GROUND SPEED PRESENT SELECT SYSTEM STEERING OR DEAD RECKONING (DR) NAVIGATION 51 DETERMINE STATUS OF INS, RADAR, AND DOPPLER RADAR 52 OBSERVE CURSOR INTERSECTION DRIFT ON RADAR SCOPE 53 GO TO T5 IF DEAD RECKONING IS SELECTED (LARGE CURSOR INTERSECTION DRIFT) OR SEGTl IF SYSTEM NAVIGATION IS MENT 3, SELECTED (SMALL CURSOR INTERSECTION DRIFT) PERFORM DEAD RECKONING NAVIGATION CBEYOND THE SCOPE OF THIS REPORT) 225 ON SEGMENT Tl T2 T3 T4 NAVIGATION TO TURN POINT 3: (TP) INITIATE TURN AT IP 51 ALERT PILOT OF NEXT OUTBOUND HEADING ONE MINUTE PRIOR TO REACHING IP 52 SET OUTBOUND HEADING ON HORIZONTAL SITUATION INDICATOR (HSI) BY ROTATING HEADING SELECT KNOB UNTIL THE HEADING SELECT MARKER IS ON THE DESIRED HEADING 53 CHECK AZ-RNG TRKG SWITCH 54 MONITOR DVRI HEADING BUG FOR MOVEMENT TO 18 0° RELATIVE POSITION (IP PASSAGE) 55 ACTIVATE COCKPIT CLOCK AT IP PASSAGE 56 INFORM PILOT OF IP PASSAGE AND OUTBOUND HEADING 57 CHECK FOR PILOT TURNING TO NEW HEADING OFF ACTIVATE SYSTEM STEERING TO TP 51 DEPRESS TGT N ADDRESS KEY HAVING NEXT TP LAT/LONG 52 CHECK COMPTMODE SWITCH CHECK FOR ACCURATE SYSTEM STEERING TO TP 51 READ SYSTEM BEARING AND RANGE TO TP FROM DVRI BUG AND RANGE DISPLAYS 52 COMPARE SYSTEM BEARING AND RANGE TO TP WITH PREPLANNED OR ESTIMATED BEARING AND RANGE TO TP 53 GO TO T5 IF SYSTEM STEERING TO TP IS CORRECT 34 GO TO T4 IF SYSTEM STEERING TO TP IS NOT CORRECT TROUBLESHOOT SYSTEM STEERING IF REQUIRED SI LAT/LONG FROM CHART OR IFR SUPPLEMENT DETERI-IINE TP 226 STEER 52 T5 (a) THROW DDU DATA SWITCH (b) READ SYSTEM TGT N ADDRESS (TP) FROM LOWER DDU LAT/LONG DISPLAYS 53 INSERT CORRECT TP LAT/LONG IF REQUIRED 54 EVALUATE SYSTEM PRESENT POSITION AND ESTIMATED POSITION FROM CHART ON CALL PRES POS (a) THROW DDU DATA SWITCH (b) READ SYSTEM PRESENT POSITION LAT/ LONG FROM LOWER DDU LAT/LONG DISPLAYS (c) COMPARE SYSTEM PRESENT POSITION WITH ESTIMATED PRESENT POSITION FROM CHART (d) INSERT CORRECT PRESENT POSITION IF REQUIRED (1) DEPRESS PRES LOC ADDRESS KEY ON COMPUTER KEYBOARD (2) THROW COMPTMODE SWITCH (3) INSERT CORRECT PRESENT POSITION LAT/LONG (4) THROW COMPTMODE SWITCH (5) CHECK LOWER DDU LAT/LONG DISPLAYS FOR ACCURATE DATA ENTRY ENTER STEER INFORM PILOT THAT SYSTExM STEERING IS TO THE TP SI T6 COMPARE SYSTEf-1 TP LAT/LONG WITH ACTUAL TP LAT/LONG CHECK THAT PILOT MAINTAINS SAFE FLIGHT AND FOLLOWS SYSTEM STEERING INSERT DATA FOR NEXT REMAINING TPs IF REQUIRED 51 THROW COMPTMODE SWITCH 52 DEPRESS PREVIOUSLY UTILIZED TGT N ADDRESS KEY 53 DEPRESS POS ACTION KEY 227 ENTER 54 DEPRESS APPROPRIATE QUANTITY KEYS IN SEQUENCE FOR TP LAT/LONG 55 DEPRESS ALT ACTION KEY 56 DEPRESS APPROPRIATE QUANTITY KEYS IN SEQUENCE FOR TP ALTITUDE 57 ALERT PILOT TO MAINTAIN PRESENT HEADING 58 THROW COMPTMODE SWITCH 59 THROW DDU DATA SWITCH 810 CHECK LOWER DDU LAT/LONG DISPLAYS FOR ACCURATE DATA ENTRY 511 REPEAT SI THROUGH SIO IF REQUIRED 512 DEPRESS REQUIRED TGT N ADDRESS KEY FOR CURRENT TP SYSTEM STEERING 513 CHECK FOR ACCURATE SYSTEM STEERING TO TP (a) READ SYSTEM BEARING AND RANGE TO TP FROM DVRI BUG AND RANGE DISPLAYS (b) COMPARE SYSTEM BEARING AND RANGE TO TP WITH PREPLANNED OR ESTIMATED BEARING AND RANGE TO TP 514 REPEAT S12 AND S13 IF REQUIRED 515 INFORM PILOT THAT SYSTEM STEERING IS TO THE TP (a) T7 STEER ON CALL CHECK THAT PILOT MAINTAINS SAFE FLIGHT AND FOLLOWS SYSTEM STEERING PERFORM SYSTEM NAVIGATION TASKS SI TUNE RADAR FOR OPTIMUM PPI DISPLAY (a) (b) (c) ADJUST PPI RANGE CONTROL (UNTIL TP AT TOP OF SCOPE) CW ADJUST RCVR CONTROL (UNTIL RETURN IS ENHANCED) CW/CCW ADJUST STC DEPTH CONTROL (UNTIL EVEN RETURN PRESENT) 228 CW (d) 52 ADJUST SCAN ANGLE CONTROL (UNTIL DESIRED SWEEP WIDTH PRESENT) .. CW/CCW MONITOR FLIGHT PROGRESS USING RADAR SIGNIFICANT TERRAIN/CULTURAL FEATURES AS CHECK POINTS (a) COMPARE RATAR RETURN IMAGE WITH PREPLANNED CHART AND QUICK & DIRTY (b) COMPARE SYSTEM PRESENT POSITION WITH ESTIMATED PRESENT POSITION FROM CHART IF REQUIRED (c) PRES POS (1) CHECK DDU DATA SWITCH (2) READ SYSTEM PRESENT POSITION LAT/LONG FROM LOWER DDU LAT/LONG DISPLAYS (3) RECORD SYSTEM PRESENT POSITION ON CHART (4) READ TIME- INTO- LEG OR TOTAL TIME FROM COCKPIT CLOCK (5) RECORD ESTIMATED PRESENT POSITION ON CHART REPEAT (a) AND (b) IF REQUIRED 53 INFORM PILOT OF SYSTEM NAVIGATIONAL ACCURACY 54 POSITION CURSOR INTERSECTION ON TP IF REQUIRED (a) PLACE LEFT HAND ON SLEW CONTROL STICK (b) DEPRESS RADAR SLEW BUTTON AND PUSH SLEW CONTROL STICK IN DIRECTION OF DESIRED CURSOR INTERSECTION MOVEMENT (c) COMPARE RADAR RETURN IMAGE ON CURSOR INTERSECTION WITH PREPLANNED CHART AND QUICK & DIRTY TP (d) REPEAT (e) PUSH CORRECT POS BUTTON ON LOWER DDU PANEL (a) AND 229 (b) IF REQUIRED 55 MONITOR NAVIGATIONAL EQUIPMENT FOR OPERATING STATUS OR EQUIPMENT CONDITION (a) CHECK NAVIGATION (INS) CONTROL PANEL FOR INS FAILURE INDICATIONS (b) CHECK VTR PANEL FOR OPERATING INDICATIONS IF REQUIRED (c) CHECK FOR COMPUTER ERROR LIGHT ON DVRI PANEL AND LOWER DDU PANEL (d) CHECK ATTITUDE REF SWITCH (e) CHECK MAGNETIC VARIATION FROM MAG VAR DISPLAY ON LOWER DDU PANEL (1) (f) 56 57 COMP IN COMPARE TO CHART MAGNETIC VARIATION AND SET IF NECESSARY INFORM PILOT OF NAVIGATIONAL EQUIPMENT STATUS MONITOR TIME ON TP (a) COMPARE RECORDED ACTUAL LEG OR TOTAL TIME FOR PREVIOUS TP WITH PREPLANNED LEG OR TOTAL TIME FOR PREVIOUS TP (b) INSTRUCT PILOT TO ADJUST THROTTLE CONTROLS SO AS TO CORRECT TIME ON TP (c) INFORM PILOT OF TIME ON TP RESULTS MONITOR SYSTEM VELOCITIES (a) READ SYSTEM GROUND SPEED (GS) AND WIND (1) THROW DDU DATA SWITCH (2) RECORD GROUND SPEED IN G/S DISPLAY (3) RECORD WIND SPEED AND DIRECTION IN WIND SPEED/WIND DIR DISPLAYS (b) READ AND RECORD GS FROM DOPPLER PANEL GND SPEED DISPLAY (c) READ AND RECORD INDICATED AIRSPEED (IAS) FROM AIRSPEED INDICATOR ON PILOT'S INSTRUMENT PANEL 230 DATA , (d) 58 59 READ SYSTEM TRUE AIRSPEED (TAS) (.1) THROW DDU DATA SELECT SWITCH (2) RECORD TAS FROM DISPLAY A 1 (e) EVALUATE SYSTEM VELOCITIES USING GS WIND, IAS, AND TAS (f) TROUBLESHOOT SYSTEM VELOCITIES IF REQUIRED (g) INFORM PILOT OF SYSTEM VELOCITY RESULTS MONITOR SYSTEM HEADING (a) READ TRUE HEADING FROM DVRI DISPLAY BUG (b) READ MAGNETIC HEADING FROM WET COMPASS (c) READ HEADING FROM HORIZONTAL SITUATION INDICATOR (d) READ MAGNETIC VARIATION FROM MAG VAR DISPLAY ON LOWER DDU PANEL (e) EVALUATE HEADING ACCURACY USING TRUE HEADING, MAGNETIC VARIATION, AND COMPASS HEADING DATA WITH "CDMVT" FORMULA (f) ADJUST MA-1 COMPASS NEEDLE DEFLECTION WITH PULL TO SET CONTROL IF REQUIRED (g) TROUBLESHOOT SYSTEM HEADING IF REQUIRED (h) INSTRUCT PILOT TO ADJUST HEADING SO AS TO MAINTAIN PREPLANNED COURSE (i) INFORM PILOT OF SYSTEM HEADING RESULTS MONITOR SYSTEM ALTITUDE (a) READ ALTITUDE (AGL) FROM RADAR ALTIMETER Cb) READ PRESSURE ALTITUDE (MSL) FROM PRESSURE ALTIMETER (c) READ SYSTEM ALTITUDE CD (MSL) THROW DDU DATA SWITCH 231 PRES POS C2) (d) READ TERRAIN ALTITUDE FOR PRESENT POSITION ON CHART (e) EVALUATE SYSTEM ALTITUDE ACCURACY USING ABOVE SOURCES (f) TROUBLESHOOT SYSTEM ALTITUDE IF REQUIRED (g) (h) SIO T8 READ ALTITUDE IN ALT DISPLAY INSERT CORRECT ALTITUDE IF REQUIRED INFORM PILOT OF SYSTEM ALTITUDE RESULTS MONITOR SAFETY OF FLIGHT INSTRUMENTS AND EQUIPMENT (a) EVALUATE FUEL STATUS BY COMPARING ACTUAL FUEL REMAINING WITH PREPLANNED FUEL REMAINING (b) INFORM PILOT OF FUEL STATUS (c) CHECK ANNUNCIATOR CAUTION LIGHTS FOR EMERGENCY INDICATIONS (d) CHECK FIRE WARNING LIGHTS FOR AIRCRAFT FIRE INDICATIONS (e) CHECK ACCESSIBLE CIRCUIT BREAKERS (f) INFORM PILOT OF SAFETY OF FLIGHT INSTRUMENTS AND EQUIPMENT RESULTS PERFORM APPROACH TO TP PROCEDURES SI CHECK FLIGHT PROGRESS USING RADAR SIGNIFICANT TERRAIN/CULTURAL FEATURES AS CHECK POINTS (a) COMPARE RADAR RETURN IMAGE ON CURSOR INTERSECTION WITH PREPLANNED CHART AND QUICK & DIRTY Cb) POSITION CURSOR INTERSECTION ON TP IF REQUIRED (c) REPEAT (a) AND 232 (b) IF REQUIRED IN ) Cd) 52 SELECT ARE 30/60 DISPLAY AT APPROXIiMATELY 17 MILES FROM TP AND NAVIGATION IS ACCURATE 53 OBSERVE ARE 30/60 EXPANDED DISPLAY AT APPROXIMATELY 17 MILES 54 TUNE RADAR FOR OPTIMUM ARE 30/60 DISPLAY (a) ROTATE STC SLOPE CONTROL (b) THROW ANT PATT SWITCH (CONTINUE UNTIL BRIGHTEST RETURN PRESENT) (c) T9.1 CCW (e) CHECK AZ-RNG TRKG SWITCH OFF CHECK ELEV TRKG SWITCH OFF f ADJUST VIDEO/DIF CONTROLS TO ENHANCE RETURN RESOLUTION IF REQUIRED CONTINUE POSITIONING CURSOR INTERSECTION ON FRONT LEADING EDGE CENTER OF TURN POINT RETURN PERFORM VELOCITY CORRECT PROCEDURES 51 GO TO T9.1 FOR AUTOMATIC VELOCITY CORRECT (AZ- RANGE LOCK-ON OR TRACK- WHILE- SCAN REQUIRED) 52 GO TO T9.2 FOR JIANUAL VELOCITY CORRECT 53 FLIR TRACKING MANUAL VELOCITY CORRECT (NOT PART OF SIMULATOR CAPABILITY) PERFORM AUTOMATIC VELOCITY CORRECT SI NEAR ADJUST SCAN ANGLE CONTROL (g) 55 ADJUST RCVR CONTROL (UNTIL RETURN IS ENHANCED) CCW (d) ( T9 PUSH CORRECT PQS BUTTON ON LOWER DDU PANEL POSITION AZIMUTH CURSOR TO CENTER OF TP RETURN 233 CW/CCW T9.2 TIO 52 POSITION RANGE CURSOR TO JUST LEADING EDGE OF TP RETURN 53 THROW AZ-RNG TRKG SWITCH ON 54 CHECK AZ- RANGE INDICATOR LIGHT ON 55 ROTATE VELOCITY CORRECT SWITCH MEMORY POINT 56 CHECK DDU DATA SELECT SWITCH 57 CHECK FLT DATA DISPLAY VALUE GREATER THAN 000 58 ROTATE VELOCITY CORRECT SWITCH (BEFORE TP WALK- DOWN) 4 A FOR SOME OFF SAVE PERFORM TdANUAL VELOCITY CORRECT 51 POSITION AZIMUTH CURSOR TO CENTER OF TP RETURN 52 POSITION RANGE CURSOR TO JUST LEADING EDGE OF TP RETURN 53 CHECK AZ-RNG TRKG SWITCH 54 ROTATE VELOCITY CORRECT SWITCH 55 DELAY FOR 10 SECOND MINIMUM/128 SECOND MAXIMUM TO ALLOW CURSOR DRIFT 56 REPEAT POSITIONING BEARING AND RANGE CURSORS TO JUST LEADING EDGE CENTER OF TP RETURN 57 MONITOR FURTHER CURSOR DRIFT AND REPEAT POSITIONING IF REQUIRED SB CHECK DDU DATA SELECT SWITCH 59 CHECK FLT DATA DISPLAY GREATER THAN 000 510 ROTATE VELOCITY CORRECT SWITCH (BEFORE TP WALK-DOWN) 4 OFF MEMORY POINT FOR SOME VALUE INITIATE TURN AT TP SI ALERT PILOT OF NEXT OUTBOUND HEADING ONE MINUTE PRIOR TO REACHING TP 234 A OFF SAVE 52 SET OUTBOUND HEADING ON HSI 53 CHECK AZ-RNG TRKG SWITCH 54 MONITOR DVRI HEADING BUG FOR MOVEMENT TO 180° RELATIVE POSITION (TP PASSAGE) 55 RECORD LEG TIME OR TOTAL ELAPSED TIME AT TP PASSAGE 56 ACTIVATE COCKPIT CLOCK AT TP PASSAGE IF REQUIRED (LEG TIME ONLY) 57 INFORM PILOT OF TP PASSAGE AND OUTBOUND HEADING 58 CHECK FOR PILOT TURNING TO NEW HEADING For subsequent navigation legs, return to Segment 235 OFF 3, Task 2 APPENDIX B GLOSSARY OF TASK SPECIFIC BEHAVIORS The enclosed glossary of 31 specific behaviors, or action verbs, was excerpted from Oiler [1968]. Each verb has a specific meaning and is acceptable in the sense that all Each action verb is used in synonyms have been eliminated. the task listing (Appendix A) the MTLA (Appendix D) , , task analysis (Appendix C) , and for the purpose of defining observable behavior that may be measured in terms of task performance. 236 . Provide the initial force or action to begin an operation of some equipment configuration. - Activate Manipulate controls, levers, linkages and other equipment items to return equipment from an out-of-tolerance condition to an in-tolerance condition. - Adjust Inform designated persons that a certain condition exists in order to bring them up to a watchful state in which Alert a - quick reaction is possible. Check - Examine to determine if a given action produces a specified result; to determine that a presupposed condition actually exists, or to confirm or determine measurements by the use of visual, auditory, tactile, or mechanical means. Checkout - Perform routine procedures , which are discrete, ordered stepwise actions designed to determine the status or assess the performance of an item. Compare - Examine the characteristics of two or more items to determine their similarities and differences. Proceed in the performance of some action, procedure, etc., or to remain on the same course or direction e.g., continue to check the temperature fluctuations; continue to adjust the controls; and continue on the same heading) Continue - ( Delay Wait a brief period of time before taking a certain action or making a response. - Depress I j ' Apply manual (as opposed to automatic) pressure to activate or initiate an action or to cause an item of equipment to function or cease to function. - Determine Find, discover, or detect a condition mine degree of angle ). - 237 (e.g., deter- I . , Judge or appraise the worth or amount, of a unit of equipment, operational procedure or condition (e.g. evaluate status of life support systems) - Evaluate Inform - Pass on information in some appropriate manner to that they one or more persons about a condition, event, etc. , should be aware of. Initiate - Give a start to a plan, idea, request, or some form of human action (e.g., initiate a new safety procedure). Insert - Place, put, or thrust something within an existing context (e.g., insert a part in the equipment, insert a request in the computer). Instruct - Impart information in an organized, systematic manner to one or more persons. Monitor Observe continually or periodically visual displays, - or listen for or to audio displays, or vibrations in order to determine equipment condition or operating status. Observe Note the presence of mechanical motion, the condition - of an indicator, or audio display, or other sources of movement or audible sounds on a nonperiodic basis. Perform - Place Transport an object to an exact location. Carry out some action from preparation to completion (It is understood that some special skill or knowledge is required to successfully accomplish the action.). - Position Turn, slide, rotate, or otherwise move a switch, lever, valve handle, or similar control device to a selected orientation about some fixed reference point. P^sh - Exert a force on an object in such a manner that the object will move or tend to move away from the origin of the - force. Use ones eyes to comprehend some standardized form of visual symbols (e.g., sign, gauge, or chart). ^^3.6. - 238 . Make a permanent account of the results of some action, test, event, etc., so that the authentic evidence will be available for subsequent examination. - Record Repeat - Perform the same series of tests, operations, etc., over again, or perform an identical series of tasks, tests, operations, etc. Apply manual torque to cause a multiple position rotary switch or a constantly varying device like a handwheel, Rotate - thumbwheel, or potentiometer to move in a clockwise or counter- clockwise manner. Select - Choose, or to be commanded to choose, an alternative from among a series of similar choices (e.g., select a proper transmission frequency) Set Move pointers, clock hands, etc., to a position in con- - formity with a standard, or place mechanical controls in a predetermined position. Change manually the setting of a toggle switch from one position to another. Throw - Troubleshoot - Examine and analyze failure reports, equipment readouts, test equipment meter valves, failure symptoms, etc., to isolate the source of malfunction. Adjust an item of equipment to a prescribed operating condition. Tune - Use - Utilize some unit of equipment or operational procedure. 239 APPENDIX C A-6E TRAM RADAR NAVIGATION TASK ANALYSIS The purpose of performing a task analysis for measuring B/N performance during radar navigation was to provide candidate performance measure metrics that may describe either successful task performance or B/N skill acquisition. Only segment three, Navigation to TP, was examined to limit the scope of the task analysis. The sequential flow of tasks, subtasks, and subtask elements (defined below), is the same as that found in the A-6E TRAM radar navigation task listing (Appendix A) The seven columns of the task analysis form . were defined in Chapter V but the definitions are repeated here for the convenience of the reader: (1) Subtask - a component activity of a task. Within a task, collectively all subtasks comprise the task. Subtasks are represented by the letter "S" followed immediately Subtask elements are represented by a small letter in parentheses. by a numeral. (2) Feedback action. the indication of adequacy of response or Listed as VISUAL, TACTILE, AUDITORY, or - VESTIBULAR and is listed in the subtask column for convenience only. (3) Action Stimulus - the event or cue that instigates performance of the subtask. This stimulus may be an out-of- tolerance display indication, a requirement of periodic inspection, a command, a failure, etc. 240 . (4) . the estimated time in seconds to perform the subtask or task element calculated from initiation to Time - completion. (5) Criticality the relationship between mission success - and the below-minimum performance or required excessive performance time of a particular subtask or subtask element. "High" (H) indicates poor subtask performance may lead to mission failure or an accident. (M) "Medium" indicates the possibility of degraded mission capability. "Low" (L) indicates that poor performance may have little effect on mission success. (6) Potential Error - errors are classified as failure to perform the task (OMIT) , performing the task inappropriately in time or accuracy (COMMIT) task steps in the incorrect order (7) , or performing sequential (SEQUENTIAL) Skills Required - the taxonomy of training objectives used for the Grumman task analysis was retained and presented in Table VII (8) [Campbell, et al., Performance Measure Metrics - a 1977]. candidate metric which may best describe the successful performance of the task or a genuine display of the required skills. The types of metrics suggested were classified as: TIME (time in seconds from start to finish of task) T-S (time-sharing or proportion of time that particular task is performed in relation to other tasks being performed in the same time period) R-T (reaction time in seconds from the onset , , of an action stimulus to task initiation) of task performance) , ACC (accuracy FREQ (number of task occurrences) DEC (decisions made as a correct or .incorrect choice depending on the particular situation and mission requirements) , , QUAL (quality of a task, especially in regards to radar scope tuning quality) and SUBJ (subjective observation or comprehension of the task execution success by an instructor) , , 241 The right-hand column of the task analysis form ("performance measure metrics") provided several hundred possible candidate measures for describing successful task performance or B/N skill acquisition. Using initial measure selection criteria as outlined in Chapter IV, these measures were reduced and combined with literature review candidate measures II of Chapter IV) (Table to produce the final candidate measure set as shown in Table XI of Chapter VII. 242 U o w 2 DJ U ^ a: M S 3 Pi S Eh o< u Pm W S a s ^ ^ W Oi ^ CO OI Cm Eh Pi Cm 1 Eh t/5 u di 1 w" ^tO Eh CO sum HUD < Cd Oi b *fo Eh <: CO H t3 <U TJ 0) 73 CU T3 0) T3 rH (D rH 0) rH OJ rH CU a e U a O e M u 0^ a o e a u e M u •H cu CO P^ (U u X 3 u 3 rH o. o S (U u M u •H cu CO CU H ^« Eh < Z « D O Eh Z CCj EH o Eh W -p p AA 4J 4J •H •H •H •H •H o 2 o o 1-q ^ in in o o 0) e cu W Eh < H Eh H 2 H iH Eh • • u a: M Eh Eh M S CJ o Z D OJ M3 E^ S U M < Eh tA < g O K CN 0) (U > > •H > •H •H 4J -P 4J U o •H •H u M o O M •H •H •H 0) 0) Q 1 W , C 03 -P M •H T3 j:; e M O (J^ G C C Eh ti< CO <: &^ CQ D CO ^ cr^ 03 -H -H -H -H -C TJ O Cn+J U 3 C 03 0) T5 Jh 03 ^ (U Oix: < -P u 03 -p T3 M C (D D s-i ^ ja T3 -H 0) 4J ro ^^ s-( (U 0) U <: i3 Chfa C/3 -h T3 5 QJ ^ 03 (U g x: 03 0) >-( X} -H -H OJ W en 0) >-i -i4 XI O M CO K OJ G > 03 QJ 0) ^ (U 03 U) x: G CJ « « ^ U G 2 « rH (U rH • o 03 3 JJ x: 4J 4J cn H H +J en G > 0) a Q e •H > a ^ • o (U 03 OJ 0) • u OJ cn •• cn CU 03 QJ U -P Oi 0^ 03 03 03 03 cn XI 73 > CU XI •H 0) . a) cn 0) M-l = -HO ^ > > O -H -p e -p •H 03 G ^ rH -i^ r- -P -i^ l+H M •H Cli CO G 03 03 OJ .. T3 U -H 03 > CJ U m 243 ^ O 3 •H < 0) •H CO n3 -H 03 en •• -H x; XI T3 -P T3 rH rH -H TJ C7> >i-P 3 W cr CO H -- G UJ rH 3 iTio 03 03 -P 3 3 J-) c i3 0^ U 4J C C -P Ti •H 3 O O ro 4J G 3 JG •H fi 0) s . U C M OJ -P -P -H rH •H C 4J (0 01 tj\ -H Cn . OJ (0 •H c O Q Q D D c •• E O K rH to Eh Z o M Eh < O M > < z s s < in CO H 73 OJ -P QJ 03 Cm CiJ u 2 PJ < 2 S D 2 01 O< W s fl4 w k U CO H4 (X Eh E-* w S cc; CJ 1 "» a w « fa * W 'fO S U CQ HUD < CO u CJ <: E-t cu q w trj 0) (U u M X a J Di J h4 H 3 w C ^y OJ H CU 6 U 3 T3 0) 'C 0) rH 0) a U S ^ •H M CO 0^ CU ^ < H4 d:: O Z « W o E-t —^ T3 3 a U •H E-i 4J 4J E-t •H •H CU o Di M M 2 c 6 e £-• &J O s K o o m H rH u ^-' CO 2 iJ O MD E^ S U M < :=3 rH Eh E-t ,a 0] fC <0 •H •H P 4J C C OJ 0) D 3 5T (T CO CO O 0) < etT» G •H cu TJ f-l G U :stj M < Eh CO CJ M > < z M-l D CO u U3 -p 4J -H n 3 < -M 'TJ rH •H Q4 C . cn G ^ O . -P CJ^ cn OJ ^ ^ M fd -P e ^ ,, m O ja M rH 4J 3 CO P-t < M 0^:3 HO o z o >i G M 3 Q) G U Xi 13 0) 0) -P fa 244 U 2 DJ < Cm S D S o< U Di S u t/J Cj^ CO U M Di E-i u S DJ cu a ^ U U < " Eh 1 « u a: fa o W ^ u u < Di fa Oh Eh O Eh q U C/3 U 2 H CH w w tJ oi X :3 t-4 0) T3 (U r-t OJ rH »-^ MD d biJ Cl, a u e e U v^ •H 0. CO EH 1-^ S w < I-+ Eh 2 w < > H u O Eh CO 0) a: D:? 4J a £h •H cr g 0) o en Eh Eh U < CN &^ M U C:: Eh m Eh M S M CN 2 D O M D U w < Eh i-J • « C C •H CO CQ 2 en W OJ O 5-< n3 T3 E•z o H m < 3 > < 2 ^^ ro tn ^J 2 J \ -P tT3 t-^ &i Eh C rd Eh CJ Eh -H 3 W fO -H cn cp oj c M a Q r-\ CO .. ^ U 03 C nj (13 ia -H TJ > 3 u m Q O g G^ X o u ^ u A U 03 ro Eh • -H > •• U ^ (U OJ (tJ o -p ia en T3 . <u • (U • fa CN CO 245 CJ 0^ Eh O Eh O 2 H o u CO z u u iCf Qi 1-4 !r r:) c:: K CO EO< w fa C4 S Ci] Cu U H CO CO CiJ hJ i-h CD (U U 3 U X P O T3 0) ^ •H Q) a rH a o e S •H )-| CO (1^ U (U u C M p 4J o 4J •H T) 0) 3 rH 0) rH a iH e 03 a. a u •H CO cu •H (U u 5-1 CO CU o G -P O -H 3 e D^ g (U o U C -P Q) -H D S cr e (U o ° K O MH X s in CN CN CO T3 e M ^ > cu fa g s o Pi -P •H 3 > -p O •H Q) U CQ CO •H •H Eh >, c P5 03 •H Cn> rH H o CO < % o rd rH cu M e -H (d X5 (d cu ^3 e QJ 0. Eh 3 en e -H (U c^> c -P >-i ja u^ P CO M Q a tn (u •H Q • >H en C (U fCJ -H -p en (u -H +J -H 4J 5 a, Eh tn QJ M QJ 0) CP (d rH c a< M fd e s >-i (U c (d (TJ J^ (d rH i3 (d Sh a C7> iCU Eh en Eh 4-1 •H 4J 4-> •• M un tT> • ^ Eh G -P Eh •H U (d ^ 73 -P (U 0) 03 Q (U C >4 iCU T3 03 OJ -P -H -P -H Oi-h X} Sh e en 4-1 c rd a u u 4J 73 T3 en (T3 -P g i+j 'd en +J Eh >i (T3 cn t3 CO •H Q en H c Eh w o (U 13 (T3 ro O T3 Eh > < •H 4J CO o > H en o Q W rt H D a w M u 2 CO Ci3 S D Di K CO Eh o< a s ci S u IJ-i Ci3 04 H O s M « w w CO Eh CO >• CO .. Eh CO Eh Eh T3 <D TJ dJ .H 0) e u ^-1 (U U 3 M 3 (D 4-» fd H 03 e U U v^ 0^ > •H w O e M •H CO o. ^H X 3 O e M CO 04 U O U 04 a o cs: ^ 9 0) 4J +J -P 3 4J p PH CJ •H •H •H cr •H •H 04 O o Eh <0 X D <]: M ^ u s H U Eh < -H e a* Cid 5-1 a u pa G CJ l-H M s 6 e o O Eh s s s M 2 U CO o CM iH CN (U (U cj a: Eh Eh ^q Q o K CO w "^ 1 s H a w o:: » ^ en I-) U X 3 Eh Eh fa r-< a Cm ^ w CO CJ Qi Eh W S CQ H D to -^ Oi MD 1 w S H - J CO « <: Q) E-t D O a ca U u U <; oi M- z o hJ O MD Eh S U M > > •H •H 4J +J U (U M o (D U <C Eh •H Q •H en CO e oj 2 o o c (U 3 rH in rH •H .3 4J 4J c 0) 3 cr Q o u s o ^ •H P e g g o CO •H c 0) 3 tr cr 0) (U < 2 •• m ^ CO <c Cn 4J C (d ^ CO Oh Eh iH 3 nj •H > • 4J 4J 1 ^ G WO) o< Eh fd en U Q) O rH rO (^ Eh enJ V J ^ \Di -M Cm M ^ O •H .. 0^^ C U Q rH D iH Q ^ Q u u • S C M J !-l M Q) TJ ^3 fd 0) Cm « ^^ ^-^ <d £ M Q 03 fO > rT3 (U (U CO tJ TD 0) Cm 3 w •H T3 ^ >i . 1-^ tJ -p \ W • ^-1 (d (d • U -P td ^DQ W Eh ^^ rH -P j:: Q) ^ 04 CD B^ (T3 <T3 U IW S S-i 5 (d K^l XI Oi Cn\T3 g C 4-1 0) e c •• td \ en 3 ^ 4-4 -P 04 •• Eh ^3 (U « •• w ^ rH 0^ 5-1 -H 03 U -H J-i -H 0) > 0) 3 >i o M cr cu (d td S-i OJ 4J rH i3 04 T3 5-1 en U 4-1 4J 03 Q) •H rH T3 (U ^ (d G 3 -P o ^ (u QJ en M 03 c M G H ^:i O +J -H CO 247 . 04 Eh •• M U (d ia TJ a> 0) Cm » u ^ RMAN SURE TRIG < tJ ^ a u OH fe ci3 s di W S PQ H D Eh CO - Q W H D Eh Eh Eh CO MD -H x; U D 0) T3 td rH CU (U 3 S u ^ ^ < a: ^ 9 2 +J CU Q) fH Q4 u g •H M (T3 M > w C/1 e Cu O u oi E-< M s u »* Et 2 O ^ S E-t 0) TJ QJ rH Ol (U 3 U rH e S-i U CO Oi e-t (U e g +J =! •H CT 03 03 M > 0. w P 4J •H •H e u CO K G '*~^ •H -P T! e cu G -H n ,— CO U X 3 n3 •H -u -P ^ 3 (U •H M ^ S <D 4J CtJ C 0) a 1-4 •H (U r-i a <-{ a. tT) C Q) w tn 1 H H f-3 s s s CN in rH rH in -H rH Q) 03 03 > 03 •H •H •H •H -p u 0) (U 3 (U - 0) 3 3 CO CO Ui C 4J -p •+-< -H •H CO 5-1 •H •• 01 (U Q 0) to U 4J E-" C 'H O4 W 4J •• E-» 0) (T3 +J (U 0) U B U -^ E-i 2 M < > < 2 •• ro fO W -u x: ^ CO < D CO e w Q) -P W i C ^ 03 <44 > >i -p •• 03 cn Q D Q Q 0) i cn cn ' >, XI T3 • cn cn M P M U U Eh rH -H -H OJ ^-^ ^^ 03 03 pLi £ p +J -P 03 > Q4 a (U . CO TJ -H OJ 03 0) Q4&L, Oi '^ c;3 248 a e cn (U -H -H S-I +j a g -H OJ >i> a cn cn 4J 0) S-4 Si cn T( 03 OJ g aM_l ^q OJ Pn U ^^^..^ 0) s-j S-i 03 03 n +J T3 cn Q4 • • ^ a,^ u -H a4-H fH uu 5 G OJ 03 g g • cn ^ aa G 03 G a x; jj 0) U S-I cn -H >i •• ^ 0) •H +J 4J 4J cn r-i cn 0) 'CO -H -H rH 73 -P Q) ^^ ^ U cn • -H -H 03 4-) x: -P (Tj OJ -P i3 4J -H T3 03 03 13 -P Oh rH *-5 -P M C C 03 G e Q) • c cn CO (d Q) cn J D a^ Q Q 5 ^ •• ijq cn th -H -P 03 e 3 (U W CO to U-l rH tion \ Q) 5 M Oi-H G sent 6^ 0) g -P T) -< J Oi J M MD s^ a en w X 3 0) rH a e o:: U n (U M Cm ^ < h4 a: E^ 2 -— ^ c •H Ci •H 0^ o W U Qi Eh M 2 -P -p DJ Pi E- o Q) o h-t e 2 E-* in ta "SJ* rH U tn Z 3 "îr E-< •• OJ M 3 ^ s 4J Ul 01 c (U U H < Eh ;3 < ^ 1 CO 0) a Cu fo e-« o E-* Z c M < C3 M > <: 2 E-t tsi CO < E-i DQ 3 01 -P -H CO U (DC M M -H -P 0-H -H ^-» T3 (U 3 C •H 4J t« P CO > . •• rd 44 —' 249 DJ U 2 <: Cii a u 1-4 s D e: K CO Eh o< w P4 Cd S Di S U o 2 H « w w f^ -CQ U H U CO < V cy EH cu 0) q J-l X 3 CO Cd ^J ci )J 1-4 Eh W MD s w Eh W W 05 Oi fc 1 0) T3 rH 0) a w a "'g e U u 04 >H CO H^ Eh < M < K Eh 04 O Eh hJ H W ] T] Q_, a-! FORM TO 2 CO H H LO Eh t« ci: ^ 9 2 « CJ (^ Eh U Eh 4J •H o e 04 u ai M Eh Eh M S ClI o E o CM CO 2 O J O M 3 H s U M < Eh to a •H CO CO •H S CO < Cn Eh •H 04 Eh td • • CJ M (d 0) rH trj 0) +J ^ CO Cn lit: CO <: £h cfl -73 3 C (T3 nH +J CO 3 -P -P XJ tH n C M S o z o M EH < CJ M > < 2 c; 1 -H fU iH -H •H rH g CO 0) CO OJ aMH (U 3 -H 4J > U > CO 4-> 0) (K 4-1 D x: tU CO +J CO -i^ CO >i CO CO 5 (U -H rH U +J IH ^ (T3 •• ^ U 03 -Q CU OJ Ii4 H CO CO 250 u 2 CJ U < a; s 3 ei S EH o< u fM w S a s w I-+ C/3 a* a a ^ U y < -a w Oi Cm •• CO * CO Oh Eh Cm 1 1 £h u < â 0:1 W CO a u ^ o 9 b CO X b ^ CO ciq Qi Eh Cm 1 1 Eh ^ U u < *a CO pq a; Eh Cm 1 1 Eh Eh O 2 H Z H < s w a t/l Q w ^-1 Di J M MD OJ M X 3 iH d TJ (U TS (1) rH (U rH <U rH e "'S 0) (U a a b^ OJ M 3 •H a S U U CO cu T3 a O u e 0) 0) <D u X 3 M U X 3 TJ 0) ^3 (U rH Q) rH (U rH a e M •H S-i Oi CO (1^ 3 <U a u e M O o (U u e •H !-j CO a. 0. T3 a U 5h PU E-t w z « o J < M £h z Cn <: Eh o [i Q H Eh Eh c C o a e: 0) O 3 Eh Oi M M S E^ td VO Eh Z O M U ir^ D J D S M <C Eh t« t/3 -P 3 -P 3 -P •H g cr •H •H cr (U s CP 6 (U S (D S (U o CO ^ 2 CM O o c to S +J (tS o en E-i w Q O g Eh rH •H •« O b3 CO rH <0 (d •H •H •H •M -P 4J 4J 4J c C 03 0) 3 3 a o 3 cr a 0^ CT" Q) (U (U CO CO CO < S D Eh •H . 3 (1) CO 0^ s o u 5 U <D •• 4-1 U G to • -Q Ti (U W -P fd rH -H -H •H -H •H -P -P -P M U U O M Q) ru b! 2 Z U • x: • 0) Eh • fe CO Eh O • • cu 0)^X5 d) TJ T3 M Ch CU a (U (UNO) Q -H Cm 0) Q »-q C T3 3 -P Eh a. ^4 a >1 •• M nj ^ T3 rH 3 CO -H CO SH > a -p C (U -P rH H rH -H 03 •r^ -P -P 5h U U Oi 03 M-l 03 0) < !h a • • M 03 CO 03 3 Eh CO >i -H ^ > CU >i 4H 0) ^ CO -P to -P -H o u C 03 O CO CO 03 to -H O U G 03 OJ -P (U i2 T5 Q) Xi n3 OJ -P CU U M G 3 M G 3 XS T3 0) a, 03 tr 0) a. Q) O. 03 tr 0) 0) 3 (U CO (U <D Q) (U 3 OJ CU D^ CO Cm Cm Q D^ Cm Q Cm Q -H CN m ^ un CO ^ CO CO CO CO CO 251 rH -H Oi CUM >-q ^ 03 C CO 03 E^ Eh 0) >i •H -i^ 03 -H 0^ CU^ CO CO n3 to CO M g n3 (T3 5h QJ • \ 0) C QJ •H n3 03 > 'H Eh Q, CT t/i 3 +j CD M >1 Q) a • c 3 6 C M ^ a U W > CO :j> W Eh 3 0) CO • CO -H r-i o o 03 CO (T3 s a^ (0 (TJ <-i hJ •H >i rH 4J O <d •H a CO ON rH 1 5 m s O V •H tj O iH 0) CO CO •H C 3 •H O CN rH U: CO (U +J •H < z o M fH < 3 M > <; z 0) :3 e-i On Eh O G c ai iH CO Q> o -p en S Cl H H 0) •H O Oi U o o c o (U q: 0) » w u U 2 < 2 ^ CO :3 oi Eh Cx^ s »-4 ttV C/l o< u Cn pq CC 2. 1 E-" s u Dj MD « ^ ^ a um U D o; 0$ <: CO H b « D ^ o CO 1 1 2 W M « Eh <C CO Eh Eh Eh fa ^ ^ 1 Eh ** ^^ a W CQ fa CO ^ W sum HUD «.tD V CO I-) a H b fci fa w fa - 0) 0) 0) <D u X a U 3 M 3 U X 3 U X 3 (U T3 0) rH a. (D rH a, e u o ^ 0) 6 u •H cu 0) T! <U tJ 0) 13 rH (U rH (U rH OJ a o a o g a o 6 V^ •H Jh CO cu CO Pla u - 2 U U H U fa Eh < Q OJ ^— 3 to ^ a w e v^ U CU o M CI. J < 0) I-+ ai ^ 9 2 ^ Oi Ci3 „-^ o ^3 cu •H 4J U El M 0i v^ Eh e 2 O M b* U CO D J CD S M £-• en -H +J C CO -P -H E-i 4J TJ W d (D < > a CO rH x; •H •• 4«J a-p o c fd -p cu ia 5h W T) 0)0)0) rH < r^ CO C > U o; 0) •• ^ o -P 03 CO J3 • Ti . 0) • *-3 rH 03 U 3 U U rH 03 3 TJ 0) ^ 3 3 O X: en cn 03 -H ^ !h > j:: rH • Q) 1+^ >i 5 U '• -P ;:«J CO 0) CO -P 03 DO Q Q « ^ o \-p 03 4J w Q o s OJ SH •H t3 5 Eh cu fC CQ D t3 +J Q U O fc4 0) > •H •H -P . 03 (U -C Jh H cr (u CO rH rH 4J w CO O U 2 a > O 0) U S 3 g cr e o o CO O •H < g s OJ J nj W 3 ^ >irH CO <D K C f• •H O +J n O o C 0) o cu 6h ^ z o H t: < o M > < z -P rH < •t E-* M S W G ^ P •H s 0) C -p •H CD u c P 03 E V^ rH 03 i4H 3 Sh CO -H ^ >i S^ C0 T3 -P .• ^ rH >, C O 0) G U ^ 03 0) 03 -P Sh <U 03 -H 03 i2 TJ 03 O rH a* a; 3 OJ to 4J (U -C -H 03 a T3 03 fa 03 JH 03 Q4 0^-P <u rd QJ (U (U 05 !h U O iH CO S 03 fa w U 2 bl U <C OC S 3 2 ca Eh o -l -H S-l -H X 3 4J 73 X 3 X D 4J Cli OJ Ti 0) Ti (T3 0) T3 t-4 -H (U rH (U 3 rH (U a a g e •â U V^ u Pu a o nj ^ > •H cu CO CM .H S w C M 3 3 HD « a ^ w a s w H « rH a u (U g M O (d 3 -H 5^ 0^ d > w J <c H4 cc; ^ 2 2 °^ a ci o n c: 4J G e o o 04 Q) •H •p H s DJ e-i ^-v 4J •H U Eh M Ci Eh 4J 4J •H •H o o K s K K ro o LD in M S U o O rH o rH .H O KD E-i 2 o M EH U < y: D J 3 c C •H 1 •H 4J +J •H •H W W C CO cn (U Eh •H tr cn s •H S M fd fO 3 3 2 cr OJ (1) CO G ' 2 e" M D^ 4-1 c C • fC •H Eh EH O u 04 Eh o e 2 O M < o M > < 2 E:i Eh 3 en iia U U ^ rrj t7> 3 rH 3 C .H a. O -H 03 g O > 5-1 (U OJ M 4J Ui iw M Q) <1) Eh fe^ 0) ^ U g Cli 0) Eh ^ OJ 5-1 Ti T3 &4 Q < (tJ WU 6 U <N rH -H 03 CO CO >i-H D (d > (0 •• H > Q ^ a cnttJ -H c (tJ (U Si +J TJ 05 ^ i2 OJ 0) CU G G -u Cn td CO C n-( >i fd g CO 5-1 (U ro 'd (T3 5-1 G Q) ja (d (d TS g Cm3 G (U 5^ d Q) fi rd U-l 5-1 Pm a. 0) ^-» oi^ >i (U ^ ;Q Pi4 253 a tU 1 G • 5-1 QJ -H 04 ra <-\ s-i i2 4J T3 Q) 4J aa g g 0) Q) a Eh (d 5-1 U U O •• Cn^ G U fd fd fd g 5-1 ja T3 -G -H g cn-P G -H U -H 5 W W TJ V4 flj CO +J o T) td 3 W cr» >i m W 0) (tl • CO CO •• X rO 4J i2 0) +J CO 0) ^ OJ n3 n3 5-1 rH C 5-1 OJ -P +J 4J +J Q) ^ cwxi • • m O n3 +J EH IT >i (U -U C -H 0) n3 0. EH +J T3 (U CO G 0) <U (d fc4 w u 2 CJ U < o: S D c^ 2 Eh O < w f^ 3 s CC S W W s M W -1^ S U CQ cu Eh Eh <; 03 i-H ^ 03 U W 1 Q Eh Di fa C/3 ^ «b 03 C (U ^ Di J H^ MD ^ a 0) (U U X D u U X d -H TS fC o; Ti u & w a u X 3 a o e u u cu u cu ^ < Hf Eh z ,_^ Eh T3 o Q) Oi c •H c Eh <• O 4J a; •H 3 -p 4J 1 •H •H u o U 05 Eh K £h M S U ro rH M u '—' VD a: 03 D H^ D X M <C Eh 03 4-) 03 Q s i-M o lO r-t <-\ (U Z O M EH U g s fd ^ •H u c 3 -H U U ^-1 <1J 03 Q) MC -P OI fl A-i c 3 •H (U CQ to cr •H s Q) m to ^4 < (t3 Eh rH 13 03 M-l cu E-i O rH 03 td 03 2 O M Eh < o H > < z 3 x: fT> en 4J t«J G C -H 03 03 > < -P < ^ 3 •H -P M QJ Eh 03 D (N rH . 03 TJ -i4 03 (U O -P M 03 -H •• 03 ^ 0)3'^ rH •H a 0) . 4-> CU Eh W s e M 0) 4-1 (D IH 03 m 0) (U (U >i OS M C fci O w •^ rH in -H 0^ 03 03 H •• ^ U Q, cr •• -H T3 M -P -H ^3 Eh -P 4J • ja rH tr>-H rH 0) rH C -P •H "H -H 03 03 UH M (U >1 W ro rH W 0) 03 jC jq -P T3 Q) CU +J li( a 03 ^ 0) 03 03 C -P C 03 03 -H ^ 03 +J S 03 4J x: (u ja C (7^-P T3 0) X: U ^-» JE 254 -H -H 03 .H e *H 03 rH 03 0) 03 3 -H 03 3 > U > Q) 4-> U 03 •• ^ Q) >i Q) 03 fa . c ^ 2 ClJ < IX S D U H+ Ci K tn ^ M O< Cd S Cti rî ci ^*^ S ^-4 CU -^ en <: D en OQ Eh ir* 1 a 1 -J D D -J en <: D en <: en Eh Eh "•^ 1 a 1 w dit^ S W CQ H Oi D w a s w H « w a f^ S fa CQ H 05 D Eh En en Eh fa Eh fa en -(-3 en <C a 1 Eh w"6if^ 2 H D a fa CDI fD fa 05 &^ ;iM m D m D ^ s H fa Di Eh fa en en « < QJ O c en Q EH tn 2 O H Eh < iJ Qi CjJ M 1-1 2a HH >^J 03 CO Sh X s •H •H X 3 a 't! W 03 a'4-i >1 rH >1 rH rH a, S M fd <d e (U >H rH U Q) (U 03 c c OJ 0^ < < H U u X 3 CU .H Qh OJ o e u o Oh n 0) u U o< > < :3 1-4 s H EH en >H en O fa Pi Ol ^ 2 9 Oh 3 c< Eh &J O 04 U CJ Eh H S U t-H E-. -p +J •H •H 1 £ o U U a: E ^ e 04 rEh • « en z s O^ MD Eh S < Eh G en O CU •H •H 4J +» o o M fl W «4-l S-l to 1X5 iH T3 nj M OJ C w o •H >i ro 2 LD fi :3 z s 00 JJ Eh > E CX5 > •H +J D u > 0( Oh P •• O.^ tn u -H (d n3 i2 H H 01 :3 •ri TJ G D ^ S U M T3 to to 3 M G -P < O G •H > "H 4-) u (0 -P X ^ • Oh Eh ^ > u S « U u •• -P (T3 •H \ s Xi 03 0) D •n u ^ JJ 4J G 3 a -^ Q s U u (U -H G CU CU 03 to Q) 3 rH en Sh 03 Oi-H 03 Q) U -H CU • 03 03 M . CU CU en • tu o 5h to 4J i2 T) • -P to -P .. ja T3 03 M 3 3 -P G 4J 0) •f-> G (U OJ Qj M T! > (U CU fa <: 0) fa i-i rH 03 3 CU 3 Oh QJ Eh Oh Cm (TJ ^-' en 255 ,-«, .^^ £ ^-^ o n U Qh rH !h U •• X u (0 -H JG XI -P 4-> T3 TJ (U 3 -H CU -n C G 73 > S U 3 G U 73 ^ < \ 4J -H S M G > to . .. CU rH Eh • CU G G U 3 (U 03 Qh rH +J t3 • . ^.^ , ^ rH (H en u OS 1 Z o u CU S o g rH H &H s •H en &H +J -H 6 O O CN w -p •H +J — . 3 — 5 fa 3 « u 2 C4 <^ ff: o s D « U M »: S La Eh o<u fM w a W 1 Eh Oi f^ CQ CO 1 D Eh - OQ CO UD U CO 1 Eh u u s H U W Eh <: Q S M Eh 1 Eh oi OJ H MD CU (U U c o a ^ Hi C !4 cu ^ e ^ OJ (U U 04 G c 04 w jj P •H s g 4J M P •H •H g g g g u u K •H 04 o u K o K ffi in rH in o r-l fNl in ro G c •H •H g e U Di E-« M S U 1-4 Eh 2 D C ^J •H •H •H CO CO CQ 01 (0 CQ CQ to •H •H •H •H U M s S en -g +J en c c 5-j U r4 0) -H td td rH fd (U •H J^ JJ +J 4-1 o >-l a Eh -H C rH en fC +J -H be: < D x: en U CO CO <4-l tJ D O fd v, tn (d i-i Cn4-i rd <; fO ;^ C Q) O -r-l v^ n3 (d 3 •H C S-i -H 5-1 (d (U cn 0) 0) Q) C -H (d cn 54 S-i a -t-) 2 04 0) JQ +J 73 M-i pm a < 2 cr j5 5 G -P -H g •• ^ C (d e -H fd cn cn td fd cn M -P TJ a. (U cn U H g 0) td (d g ;a U 'IJ 0) (1) fa U ^— ^^ (d £^ ^-' <N CQ 256 5-1 jj i: u c >i;Q -H TS td -P Q4 +J 6 cn tn 0) U £ > H tU +J U-l 4J -H -H oj • td D >i 4J t3 •H fd t7> 5-1 n3 !-i +J S O .H s-( U tT> 3 -H O > u td 5-1 -H TJ U X^ P •H 3 u o > o fd ^ C/2 • • (U a U 04^ (U o -H ^ (d Oi 3 4J cr> •H G a o^-H > < 2 0) G CO < s 5H O4 o S 2 O g U 01 ^ 2 2 or Eh 3 O Eh u u Q (U - c aM-i U -P &H u u s H U W Eh <: Q < D td •H 1 « CX4 t/1 T3 CO U 2 u «— a w • •• TJ r^ u H g P • ^ T3 fd 5h (U —'-H D :J -f^ • a w «» «> u CQ U W D U Q (^ < ^ - ^ cn «k e^ 1 u cn fa CU Eh <: en Eh Oi fa Eh fa ^ a«w s fa T UD u cn < ^ ^ pq o X s CU u e U "'S CD! &^ ^ Nk h:i 1 D sum HUD -f^ cn 1x1 Eh Oi fa cn 1 « w -a Eh fa < S H a D W V o 1^ m CU 73 rH a, CU e ^^ CU •H (13 X S CU S-i H aiH M U }-i atH e M cn &< Oi a 03 X e CU H o < D s n CU u CU Q) c^ CU So; Eh O 2 •H Cr: 'a c -P C 4J e 3 •H !T •H ^ ou ° 1-:) K K 4J -P •H •H 04 M Oi Eh M S W o Eh g e g U •H 0) p bj (H Eh lil o <-* c 4J g u rEh • « 2 O M H U < cn D J D S M Eh tn O X K LO in o H o CU ^-^ g CU cn > H (T3 C •H c •H -P -P U U •H W CU u •H Q (T3 cn T3 •H (0 « S c CU a •H 3 0} CQ er •H CU s: CU u cn cn en 1 < Eh «« a u en CU >i w H r> • fd (XS in < EH s D cn -P rH >i e T3 -P -P cn d c 0) (T3 M M 3 3 -H < CO M-l M 3 a •• ^ -P o c 03 (d •H -H T3 •H CU 01 -H 0^ &H > (T3 > C d H C CU -H fa Oi 4J fa •<!3' cn cn cn Sh 3 (T5 a. ^ il_l 4J Q) C H Ti (t3 ., M CU O (13 [5 CU s-i cn (T3 cn JQ -d CU to CU (U fa O H H p cn -P 03 cu CU O 3 u TD CU cn -HO cn ^ c M u CU Eh: U >ir (t3 s-i 03 • •• -i4 cn S-l G U CU 03 M u a e CU a-p c u M > H 3 (U 3 Q J3 'O U e 0) O CU g 03 Sh g (U uH ,»^ ^M» £ u ^:5 S-I H TJ* ^ i^C u a G ^jC G UT CO (d 257 -P T3 2j cn fa V M 5 3 03 o G x: U G-H 03 ^-s ^^ s^ j:: ja "3 CU T- c cun 03 T! p aâ cn -H 4J CU •H -H -P -p c G -P > u <-{ -r-i -P -P H cn <H T3 -P c 1 -H (0 U HH ^ G CU M a C U 1 a T3 CU ro x; -H -C rH Eh -P U c CT3 O .. ^ c u G o cn (13 &4 M-l cn 4J -H c: CU CU H •H CTrH t3 CU S O -H •H -H CU S H ^ CO C Q4-H • T3 03 rH •H -n CU >i M M W 3 u -u i+H O -H i< • to U -M CU o H z o M H <C O M > < 2 e Q) CU +j 03 H -H a D Q a. cm. ^ 1 w CJ en 1 METRIC aiT) W cq Eh oi D (J4 CO 'ERFORMANi MJIASURE &H <: CO oi &H Oi a CO u u Q S U CQ HUD Eh < CO < QJ Eh 1 05 OJ O a n X g 0) s^ n3 X e 0) u p REQUIREID r-i CD T3 Q) rH 0) r-^ a a«4-i e u o U D « u G SKILLS >D CQ w 1 h sou H U W l-U a u O g M •H M X 3 U 0) T3 rH Cu (U Q44-I g u g U cn 0^ 0^ 0) CD u 04 u cu ^ < Di ^ 2 z a: 1-4 M ^-» •H 4J •H g S 05 o P P +J •H •H •H ^ ° g uo M U 0:J E-i M 2 4-> -P -H E-i K K 3: 3: 'O CN 00 CN o DJ g o •-H rH ^ fC CO c Z D O hJ c c •H •H CO cn Q> cn CO CO •H •H 4J r^ Eh C •H •H U H •H CO S 3 en S 0) CO f4 CO < — ^ E-" E-» -p > O Ij ^3 =5 •H cn T3 <C U U c 4J CU 03 O &-* • T3 < Oi c • r-i •H M > <c z " CO <c a D CO (d ^^ --•H —v O cr fd u (t( 0) 0) ^ J3 TJ ^ ^ Q) 3 c +J (U u 5 CMC (T3 (U Eh <-i c Oa <d -p CO 4J Q) U-^ Q) 3 3 Oi -H t4 04 J3 .o^ <«« 2 0) c m u ^^ O u ^ C -P U c U =3 n •H •• ^ c r-l rM 4J S-l (tJ (TJ *"* t« -i > C -P S-4 C c •• C 3 2 03 U CU 4J 03 x: C to 0) 0) u fa -CO -H D^-P 0) ^ u , — 03 '^ 3 CO 3 •H -H • -H CO 03 t+-i > c •• •H C U -i4 g W a3 03 (U S^ 03 Q) -H XI •H -H -P 13 T3 +j S-i CO > cr MO) LD cn CO tn fd H -H >U^ 0) -P -H n3 C fH M D •H nj (U (U •H +J U ^ n •H rH 03 CU -P •nag p a 03 bei o 1— rH C71 Q) CO M C O D Q r^ (U . ^ cn +J z H Z M H 03 03 U J3 •H TJ T3 0) C --^^ •H Q) fa 1 ' CJ u 2 < si ^ U DJ a; H+ D ^ 1-3 a a D ffi u u < CO SW o <: a t^ E-i CJ S oi ^ S D CiJ cu 1 X cn Q ^ MID a K D a T3 CD T3 Q) iH (U u U a 6 u u H 0. CO 04 ^ •H Cd ^ 1^ Eh CO sum MUD < D Oi CO 0) u 3 (U -r) rH a ao 6 3 0) 1 0) 5h 1 Q) U X 3 >H X 3 0) T> rH 0) a e Sh O CO 04 ^ O a Eh Oi " 1-3 &H CQ P5 t4 CO rH a, g a M -tr5 Eh CQ ^^ CQ CO .. T! rH a o Q) u e M U 04 ^^ 04 J < M Eh z c:: -p +J +> -M 4J CJ tf &H •H •H •H •H •H a o „-^ n 05 O g e o Q ^-q S D4 g s o o s ^ ^ o O 0} C U •H 4J Eh Ci M Eh in M S M o CN O e Q o fi o »— r^ Eh C/5 2 hJ O M D Eh S U M < Eh :i5 Q) 0) (U OJ > > > > •H >1 •H •H -H -P H a 10 4J 4J -P O O CO 0) V4 iH u a u •H •H •H •H U Q) U •H Q cn < M E-i Q W C H o i- 4J rH (T3 ra U 3 (U -H 'O 0) a -H e •• U U fO c . > Oi-H TJ "Z o M H <: a M > < z ;^ CO OJ Oi Eh < &: ca D in ^-^ T3 0) M -H 3 O -P CP 03 nJ CU i3 -P -H U ,_^ u U ^ "O—1^ ^ -H •H *««-l^ Cd cn C O U U Q) Q) auH j:: D fi •• m > ^ tJi Q 3 0) 15 •H T! > S-l iH ^ 0) (T3 > u • (T3 CJ < tU rH rH ja 0) 0) T3 0) U O fd 0) x: (T3 a aci^ • U ,^ H +J G • • M cp Sh fO > (T3 s x: 03 14H J3 3 W 0) Qj Cn M M C G 0) (T3 rH 4H P +. a •'- r^ •H 3 = < CT G >i 03 • r-{ r^ x: -H -H -P HD OJ -^ •H QjiH u n (T3 j:^ -r! o-H 0) UH -p Q) OiPh ^ . c e c w o: M 3x: G M ^^ ,_^ OJ U-l ^•—l^ g r-^ o G a (U W G O U > U c 03-P r-i e e M -P TJ > fd o -H •H cn • î 3 G > U 0) -P -P • p w a-H •H • 2 259 C d e o in Ol (TJ •H „^ ^^ r-\ -i^ • QJ . a !T>'d O u oi D w x: ^ Ci, (tJ 0) -P 0) ^ 0) -P iH -H UH rH M MH V^ c Ti Q) •H rH 1 D Q Q n3 H D M OS Q a > Q -P Q rH G Q 1 -H 4-1 Oi Q N^ 4Jt: +J a a acn &. 03 03 DJ U _. , .. en 1 METRIC ^ •• a w Eh 05 -<c U D ua < :rforman NiEASURE W -1^ S U OQ MUD Eh <: W uu Q^ Eh 1 « Eh Cn CO l-t >-} C 5^ rH ^ o ^s 05 -P X g u X 3 03 0) 3 CJ TJ rH i-\ Q) )H Q4M-1 rH OiMH S M Q* 03 g u g U > CD w (h u (U c •H c U a. P O OJ ?5 Dd M 4J •H •H g g s U E K in in rH 6 U Di Eh M S U E-. C K CM -P 3 4J -H CT" •H g (U CO ^-5 LD a; cn Eh • U flH •H P u_ r^ U Q) < Z S Eh O (U •H J Eh S U oq H CJ D Eh < CO CO 0) G (U W cq « D li. 1::^ G (0 IIRED 1 Eh 05 Cm âf-3 w « s H 05 D O ;lls ^ CO 01 yS >» UU u w <Q > Z D k3 O MD &4 X U M < Eh C G c •H •H •H CO (Q CO 0] (0 Cfl •H ^ •H •H C/1 s •H 4J S s Q W rH Eh iH E-« 03 P <d c T3 0^ 0) E-i O Z o M Eh < z (U D^4J Sh < ^ 03 ZD CO > OJ e < z •H -P u en (U M •• M -P i3 •H -n Q) ro S Cu O W ô (1) CT> (U 03 g 03 U 3 -P CU-P Eh ;-( 0) 1 M-i cu g -P 03 a-P CO ^ !h 5 en C U 3 -P rH -P rH -P •H 04 )H x: -P -P 03 (U g cn -P CO 03 3 -P CJ u ^ u 4J 3 -P M to -r-i C M C -73 H 03 U U V< QJ OjCm ^-v „^ s ja C U IH 4J -r^ C CO tn -H • -H g CO TJ -H -H -P -P Eh rH r-i •• M •H CU U 03 JH 0) Pn CO 260 Q) M •• M 03 cu ja Eh T3 C C »»»• 0^ CO MH ,_^ 3 < 3 u g M -r) OJ >i -P cn < 2 -H 3 U (DO) -P 03 Q) •H OJ 4-1 -P rH +J -H XJ x: 4J > T3 -H -H 03 s .0 rH -H Oi Sh rH G, rH -P 3 CI. Q) 03 a -P g • CM Eh 03 T3 C > a o; M g (T3 ^-1 rH 0^ U -H n3 iTJ C •• -H -H +J 0) U-i G lili CO 4-1 -H C U 3 E-i a 03 M .)A CO 0^ (U -H Eh +J 4J (d QJ -H (U 0) Em CJ ^ ^ {J 2 W U < QC S 3 C? to w o< t: s s w 1-+ o:: E-' P-t ce; cu ^ cn 1 Eh ^ ^ CQ u D U CO N' ' . a M a w w « oi fa ^ ^ a a w . « fa oc> fa fa <: ^ w S H ^ *» EH a w oi Eh D5 Ch 1 ^ « ^ W 1^ CQ S H D W CQ S H D CQ S M D W Tj CQ S H D Eh CO Eh CO Eh CO Eh CO (U 0) (D 0) 3 U 3 X 3 1-3 fa I-: 0) O Q tn cj ^ 05 hJ t-4 MD J-l X s dJ u X 3 rH e u i H+ c o '—' r^ t^ t« M Oi E-" M S 2 O M U < E-t a 73 (U 73 OJ rH 0) rH Q) U S u •H Sh •H U a a e e CO Oa CU a e M u CO 0^ 4J 4J 4J -p +j •H •H •H •H •H g U •H 73 rH O u a< O W o a. 3 C O 0) £2: E^ •z en Oh ,.-^ TJ a eu Sh (U s^ C D^ D^ cr 2 ^^ m O M CO U (U > t^ Ul <; E-* <: C C -H 3 5 e (u CO rH T3 O M Oj en 73 - TJ rH +j rH 4h tt3 •H e M +J CO Q rH ftJ 3 M T3 to CD C -H nj e (U CQ > ^ 4J CO O T3 M O^ 73 >. CO (TJ 03 C/1 M u ^ tU 0) fO o a (U -P i3 05 en fa H rd C S . OJ &4 — (T3 ^^ rH n3 03 OJ OirH rH nj (U 03 rH 3 73 CL CO >H CO M -H CU G 3 CO to 3 M Sh -H n3 -H u -H 03 0) 03 . aa CO -H -H 73 > OiTD •• rH c TS -Q 73 OJ 73 • C M O .. (U 0) >i-- w 0) 0) CO 3 0) >irH O M .. 0. 0) 73 Cli e 03 OJ ^ a u 73 M a oi MH ^-v ^ — r^ CO 261 03 73 73 G 03 c e Sh en 73 +J 73 0) CU Sh 03 ^ 03 X! O -- 73 73 -H CO -H 03 73 i< 73 (U (U 0) G cnfa (Ij .H a> -P !h +J MH OJ G 0) g 3 fH ^ • 3 tn -P CO -H <C e Eh > dJ •• 03 — — 4J to 73 ^ CJ >i QJ U 03 to (U 03 ai2 4.) X! 73 73 tn <U 03 M OJ OJ -H (U ..H -H fa Oi 03 fa Sh H C G — -— ^— —u s tn 73 0) » « w u 2 CJ U < Di s D S 05 < u O ^ t4 'Z a; g u ^ ^ U cq U D < CO t-4 ct; . « *• u cn u Q 1 1 Eh ^ w o s w H W fD S CQ H D sum HUD 2 H Eh Eh Cn Eh <; cn Eh ce; cu w ^ * «k cn £h Oi fa £-• «k ^ a Cl4 ^ ^^-) fa Q u tn fj Oi ^ HD « O y-* O G CO 03 4J •H X S u X 3 (0 to 0) (u -d 0) r-i Q) rH D â M >i'H .H 0,4-1 e M coo w 1 a fa Pi fa c <a •H rH », -a Eh O o (U C ^ U D U cn < ^ CQ CU OiMH S e ^ u Pn 03 X E M u U cu 0) &i h:} < HH o: p ^ o 2 IX *-» 0} U -P c 4J Fh Du •H 04 o o T3 •H •H E-i Di M M 2 e Eh DJ u ^-^ H •« -p -p -H •H o K K ^J o K LD CN r-l U3 o 0) H> C •H +J C/3 Q in 03 2 D iJ O M Z2 ^ s U M < E^ M •H e e uo rH en I^ g 6 C 4-> C •H o to CO a; 3 ^4 to (Q cr •H •H •H 0) s: •H Q s en t/3 c <: . 03 CJ CO -H E-) tji 04 Eh P o CO P-i U CO CO <: Ch oa U Ci] CO -p 0) CO 03 03 CO C 03 D 03 > o crj CO 0) 0) -•^. ta 0) p c •H -P ,, ro -p a rH -H T3 rH Qj >i i«5 ""^ -H ^ en •• <^ U 03 -H '^ n3 T) 03 rH > C (U -H W > 5 4-1 -H <; cn (1) . •• to -H -a -i^ (U -P x: (U 0) OJ Sh Q) Q (U fa Eh > 5H fa <— OJ UH ^-^ '-^ r^ en ai rH 4J V4 C 03 •H fi UH +J > •H a u rH -H S-l 03 Xi O -H Xi d D T3 rH cp a '— >1 >i +J -P •H -H ^3 U D (U > e e M 0) • -H M <; cu CO o 03 ^ 3 G >. OJ 0) CO M fa j:: CO >i CO 'X3 -P U 03 -p rH -r) <: rH rH CO -H a o to 4-) , •H e-H u (U o 3 P O CO <-t O >, Q) >i S W U -P 4J Eh 03 OJ -P 4J 6 C 0) 2 o H Eh < O M > < 2 < H to e (U Q) 0) fa 2 CO > . u Z - u <j; u u DJ CJ Di M. E-< * s k4 a C/2 Eh CO &H CO Eh fa -i O <: ^ S CQ H D Q u M HD fa S H D E-t tn vJ Oi iJ u u W S H D ffl CU « w « ^ 1-3 M i< Oi fa < s D »: S to < u o Cm C4 S Di S - a a w •. » •H a u e U •H fa fa 1-:) M 3 T3 OJ 73 <U 13 CU rH 0) -H CU a o e U en CU CO CU JJ H •H a o o e M •H CO CU i-l CO CU « H O z 3 cc; y-^ 'O 0) •H 4J G o Eh «• e 6 04 O Oi M H M s u E-1 C/J r» 4J •H E-" z o iJ O MD S U M <^ in o K s s s O O o o rH rH rH m (d (d •H •H •H -p -p c c G G Q) CD CU (1) 3 P a 3 3 cr CT CT" ,rH 03 • rH C nj fd c/3 t;J •H 73 r-( x: cn CU a W O fO 03 -H 0) -H jG 'O > Ph S D CO (U V^ t3 -P M «^ > Q 3 C 'd -Q n3 T3 (U cd •H 0) >4 (U Q) Oi "4-1 0) CO U (T3 •H fi CP-P ^ 0) U g 3 "3 (T3 e i2 73 U 0) Q) Oi iw fa G >-v ,, O *-' <TJ ^-^ D Q Q 1 •H •H o CO cr" c c i3 P-t u Ci] (Ti 3 CU n O rH (0 •H E-4 < O M > < 2 g o o^ C-t g o < z o JJ •H O P E-< -p •H e -p to H 3 -H 4J CO •H -H U-l 0^ •H rH 13 (d (d (U ^ J3 00 CO 263 C N td 3 > S-t CU cci G f^ S G g •• ^ 3 w -H > r-\ G ^ Di td x: -H fa Id Cj) -P CU (U M > a •ri P u fd <: 5 . x; 13 -H -H 13 (d M 13 W C — u >-» ^-» •H fd M g "H 13 C O >i fd • en C -H -H Qi +J 13 td td (U — , 13 — td rH rH j3 a. 0) 13 (U O-fa s » W u Z CJ CJ '^ rt 5 xpo: a O < CJ fa w 2: OS S u tyj ^ 01 u W Q ^ W "h) S U CQ MUD Eh < 01 - 1 1 04 U w U u w &H U Q < ^ ^ w -a Eh W S M a; ^ on ^ Eh E-i ^ ^ 01 ^ Q ^ W S CQ H D f-j 1 Eh 01 0) Q) Q w en J C X 6 M M MD ^ a ^ S3 a i rH •H rH CO W M •H £h s 1 rtC •H a CQ w CQ -H •H S Q s C/1 0) (U 5-1 >4 -P C •. . c; 13 -HO) 0) en -P +j 03 1 u S x: -H -P •H cno; CUT) T3C01GC 03 D [SrHrH Q)cn3(T3 x; 3 e « U5 < ^ ca D 01 'd 0) D C •H 4-) C 'O M-i •• U ^-^ 1 S 03o3(UC-PÊho3 -p >i ^0^3 03 •> Cn-H (Us u > XJ rH3aT3-HMS'T3 3 M G -H W 3 03 03O0303MWQQJ >U-H(U0303U<1) w 03 cux; > as fa CO en -p •H 03 CO H 4J >^ :3 CO CP w -H 3 H > <4H rH •H T3 -H a •• X r-^ CO H -Q T3 JG -P T3 3 < C C 5 U OJ (U fa -^ T3 <c s M (T3 cu .. ;i<J a> u C 03 ^ i3 T3 05 (U (U (U H Eh x: fa ^-^ C7^ ^^ C u 3 M MH x; -H (U 03 03 cu -P MX! -H — tn (U OJ OJ MH CT +J TD , 3 3 U +J "-^ 264 (U ui (U Q) 03 01 0) •r-i ^-' cn 0) -p MH CU 'X3 H -H HH 6 ^ 0) a Q) <q . t3 -P X (U ,— e 3036 CMPaU ^-H C a rH Oi n3-HS^cnracn6>0 ^-^ O u to o ro •H v^ c cn cu t^ •• P • 01 &H z CJ cu •H < > < u cu Q) H z O M H <c o M in cu c a 3 xi • • U <u ,-» Eh e 0) cu HV QJ a: z 9 [^ X 6 0) u ^ auH e u < E-^ •H a aM-i U c: fO M X 3 (d Pi iJ 0) ' W u 2 < i« ^ CJ Oi U M 03 1 s 3 e: S 07 Eh O < u fm w s (X s Eh W ^ «. O U 01 cy 1 « w Oi 01 w 1 Eh 05 fa -h) w" S U OQ HUD Eh < 01 Eh -»-:) S H ffl H-+ ;^ a "'S s H4 Eh Z 3 i-< o '-N n3 U 1 O fa 05 fa 05 fa u fa 1^ fa Id S CQ H D S CQ H D Eh 01 Eh 01 0) Q) M 3 M 3 O G 0) u X 3 X s 0) M -H atw S M ^ MD W a Q) fO (x; -O Eh Pi Eh 05 fa O •-3 U D U 01 < W*" S U HUD Eh < 01 O c q ^^ - CQ «i fd X e OJ ^ H TJ ao (U S (U u Pu <U H 0) H a am s ^ u U a^ e •H CJ 73 O T3 Q) rH 0) a u e ^ •H 01 04 0^ u M 01 0^ 0) o a: c O 01 OJ oi u -P D M •p -p •H •H •H P Cr •H •H 6 0) s e s O 04 o o n: J hj in in 01 s o o (D C u •H +J c Eh ci: M S o -"^ r-- Eh • • M Z o M EH U < Eh DJ o CN (U w > 3 ^ D S H •H C5 o o nH iH (13 (0 •H •H C 0) 3 O •H G cn 0) CO a; 3 w ^ cn •H •H •H Q P P •H S 01 K IT) G 4J Eh K in rr s cr 0) 0) Ol rn s • < cn • 0) Eh (tj 0) -P -. OJ 73 cn ^-3 4J >i U ^^ +J CU Eh •P Eh < o M > < 2 •H O DJ CO VI o cn D to o •• ^-» e-4 X) OJ D C •H 4J U 3 M -P C O C 3 e C cn rO cn -r-i G H -73 4J -H -73 03 4J X O fd JQ T3 -H T3 cn G -H > 4J U-l tn u D 4J J3 4J -P -H 3 T3 <i; rH (T3 •H 0) ax: . >1 cn •H P H (t3 cn 4J (T3 0) rH i2 -P 3 T! tn e s d) ftJ >i -P 0) 03 X 0) (w cn tn oj G u u (U >i CU (U (T3 cn s-i (JL, -P i3 -P H 03 (D 03 3 -P M -P •H 03 (U M 3 u ^^ --«. <>^ x: •H ^-^ *— CO G (U S fa a> W 01 265 (U M H Q) 03 4J e cn -H H -H < e — '-H 3 P rH -H T3 > VI m cn e -H -H > -P H cn -^ cn 1-5 -P T3 -^ H 0) 01 0) 03 O VI s vi 03 S-( 03 a— 3 0) •• •• (T3 H 3 < < rH Oafri Q) -P 13 M e 3 U -P O . tn <U o cn S-l X a +j 1 •H oj 3 ^ o 03 e ja T3 T3 <U tn 0) 03 T3 0) CU (U S-I QJ a 3 M 0) 05 M-i fa T) 03 tn 05 -P •H Ti ,, ro . -TJ 0) 03 03 (U (TJ be: < S D u< (13 1 C a ^ -H G a -H o t-( z o M cn M ^^ 03 "«^ ^^ £^ i3 no Otfa s ^ dJ u 2 ^ CJ U Di )-+ a w D a: S tn o< w f^9 s a s 2: ^ ^ •-:) « U CQ U D Cm <: en ^ 1 £-• u a D ce; ^ «» la cu ^ W ^^ S OQ M D w a 2 W H « en Eh &4 E-t Ciq - ^ cq 1^ CQ u S MuU W tn < Q w ff; CU ^ X 3 M X 3 t-f cu 'O (U T3 O rH <U rH a, cu MD lii cu a "§ e u J < M ^ 9 Z CJ a OS U o u e u U cu cu c c CU (d fd i^ X g cu 3 C •H u -P c o rEh •• +J -P E^ cc; M •H •H g e o E" M S U X IT) in o e M g u g Q) cu u cu u cu g g o u .J J •H •-' -P -H •"-I g g rH 2 D O hJ M 'D ^ s U H <: H •H •S -p -P 4-) c C c cu CU <u Cfl 3 3 3 CQ (0 tr CT 0* •H •H cu CU 0) en en en CO iH (d E" ^3 c c •H •H 3 r-l -H -P ^ n3 tn td cu 3 g 3 :2 en «»«* . ^Xi S c CO >i .. > S-l TS CU •H S-^ ^— 03 a ;^ D §:: W •H -P w HD H (d ^c: C Ij-h • •H « p >i w u Q. en (d > g cu w u 33.. (J en s-i 5^ U O -P U u (d fd fd (d rH >i cu ^ . fd •• M s , 1 4J m s •H 04 u IT) •H en g g ^q in iH o u a. 4J H rH cu m O cu CWU H o hJ < O H Z o M fi < o M > < 2 T3 au-i p 04 0) CU +J E-« t3 X 3 ^-1 <-\ cc; .» Ctl iH Q4 u cu D M U E^ < Q Eh en X e cu u M 1-3 sou S H D rH o W CQ Eh 05 Cm en ^ - Cd 1 « *» (U ij en 01 Cm en E-' 01 ij *» u H Q en cDJfo ri^ en td ^ g cu cu >^ 1 •H cu cu 4J -P -H +J TD rH • rH g 3 (d ^3 fd en 3 >i er en cu Id M 3 ^1 -U u-i 0-H < en cu ^3 -H 3 fd -Q -p XI 3 cu CU fd TJ cu > 3 X) irH ej cu en <: sh cr u cu 3 IH -p -H ^ > •• ^ U td cu XJ 'd cu en -73 cu Q) G 3 cu 4-) Cm 3-H 'TJ -P r-i cu H _^ ^^ U^ 3 ^^ 3 i-( 44 rH (U -P -H en -P cu !-l cu n3 > 4J j:: Q^jC jq O cu Uj u Z *k u <C Oi ^ ^ ^ S to O< w Cm C4 S en Cj I-+ 1 Eh •. •«> a w Cm E-" DJ * 1^ sum MUD < W u CU Ctl CQ D Oi Eh 05 Cm CO 1 X HD w a Q) n r-l 0) a e â U H « &a a ClI ^^T) s u pa HUD Eh < CO s w H tf Eh Cm C c •H •H 4J +J (0 to Q) 3 3 n r-i Q) rH rH Oi (d rO e (I) ^ 1-4 >» a Cm 0) 01 cq ^q E-t 1 Eh C/2 -Of^ CxT S M Eh q U PQ U D < 1 « » cn -•d <k -u u m Eh U Q ^< CO u U > > CU cq cq M X 3 U U M 04 -3 < (-4 ff: o E^ z ^.^ OC C4 d:; 4-> +) P -p E-" DJ •H •H •H •H g g s n3 o Qi 0« c •H C e M E-i ^q E-t M s u in X in c: C c: U •H •H •H 0) CO M CO CO CO CO CO •H •H •H ^ cn Q) > •H O kJ M3 E^ S U H < • :r in o Z D • en a; en Eh o u u r- o +J •H Eh < 3 i:: >i fH en -P 1 •H Q4 rH -H (U +J E-t 73 -H T3 o -P 4J P -H (< U^ 04 d E-t z o H H < CJ M > < z •• ^ < S D H ^J •H tH a nj C/1 e-i CO ^_^ T! (U 3 C H -P u CO -U M-( C H c o <4-i >1 . W +j ^ (T3 rH XI 3 T3 t/J c/3 0) >. 0) 0) W M Cm -P x: CP P 3 -H -P -P 5 (T3 CO CJ Cm rH c OJ cn-H :3 c T3 (U C M-4 ft! 03 sh i 14H (t3 W M 3 4J ciq M-l Q) OJ 4-1 UH (TJ cn Cn rH 03 a g C en •• g G C ^ 0) 03 -H s-1 rH c: rH >i W^ 1 Q) 3 M MH Qj a x: C W C C OJ 2-H S OJ --^ 0) fO Cm rH in 267 ^^ 4J U-l -H T3 e u 3 +J < rH . td Q) oj > > •H X a • 03 Ot-H X} U rH -HO) (0 -H rH (U -p M^ +J Cr a\ w +J C 03 TJ e 0) <D 0) Sh Cm g W 3 )-l u 03 j3 -p 'd OJ M-l 03 C -P (U 0) Cm H •H JJ 4J T3 ^.^ ro O g e M CU M-l s S s Q t/1 ^-% XJ ^^ u u Z C4 CJ r^ D:i *k H4 f^ OQ Id 0} l-D D m D cn cn cn ^ ^ Eh Eh Eh 1 1 D "Z Z2 (H K UJ ^ o< u b i ^ OS g « cu 3 iJ -1 1 •» o u Eh OJ &4 ^ .. CJ tn cn 0) <U (U <0 u 3 ^ P U 3 u X 3 0) T3 l-H -H 0) a W O wg 6 •H 1 X « Cii M3 W îT) S U CQ HUD Eh < cn ^ a 73 0) 0) rH •H M OJ g M a u rH 04 0) o e e •H cn 04 cn 04 a)n3 TJ )H U cn o. 0) M 04 ^ < H4 ttl ^ 9 z »: gs o ^-^ -o c •H p c u Eh M a; +) -p JJ •H •H •H g s 04 <u +j •H Eh M 2 U e s o o o o X Ml K x CN 00 o O o o o r^ Eh «t Z o M EH U < cn 3 J D S H c C G G H H •H •H 01 (0 CO CO to •H •H w W •H S Eh CO H S s: s C/1 to iH < -p M Eh V^ P 04 Eh o C 3 G &-" 2 O M Eh <C O H > <: z • • s*: en < &: CQ D cn ,-^ T^ CU 3 C H -P G n 3 O U O en fd -H •H +J t3 -G C nj CO -H G td •H 3 w 5-1 -H 03 > cox ai ^ O >1 C C 0) •H Q) -P -H -H -H rH X! •H ^ -P CO 03 03 G Eh 03 O o a u Xi f. 03 -P (T3 CO Q) Sh -H UH 03 ja M G '13 o 0) (U JZ (U S -H 0) Q) +J 1*4 > ^ U ^^ »>^ u 3 q-i -H T3 .. ^ CO GO -H -P Xi X (U 0) x: 03 Q) cn n iH '-{ -r^ Q) rH tw Clh t/3 268 X U > n CO 4J a 03 3 U U e J3 73 Sh Sh G O H jG -h U • a; OJ fc JJ 4H X: T3 rH 0^ G 3 •H 03 CO -P rH Q) 4H CO M 4J rH •H C -P •iH «-«» r^ u • CO OJ M O • O ^-1 0) fd Q) to (T3 ^-' cn fT3 o^ x: rH -H rH S 4J 3 CO 3 . M G O G a CO M -H (D U nj Cr>-H -H rH (d cn<w 1 M-i • m >i -P 0) >H MH G H ^-v OJ UH ^-' •-^ c e oj 3 e 0) Sh Q4 -P -H CO (d G CO -H 3 CP (U U u 2 U < a: 2 D K tn o< u a i! w Ci3 V-+ ti: a CQ 175 cn Cc; D Eh IJM cn W 1 Eh E-i u ^ - 1 •» sou HOWQ ^ W J 2 ^ <: H D S H Eh 05 Eh •» 1 Q) tn ly »J Oi M MD iJ « en CU en ttJ 2 H fa Eh fa cn fa CQ cc; U G X S fd X 6 a ^ X 6 0) u X s u S-i -^ e ^ u (U u fl "H <-i aM-i ai+-i e M e u u cu D 0) fd 0) a: 1 « ^ 01 1^ (d ^'U w •fc fa Eh fa nj -H a Eh *» u u Q c Q Pi P^ (U U c 1 Eh en « D Q W u o H cn ^ t-j CJ fa UQ < ^ ^ W ^CD< en <: ^ w ^ en ^ CQ 1 &H <C cu ^ U D » ^ cn u cu CU^H S M 0) u a< a cu o Eh < M K U <: o Eh Z U Eh O D5 &, W 0^ 00 Eh 0:5 4-> CtJ •H +j •H £ S ^ u u a; Eh M S U i-H Eh E cn rH c c: •H •H CO cn n cn CO ca CQ CQ •H •H •H •H S cu U O M Eh 2 iii cn <c Eh 03 D cn p J2 !T> p -p C C rcj -H id'H a. ^ ^ 0) 4J ^-1 +J -H fd ^-1 +J 4J M -H u 13 rd fd (d OJ i-i Di 73 M j:: p -H cn 3 u > -p u (d M rH cn O s fd 3 (d rH (d ^+-4 M C \ •H U 0) M M rd C M -H M cn rd U ^ cn M c 3 U -H +J C (U 0) cn x: -H T3 td M 3 u 13 --a <; OJ G G Ai (d -H cn (d S-I 2 G U i2 ^ fd cu 0) a (U 3 4J 1+^ fa ,—» • G . W T3 -H G > Tl -H +J ^ -H OJ G •H cn tj* -p !^ a •H (U iq cn -p G <w fd f-t cn U3 269 'j 1 .. ain G • ^ " — -Hu ^ G (Ti (d -p cr fd X5 XJ fd 0) 0) M ja 73 OJ cu CD OJ (U U-l 0) cu -H -H fa OJ -H fa ^^ ^^ u > G.t^ QJ ^ (d ^ S -H 2 73 <3 cu OJ G U ^ (d ^-^ fd 5-1 -P T3 cn -P G -H 0) fa r-{ fd fl •• ^ rH 3 ftJ Ch cri3 Qj Ti <c 2 e CD e g •H Eh < a •H in < Eh -p •H e E in Z o M 4J en 2 3 h4 O M 3 Eh S U M < ir^ • • -4J •H Ci 04 < O d:: O fd ' » t DJ 2 W U < d: M s 3 a: S ta H o4 a 0) TS rH a. (U •H M X 3 -P 03 0) 3 Ti rH (U a fH e 6 ra > >^ w cn Oh U G 0) U rd X S Q) U rH 04 1^ u g U M U 0^ 0) 04 i-:i < ^ 9 2 S ?1 c:J ^ •H Cu •H p 4J •H •H e § -tJ •H e g E t-" K E-" M S U CN O CN O o r-i c» &H t > U < •H •H -P 4J G D 1— U CJ (d 3 TJ D^ td « E-" •• -H d) (U hJ g o ffi O 1— -P ou &H t/U •H o u Pi — e g o U +J G M •^ •H e O 0) 4-» OJ a u c o •H (U cn u cn •H -H cn Q CO 't^ s: CO Ui w >i D Q Q 04 (d <-{ (U ^1/ •H CO M E-t o< O -p 0) o :5 fd 4J z o M s o M > < 2 *• oo O U cn ^ CO < s D CO U Eh f-i 1-H ^M > \ O +J G m g -H rH rH -H Id -P 3 W • (d -H 13 a Eh G -P fd « 0) 'tj 0) <: (d 0) -P CUEh afr4 u a. -P OX vo \ o a rH S-i 00 w S < 0) cn nj 3 .H O. cn > +j n3 U cn M-4 5-1 • cn a > M td -H Oi rd •• ^ Id T3 oj o .-1 (d M rH -H X3 g TJ '^ >i fd a cn • CU >i oj (d -H T3 o ^ \ o cn G p X! -H r^ (U T3 rH fa O EH ro •^ cn cn r. W « < D a ' » ^ w u •» 2: cJ < « s u 02 l-H 1 -I-) CO CO Eh CO Eh 1 tyj w 1 «» «» w a s w fa S M t-4 iJ t-f Jxi Q M <0 u 3 Oi M D o c: l-H — - ^*^ CO Eh rt Eh tJd O t3 c O Ci3 M U CJ Eh M 2 e O 2 D J O )— ID &H S U M < Eh t u >1 •H M HS U CO CO CI, fl^ oj 4-1 e (u O CO Eh J o O (D cu 0) (U 0) > > •H > > •H •H > •H •H 4J 4J 4J +J O 4J o o Q) 0) 0) •H •H U !h Sh •H •H •H Q Q o +J > • +J l-q [2 U T3 03 C Eh 03 u •— wJ O P C rH . <-^ -rA ,-^ •• M r-^ Eh S < G o iH 03 4-1 a -H > cn OJ !h U o 05 •• a^ ^ C U C D c^ CO CO >-.H 03 (T3 — P r-^ > u Di CJ s U U (J CO G 4-» V4 r-{ 03 -P J3 CO T) G i2 T3 5 OJ rH -H U C U P 0) • 4-1 4J Q) c Q) p U oG Q) T3 fa (d Eh . . • cn C ^ G Q) < M 4J 3 E-t CO • O 2 04 -H > *4H 05 1 . ISJ rH < • •• • X • u x: fd Sh rS<J 3 i3 4-» 4J â •r-i C u 0) -H CU T3 O ^ < O ,_^ ^^ 2 271 03 05 CO —u W « a Xi '^ . CJ 4J .-^ "— M > U 2u <\ s 2u < -H M (U x: -^ CO Q • fa ^^ 12 CJ • 4-) fa c 03 r • ttJ Oi OJ • 4J ^-^ U ^~^ u • . 4-) Eh -H CO Eh 4J (U <: C -l-l c^ x; • 0) p C • 4J •H 4-> U U CO Q 4J T) CO rH -H •H o Eh cn t3 < O J CO <: 0) CO K cu tii cr E O CO m Wf Eh +J •H OJ (U o G (U 4J 3 •H 0^ e O ^ •H o D •H tr u CO cu 0) o c 0) G tu P a ^ 03 cu Eh S o o c e 0) m O 0) c 0) H > T3 rH 4-1 E- 2 O O Q) •H tr Q .. t3 cr pg t/5 OJ rH Oj -P s Eh CD x: 4J •H e M 3 (U Q) O PQ u o G CO 0) ^ Id T! o c 0) 3 O <; CO CO <u Di Eh a CO a^ Eh fa (D 0) S-i o:; U U 3 rH < « w a w s H Oi CO 0) S a fa •H T3 •H ^ a fa «• «i» M X P 0) o "N (D rH a D a 1 E^ £h fa Eh fa tn a •* Eh Pm D4 -^^ CO <; fa W Oi w s H oi (-*H -^ < D m D Eh cc: S Eh o— (U 5 OJ CO fa ^' — » ' a u 2 W U < 2 D c: S CO t^ o< u W 2^ Ci S u ^ t-4 cc: -f-^ en <: a w 1 « Eh fr^ 04 D a a 1 Eh «.<; UD Ua < ^ Ehâô U CQ U D w s w H « S H <: en Eh Cm Eh D5 -1^ Cj-i h:1 en Cq CQ 1 cc; D li, en (U Q 03 1x1 (J 05 hJ M HD W u •H (0 w X e (U M 0) 'd 0) 0) jC u 4J a.M-1 G u e M >^ O S •H or cn r-^ a CO PJ c Q) en cu M U en (U 04 v-5 <: M -p d:: H o •z CJ — •H ff| 6 00 H M U Ci E-« M S U Eh O ;-:i X o in in 2 3 >J O M D S U M < (D > > •H •H -P 4J •H CQ (U 0) V-i 0) •H •H •H Q Q u « cu 5si CO < a p CO >-v TJ QJ P C •H +J G ^^ o td ' « H > w J w X p w . M-l -H > M-l • •• . X • u -C 03 O Xi O -M T3 (U -H U S (U OJ cn Cm ^ — U-l — G U c rH E-« d S (U E-t M > G U !-l E-* en o rH < o H 2 o g o u o ^^ E-t • • 4-> •H s O W ^ • U-l • >H -P (U • X 0) 5-1 (13 tT3 M cn D P -H 01 O > •H O fO -H +J (T3 -H (TJ G ^ a 01 G -P G ^ G -P 3 M OJ 0) U U M-i U M in cn en rH G Cn OJ -p -p 0) -H CT 0) fO M G a D M iH -H G -H cn -P TJ CL -H TJ G ^d G OJ G -P , 4J cn (t3 H£ Q G u (d i2 T3 0) cu &4 <: 2 . U u 2 <C 2 CJ S o < w fats tyj w w ci 05 cu w fa hJ oi hi h4 HD u a « "g « o u « c; C •H •H +j 4J fO fC 3 3 iH rH 03 > > O 6- Z W •H fa Pi e e o 04 u a; M E-" E- M S fa cu Z O M U < £-• w D J 3 2 M &H 01 +J +j D5 w ^ M < 1 Di 1-4 cr: H •• •» Eh ^ EH ^ fa >H E^ a\ fa Q 1 t/i ^ O U fa Q E-t q Di C^ fa E-i § Q W u o o U Di hH Z) Oi H 6 B -u -H g -P -H e a o uo B E o O C G •H •H CO CO w CO •H •H S S ^-1 u •H e-t +J 1 (d 03 6 CU E-" -P o P E-i (d •z o M en < o M > »< z ii^ M Ul < S D M-l '- M 03 -P (U ^ -H >-( U 3 cr (u c M o; rH -P -H D G (T3 c e D — -H f3 • e +J O U Q) U ^ U-i G CN o (T» >, Eh -P M U r-( •H U S-i rH o W C • O a> to Eh 4J >i •H o CD 4J -H x: a > ^ 5 (Tj O (t3 1 OJ a nj CO +J S U (^ G ^ CU (J (N C/1 ^ U U (T3 > -P !h 03 • — >, a-P -H >i S-l -P -P rH -P -H G -H O a; rH "H (U en •H tJîH -P D rH rtJ 1 fH 171 • fO 4-J r-i ^^ m O N < ^ s-^ E-t CO 1 H t^ fa rH 0) i3 ^ (T3 s^ a 3 03 > u o (U PO 01 273 + u u 2 •. bJ <! Cu u w Pi U CO •— D « U> Eh o< u P4 CJ S IX s :s &H t cr: !JJ Oc 1 Eh W -J 2 U < HUDa n3 X e 0) 0) a 1^ H Eh < s o £h D < ^ < HV DC ^ o z cc ^ s o Sh m D CO Eh CO CO 1 H a<4H a.M-1 e 0) U 04 «» ^-3 W a; fa fa fa s H Eh fa (U 0) u 3 5h Eh ^ 3 3 0) 0) ID 0) ^3 dJ 73 0) rH a. (U r-{ <i) r~{ 0) rH M E w •H a S <D •H a u u CO 04 a Q4 6 •H V^ e •H Sh CO 0, CO O4 CO p 4J +J -p -P 4J •H •H •H •H •H •H E e g E s E O G QJ ^^ 04 ^ «k «b w a S fa H OJ 01 « >H e ^ U m D ^ a w c 03 X S "S " h) CO D CO W »J 2 u < HUD Eh < a hJ Oi iJ ^ f^ CQ fa o c M3 m D w « Cl3 M > u tH >-:> *». «k Q t-5 1 « Eh <; t/J H u o w »* en P4 o u >^ Eh •. a w Oi a a. O u Eh M a; M S tH CJ HM n o ffi K E s E CN CM CN CN CN 0^ H • (T> EH •• 1— 2 O M EH U < in n J D S H &H cn < c W z o H U 3 O cn < EH C (U 3 c G -P CO (U rH n3 3 S •H N CO -r* 1+^ > d) u •H C td +J CU J3 T3 3 3 3 CT" CT IT cr Q) (D -i 4J -P a) (U 5 o^ 73 CO rH OJ 3 G « a (U CO Eh C7>-H C ^3 03 fCJ 0) . G M 3 -H 4J C OJ -P •H i^ CO -H > •• ^ u •H T3 rH C 03 •H 3 CO •H U 2 « > ^ H -H c 03 U 0:; > Q M (U ^ &-I Eh en CO cn 274 • 0) . a . Cl4 1 M < ^ U C U > G •H CO 4-1 •H H CO e a > •p 0) E -H > • •• • M • -P JQ j:: T3 0) <-{ 03 0) jG -H (U rH fa Q) r-i >1 >i 03 -H 4J ^ 3 -P rH u G 03 a rH 0) ^ -p T3 Q) CO 03 cr> , p rH 03 • >• 1 to 03 CO U 03 CJ Cn 0) CO QJ CJ H (tl •• X -P •H U 3 M 5-1 c G Q) 4J -M O4 3 Eh CO U JJ u J5 • • m O +J . c U 03 03 ID 01 -P P u fe; <C O H > < 2 •H 0) Sh i«5 03 •H ^ s CO Eh 03 •H CO M o 03 •H X •H Eh 04 Eh 03 CO •H rH •H rH r-l r-l -P ^ u 03 u ia 03 -P TJ -P -H 0) 5 0) CO fa 03 3 03 03 Eh T3 D Q Q ^ U 0) x: U Oi ':!• in vD CO CO CO U > • << • X • x: 03 u ja -P T3 -H 0) 5 Qj CO fa u u is^ S D a w c^ W S s w a 03 fr. E-^ o<u Pm c; 1^ cu s H w" W f^ m s M D Eh Eh 03 w u a tn ^q Pi 1-3 1-^ MD « C en y Q) Tl 0) rH a, 0) rH e •H TS a o e U M •H 03 0^ Q) o 03 a, J < M 6^ ^"^ z U o 'O <D E-I D C C O a; Oh CJ u E-t ct; 4J +J •H •H e s o cu •H -U o: M E-I M S W o K w CM O o rH <-{ rH • cr> Eh .• crj z s J o MD H s U M < Eh cn tT3 (C3 •H •H 4-» 4-» c C i S (tJ nH Q. Cu E-i Ui o o M E-i !< O M > <c 2 •• r»l O W CO t< C/l <C to 4J ftJ D Q Q -H j:; 03 -P D Vj (1) 4J 0) r-^ CUU ^ M a (U oi OJ fi U^ r^ 03 ^_^ M u C 5 CU > >1 CO p -H •H > U D 03 CO y-l - CU TJ iH rH I fC3 -H 3 -M -H cn fd -H ^ 5 > td Eh M-l rH fO 3 (U ^ 0) -P T3 s 3 M-i O (T3 p-i C/1 -^-'f U G •H Ti u^ E-) •z, CO . o o O .. CU cu . ^ M > • CU (0 cu x: -M O M o ^ .. ^ o (tj ^ +J IH x5 T3 (TJ rH 0) 4-1 -H CU nj OJ > « S i3 Ix, en — CU O Ci4 00 03 275 • w u 2 DJ U ^ DC H+ S 3 S CO E^ o <; W Ph U S w V en 1 - ^^ CO CO ^ D w 1 W S U < HUD Eh < a CU Q) G (13 td o M 3 Q) (U '-i rH CU Cuu-i aM-i e u e M e OJ 0) •H U u Oi «k a C4 CO CO B^ Pi 1 ^ S H Eh CQ 1 (D M 3 3 CU n:3 (U O o T3 rH 0) rH a. cu M •H ^ •H a S P Eh Pi CO !h T^ -1^ tq cu S CO cu CO cu a. ^ w a W S H Pi ^ fa Pi &4 X s U ^ MD w a "'g »k a w ^hJ o c •k m D Ci^ W -J S U <C HUD Eh < a X e QJ U en »j Pi ^q H-V H u o ^^ OQ » q w >l Eh 1 Vk, Oi Eh Pi Pn PL4 cu « « o u »k W Eh Pi o:: Eh ^ O w u M CO P^ ^-; > J < y^ < D 2 < s t-f Eh 2 W Eh o Pi O PC -U Pi •H W -p H P 4J •H •H g g s e s o o o o -p •H o cu o U Pi Eh rH . a> E-t Pi Eh M S U 2 O M EH U <C •« M K w K M-t n: 00 fN (M cN o o o 05 D hJ D S M •H i-* •H C s < M • G u a d -P U Q) U a. Eh o Eh 2 O M ^ < O M > < 2 x: -p 04 D Eh i^ m < •H Eh 03 (T3 e N D a cn •H iH 4J 4J •H G 4J M Q) •H C -P (U -H U cn 3 G 0) 3 (U cr CP J5 U 0) 0) CO CO -P ô 0) (d a W OJ -H en > •• M 4-) U G U 5 H 0) d d o^ Ti CO 0) O rH ;:4 rd 3 Pi ^ en H -H > en G -H ^ (tJ > CJ 2 c M ^ 3 U •• • • 1 M tS] < • U U-l (d M-l XJ H^ (d XI Tl -p CI) ja ri»:J t3 O OJ 0) cu . x: . • Cu •H -P M cn 3 cu cu -nE-t IJu u u <d EM G cd a. H >1 >i JJ 5h fd 3 -P •H cn o -H fd rH e CU • > • CU P > •• ^ • U x; (d Eh T3 G M 3 u u CO (U O CO M y-i >t H \ e d e (tJ H -H cu cu S cu cu -H en Cu Pi -P -H a X U 5 U tH 00 rH CN i-{ XI fd -P TJ 4J -H cu n U CO CJ CU cu -H -H U 6 -H -73 Q O U rH CN ro 'ST in c/3 cn cn CO CO CO 276 ^ 1 cu Pi • •H fe cu • CQ CD tn cr o cu -P u G ^ u •H G nj dJ Oi C 3 +J •H <4-4 W U d u G • l+H H 1^ •H 4J r-i rH rH M to E-t o -p ^ W W tfi o rd -p X . (d -p xj g G 3 -H a e M cu e -H T3 Ct^ IW rQ » > u 2 ^ o -1-0 - U Ci4 hH a: O:: ^D in ^ K a< a fM w s « s u 04 ^ worn 1 u CO W D w Q 1 Eh Oi 01 &4 «k K E-* ^ W -i-^l s u <c MUD Eh < a ^ ^ CQ D OQ a u en en OJ fa (-D D w a ô S W CQ H « D V ^ w s H u s Eh Cm en EH Eh «> ^ t-3 fa 2 m H D i-i Eh en OJ U 0) w 3 s ^ Q) U .J OS -^ l-V MD !^ o X 6 0) u a a^-i "'S O 3 -H r-i OJ 0) T! 0) T> Q) -H (U r-H (U S U e 0) •H Ph en U Oi G XJ a a^-l 3 QJ rH a u o CLi e M •H A O S •H 5h en &i Sh cn cu ^ < M - e^ — tt; o -P 2 « M a; Eh w o Ti 0) 3 •H o M -p •H •H e S o Cu •H 4-1 +J u cc: &-< M s u E-« e E g o 4J H •H o 5 s g O U O K E X un rH (N ro CN X un +J -P H O 1— i-> rH rH CN • C\ E-i «• 2 o M fH U < en :d (T3 •H J D s M 4J Eh cr CO 0) M C -P 0) d eo <+-( i-< -H 3 M ^ Cn W C •H 3 C O 5-1 o Eh ^ Ul < Eh ca 3 01 TD 0) tT> G ^ d M 4J ftJ (U C U •H (U -P Cn^J Oi •H W -H C (13 03 0) 5-1 iH T3 4-> a. en -P fd C W d 03 c 0) 3 m C cr <D a en rn U Q) CO -H !T> O a D M-l -P ^-1 S-i D U iH rH 0) 0) a fH cr (U 03 U -P 03 (y rH en 03 •H T3 3 i^ 03 (li (U 0) M-t -H a-H 03 03 U jG -C 4-1 •H 4J -P (U 5^ M > •• OXO S-i (T3 (U X5 Ô GO) <D 4-) ^^ DOW ^ 3 U C y-j 'a -H M G c 03 -H -P -P -P S Q •• ^ O • • -i4 -G n3 O i3 73 4-1 (U CD -H 03 03 ^ •H Q) u-i eT> P iH 4J >i -H Eh -n+J 0) Cli «• G 0) 3 OJ ^ e-i •z c (U 3 "0 fO cu E-" 4J 4J 1 < ^ 2 O M < O M > < •H &• U Q en to rd H +J 03 o OJ 0) 4-1 u 03 G (U 03 H <-i Cli tji 03 G D Q D Q rH a, 03 •H > 4-1 rH cjH X 0) a a 3 0) x: U o^ en a e o o O ^ "^ UH G ,, ;^ O > 1 " rH 03 •H 03 03 Mh Uh (U > 5-H OOi • Eh . • ^ 3 > •• X 0) 6 !h 03 03 (U ja 4J U OXl 'TI Oj 4-> 4JUH t3 -H 0) 0) 03 Q) -G 0) 4-» fa 5ja Q) 05 03-^fa o rH cn 277 0^ (l)t3 >. 03-^ 4J U3rH fH 03 U C (U u 03 ^ > 0) fa U u 2 ^ o •. CJ u c: 1-4 s D 2 to E^ o < a f^ w :s oi S ct: w 3 J « ^ H4 tn Eh Si &H V Eh Di fe W 1 w a w w 1 *k a m CO ^ O 1 fa B^ Qi fa Oi X Pi^ fa w s u H U Eh < -^ U CQ U D Eh <C to PH Q) 0) •. a CO a e u 73 0) T3 0) T) CD .H 0) rH <U e u ^ a U S u u cu a o u •H cu to Oi ^ W s *» rH a, e U ÎD HUDm 1 fa Eh <: to 0) OJ T3 rH QJ a u g u U 0^ u M 04 eu tn ^ Eh < S « D < H Eh H H O M Eh «• -p +J 4J p 4J Eh PJ •H •H •H •H •H 04 o o Eh Eh ^ O 2 £4 u rt e M g g g O o ^ o U Di Eh 2 tJ h:i K K Eh M S U lD o LT) o CM 00 LO 0) (U (U CO 2 D J O M D EH S U M < Eh to o > > •H > •H •H -P +J +J U o dJ S-l 0) V4 •H O O U •H Q <-) c •H 03 CO (0 W H H •H Q Q C •H S s; to <c Eh 1 OJ G -P -P 3 d c 4J g o 0) Q) Eh C! C C 2 O M Eh < 2 * • m O i^ CO Eh ca 4-> < D CO rH M T3 rH (« (TJ Q) a U « « CO Eh xi 3 T3 (T3 QJ U < -i^ 0) -p (13 -p -d M G O d rH C d v^ -H 0) S-l 0) acn ^ -P O d (T3 -Q ^ H • -P rH TJ iH itl d 0) •H > tJJ 2 ;«! ^ . ja +j t3 •H (U OJ . OJ (U x: • Q) pcH • t. fO en 278 (T3 i-i CO CN CO -H 0} en > > en -H (d a; <+-i O u d w -H H +J W ^ > OS C > o a Q e (T3 g G M S ^ to 0^ Eh H (0 ;C -P -P •H y-i CO a: OJ rH 5j> 1 to <: !-l C (0 (U -P g d (T3 cn •H -P (13 +j a^ (t3 T3 > D^-H (U +J rH T3 M . M O TJ OJ 0) tn (T3 cu cn XI T3 tri (d rH O. OJ o (T3 en (u Eh 0) (U rH (13 cu fa o: OJ QJ "^^ •• ^ -H q; g (U Oi X5 £1 TJ •H 00 (13 +j O • en •H (T3 cux: rH tJIO C CO •H ^ G H cn 5 o^ c >i -H +J -H tT> O 13 -H Ti d . Eh X M-i tJ> o x> •H CU 0* Eh x: S-i — in cn afa V . U o 2 DJ U ^ Di S D S cn Eh O < DJ tn CJ S cc S W t-H £2: cu w p w u Oi k4 M M :3 j^ a en w q: ^ CO a w 1 « Eh Cl< a. a u « fr^ W -^:) S U CQ HUD Eh < CO V £h 1 « 0) Q) u X 3 U X a 0) T3 Q) T3 rH Q* OJ <-\ 0) e U a O e u U 0^ U U < (U v^ 3 u H a -u M •H u 0. CO &( OJ e a J < K-+ ct; £h z — Eh TJ o QJ D^ C •H U -P C Eh rH Eh • • o:: Cii M Eh M 2 U CC; u ~^ o o Z O M Eh U < 03 D hJ D S M E" CO 4J p +» •H •H •H e e o e o o re s K o in o OJ rH r-i > nj •H •H -g -P H +J u C cu M (U :3 •H cr o Q (0 C 9 0) tr (U rn en CO <: Eh . ,— >i ^ cn a G aEh r-l U o ^ z o M Eh < o M > < 2 •• ro (^ Eh e 0) -H rH rH •H -H +J -H 03 Eh tic: CO <c ^ ca D CO O u en CT> 0) fO -H rH ^ Eh n3 a p (T3 > w w fcj a. !T> T) -^ > -P e M 03 -P 3 3 c .H 03 cu -P < rH XI -H DJ 01 .-H C •H -H a-T^ 03 03 • cn 03 MH W C 03 tT> C M O 03 -H ja T3 TJ OJ 0) a^ U Q) M 0) x; U 0) > " M 5 03 U Xi U C d) CO Xi UH (U lii > CJi-P CJ U3 CO 3 W -P C cn H -H 03 TJ 0) fO M Sh 3 3 rH •H 04 C -H TS +J i T3 fd CD ro ^ CU a -n C 03 •H JJ Di •H e C7> > CO r^ CO Q) 03 CO CO tn -H Eh 279 APPENDIX D RADAR NAVIGATION MISSION TIME LINE ANALYSIS The Mission Tine Line Analysis relates the sequence (MTLA) of tasks to be performed by the operator to a real time basis, and can be used to identify critical tasks within a maneuver that are important for performance measurement [Matheny, et al., Using the segment three portion of the A-6E TRAM 1970]. radar navigation task listing (Appendix A), each task/subtask was listed along the vertical axis of the time line. The estimated time to perform each task and subtask was then extracted from the task analysis (Appendix C) and plotted along the horizontal axis, which represents in this example a seven- minute radar navigation TP-to-TP "leg." (1) Time is coded as: dark if the task must be executed for maneuver Success or if the task requires complete operator attention, or (2) shaded if the task is one of monitoring or troubleshooting and can be performed simultaneously with other tasks. The MTLA is a large graph but is presented here as two task pages (Tl to T7, two time pages (0 S3; to 3+30, and T7 , each followed by S9 to TIO) and 3+30 to 7+00) . By removing and appropriately arranging the six pages, the full MTLA graph will result. 280 SEGMENT Tl 3: NAVIGATION TO TURN POINT INITIATE TURN AT IP ALERT PILOT OF OUTBOUND HEADING SET HEADING ON HSI CHECK AZ-RNG TRKG SWITCH MONITOR DVRI FOR IP PASSAGE ACTIVATE COCKPIT CLOCK INFORM PILOT OF IP PASSAGE CHECK FOR PILOT TURNING ACTIVATE STEERING TO TP 51 DEPRESS TGT N ADDRESS KEY 52 CHECK COMPTMODE SWITCH CHECK FOR ACCURATE STEERING READ SYSTEM BEARING AND RANGE SI S2 COMPARE BEARING AND RANGES TROUBLESHOOT STEERING IF REQUIRED SI DETERMINE ACTUAL TP LAT/LONG S2 COMPARE ACTUAL/SYSTEM TP S3 INSERT CORRECT TP LAT/LONG S4 EVALUATE SYSTEM PRESENT POSITION INFORM PILOT OF STEERING TO TP INSERT DATA FOR NEXT TP(s) SI THROW COMPTMODE SWITCH S2 DEPRESS OLD TGT N ADDRESS KEY S3 DEPRESS POS ACTION KEY 54 DEPRESS QUANTITY KEYS 55 DEPRESS ALT ACTION KEY S6 DEPRESS QUANTITY KEYS S7 ALERT PILOT TO MAINTAIN HEADING S8 THROW COMPTMODE SWITCH S9 THROW DDU DATA SWITCH SIO CHECK FOR ACCURATE DATA ENTRY Sll REPEAT INSERT IF REQUIRED S12 DEPRESS TGT N ADDRESS KEY FOR TP S13 CHECK FOR ACCURATE STEERING 514 REPEAT S12/S13 IF REQUIRED 515 INFORM PILOT OF STEERING TO TP PERFORM SYSTEM NAVIGATION TASKS SI TUNE RADAR FOR OPTIMUM DISPLAY S2 MONITOR FLIGHT PROGRESS 53 INFORM PILOT OF NAV ACCURACY 54 POSITION CURSORS ON TP S5 MONITOR NAVIGATION EQUIPMENT S6 MONITOR TIME ON TP S7 MONITOR SYSTEM VELOCITIES S8 MONITOR SYSTEM HEADING SI S2 S3 54 55 S6 S7 T2 T3 T4 T5 T6 T7 281 -0+30 (TP) - — « ////// //////// 1+30 1+00 0+30 2+00 /////// /// — /// / V// '//////// '//////// V///// / / // / //// /// f 7 /// ////// /////// / /// /// // /// — 282 2+30 3+00 3+30 3+30 4+00 4+30 5+00 5+30 • 283 6+00 6+30 7+00 SEGMENT T7 -0 + 30 (continued) (continued) S9 T8 3: MONITOR SYSTEM ALTITUDE SIO MONITOR FLIGHT SAFETY INSTR. PERFORM APPROACH TO TP PROCEDURES 51 CHECK FLIGHT PROGRESS 52 SELECT ARE 30/60 DISPLAY OBSERVE ARE 30/60 DISPLAY 53 TUNE RADAR FOR OPTIMUM DISPLAY CONTINUE CURSOR POSITIONING T9 1 PERFORM AUT0I4ATIC VELOCITY CORRECT SI POSITION AZIMUTH CURSOR S2 POSITION RANGE CURSOR S3 THROW AZ-RNG TRKG SWITCH S4 CHECK FOR AZ-RNG LOCK-ON S5 ROTATE VELOCITY CORRECT SWITCH S6 CHECK DDU DATA SELECT SWITCH S7 CHECK A-4 DISPLAY S8 ROTATE VELOCITY CORRECT SWITCH T9 1 PERFORM MANUAL VELOCITY CORRECT SI POSITION AZIMUTH CURSOR S2 POSITION RANGE CURSOR S3 CHECK AZ-RNG TRKG SWITCH S4 ROTATE VELOCITY CORRECT SWITCH S5 DELAY FOR 10- SEC MINIMUM REPEAT CURSOR POSITIONING S6 S7 MONITOR FURTHER CURSOR DRIFT S8 CHECK DDU DATA SELECT SWITCH S9 CHECK A-4 DISPLAY SIO ROTATE VELOCITY CORRECT SWITCH TIO INITIATE TURN AT TP SI ALERT PILOT OF OUTBOUND HEADING S2 SET HEADING ON HSI S3 CHECK AZ-RNG TRKG SWITCH MONITOR DVRI FOR TP PASSAGE S4 S5 RECORD TIME OF TP PASSAGE S6 ACTIVATE COCKPIT CLOCK S7 INFORM PILOT OF TP PASSAGE RETURN TO SEGMENT 3, TASK 2 54 55 284 0+30 1+00 1+30 2+00 285 2+30 3+00 3+30 3+30 4+00 4+30 5+30 5+00 6+00 6+30 7+0( " — — — - -- // 7 // //// /// // V // // /// 286 //////// APPENDIX E SEQUENTIAL SAMPLING DECISION MODEL This appendix presents the sequential sampling decision model and its parameters in sufficient detail for the reader unfamiliar with the background theory of the model. Much of the material is excerpted from Rankin and McDaniel [1980] a in method proposal for achieving improvements in the precision of determining FRS student aviator proficiency using a Com- puter Aided Training Evaluation and Scheduling (GATES) system, GATES provides a computer managed, prescriptive training program based on individual student performance, and could be utilized for the evaluation portion of the model to measure B/N performance by either simulator software incorporation or by desk-top minicomputers. 287 . I. GATES DECISION MODEL One sequential method that may be used as a means for making statistical decisions with a minimum sample was introduced by Wald [1947] . Probability ratio tests and corresponding sequential procedures were developed for several statistical One of the tests, the binomial probability distributions. ratio test, was formulated in a context of a sampling procedure to determine whether a collection of a manufactured product should be rejected because the proportion of defectives is too high or should be accepted because the proportion of the defectives is below an acceptable level. The sequential testing procedure also provides for a postponement of decisions conThis deferred decision is cerning acceptance or rejection. based on prescribed values of alpha (a) (a) and beta (3). Alpha limits errors of declaring something "True" when is is "False" (Type I Beta error). (3) limits errors of declaring something "False" when it is "True" (Type II error) In an industrial quality control setting, the inspector needs a chart similar to Figure El to perform a sequential test to determine if a manufacturing process has turned out a lot with too many defective items or whether the proportion of defects is acceptable. As each item is observed, the inspector plots a point on the chart one unit to the right if it is not defective, one unit to the right and one unit up if the item is defective. If the plotted line crosses the upper parallel line, the inspector will reject the production lot. 288 ; If the plotted line crosses the lower parallel line, will be accepted. the lot If the plotted line remains between the two parallel lines of the sequential decision chart, another sample item will be drawn and observed/tested. The GATES decision model focuses on proportions of proficient trials (analogous to nondef ectives or correct responses) whereas, in previous applications, proportions of defectives or incorrect responses were the items of interest. This approach does not alter the logic of the sequential sampling procedure or the decision model. It does enhance the "mean- ingfulness" of the procedure in decisions concerning proficiency because the ultimate goal is to determine "proficiency" rather than "nonprof iciency. " It should be noted that in the industrial quality control setting, sampling occurs after the manufacturing process. In educational and training applications, sampling occurred after the learning period. sequential In the GATES System, the sequential sampling occurs during the learning period and eventually terminates it. The GATES decision model can be described as consisting of decision boundaries. Referring to Figure lines represent those decision boundaries. El, the parallel Grossing the upper line, or boundary, results in a decision to "Reject Lot" crossing the lower line, or boundary, results in "Accept Lot." a decision to In the GATES system, these decision boundaries translate to "Proficient" and "Not Proficient." Galculations of the decision boundaries require four parameters. parameters are: 289 These four -p u x: u en c •H .H a e td CO (d •H C 0) td o •H P Q) x: o a >i u en > M &^ u CO S O w Eh H Cm Z W Q 290 — . p. Lowest acceptable proportion of proficient trials (P) required to pass the NATOPS flight evaluation with a grade of "Qualified." Passage of the NATOPS flight evaluation is required to be considered a trained aviator in an operational (fleet] squadron. P- Acceptable proportion of proficient trials (P) that represent desirable performance on the NATOPS flight evaluation. Alpha Beta (a) (3) The probability of making a TYPE I decision error (deciding a student is proficient when in fact he is not proficient) The probability of making a TYPE II decision error (deciding a student is not proficient when in fact he is proficient. Parameter setting is a crucial element in the development of the sequential sampling decision model. Kalisch [1980] outlines three methods for selecting proficient/not proficient performance (q^v/q-, values) as: — Individuals are classified Method 1 External Criterion as masters, non-masters, or unknown on the basis of performance on criteria directly related to the instructional objectives. These criteria can be in terms of demonstrated levels of proficiency either on the job or in a training environment. The mean proportion of items answered correctly by the masters on an objective would provide an estimate for qSimilarly, q, would be the proportion correct for the non-masters. . . — Method 2 Rationalization Experts in the subject area who understand the relation of the training objectives to the end result; e.g., on-the-job performance, select the q^ and q, values to reflect their estimation of the necessary levels of performance. This method is probably the closest to that now used by the Air Force. The procedure may provide somewhat easier decision making since specifying two values creates an indecision zone neither mastery nor non-mastery. This indecision zone indicates that performance is at a level which may not be mastery but is not sufficiently poor to be considered at a non-mastery level. . 291 The scores of prior Method 3--Representative Sample trainees, who demonstrate the entire range from extremely poor to exemplary performance on objectives, are used The proportion correct for the to estimate q^ and q, entire sample is used to obtain an initial cutting score Scores are separated into two categories: (a) those C. scores greater than or equal to C and (b) those scores For each category, the mean proportion less than C. correct score is computed. The mean for the first category equals q^; the mean for the second category equals » . ^1- The selection of alpha (a) and beta (3) should be based on the criticality of accurate proficiency decisions. values of alpha (ct) and beta (3) Small require additional task trials to make decisions with greater confidence. Factors that are important in selecting values for alpha and beta (a) (3) are outlined below: (1) C2) Alpha (a) values — (a) Safety potential harm to the trainee or to others due to the trainee's actual non-mastery of the task. (b) Prerequisite in Instruction potential problems in future instruction, especially if the task is prerequisite to other tasks. (c) Time/Cost potential loss or destruction of equipment either in training or upon fleet assignment. (d) Trainee's View of the Training potential negative view by trainee when classified as proficient although the trainee lacks confidence in that decision. Also, after fleet assignment if previous training has not prepared him sufficiently the trainee may also have a negative view of the training program. Beta Ca) — — — (3) values — Instruction requirement for additional training resources (personnel and materials) for unnecessary training in case of misclassif ication as not proficient. 292 — Cb) Trainee Attitudes the attitude of trainees when tasks have been mastered yet training continues; trainee frustration; corresponding impact on performance in the remainder of the training program and fleet assignment. (c) Cost/Time-- the additional cost and time required for additional training that is not really needed. After the model parameters have been selected, calculation of the decision boundaries may be accomplished using the Wald Binomial Probability Ratio Test. A formal mathematical discussion of this test follows. WALD BINOMIAL PROBABILITY RATIO TEST II. The Wald binomial probability ratio test was developed by Wald [1947] as a means of making statistical decisions using as limited a sample as possible. The procedure involves the consideration of two hypotheses: and P is the H. 1 : P > - P^ 2 where proportion of nondefectives in the collection under consideration, P, is the minimum proportion of nondefectives at or below which the collection is rejected, and P^ is the desired proportion of nondefectives, at or above which the collection is accepted. Since a simple hypothesis is being tested against a simple alternative, the basis for deciding between Hq and H^ may be tested using the likelihood ratio: 293 R : <^2'''" ^2n Where: - ^Z'"""" Minimum proportion of nondef ectives at or below which the collection is rejected. = P, (^ P^ = Desirable proportion of nondef ectives at or above which the collection is accepted. = Total items in collection. n dn = Total nondef ectives in collection. The sequential testing procedure provides for a postponement region based on prescribed values of alpha (3) and beta that approximate the two types of errors found in the statistical decision process. H-.: (a) P = P, To test the hypothesis calculate the likelihood ratio and proceed as , follows P ^^^ ^^ P^ In If -^ - T^ > i:i^ ' accept Hq , accept P (2) În °^ P 6 (3) If T-^— 1-a < — — 2n Pt„ In t; < 1- a ^ , H^ ^ take an additional observation. These three decisions relate well to the task proficiency problem. (1) We may use the following rules: Accept the hypothesis that the grade of P is accumulated in lower proportions than acceptable performance would indicate. 294 Reject the hypothesis that the grade of P is accumu- (2) lated in lower proportions than acceptable performance would By rejecting this hypothesis, an alternative indicate. hypothesis is accepted that the grade of P is accumulated in proportions equal to or greater than desired performance. Continue training by taking an additional trial (s); (3) a decision cannot be made with specified confidence. The following equations are used to calculate the decision regions of the sequential sampling decision model. log dn < j^ i-a — =-— 1-P ;r P log + n 2 1 log p- + log .TTp^1 ^ ^2 log dn > =: 2 log — =r 1-P — + n 1 log p- + log y;-^^1 ^ ^2 Where: P 1-P 2 1 log p- + log ^Tp" ^ 1 i^ :; P j^ ^2 P 2 ^2 "^ 1-P 1 log p- + log yZp^ ^1 2 dn = Accumulation of trials graded as "P" in the sequence. n - Total trials presented in the sequence. P- = Lowest acceptable proportion of proficient trials (P) required to pass the NATOPS flight evaluation with a grade of "Qualified." P^ = Proportion of proficient trials (p) that represent desirable performance on the NATOPS flight evaluation. Alpha (a)= The probability of making a type I error (deciding a student is proficient when in fact he is not proficient. 295 . Beta (3) = The probability of making a type II error (deciding a student is not proficient when in fact he is proficient) The first term of the two equations will detejrmine the intercepts of the two linear equations. The width between these intercepts is determined largely by values selected for alpha (a) and beta (6) . The width between the intercepts translates into a region of uncertainty; thus, as lower values of alpha (a) and beta (3) are selected this region of uncertainty increases. The second term of the equations determines the slopes of the linear equation. Since the second term is the same for both equations, the result will be slopes with parallel lines. P, Values of P, and P^ ^^ well as differences between and P2 affect the slope of the lines. translated into task difficulty. This is easily As P^ values increase, in- dicating easier tasks, the slope becomes more steep. This in turn results in fewer trials required in the sample to reach a decision. As differences in P, and P^ increase, the slope also becomes steeper and the uncertainty region decreases. consonant with rational decision making. This is When the difference between the lower level of proficiency and upper level of proficiency is great, it is easier to determine at which proficiency level the pilot trainee is performing. of differences in P, The concept and 2^ is analogous to the concept of 296 . effect size in statistically testing the difference between the means of two groups. alpha (a) and beta (3) In such statistical testing, when remain constant, the number of observations required to detect a significant difference may be reduced as the anticipated effect size increases [Kalisch, 1980] 297 INITIAL DISTRIBUTION LIST Copies No. 1. Defense Technical Information Center Cameron Station Alexandria, Virginia 22314 2 2. Library, Code 014 2 Naval Postgraduate School Monterey, California 93 94 2 3. Department Chairman, Code 55 Department of Operations Research Naval Postgraduate School Monterey, California 93940 1 4. Asst Professor D. E. Neil, Code 55 Department of Operations Research Naval Postgraduate School Monterey, California 93940 2 5. LT Ted R. Mixon 4 61 Lilac Rd Casselberry, Florida 32707 5 6. Mr. Don Vreuls Vreuls Research Corporation 68 Long Court, Suite E Thousand Oaks, California 91360 1 7. Dr. Jack B. Shelnutt Seville Research Corporation 400 Plaza Building Pensacola, Florida 32505 1 8. A. F. Smode, Director TAEG Training Analysis and Evaluation Group Orlando, Florida 32813 1 9. LT Gerald S. Stoffer Code N-712 Naval Training Equipment Center Orlando, Florida 32813 1 10. Don Norman Code N-71 Naval Training Equipment Center Orlando, Florida 32813 1 298 Copies No. 11. COMNAVAIRPAC Commander Naval Air Force US Pacific Fleet Box 1210, Naval Air Station, North Island San Diego, California 92135 1 12. COMNAVAIRLANT, Commander Naval Air Force US Atlantic Fleet Norfolk, Virginia 23511 1 13. COMTACWINGSLANT, Commander Tactical Wings Atlantic Naval Air Station, Oceana Virginia Beach, Virginia 23460 1 14. COMMATVAQWINGPAC Commander Medium Attack Tactical Electronic Warfare Wing Pacific Naval Air Station, Whidbey Island Oak Harbor, Washington 98 27 8 2 15. MATWING, Medium Attack Wing 1 Naval Air Station, Odeana Virginia Beach, Virginia 23460 2 16. Commanding Officer, Attack Squadron 42 (VA-42) Naval Air Station, Oceana Virginia Beach, Virginia 23460 2 17. Commanding Officer, Attack Squadron 12 8 (VA-128) Naval Air Station, Whidbey Island Oak Harbor, Washington 98278 2 18. CDR W. Moroney, Code 55Mp Department of Operations Research Naval Postgraduate School Monterey, California 93940 6 19. NAVAIRSYSCOM Headquarters Navy Department Washington, DC 20361 - AIR 5313 1 20. NAVAIRSYSCOM Headquarters Navy Department Washington, DC 20361 - AIR 34 21. NAVAIRSYSCOM Headquarters Navy Department Washington, DC 20361 - AIR 413 5 A 22. Leonard Ryan Naval Training Equipment Center Orlando, Florida 32813 , , 299 F 1 1 1 No. 23. Dr. Sam Campbell Code All-04 Grumman Aerospace Corporation Bethpage, New York 11714 24. Professor Glenn Lindsay, Code 55Ls Department of Operations Research Naval Postgraduate School Monterey, California 93940 25. Dr. Tom Killion Air Force Human Resources Laboratory Code OT(UDRI), Bldg 558 Williams AFB, Arizona 85224 26. Ira Goldstein Code N-71 Naval Training Equipment Center Orlando, Florida 32313 27. LT Alexander Bory Human Factors Engineering Branch Pacific Missile Test Center Point Mugu, California 93042 28. CDR Paul R. Chatelier Office of Director Defense Research and Development 3 D 129 Pentagon Washington, DC 20350 29. LCDR Wade R. Helm Human Factors Division Naval Air Development Center Warminster, Pennsylvania 18 07 4 30. CDR Robert S. Kennedy Chief, Human Performance Science Division Naval Biodynamic Laboratory Box 29407, Michoud Station New Orleans, Louisiana 7 0189 31. CDR Norman E. Lane Human Factors Engineering Division Crew Systems Department NADC Warminster, Pennsylvania 18074 300 Copies No. 32. LT Michael G. Lilienthal Code 5 Human Performance Sciences Department Naval Aerospace Medical Research Laboratory NAS Pensacola, Florida 32508 33. CDR Joseph F. Funaro Code N-82 Naval Training Equipment Center Orlando, Florida 32813 34. CAPT Thomas J. Gallagher Naval Air Development Center Warminster, Pennsylvania 18 074 35. LCDR Richard H. Shannon Naval Biodynamic Laboratory Box 29407, Michoud Station New Orleans, Louisiana 7 018 9 36. CDR Patrick M. Curran Head, Aerospace Psychology Branch (3C13) Bureau of Medicine and Surgery Navy Department Washington, DC 20372 37. Stan Collyer Human Factors Laboratory Naval Training Equipment Center Orlando, Florida 32813 38. Larry R. Beideman McDonnell Douglas Corporation Box 516 Saint Louis, Missouri 63166 39. William C. McDaniel Training Analysis and Evaluation Group Orlando, Florida 32813 301 Copies Thesis 1 1652 7 c.l <r 192347 Mixon A model to measure bombardier /navigator performance during radar navigation in device 2F114, A-6E weapon system trainer, , thesM6527 " -.?;iei to measure bombardier navigator 3 2768 000 98471 DUDLEY KNOX LIBRARY MP' R V . E \ i ' »» .. -/• • •! ..ij v-.-ikl'V- M ;,.-• k' 0. A '"•••S'xVV.- •';,v.'.,-';|v-H.i^. . l<'''-V/jn| ';/ •s\'<S'iJ A\ , .'.1- fcl

UNS A-6 Trainer Effectiveness

Related documents

Products

Support

UNS A-6 Trainer Effectiveness

Related documents

Add this document to collection(s)

Add this document to saved

Suggest us how to improve StudyLib