What’s Up DOT? IARP Occupational Database Committee IARP Forensic Conference Hyatt Regency Bonaventure Weston, Florida October 31, 2008 Scope of Project IODC → formed spring 2007 Purpose: • Identify replacement of DOT Focus: • research past efforts to update DOT • current uses of DOT & O*NET • identify & evaluate other occupational databases • develop support to update/replace DOT Scope of Project Identified two occupational databases: eDOT, a product of Economic Research Institute; McDOT, a product of Vocationology, Inc. Third database, based on the Common Metric Questionnaire (CMQ) may be ready in few years Arranged to have training and use of these databases for study purposes. First, needed to learn more about DOT History of the DOT Following Civil War, U.S. and Census Bureau began to classify occupations First occupational dictionary: 1927, A Dictionary of Occupational Terms, Great Britain Ministry of Labour Wagner Peyser Act, 1933: United States Employment Service (USES)-- match workers with jobs. History of the DOT Work on the DOT began around the same time; 1st Edition 1939 Subsequent editions published 1949, 1965, 1977 Supplements came out 1982, ’86, ’91 20 new occupations added in 1998 Intended purpose: to assist USES in placing workers in jobs. 1st Edition: skilled, semi-skilled or unskilled History of the DOT 1965 edition: worker traits, worker functions (Data, People, Things), Occupational Group Arrangements 1977 edition: ~75,000 job analyses; 2,100+ new occupations; eliminated gender bias in job titles and descriptions; included detailed worker characteristics (SCO) 1986 ed. added 761 new occupations 1991 added 844, deleted 208; added GOE codes, GED, SVP, DLU History of the DOT Each definition includes: code, title, industry designation, alternate titles, body of definition, undefined related titles, and definition trailer (GOE, strength, GED-RML, SVP, DLU) Basic concepts described in the DOT occupational definitions: element, task, position, job, and occupation. History of the DOT Major elements of an occupation: • What the worker does (data, people, things) • What gets done (work fields-WF) • Skill level to perform this work (SVP) • End product (materials, products, subject matter and services-MPSMS) History of the DOT DOT → 4 classification systems: • • • • by job content (OGA) by worker function (DPT) by industry affiliation (Industry Designation) alphabetically by title History of the DOT Job analysis → basis of the DOT National Research Council (1999) defines occupational [job] analysis: "the tools and methods used to describe and label work, positions, jobs and occupations" DOT defines an occupation as: “a collective description of individual jobs performed, with minor variations, in many establishments” History of the DOT The Handbook for Analyzing Jobs → job analysis methodology used in creation of DOT. HAJ first published in 1944; 4th Edition in 1991 (RHAJ). 1972 Edition published halfway through completion of 1977 DOT. Two primary formats in job analysis: workoriented—what gets done; and workeroriented—what the worker does. HAJ/RHAJ uses both formats History of the DOT The Selected Characteristics of Occupations Defined in the Revised Dictionary of Occupational Titles (1993) 1st edition 1966, updates in 1968, 1981, and 1993. Includes DOT titles arranged by: GOE; SVP; strength; physical demands; environmental conditions; index of titles with DOT codes; definitions of the worker traits and functions History of the DOT Positive factors of the DOT: • use of skilled job analysts • task level descriptions • foundation built upon data people things • Worker traits, characteristics • Useful for TSA: MPSMS, WF, SVP • Attempt to cover national economy • Depth of information Development of the DOT Sample for DOT: all jobs in the US economy County Business Patterns/Thomas Registry → used to identify # establishments in each industry. Industries assigned to one of 11 field centers → average 42 industries/field center “any industry” designation given to one field office Development of the DOT Each analyst identified which establishments to contact Attempt to include one small, one medium and one large size establishment for each industry, and to focus on new and emerging occupations. No clear supervision during process. Appeared primary criteria for selecting establishment → proximity to field office Employers: right of refusal; no incentives Development of the DOT Employers controlled which jobs were analyzed, and which employees were chosen for analysis. Analyst chose which jobs to observe. If job analyzed for the 3rd Edition, may complete abbreviated analysis or none at all. If another analyst had recently completed a similar analysis, none would be done. Observe 1-2 workers per job. Development of the DOT Variety methods to record data: HAJ format; abbreviated format; or simple notes. Analysts not allowed to bring in tools or equipment to measure certain aspects of jobs— estimation and observation only. Methods for rating worker traits were vague— particularly for aptitudes, temperaments and interests. Updates of the DOT Much time has passed since any onsite job analyses of DOT occupations Date last analyzed 1977 and previous 1978 1979-1982 1984-1988 1989-1991 1993 N % Age 10,288 355 520 1,187 80.6 2.8 4.1 9.3 30 years 30 years >25 years >20 years 390 20 3.1 0.2 >15 years 15 years How the DOT is used Basis for transferable skills analysis Primary uses: career & voc counseling, SSD Secondary uses: library reference, personnel management, employee placement, govt. uses, research, others O*NET Outgrowth of the Advisory Panel on the DOT (APDOT) from the early 1990s Dept. of Labor “replacement” for the DOT Skills-based database rather than task-based dictionary Preliminary version released Dec. 1997 with first version (O*NET 98) out Dec. 1998 O*NET Utilized SOC coding rather than DOT Current version is O*NET 13 The 12, 761 DOT occupations have been aggregated/collapsed to 812 groups Composite information from many jobs; not intended to describe a particular job Uses mean data rather than mode used by DOT Position Analysis Questionnaire (PAQ) PAQ’s business: create useful work and labor economic reports Structured job analysis; 187 items (plus 8 items re: comp.) called “job elements” Job elements are worker-oriented;6 domains: information input; work output (physical activities and tools); mental processes; relationships with others; job context (physical and social environment); and other work characteristics (such as pace and structure) PAQ PAQ dedicated solely to job analysis services since 1972 Terminology, definitions and rating scales different than that used in DOT Requires post-college graduate reading level Job analysts or supervisors usually complete PAQ PAQ PAQ trains subscriber employers in job analysis techniques—in person and online Limited coverage of managerial, supervisory, executive and professional work led to development of separate questionnaire Average reliabilities—item and re-rate—range .68-.80 PAQ/DAQ DAQ: Disability Analysis Questionnaire. Developed by PAQ Includes 99 questions from the PAQ that are most closely related to items from the DOT/HAJ ERI modified DAQ slightly—made certain the questions best-matched those of the DOT The DAQ questions are the ones asked of incumbents when visiting websites eDOT The “enhanced DOT” -- database and software program developed by Economic Research Institute (ERI) ERI began as compensation information provider in 1987 Started the eDOT Skills Project in 2002 to collect data and update the DOT In 2004, ERI purchased PAQ; PAQ operates eDOT Skills Project under a license with ERI Database has two parts: archived DOT and new eDOT which includes the old DOT eDOT Includes 64 SCO characteristics + 35 new factors such as keyboarding, education, mental and cognitive factors, etc. Includes 20 measures from SSA’s Mental Residual Capacity paper Various people contribute data: subject matter experts (job analysts trained in the use of the PAQ; voc rehab counselors; major disability carriers; workers compensation analysts; and primarily incumbents visiting websites Career Builder, SalaryExpert, SalariesReview. eDOT Each data point has associated reliability, standard error and deviation calculated 1,000-1,500 job analyses done per year, including those by incumbents Over 1 million PAQ job analyses included in eDOT (completed since 1974) Sample is one of convenience Control for sampling error by using multiple sources of data eDOT Differences between eDOT and DOT • Different rating scales, definitions, terms • eDOT→ uses revised version of SIC which it developed, called eSIC • eDOT→fluid database; DOT→fixed • eDOT→convenience sample; DOT→attempt to capture all jobs in the national economy • Electronic v. paper • PAQ→ interviewing worker, not always observing; DOT→observation eDOT Examples of new jobs added to eDOT computer sys admin 030.162-500 call center rep 299.357-201 asst. mgr retail store 185.167-505 executive asst 169.167-911 sales assistant 209.562-800 maint. helper 806.687-011 CAD/CAM Tech 017.262-519 eDOT ERI/PAQ “masks” jobs Criteria for exclusion: • job analyses over 15 years old • face validity (abalone diver) • 10 or less requests for the job • not listed on job boards or salary surveys • not mentioned on any loan applications • not on any labor/proxy/form 990 • not mentioned on Calif. state work comp eDOT Examples of “masked” jobs in eDOT: animal breeder 410.161-010 military recruiter 166.267-026 pres., financial inst. 186.117-054 police commissioner 188.117-118 feed blender 520.685-094 collator operator 208.685-010 eDOT As of January 2008, ERI had • Added 717 new occupations • Removed 4,103 occupations (no/low frequency) • Verified the existence of 8,658 occupations • Total of 9,375 occupations in eDOT, compared to 12,761 in the DOT and 812 in O*NET • As of July 2008, have added WF and MPSMS to all jobs in eDOT McDOT History McCroskey Dictionary of Occupational Titles (McDOT) Part of the McCroskey Vocational Quotient System (MVQS) Based on VDARE McDOT includes both the old DOT and O*NET. Data has been fused from both sources, identifying 24 most significant worker traits UNUM: Project with eDOT Methodology for selecting eDOT Findings Future Directions Issues with DOT Miller et al. Study 1980 Work, Jobs and Occupations: A Critical Review of the Dictionary of Occupational Titles Comprehensive review & evaluation of the 1977, 4th Ed. DOT, conducted on behalf of the National Research Council, at the request of the Department of Labor Issues with DOT Miller et al. Study 1980 “The comprehensiveness, reliability and accuracy of the DOT are in large part a function of the data collection and analysis procedures used to produce it.” p.114 Issues with DOT Miller et al. Study 1980 Data Collection/Methodology Issues: • Lack of written procedures on how DOT produce • Majority of principles used to create DOT were established in 1939 & 1949 • Sampling plan “complicated and indirect,” did not include all jobs in US economy Issues with DOT Miller et al. Study 1980 Data Collection/Methodology Issues: • Heavy orientation to manufacturing sector • Limited review of jobs requiring cognitive skills over physical skills • Inadequate update schedule Issues with DOT Miller et al. Study 1980 Found analysts often unable to provide concrete explanation for how they rated worker traits besides subjective means or past experience Some occupations not reviewed at all; others reviewed excessively: material handler → 652 job analyses 1979 study random job titles found 81% of 4th Ed. descriptions exactly the same as the 3rd Edition Issues with DOT Miller et al. Study 1980 Data Collection/Methodology Issues: • 16% occupational descriptions completed without a single job analysis • 29% based on one job analysis • 19% based on two job analyses • 36% based on three or more analyses Issues with DOT Miller et al. Study 1980 Reliability/Validity Issues: • Validity: measurements of worker traits/functions found to be fairly unreliable: “vague and ambiguously defined. Not readily apparent what the variables are intended to measure” (p. 164) • Worker traits/functions developed in the 1950s— questionable validity for today’s labor market • Question whether GED and SVP measure prestige or social status of occupations Issues with DOT Miller et al. Study 1980 Reliability/Validity Issues: • GED scale validated against curriculum content, not validated for occupational performance • Working conditions & physical demands appear to be developed for unskilled, physical jobs • Reliability: items are scored subjectively; raters had trouble assigning some factors, particularly SVP and aptitudes • Reliabilities higher for manufacturing jobs than for service jobs Issues with DOT Cain et al. (1983) study on reliabilities of different ratings. • Two job descriptions per 24 occupations; experienced analysts rated the factors • Found acceptable reliabilities for: data, people, GED reasoning, GED language and SVP • Modest reliabilities for things, GED math, strength factors and location • Reliabilities higher for manufacturing jobs Issues with DOT Botterbusch (1993): DOT weaknesses: • Data people things not actually a hierarchy—data is, people is not and things is two hierarchies • GED not directly related to education • SVP does not distinguish between formal and informal training • Difficulty defining, using and defending temperaments • Too many titles Issues with DOT Dunn & Growick (2000): weaknesses: • Failure to include variables such as org. setting and worker responsibility level • Low reliability of worker traits • Redundant classification system • Skills that are psychometrically questionable, such as GED and SVP • Ambiguous definitions of worker traits • Questionable validity of worker traits Issues with DOT Harvey (Fine et al. 2004) addressed weaknesses of DOT: • Legal and psychometric defensibility in the use of holistic scales to rate worker traits— tend to show low inter-rater reliability and low discriminant validity • Lack of reasonable update schedule Issues with DOT National Research Council (1999) • “Unwieldy size” • Growing differences between job descriptions and jobs as they actually occur in the labor market today • Too much detail in each definition • Jobs described by task Issues with DOT National Research Council cont. (1999) • Lack of information on factors such as skills, abilities, and knowledge requirements that are either not collected or not reported • Little or no reported information on cognitive factors • Expense of updating • Difficulty of linking it with other databases Issues with DOT • APDOT (1993): need to reinvent the DOT • Representative of economy • Develop new JA methodology: found existing ones to be lacking, particularly with cognitive skills • Create one better at job matching, skills transfer Issues with DOT 1980 Miller et al. study issued several conclusions and a number of recommendations (p.214) Conclusion #1 – “There is a strong and continuing need both within and outside the U.S. Employment Service for the kind of information provided by the Dictionary of Occupational Titles and certain other products based on it.” Conclusion #2 – “Substantial improvements in the procedures and products of the occupational analysis program are required in order to meet the national need for occupational information.” Issues with O*NET “…O*NET as currently implemented arguably fails to provide (a) occupational data expressed at the appropriate level of detail and defensibility needed for many functions and (b) occupational title taxonomy that is sufficiently detailed and phrased at the desired level of analysis” (Fine et al., 2004, p1). Depth of material in O*NET: telling someone which state you live in but not giving your address (ERI) Issues with O*NET Terminology and measurement scales are poorly defined Does not provide correct type of information on physical demands, training requirements and other information essential for TSA Job demands difficult to measure (e.g. cognitive) Use of numerical scales over interval scales Issues with O*NET Completed by incumbents, except for abilities section Problem with incomplete survey instruments Uses mean data rather than modal data as found in the DOT On the + Side: O*NET Many new variables to describe work-related characteristics and behavior Includes relevant aspects of work not in the DOT, i.e. work context Expanded knowledge requirements, training and preparation needs IODC Evaluation of eDOT ERI provided complimentary eDOT software plus two training sessions Met with ERI staff at 2007 Forensic Conference to discuss issues/concerns ERI staff available to answer questions throughout process Separate training provided by user of eDOT at UNUM Provided beta versions of OccuBrowse database with integrated eDOT IODC Evaluation of eDOT Intention to update DOT positive Creative in data gathering processes Inclusion of cognitive elements Staffed with I/O psychologists Presentation of raw data and data sources for professional study Separation of physical demands IODC Evaluation of eDOT New ratings gathered from convenience sample Industries and occupations do not always match up Coding of occupations different than DOT Uses mean trait ratings rather than mode used in DOT Method for removing occupations unsatisfactory IODC Evaluation of eDOT Method of data aggregation to define new occupations are unclear Method used to translate PAQ data into DOT terms is unclear Several listings for same job--confusing Unclear and excessive number of rating scales Based on the flawed DOT IODC Evaluation of McDOT IODC provided with complementary software One training session Vocationology available for questions, issues throughout process IODC Evaluation of McDOT The new job analysis information in McDOT comes directly from O*NET—positive and negative Analyses of inter-correlations among various worker trait characteristics Reliability and validity data No independent job analyses were conducted by Vocationology, Inc. IODC Evaluation of McDOT Since O*NET was initially derived from DOT data, O*NET is a confounded data source as an “update” to the DOT. Incorporates all problems of both DOT and O*NET SSA has reported it will not use a database with O*NET in it Difficult for many to understand or explain to judges, juries, attorneys, etc. IODC Findings IODC was unable to identify and endorse a ready-to-use database that fits members needs and needs of SSA eDOT has potential to be that database, but many methodological problems which need to be solved before it is usable for transferable skills analysis and in court settings IODC recommends continued use of DOT until better product is available Moving toward a better DOT Job Analysis Issues: • Identification of an agreed-upon job analysis format: reliable & valid, standardized format • Re-evaluation and redefinition of worker characteristics; e.g. what constitutes a “physical demand” • Develop improved definitions and indicators for attributes; scales for each • Use of decomposed rather than holistic ratings Moving toward a better DOT Job Analysis Issues: • Expanded physical demand ratings: sitting, standing, walking, reaching (above shoulder v. forward v. downward), vehicle operation, lift, carry, push, pull, use of controls (hands, feet), bilateral/unilateral upper extremity use, exposure to chemical agents, biological hazards, latex, animals, potential food allergens, photic triggers (seizure potential) Moving toward a better DOT Job Analysis Issues: • Addition of “basic” skills including: keyboarding, technology use (computer user, familiarity with multiple applications, train others, troubleshooting, etc.) • Addition of attributes, including cognitive requirements; deletion of others Moving toward a better DOT Job Analysis Issues: • Preferred personal qualities of workers for successful performance: attitude, initiative, persistence, resourcefulness, honesty, flexibility, team orientation, communication skills, tact, organization, leadership, efficiency, versatility, reliability, quality orientation, attentiveness, etc. Moving toward a better DOT Job Analysis Issues: • Acceptable methods of preparation for entry into occupations: formal education, vo-tech, apprenticeship, in-service training, OJT, certification, licensure, registration, prior experience • Barriers to hiring for a specific occupation: criminal history, appearance, personal hygiene habits, monocular vision, etc. Moving toward a better DOT Data Collection/Methodology: • Comprehensive, multi-level stratified sampling plan • Audit to identify where occupations are found within businesses /industries of all sizes (e.g. small employers with <25 employees, selfemployment) • Use only trained/qualified professionals as job analysts Moving toward a better DOT Data Collection/Methodology: • Determine how job analysts would be trained and retrained: what, who, where, when • Identify whether JAs would be done solely by resurrected Field Offices, public sector or with help from private. • Improvement in technical quality of data and methodologies used, must be ongoing in order to ensure its defensibility Moving toward a better DOT Data Collection/Methodology: • Coding methodology used must make sense • 4 raters produces good reliability; 4 raters should be goal • Avoid use of incumbent ratings due to reliability problems with this data • Online system for input of job analysis Moving toward a better DOT Data Collection/Methodology • Ensure database reflects the economy/labor market with at least 90% (preferred 95%) confidence – tied to number of analyses performed and variability in results obtained • Attributes must have multiple measures • Appropriate scaling for each attribute and submeasures must make sense Moving toward a better DOT Data Collection/Methodology • Proper instrumentation/equipment must be available to conduct an objective, measurable rather than strictly observational job analysis • Total number and range of attributes should be limited to what an analyst can handle without diminishing quality of data being gathered Moving toward a better DOT Data Collection/Methodology • Identify variables needed in order to do a TSA-job match; once identified, develop scales, use accepted psychometric practices— this would increase reliability Moving toward a better DOT Use of Data/DOT • Methodology needs to be explained in as simple terms as possible for ease of customer use • Processes & methodologies, as well as results of data usage, must be easily explainable to ALJs/judges, juries, attorneys, etc. • Software used to access data and perform TSAs, etc., should be simplified wherever possible to reduce errors/improve understandability Moving toward a better DOT Use of Data/DOT • Needs to be available as stand-alone product, not solely as part of a transferable skills analysis product • Allow end-users to search, compare, and retrieve information • Development of crosswalks should be wellexplained Moving toward a better DOT Use of Data/DOT • Capable of generating printed reports • Data available to end-users in variety of formats, including online and in print; particularly important that data not require Internet connection so it can be accessed during Social Security hearings Moving toward a better DOT Use of Data/DOT • Author should have vocational rehabilitation person(s) on staff for development phase and customer support Moving toward a better DOT Updates: • Continuously updated • Monitor changes in labor market; reflect changes within the database • Work closely with OES Long Term Projections Survey to learn of new emerging occupations • Sufficient funding must be provided to develop improved database and ensure maintenance into the future Moving toward a better DOT Integration: • Method for integrating new DOT with existing occupational classification systems (SOC, O*NET, NAICS, etc.) • Decisions must be made on other Labor Market Information databases that should be tied into an improved occupational database to replace the D.O.T. The IODC Committee Angela Heitzman, MA, CRC, CLCP, MSCC John Meltzer, MS, CRC, CDMS, LPC Sonia Paquette, OTD, OTR/L, CPE, ABVE-D Gerald Schneck, PhD, CRC-MAC, FVE, NCC Jeffrey Truthan, MS, CVE With assistance from Kelly Beery-Marsiano, MEd, CRC; Betty Morris, MS, CRC; G. Shannon O’Kelley, MEd, CRC; and Sharon Shou, CRC, LCPC, ABVE, all of UNUM References Botterbusch, K.F. (1993, Fall). Suggestions for revisions in the dictionary of occupational titles. Vocational Evaluation and Work Adjustment Bulletin, 101-110. Cain, P.S. & Green, B.F. (1983). Reliabilities of selected ratings available from the Dictionary of Occupational Titles. Journal of Applied Psychology, 68(1), 155-165. Dunn, P.L. & Growick, B.S. (2000). Transferable skills analysis in vocational rehabilitation: Historical foundations, current status, and future trends. Journal of Vocational Rehabilitation, 14, 79-87. References Economic Research Institute. (April 2008). Methodology/Disclaimer: Occupational Assessor™ and Survey (OA/OA+) -- eDOT®, Software and Databases (Professional and Consultant Editions). Retrieved from: www.paq.com/pdf/e-dot-methodology.pdf. Fine, S.A., Harvey, R.J. & Crenshaw, S.F. (2004, April). FJA strategies for addressing O*NET limitations in a post-DOT environment. In Fleischman, E.A. (Chair), Things, Data, and People: Fifty Years of a Seminal Theory. Symposium presented at the Annual Conference of the Society for Industrial and Organizational Psychology, Chicago. References Mariani, M. (1999, Spring). Replace with a database: O*NET replaces the Dictionary of Occupational Titles. Occupational Outlook Quarterly, 43(1), 3-9. McCroskey, B.J. (2008). Volcano 8.0. Brooklyn Park, MN: Vocationology., Inc. Miller, A.R., Treiman, D.J., Cain, P.S. & Roos, P.A. (Eds). (1980). Work, Jobs and Occupations: A Critical Review of the Dictionary of Occupational Titles. Washington, DC: National Academy Press. References National Research Council. (1999). The Changing Nature of Work: Implications for Occupational Analysis. Committee on Techniques for the Enhancement of Human Performance: Occupational Analysis, Commission on Behavioral and Social Sciences and Education. National Academy of Sciences: Washington, DC. United States Department of Labor. (1991). Dictionary of Occupational Titles (4th Ed.). Washington, DC: United States Government Printing Office. United States Department of Labor. (1991). The Revised Handbook for Analyzing Jobs. Washington, DC: United States Government Printing Office. References United States Department of Labor. (1993). Selected Characteristics of Occupations Defined in the Revised Dictionary of Occupational Titles. Washington, DC: United States Government Printing Office. U.S. Department of Labor (1993). The New DOT: A Database of Occupational Titles for the 21stCentury. Advisory Panel for the Dictionary of Occupational Titles (APDOT). United States Department of Labor. (2008). The O*NET Content Model. Retrieved from http://www.onetcenter.org/dl_files/ContentModel _DetailedDesc.pdf.