INTRODUCTION TO DESIGNING NEEDS ASSESSMENT SURVEYS An AEA Mini Workshop presented by James W. Altschuld Yi-Fang Lee Jeff White Hsin-Ling Hung The Ohio State University National Chi Nan University University of Louisiana at Lafayette University of North Dakota Annual Meeting of the American Evaluation Association November , 2010 San Antonio, Texas OBJECTIVES/AGENDA Objectives Understand Needs Assessment Learn where versions of surveys fit the picture Quick overview of survey parameters in NA Get a hands-on feel for the process Have the opportunity to discuss/comment on the experience Hopefully to have some fun on your brief journey into the world of NA Also to provide you with a glimpse into the Needs Assessment Kit (Altschuld, Eastmond, King, Kumar, Stevahn, and White, 2010) AGENDA Introduction (1 minute) Questions (3-5 minutes) Basic terms and concepts (8-10 minutes) Survey overview (15 -18 minutes) Types and where they fit the process Samples of questions (scaled or otherwise) General analysis strategies Wording problems Scenario overview, hands-on work, discussion, etc. (rest of time allotted for the session) BEGINNINGS Questions for you How many have done NAs? Have you used surveys in the process? For what purposes? Briefly describe the survey? How did you analyze the data? Overall success of the effort? Did you use other methods as well – what were they? Any other comments about what you did? Terms/concepts Need -definition with attention to use of words (need & needs) -not mixing solutions with needs -definitional problem immediately for wording survey items -examples of needs (see partial table from book 1) -types of needs (book 1) -levels of needs -Wow, issues at the outset Needs assessment -three phase model (see schematic) -types of surveys and where they might fit -why you still might have to include solutions in survey despite above admonition DEFINITIONS & ISSUES Need: the measurable discrepancy between “what is” or the present state of affairs in regard to the group and situation of interest and the “what should be” or desired state of affairs (Witkin & Altschuld, 1995). Issues: measurable discrepancy is the key -needs not solutions (premature closure on solutions) -verb vs. noun concept (misuse of the word) -‘desired’, ‘likely to occur’, ‘ought to occur’, etc. -wish and want lists -many types of needs MORE TERMS AND CONCEPTS NA is a systematic set of procedures undertaken for the purpose of setting needs-based priorities and making decisions about organizational improvement and allocation of resources (Witkin & Altschuld, 1995). Issues -context for the NA -readiness for an assessment -NA is an organizational activity -political aspects to the activity -systems concept and how to think about it Lots of subtle aspects of need and NA Examples Types Why do it and how does it relate to evaluation Source: from Needs Assessment Kit Ι, by J. W. Altschuld and D. D. Kumar , 2010, Thousand Oaks, CA: Sage. Source: from Needs Assessment Kit Ι, by J. W. Altschuld and D. D. Kumar , 2010, Thousand Oaks, CA: Sage. Source: from Needs Assessment Kit Ι, by J. W. Altschuld and D. D. Kumar , 2010, Thousand Oaks, CA: Sage. SURVEY CONSIDERATIONS Sampling Level 1, 2, and/or 3 Wording issues Where in the process For NAC? Phase 1, 2, 3 What type of need Short term vs long term Maintenance Collaborative Other Double, triple scales or more Content Behaviors Attitudes Content (continued) Degree of achievement Satisfaction with services Importance/value of services Motivation to resolve problems Frequency of use Observations Feasibility of resolving problems Literature based content Impediments to solutions List goes on and on Formatting Scales (what type) Open-ended Bars Examples of Surveys but first Table 3.1 Comparisons of Survey Usage for Organizing the NAC and for Conducting Needs Assessments Table 3.1 (Continued) Table 3.1 (Continued) Source: from Needs Assessment Kit 2, by J. W. Altschuld and J. N. Eastmond, 2010, Thousand Oaks, CA: Sage. Table 3.2 Four Basic Survey Formats for Kicking Off Initial NAC Discussions Source: from Needs Assessment Kit 2, by J. W. Altschuld and J. N. Eastmond, 2010, Thousand Oaks, CA: Sage. Table 3.5 Structure of the NAC Survey Table 3.5 (continued) Source: from Needs Assessment Kit 2, by J. W. Altschuld and J. N. Eastmond, 2010, Thousand Oaks, CA: Sage. Humann (1997) Example of First Student Survey OSEA (2005) Example of Follow-Up Survey Lee, Altschuld, & White (2007) NOW THE HANDS-ON EXERCISE Scenarios/orientation Pretend that you are the needs assessor Individually and then collectively What can a NAC tell you? Questions Content Discussion focusing Sketch out areas where they could give you guidance/input So you need a survey for other groups Review item writing rules Review examples Considerations for NA surveys What content Levels 1, 2, and/or 3 Relation to other methods Use of multiple scales Impediments Open-ended questions