July 2014

July 2014

INTRODUCTION TO THE RESEARCH AGENDA WORKSHOP ..................................................................................... 1
W HAT IS THE R ESEARCH A GENDA W ORKSHOP ? ................................................................................................................... 1
W HAT IS THE PURPOSE OF THE R ESEARCH A GENDA W ORKSHOP ? ............................................................................................ 1
W HY THIS WORKBOOK ? .................................................................................................................................................. 1
H OW TO USE THIS WORKBOOK AND MAKE THE WORKSHOP YOUR OWN ..................................................................................... 2
F INAL THOUGHTS ........................................................................................................................................................... 3
RESEARCH AGENDA WORKSHOP: FACE-TO-FACE MEETING (FIVE-AND-A-HALF HOURS) ........................................ 3
P REWORK : R EAD THE SUMMARY FROM ONE OF THE FOLLOWING RESEARCH PUBLICATIONS : .......................................................... 4
S EGMENT 1: W ELCOME AND PURPOSE ............................................................................................................................... 4
S EGMENT 2: A LLIANCE GOAL AND POSSIBLE RESEARCH TOPICS ................................................................................................ 4
S EGMENT 3: T YPES OF RESEARCH AND EVIDENCE .................................................................................................................. 5
S EGMENT 4: I NVESTIGATING RESEARCH EXAMPLES ............................................................................................................... 5
B REAK (15 MINUTES ) ..................................................................................................................................................... 6
S EGMENT 5: R ESEARCH QUESTIONS AND AGENDAS OVERVIEW ................................................................................................ 6
S EGMENT 6: P RIORITIZING RESEARCH TOPICS ...................................................................................................................... 6
S EGMENT 7: G ENERATING RESEARCH QUESTIONS ................................................................................................................. 7
L UNCH (45 MINUTES ) ..................................................................................................................................................... 8
S EGMENT 8: S HARING RESEARCH QUESTIONS GENERATED ...................................................................................................... 8
S EGMENT 9: R EFINING QUESTIONS .................................................................................................................................... 8
S
EGMENT
10: P
RIORITIZING RESEARCH QUESTIONS
.............................................................................................................. 8
B REAK (15 MINUTES ) ..................................................................................................................................................... 9
S EGMENT 11: R ESEARCH AGENDA ..................................................................................................................................... 9
S EGMENT 12: C LOSING AND NEXT STEPS ............................................................................................................................ 9
APPENDIX A. DIRECTIONS FOR THE AFFINITY GROUPING EXERCISE ..................................................................... 11
APPENDIX B. DIRECTIONS FOR THE FOCUS FOUR EXERCISE ................................................................................. 12
APPENDIX C. SUGGESTIONS FOR DOING THE WORKSHOP AS A WEBINAR ........................................................... 13
REFERENCE .......................................................................................................................................................... 14

What is the Research Agenda Workshop?
The Research Agenda Workshop is a process for helping a team or group of individuals focused on establishing a set of real and relevant questions to guide their research in a topic area. Establishing the research focus of the group and the questions that make up a coherent research agenda is a collaborative process. For the process to succeed, researchers and practitioners need to establish a common research focus and to identify targeted research questions to guide the projects that will support educators in their own work as they make decisions on education policy and practice.
What is the purpose of the Research Agenda Workshop?
Beginning in 2012 the 10 regional educational laboratories (RELs) began building and supporting research alliances focused on topics deemed important to states and local school districts. Research alliances are composed of state and local administrators, practitioners, education policymakers, and researchers. Sample focus areas include dropout prevention, urban school improvement, and college and career readiness. Each research alliance is establishing a coherent, multiyear research agenda that will include questions to guide their work over the next three to five years.
Regional Educational Laboratory Northeast & Islands (REL-NEI) has guided the development of eight research alliances across New England, New York, Puerto Rico, and the U.S. Virgin Islands. The research alliances are intended to be sustainable collaborations focused on an education priority area for the purpose of increasing state and local capacity to use data and research to inform decisions affecting policy and practice. REL-NEI developed a Research Agenda Workshop as part of its work with the Urban School
Improvement Alliance. The workshop walked alliance members through determining the education issues most important to the member communities, developing researchable questions to address these issues, and establishing a coherent research agenda tied to the questions. Although the format and focus of each research alliance may vary, a common goal is to focus the energy of the proposed research on the areas of greatest interest to the practitioners and policymakers who know best which issues are most pressing in their communities. Some examples of research partnerships outside the REL system include the
Consortium on Chicago School Research ( http://ccsr.uchicago.edu/ ) and the Research Alliance for New
York City Schools ( http://www.ssrc.org/programs/research-alliance-for-new-york-city-schools/ ). For more information on the REL-NEI alliances, see www.relnei.org
.
Why this workbook?
This workbook was developed as part of the research agenda process and provides support and consistency across all eight REL-NEI research alliances. The materials were used as the REL-NEI research alliances defined and refined their foci and developed their research agendas. REL-NEI offers this workbook as a resource for others establishing similar research alliances.
The intended users of this workbook include other RELs, school districts working with university partners, and external researchers who would like to establish a common research focus in collaboration with
1

educators in their area. The workbook provides a structured “workshop” format to introduce the concept and process of developing a grounded research agenda that includes the perspectives and interests that each member brings to the alliance. The structure of the workbook allows members to build a common understanding and consensus around the questions that should be the focus of the alliance.
How to use this workbook and make the workshop your own
This is a generic workbook that you can tailor to your topic and alliance member needs. The workbook outline is intended for a five-and-a-half-hour face-to-face meeting. However, users may divide this workshop into two or more sessions. An advantage to offering the workshop in more than one session is that between sessions participants can discuss the priority topics that emerged from the first session and collect feedback and ideas for potential research questions that they can share during the later sessions. A disadvantage, however, is the difficulty of getting the same people to attend all the sessions. Advantages to conducting the workshop in one session are that all participants are present for the full research agenda–setting process, momentum is not lost by splitting the work into multiple parts, and the day ends with a concrete product—a draft research agenda. We recommend you consider your group’s needs and circumstances when determining the format for your workshop.
Here are some recommendations and ideas for tailoring the workbook to your needs:


The facilitator guide provides instructions to the organizers of the research alliance and includes details on what the workshop entails and ways to proceed. As you prepare your own workbook, delete the introductory materials before sharing the workbook with alliance members. The introductory materials are designed for workshop facilitators.
The participant guide is meant to be distributed to all workshop participants. Again, we recommend you tailor this material to the needs of your alliance members. For example: o o o
The prework assignment (page 1 in the participant guide) suggests that participants review summaries of several kinds of research studies to increase their understanding of the connections among research questions, study designs, and data sources. Experienced researchers and others who already possess this background knowledge can skip this activity.
However, if your alliance members have varied research backgrounds and you want to ensure that all members are prepared to engage in a productive research agenda–setting session, you may want to provide them with several studies that demonstrate the use of research methods similar to those presented in the workbook and that focus on the alliance’s area of interest.
Similarly, you may decide to place more or less emphasis on the information the workbook provides on various types of research (pages 6–9 in the participant guide). This decision should be based on your alliance members’ familiarity with research design.
We recommend that you customize the parts of the workbook that guide participants in generating research questions, organizing these into affinity groups, and prioritizing topics.
One tip is to make sure you have sufficient time to engage in a fruitful discussion that leads to clarity among all group members on each idea presented. This may require you to shorten other workshop segments.
2



Although the appendixes in the facilitator guide are meant primarily for the group facilitator(s), you may want to share these with all workshop participants. These materials include directions for the
Affinity Grouping exercise (appendix A), directions for the Focus Four exercise (appendix B), and suggestions for how this workshop can be adapted for an online environment (appendix C).
A set of PowerPoint slides that include a suggested script and complement the workbook are also provided. These materials are intended for use during your workshop. Any changes you make to the workbook will also need to be made in the slides. For example, if you change the order of activities or delete some workbook segments, you will need to make corresponding changes in the slides. Similarly, the workbook refers to slide numbers, so you will need to adjust these numbers if you add, delete, or move slides around.
Final thoughts
There are many ways an individual or group can generate a research agenda. REL-NEI has used this structured workshop approach with a variety of alliances that address different topics and have members with varied research backgrounds. We understand that many alliance members have limited time. However, to develop a coherent, long-term research agenda that addresses the most pressing issues facing the education communities represented by your alliance members, your alliance members must be provided ample time to discuss all viewpoints and ideas presented. Shorter versions of this workshop did not get to the research prioritization step and left members without a sense of direction. Using these types of research agenda–setting activities coupled with the workbook should help your research alliance develop a research plan to achieve your intended outcomes. REL-NEI hopes that these materials can help your alliance develop and continually adapt a coherent and focused research agenda.
This agenda is designed for a one-day workshop session. If a two-day session would better suit your group, we suggest you complete segments 1 – 6 on day 1 and segments 7 – 12 on day 2. Segment 7, focusing on generating research questions, can be done between sessions, with participants emailing questions to the facilitators. Day 2 would then begin with a review of the submitted questions. This agenda specifies an approximate length of time for each segment. Feel free to vary the time dedicated to each segment based on your participants’ skill levels. For example, if your participants already know a lot about types of research and evidence, you can likely reduce the time spent on that segment. We recommend that you do not cut back time reserved for the segments focused on prioritizing topics and generating questions.
Supplies needed include:






Newsprint/chart paper and easels for group note-taking.
Large sticky notes.
Colored “dot” stickers for prioritization activities.
Markers.
Printed copies of worksheets from the workbook.
Printed copies of research summaries.
3

Prework: Read the summary from one of the following research publications:
These are examples of Institute of Education Sciences (IES) products that illustrate research along the continuum of rigor:




Are Texas’ English language arts and reading standards college ready? (REL 2010–091): http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2010091.pdf.
Reviewing the evidence on how teacher professional development affects student achievement
(REL 2007–033): http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/rel_2007033.pdf.
New measures of English language proficiency and their relationship to performance on large-scale content assessments (REL 2009–066): http://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/REL_2009066.pdf.
Evaluation of Program for Infant/Toddler Care (PITC): An on-site training of caregivers (NCEE
2012–4003): http://ies.ed.gov/ncee/edlabs/regions/west/pdf/REL_20124003.pdf.
Segment 1: Welcome and purpose



Duration: 15 minutes.
Handout: Workshop agenda.
Slides: 1 – 5.
Purposes for and outcomes of today’s workshop:



Engage in a collaborative process to identify research priorities for the alliance and develop a set of research questions on which to build a coherent research agenda for the work the alliance would like to do in the next three to five years.
The morning will focus on reviewing different types of research and exploring such issues as levels of evidence, making causal claims, and linking questions to methods; identifying and prioritizing possible research topics; and generating related research questions.
The afternoon will focus on refining and prioritizing research questions to form an initial research agenda for our alliance. As an alliance, we will be revisiting this research agenda over time.
Review of agenda for today:
 Facilitator walks through today’s agenda.
Introductions:
 Participants give a brief introduction of themselves: name, agency, and role or title.
Segment 2: Alliance goal and possible research topics



Duration: 30 minutes.
Handout: Alliance Goal activity sheet (page 5 in the participant guide).
Slides: 6 – 9.
4

Review alliance goal:




Goal: [Insert goal of your research alliance or group].
The purpose of this section is to reflect on the goal of the alliance and to brainstorm possible research topics related to this goal.
Facilitator reads goal.
Activity: o o
5 minutes: Participants work individually to fill out a two-column sheet (page 5 in the participant guide) with topics related to the goal/framework and topics that possibly fit into the goal. The focus is on which topics/areas of the framework are important to you and what you want your alliance’s research to focus on. Participants should not be concerned with the
“grain size” of the topics at this point. They may have very specific topics or very broad ones.
All are okay.
15 minutes: Divide the whole group into 2 – 3 breakout groups and have people share their topics. In the breakout groups, participants should write on sticky notes the topics that they would like to bring back to the whole group and post the sticky notes on the chart paper or newsprint.
 Affinity Grouping exercise: o o
The facilitator engages the participants in an exercise of “affinity grouping,” where the participants will group the topics into like categories and name these larger categories. See appendix A for directions for the Affinity Grouping exercise.
Note that topics will be prioritized later in the workshop.
Segment 3: Types of research and evidence


Duration: 20 minutes.
Slides: 10 – 15.
For the next three segments you will step back from the research topics and agenda and focus on types of research and the definitions of research agendas and questions.
This segment addresses types of research designs and subsequent types of claims that can be made. It examines such issues as making causal claims, types of evidence needed, and levels of evidence. Led by a facilitator and cofacilitator, it introduces major types of research designs and the types of claims (causal, correlational) that can be made from them. Strengths and limitations of each major research approach are addressed.
Segment 4: Investigating research examples
 Duration: 30 minutes.
5



Handouts: Excerpts or summaries from IES/REL publications used for prework; the Investigating
Research Examples activity sheet (page 2 in the participant guide).
Slide: 16.
In this segment participants delve into one or two studies that use different research methods. The purpose is to get participants thinking about different types of research and the types of questions one can answer with different types of research designs.
Activity:



5 minutes: Participants review summaries of the IES/REL publications assigned as prework.
15 minutes: Participants work in small groups to examine the study summaries and identify the research design, research questions, and data sources used in each study. Participants discuss what is not answered by each research question and design and what other questions could be asked.
10 minutes: Whole-group discussion.
Break (15 minutes)
Segment 5: Research questions and agendas overview


Duration: 15 minutes.
Slides: 18 – 22.
Researchable questions:
 Facilitator discusses what researchable questions are.
Developing research questions:


Facilitator discusses strategies for developing research questions.
Facilitator asks audience to draw on their experiences doing so.
What is a research agenda and what does one look like?


Group discusses different types of research agendas (linear, topical).
Group views examples from slides.
Segment 6: Prioritizing research topics



Duration: 25 minutes.
Reference: List of grouped affinity topics and subtopics from segment 2.
Slide: 23.
Review of grouped topics:
6

 Facilitator will share the list of topics generated and grouped in segment 2, the brainstorming/Affinity Grouping exercise session. Participants clarify the topics by asking questions and making comments. The facilitator will seek feedback about groupings, making changes as needed. All participants together will then prioritize the topics using a modified Focus Four (see below).
Activity: Modified Focus Four prioritization. See appendix B for more specific directions for the Focus Four exercise.



Refer back to the concepts of what constitutes good researchable questions and a research agenda.
Remind participants that they are prioritizing topics that will inform their research questions to guide the alliance’s work over the next several years.
Provide an overview of the process, based loosely on the Focus Four prioritizing process (Garmston
& Wellman, 2000), and explain the steps: o o o o
Brainstorm—(done in segment 2).
Clarify—(done in this segment).
Advocate—Have participants advocate for topics in a brief, positive way. Have participants think about narrowing the list of topics to one or two; they should think about which topics could make up a good agenda when taken together.
Canvass—Suggest that alliance members prioritize so that you end up with two priority research topics. Participants should each be given three colored “sticky dots” to place on the newsprint next to the topics they think should be the top priority for the alliance. After the canvassing process you should be able to count the sticky dots and identify which priority topics received the most votes.
Segment 7: Generating research questions




Duration: 30 minutes.
Reference: List of prioritized topics and subtopics from segment 6.
Handout: Generating Research Questions activity sheet (page 16 in the participant guide).
Slides: 24 – 27.
Break the group into pairs. For the two topics identified as top priorities in segment 6, ask participants to consider the following questions:


What are some possible research questions for these topics?
What data are needed to answer these questions?
Activity:
 Participants should write possible research questions on hanging newsprint to share and discuss in the next segment.
7

Lunch (45 minutes)
Segment 8: Sharing research questions generated



Duration: 15 minutes.
Reference: List of questions by topic and subtopic displayed on newsprint around the room.
Slide: 28.
Have participants engage in a gallery walk around the room to review the research questions generated in pairs. The questions will be posted on newsprint hanging on the walls and arranged by topic and subtopic.
While participants are doing the gallery walk individually, ask them to make notes on the newsprint about things they want clarified, including things they want deleted or added. They should also make notes to themselves about which questions they think are important. Reserve a few minutes for participants to discuss their thoughts in pairs.
Segment 9: Refining questions



Duration: 20 minutes.
Reference: On projector or newsprint, display the list of questions by topic and subtopic.
Slide: 30.
Take 15 minutes for a facilitated whole-group discussion of the generated research questions. Go through questions under each topic/subtopic to address suggestions from individuals.



Ask whether people have suggestions for additions, deletions, or combinations of questions; incorporate these edits into the list of questions.
Keep this segment focused on tailoring the questions rather than advocating for any particular topic/set of questions.
Look for patterns among the questions (for example, are they all descriptive?).
Take the final five minutes of the segment to allow the pairs to reflect on the questions. Prompt them to think about the following: What is important to them? What research topic and questions can inform a meaningful research agenda?
Segment 10: Prioritizing research questions



Duration: 25 minutes.
Reference: Refined questions from segment 9.
Slide: 31.
Activity: Modified Focus Four.
Prioritize questions using the Focus Four techniques. Do this in two rounds, once for each priority topic.
 Brainstorm—(already done).
8




Clarify—(done in segment 9).
Advocate—Have participants advocated for questions in a brief, positive way?
Canvass—Give each individual enough “sticky dot” votes to cover one-third of the proposed questions (for example, if there are 12 questions, each person would have 4 votes). The purpose of the canvassing is to gauge participant interests. When everyone is done voting, review the results.
Ideally, you will end up with two or three priority topics that each has about three to five questions.
If a small number of clear priority research questions cannot be determined from the initial vote, identify the 6 to 10 questions that received the most votes and conduct another vote with only those questions. This should help the group narrow the priority research questions to those that most people feel are important.
Break (15 minutes)
Segment 11: Research agenda


Duration: 20 minutes.
Slides: 32 – 33.
Engage in a whole-group conversation about the results of the prioritization exercise. Discuss how many of the top-rated questions should make up the research agenda. Ideally, the alliance will end up with two priority topics that each has about three to five questions. Reinforce the idea of a coherent research agenda.
Ask people to consider the following:



Types of studies (descriptive, correlational, impact evaluations).
Sequence.
Short- and long-term priorities.
Also, let people know that refinement will continue as information on available data is known and other developments occur.
A research agenda template is provided (pages 19–20 in the participant guide) for the group to document and summarize the main topics, subtopics, and research questions identified during the workshop. Ask the group to fill out this template at the end of the workshop, if there is time. If there is no time, the facilitator should complete the template and email or mail it to alliance members for feedback.
Remember that the research agenda should encompass the work most important to the alliance, should align with the alliance goal, and is the work that the alliance will do over the next three to five years.
Segment 12: Closing and next steps


Duration: 10 minutes.
Slides: 34 – 35.
The facilitator should summarize the next steps for the alliance in the development of the research agenda.
The alliance members will receive a summary document from this Research Agenda Workshop and may be
9

asked to refine the research questions. After an iterative process of refinement the alliance facilitator and researcher will report back to the group on the final list of priority topics and questions, providing information on some of the following:





Existing research on priority topic areas that may be brought to the group for discussion.
Additional data collection that may be required to address proposed questions.
The feasibility of proposed questions and studies given existing data.
Topic areas or questions for which workshops or other training may be appropriate.
Questions that are appropriate and feasible for immediate research projects.
10

After initial priority topics have been listed on sticky notes, ask all participants to reconvene and review the topics generated in each group. Put the sticky notes on the wall so that everyone can see them. Then, ask participants to suggest “like” and related topics and take the lead in grouping topics together. This process establishes “affinity groups” (10 – 12 minutes). When doing this:




Invite participants to make grouping suggestions.
Try to facilitate the discussion so that all can hear and understand why a participant thinks a topic might belong with another topic(s).
Try to keep things moving and not be too exacting.
If there are “outlier topics” that cannot be grouped with other like topics, include them by themselves as a group.
As the momentum for grouping dies down, ask the participants to give a name to every affinity group they created. The name given to each grouping will become the research topic, and the items within the group will be the subtopics.
11

The facilitator is going to use a structured process to identify participant preferences for the research topics and questions based on a process called “Focusing Four” (Garmston & Wellman, 2000). This process has four steps, so take two to three minutes to introduce the process.
The four steps are:




Brainstorm—Participants identify topics of interest and research questions related to the overall goal of the alliance.
Clarify—Participants clarify topics and research questions by asking questions and making comments about the topics or questions. At this point the facilitator should seek feedback from participants, making changes as needed to the topic groupings or research questions.
Advocate—Participants advocate for topics or questions in a brief, positive way. They should not be advocating against a topic or question but for a topic or question.
Canvass—The final step is prioritization using canvassing: o o
For prioritizing the research topics, each person will be given three votes (“sticky dots”) to show where they think the alliance’s research priority should be. They can use their dots any way they want—by placing them all on one topic or on multiple topics.
For prioritizing the research questions, participants should do the same canvassing process for each generated topic area. For example, if the group identified two priority research topics, canvass once for each topic. Give each individual enough votes (“sticky dots”) to make up onethird of the research questions generated (for example, if 12 questions were created, each person would have 4 votes). The purpose is to gauge participant interests.
After the voting process the facilitator should share how many votes each topic or question received.
Highlight the topics that are clearly of the most interest to group members and decide as a whole group which topics are the priority. Discuss the results of the prioritization.
12

If you decide to conduct the workshop online, you will have to modify some of the activities. For example, working in pairs may not be possible. Most webinar software allows for breakout rooms, where subgroups of participants can talk and take notes on a common “virtual whiteboard” space. Consider using breakout rooms for the topic discussions, for investigating research examples, and for refining questions. Instead of using sticky notes and newsprint, use the whiteboard space to write down topics, conduct affinity grouping, and note questions.
Another activity that needs to be modified for an online environment is the Focus Four prioritization activity. A suggestion for doing this activity online is to use the “polling” function common in webinar software. After your participants come up with their list of possible research topics (or questions), your webinar producer should create a webinar poll that lists the topics (or questions). The poll can be projected on the webinar screen, and participants can cast their votes for their favorite topics (or questions) by clicking or selecting a button or box next to the topic. Most webinar polls only allow one vote per person, so your webinar producer will need to run the same poll as many times as the number of votes you will be giving people. Participants should cast a vote each time the poll is run; they can choose to cast their vote for the same or a different topic (question) each time. After the polls are completed the webinar producer can tally the number of votes each topic (question) received and share the results with the group.
13

Garmston, R., & Wellman, B. (2000). The adaptive school: Developing and facilitating collaborative groups. Norwood, MA:
Christopher-Gordon Publishers, Inc.
14

July 2014

PURPOSE OF THE WORKSHOP ............................................................................................................................... 1
PREWORK ASSIGNMENT: INVESTIGATING RESEARCH EXAMPLES .......................................................................... 1
AGENDA................................................................................................................................................................. 3
ALLIANCE GOAL AND POSSIBLE RESEARCH TOPICS ................................................................................................ 4
TYPES OF RESEARCH AND EVIDENCE ...................................................................................................................... 6
D ESCRIPTIVE RESEARCH —W HAT ’ S H APPENING ?
................................................................................................................. 6
D
ESCRIPTIVE RESEARCH
—W
HAT
’
S
K
NOWN
?
...................................................................................................................... 7
C ORRELATIONAL RESEARCH —M AKING C ONNECTIONS .......................................................................................................... 7
I MPACT / CAUSAL RESEARCH —M AKING AN I MPACT ............................................................................................................... 8
INVESTIGATING RESEARCH EXAMPLES................................................................................................................. 10
RESEARCH QUESTIONS AND AGENDAS OVERVIEW .............................................................................................. 11
R ESEARCH QUESTIONS .................................................................................................................................................. 11
R ESEARCH AGENDA ...................................................................................................................................................... 12
PRIORITIZING RESEARCH TOPICS ......................................................................................................................... 13
GENERATING RESEARCH QUESTIONS ................................................................................................................... 14
SHARING AND REFINING QUESTIONS .................................................................................................................. 18
PRIORITIZING RESEARCH QUESTIONS .................................................................................................................. 18
RESEARCH AGENDA ............................................................................................................................................. 19
RESEARCH AGENDA TEMPLATE ............................................................................................................................ 20
NEXT STEPS AND THANK YOU .............................................................................................................................. 22
REFERENCES ......................................................................................................................................................... 22
NOTE ................................................................................................................................................................... 22
Activity sheets
A CTIVITY SHEET 1.
I NVESTIGATING RESEARCH EXAMPLES ............................................................................................................. 2
A CTIVITY SHEET 2.
A LLIANCE GOAL ......................................................................................................................................... 5
A CTIVITY SHEET 3.
G ENERATING RESEARCH QUESTIONS ............................................................................................................ 17
Tables
T ABLE 1.
T HE CONTINUUM OF RIGOR IN IMPACT EVALUATION DESIGNS ......................................................................................... 9
T ABLE 2.
G ENDER IN S CHOOLS R ESEARCH A LLIANCE : M OVING FROM TOPICS TO SUBTOPICS ............................................................ 14
T ABLE 3.
G ENDER IN S CHOOLS R ESEARCH A LLIANCE : M OVING FROM SUBTOPICS TO QUESTIONS ...................................................... 16

The purpose of the Research Agenda Workshop is to engage in a collaborative process to identify alliance research priorities and develop a set of research questions that together make up a coherent research agenda for the work the alliance would like to do over the next three to five years.
This morning we will focus on reviewing different types of research and exploring levels of evidence. We will also identify and prioritize possible research topics and generate initial research questions.
This afternoon we will refine and prioritize research questions to form an initial research agenda for our alliance. As an alliance, we will revisit the research agenda over time.
Supporting materials for this section include:
 Slides: 1–5.
Directions: Read the summary from one of the research publications listed below. Be prepared to discuss the research summary during the workshop. Feel free to use the Investigating Research Examples activity sheet on page 2 to take notes on the study summary you read.




Are Texas’ English language arts and reading standards college ready? (REL 2010–091): http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2010091.pdf.
Reviewing the evidence on how teacher professional development affects student achievement
(REL 2007–033): http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/rel_2007033.pdf.
New measures of English language proficiency and their relationship to performance on large-scale content assessments (REL 2009–066): http://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/REL_2009066.pdf.
Evaluation of Program for Infant/Toddler Care (PITC): An on-site training of caregivers (NCEE
2012–4003): http://ies.ed.gov/ncee/edlabs/regions/west/pdf/REL_20124003.pdf.
Topics that will be discussed during the workshop that relate to these summaries include:





Research questions.
Type of study.
Data source(s).
Findings.
Possible next steps—future research questions.
1

Activity sheet 1. Investigating research examples
You may wish to use this organizer to keep notes on the study summary that you read.
Title of study: _________________________________________________________________________
Research question(s) Data source(s) Findings Study design
What else there is to learn/future research questions
2

15 minutes
30 minutes
20 minutes
30 minutes
15 minutes
15 minutes
25 minutes
30 minutes
45 minutes
15 minutes
20 minutes
25 minutes
15 minutes
20 minutes
10 minutes
Welcome and purpose
Alliance goal and possible research topics
Types of research and evidence
Investigating research examples
Break
Research questions and agendas overview
Prioritizing research topics
Generating research questions
Lunch
Sharing research questions
Refining research questions
Prioritizing research questions
Break
Research agenda
Closing and next steps
3

Alliance goal: [This is an example, please insert your alliance goal here]. The U.S. Virgin Islands College and Career Readiness Research Alliance will support the Virgin Islands Department of Education in its efforts to prevent students from dropping out of schools (and reduce the number of dropouts) by providing applied research and analytic technical support on how best to use available data to both establish robust early-warning systems and identify interventions to help improve outcomes for students at risk.
The purpose of this section is to reflect on the goal of the alliance and to brainstorm possible research topics related to this goal. Brainstorming will happen in two steps:


Individually (5 minutes): Working on your own, use the Alliance Goal activity sheet on page 5 to generate lists of topics that fit under the goal or possibly fit under the goal.
Small-group discussion (15 minutes): Divide into small groups and discuss the lists of topics generated by the individual brainstorming session. Large Post-It notes will be provided for recording the topics that your group feels are related to the alliance goal. Write down one topic per
Post-It note. Discuss the topics that your group members felt might fit under the goal and decide if they should be included in your Post-It notes, which will be shared and used later in the workshop to identify research priorities.
When the whole group reconvenes, you will put all the Post-It notes on chart paper on the walls, examine all the topics identified by the groups, and engage in a process of affinity grouping where “like” or related topics are grouped together.
Your whole group will then give names to the affinity groups (10 minutes): The facilitator will ask participants to suggest “like” or related topics and will lead the group in putting related topics together.
This process establishes “affinity groups.”
Supporting materials for this section include:


Slides 6–9.
Alliance Goal activity sheet.
4

Activity sheet 2. Alliance goal
Work independently for five minutes to list research topics that fit under the alliance goal. Categorize your list by the two column headings.
Topics that fit into goal Topics that possibly fit into goal
5

In this section we will do a quick primer on different types of research and discuss how they allow us to answer different types of questions. The terms and language used in this section are based on IES guidance to RELs.
There are many ways to classify research. We will discuss three main types of research:



Descriptive ( What’s Happening and What’ s Known reports).
Correlational (Making Connections reports).
Impact/causal research, including program and policy evaluation (Making an Impact reports).
Additional resource: http://www.ecs.org/html/educationIssues/Research/primer/appendixA.asp.
Supporting materials for this section include:
 Slides: 10–15.
Descriptive research — What’s Happening?
Descriptive research looks at what is happening by examining trends, baselines, and experiences of individuals, groups, or programs. What’s Happening is the name that the REL program has given to descriptive research reports. Descriptive studies cannot inform cause and effect or explain why something is happening. But they are often a useful first step in a larger research agenda because they help establish a baseline for the current state of affairs. Descriptive studies can point out areas for further investigation. This type of research can be quantitative or qualitative and includes survey, case study, and noncausal statistical research.
Descriptive research can use primary data collected for the research but frequently uses secondary data analysis and existing datasets, documents, and records:


Primary data: Data collected for the current study
Secondary data: Data already collected for another purpose or another research study
Descriptive research tends to use simple descriptive statistics to report findings; these include averages, frequencies, and percentages.
Example What ’s Happening questions:




What is the trend in science test scores over the past three years?
What are the certification characteristics of beginning teachers in the region?
What are the features of district teacher evaluation systems?
What is the enrollment of students eligible for free or reduced-price lunch in districts in the state?
Supporting materials for this section include:
6

 Slide 11.
Descriptive research — What’s Known?
The REL program classifies another type of descriptive research product as What’s Known.
This type of research is often formed as a literature review, exemplified by such work as meta-analysis and What Works
Clearinghouse reviews. A meta-analysis is the use of statistical techniques in a systematic review to integrate the results of included studies.
1
The What Works Clearinghouse is an IES-funded initiative with the goal of being a resource for informed decision-making. It reviews research on education programs, products, practices, and policies and evaluates the credibility and reliability of the research evidence. And it includes an online searchable database with more than 700 publications and more than 6,000 reviewed studies.
A literature review of what is known can be an early step in a research agenda, as it provides information on what is known about a research topic and what is not (and maybe even why certain questions are difficult to answer).
Example What’s Known questions:



What does the research say are components of an effective preservice preparation program?
Which interventions are effective in increasing the learning of mathematics content and skills among elementary school students ( http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=21 )?
Are some interventions more effective for certain types of students, particularly students at risk of failure in mathematics ( http://ies.ed.gov/ncee/wwc/documentsum.aspx?sid=21 )?
Additional resource: What Works Clearinghouse ( http://ies.ed.gov/ncee/wwc/ ).
Supporting materials for this section include:
 Slide 12.
Correlational research — Making Connections
Some research questions ask about the relationship between two or more variables. This type of research is known as correlational research and falls under the REL program’s Making Connections product line.
Correlational research can describe trends, patterns, and relationships between two or more variables, but it does not imply causality. Researchers should note the limitations of this type of research and discuss results using such language as “association” or “related to” rather than language than implies causality.
Correlational research usually implies that one is testing for statistical differences between two or more variables or groups or testing to see how the variables move in relation to one another. For example, descriptive statistics can be used to look at the achievement of boys and girls on the science New England
Comprehensive Assessment Program, but correlational research would go further to test whether any observed differences are “statistically” different—that is, not likely due to chance.
7

Correlational studies employ quantitative analysis of newly collected or already collected (secondary) data.
Existing state and district databases are good sources of data for correlational questions.
Statistical techniques include: difference in means testing, chi-square testing, ANOVA, and regression models.
Example Making Connections questions:




How do math course-taking patterns vary by race and gender?
What factors are related to differences in graduation rates across districts?
Does science New England Comprehensive Assessment Program achievement in grade 6 vary by gender?
Are characteristics of teacher preparation programs related to observed teacher characteristics?
Be careful of two unrelated phenomena that have a spurious relationship.
Additional resource: What Works Clearinghouse ( http://ies.ed.gov/ncee/wwc/ ).
Supporting materials for this section include:
 Slide 13.
Impact/causal research — Making an Impact
Impact/causal research encompasses designs that allow the researcher to answer questions about cause and effect. In education and social/behavioral research, this type of research often involves the study of whether an intervention had an impact on the group receiving the intervention. Rigorous program and policy evaluations can fall into this category. This type of research is part of the Making an Impact REL product line.
To make causal claims or inferences, a research study must include certain design elements. REL-funded impact studies must meet What Works Clearinghouse standards. Randomized controlled trials are considered the “gold standard” in impact studies; there is randomization of treatment and control groups
(for example, those who receive the intervention and those who do not).
Table 1 shows the continuum of rigor in impact evaluation designs, from the least rigorous to the most powerful. Other designs, including quasi-experimental and matched comparisons, may not yield unbiased estimates of impact.
Impact/randomized controlled studies can be large or small. They also can be formative or summative. REL program guidance states: “Ideally, impact studies should test the effect of policies, programs, or practices that respond to concerns that have been thoroughly examined in descriptive work” (Maynard & Neild,
2012, slide 5).
8

It is likely that most alliances will have some questions of causal inference. Alliances may want to consider
“opportunistic” possibilities for impact evaluation or studies. Such a situation might include capturing the rollout of a new intervention in a subset of schools in a state and having the opportunity to help design the rollout strategy so that it can be appropriately studied using an experimental design.
Example Making an Impact questions



Does this afterschool program increase student engagement?
Do new teacher evaluation policies improve student achievement outcomes?
Do smaller class sizes increase student achievement?
Additional resource: What Works Clearinghouse ( http://ies.ed.gov/ncee/wwc/ ).
Supporting materials for this section include:
 Slides 14–15.
Table 1. The continuum of rigor in impact evaluation designs
Evaluation design
Matched comparison group
Example
Type of questions the evaluation can answer
Schools are selected to implement a new program through some nonrandom process (say, they volunteer). Before the program begins these schools are matched on important background characteristics (say, student demographics and average test scores in the prior academic year) with other, nonparticipating schools. After the program has been implemented in the participating schools, the outcomes for the two groups of schools are compared to estimate the program’s effect.
Did outcomes differ between the matched groups of participating and nonparticipating schools?
Rigor
Least rigorous design (lowest confidence that results can be attributed to program)
Most rigorous design (highest confidence that results can be attributed to program)
9

Comparative interrupted series
Regression discontinuity a time
Schools are selected to implement a new program, again through a nonrandom process. Before the program begins these schools are matched to comparison schools with similar histories of background characteristics and outcomes. After the program has been implemented in the participating schools, trends in outcomes over time are compared with the outcomes for schools that did not participate in the program.
Schools are selected to implement a new program based on a predetermined “cutpoint” on a well defined and easily measured criterion (for example, proficiency rates below 25 percent). The outcomes for participating schools are then compared with the outcomes for schools that “just missed” being selected.
Random assignment A set of schools is selected to implement a pilot program based on a random process (for example, a lottery is used to select 20 pilot schools from among interested volunteers statewide). At the end of the pilot implementation period the outcomes for pilot schools are compared with the outcomes for the other interested nonparticipating schools.
Source: Perez-Johnson et al., 2011.
Did outcomes in the program schools improve more than would be expected given trends in similar nonparticipating schools?
What is the impact of the program on outcomes?
Or, are outcomes in program schools different than they would have been absent the program?
What is the impact of the program on outcomes?
Or, are outcomes in the pilot schools different than they would have been absent the program?
The purpose of this section is to dig deeper into the research summaries reviewed by participants in their prework assignment.
Directions:


Individually skim the summary you read to refresh your memory.
Organize yourselves into pairs or small groups to discuss and identify in each study: o o o
The research design.
The research questions.
The data source(s).
Use the Investigating Research Examples activity sheet on page 2 to record your answers and take any notes.
You might also want to dig deeper to discuss:



What questions could NOT be answered by these studies.
What additional questions you have based on the study findings.
What you would need to answer these next questions (additional data, rigorous research design).
Refer to the activity sheet that you used in the pair/small-group discussion. Be prepared to report out to the whole group.
Supporting materials for this section include:
 Slide 16.
10



Research summaries that were assigned for prework.
Investigating Research Examples activity sheet (page 2).
Refer to the research summaries from the prework assignment. They are listed again here




Are Texas’ English language arts and reading standards college ready? (REL 2010–091): http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/REL_2010091.pdf.
Reviewing the evidence on how teacher professional development affects student achievement
(REL 2007–033): http://ies.ed.gov/ncee/edlabs/regions/southwest/pdf/rel_2007033.pdf.
New measures of English language proficiency and their relationship to performance on large-scale content assessments (REL 2009–066): http://ies.ed.gov/ncee/edlabs/regions/northeast/pdf/REL_2009066.pdf.
Evaluation of Program for Infant/Toddler Care (PITC): An on-site training of caregivers (NCEE
2012–4003): http://ies.ed.gov/ncee/edlabs/regions/west/pdf/REL_20124003.pdf.
Research questions
A research topic is a general statement about a research interest. A research question is more specific. How we phrase a question gives hints about the method or approach that will be used to explore the question
(descriptive, correlational, inferential). Our interest here is in identifying practical questions to address real problems or issues observed in practice or everyday occurrence (as opposed to theoretical problems).
Characteristics of a good research question (Killian, 2008):




Reasonable — It has a realistic scope and can be examined given time and budget.
Appropriate — It fits with the program or issue being studied.
Answerable — It can be answered with appropriate data.
Specific — It focuses on clearly defined and measurable indicators of success or the desired outcome.
Questions that are too general are not research questions but more likely problem or topic statements.
How does one come up with research questions?
This is usually a multistep or iterative process that starts with the identification of a topic or problem that you, or in this case the alliance, would like to address through research. You may not know if your research questions are “good ones” without doing some preliminary work. For example, you may have great questions, but there may not be existing data that can be accessed to answer your questions. So, one of the first steps is to determine if you can collect the data that will answer the question(s).
Descriptive and correlational questions (examples):


How do math course-taking patterns vary by race and gender?
What are the features of district teacher evaluation systems?
11

 What factors are related to differences in graduation rates across districts?
Inferential/causal questions (examples):



Does this afterschool program increase student engagement?
Does this math intervention lead to improved math achievement?
Do new teacher evaluation policies improve student achievement outcomes?
Supporting materials for this section include:
 Slide 18.
Research agenda
There is no clear definition of a research agenda.
For the REL research alliances we define a research agenda as two to four focused research topics each with a set of three to five coherent research questions that will help the alliance achieve its goal(s). A research agenda also can include a set of current or future projects that align with the research questions, as well as the projected timing of those projects.
Research agendas may be:


Topical — focused on one topic (with a set of questions) or a set of related topics (each with a set of questions).
Linear — a question or set of questions are tackled first to discover descriptive information, which then leads to more specific questions to be researched using an experimental or impact design.
For your alliance’s research agenda, think about the kind of work you would like your alliance to do over the next three to five years. The ideal research agenda would include a mix of types of studies, with both short-term and long-term items.
Example of a topical agenda for an alliance with the goal of increasing mathematics achievement in the
Northeast & Islands region:


Topic: Mathematics learning
Research questions: o o o
What is the effect of grade 8 participation in algebra I on students’ math achievement in high school?
What practices are schools in the Northeast & Islands states and jurisdictions using to teach math to students with disabilities?
What are the math performance patterns for students with disabilities in the Northeast &
Islands states and jurisdictions?
12

 Studies: o o o
Randomized controlled trial of the impact of student access to algebra I in grade 8.
Descriptive study of math education practices for students with disabilities in six schools in two Northeast & Islands states and jurisdictions.
Descriptive study of math performance patterns for students with disabilities in Massachusetts and New York.
Example of a linear agenda for an alliance with the goal of increasing the percentage of district X students who graduate from high school:


Topic: High school graduation
Research questions: o o o
What are the high school dropout rates for particular subgroups of students?
What are the re-enrollment rates of high school dropouts in traditional high schools, and what are their education trajectories?
What are the effects of alternative high school programs on students’ degree completion and high school graduation?
 Studies: o o o o
Descriptive study of high school dropout rates for key student subgroups.
Descriptive study of re-enrollment rates in traditional high schools and education trajectories of re-enrollees.
Descriptive study of the types of alternative programs in the district.
Randomized controlled trial of the impact of alternative programs for degree completion on high school graduation.
Supporting materials for this section include:
 Slides 19–22.
The purpose of this section is to narrow and prioritize the list of research topics generated earlier in the session. As you engage in this prioritization exercise keep in mind:


What makes a good research question.
What makes a good research agenda.
Directions: The goal of this exercise is to narrow the list of research topics to two to four priority topics for the alliance. The facilitator would like you through a modified Focus Four process to:
 Brainstorm — Review the brainstormed list of possible research topics provided by your facilitator.
13




Clarify—Participants will be asked if any topics need to be clarified.
Advocate—You will be given time to advocate for any items you feel strongly about. Think about narrowing to two to four topics at most and selecting the ones that work together coherently.
Canvass—Vote for the topics you feel are a priority to the alliance and would make the most coherent research agenda.
At the end of this exercise the facilitator will summarize for the whole group the topics that received the most votes and that are a priority for the alliance.
Supporting materials for this section include:


Slide 23.
List of possible research topics generated earlier in the session (your facilitator will provide this).
In this part of the workshop we move from prioritizing research topics to generating research questions. See tables 2 and 3 for an example of how to move from topics to research questions.
Directions:




Review the list of topics identified in the previous section as priorities for your alliance.
Identify the topic or subtopic on which you will focus.
Break into pairs and generate some possible research questions related to the topic/subtopic, keeping in mind what data would be needed and what data are available.
Record your questions on the Generating Research Questions activity sheet provided on page 16 and on the newsprint provided.
Table 2. Gender in Schools Research Alliance: Moving from topics to subtopics
Alliance goal: Provide research that informs and promotes gender equity in schools
Question Timing Topic
Girls in science, technology, engineering, and math
(STEM)
Attainment for boys
Subtopic
Tracking gender achievement differences
Encouraging interest and enrollment in STEM
Encouraging persistence in
STEM
Structured inequality
Source: Authors.
14

15

Table 3. Gender in Schools Research Alliance: Moving from subtopics to questions
Alliance goal: Provide research that informs and promotes gender equity in schools
A fully specified research question
Subtopic
Tracking gender achievement differences in science, technology, engineering, and math (STEM)
Questions
What are gender differences in STEM achievement in K – 12, and how have they changed?
Do gender differences in STEM achievement vary across districts and schools?
Encouraging interest and enrollment in
STEM
What can teachers do?
Are single-sex schools and classrooms better?
Persistence in STEM majors
What programs help promote STEM for girls?
Source: Authors.
A general question that can be used to develop a set of more specific research questions
Supporting materials for this section include:



Slides 24–27.
Prioritized topic(s).
Generating Research Questions activity sheet (page 16).
Timing
16

Activity sheet 3. Generating research questions
Alliance name: _________________________________________________
Your name: _______________________________
Your email: _______________________________
Facilitator email: _________________________________________________
Please spend time thinking about the prioritized research topics for our alliance. Consider research questions that you feel will be important for our alliance to work on and note them below. Then think about what data would be needed (and are available) to investigate these questions and what type of study design would be indicated by your questions. At this point in the process we are looking for good and important research questions, not a brainstormed list. Once you have filled out the following form, please email or hand-in this page to your facilitator.
Priority topic 1: __________________________________________________________________
1.
2.
3.
Questions Type of study Data available
1.
2.
3.
Priority topic 2: _______________________________________________________________________
Questions Type of study Data available
17

The purpose of this section is to review all the research questions generated by the pairs in the previous segment.
Directions:
Sharing research questions (15 minutes)
 Gallery walk: o o o
Review the compiled list of research questions by topic posted on newsprint around the room.
Individually make notes about the questions: edit, add, or delete research questions and generate any new ones.
Reserve a few minutes to talk with a partner about the questions.
Refining research questions (20 minutes)
 Whole-group discussion: o o
Are there suggestions for additions, deletions, or combinations?
Discuss what kind of study would result from each question (descriptive, correlational, impact evaluation).
 Paired discussion to reflect on questions: o o
What is important to each person?
What topic and questions can form a meaningful research agenda?
Supporting materials for this section include:


Slides 29–30.
Compiled list of research questions.
In this section we will prioritize the generated research questions within each priority topic. Ideally, the alliance should identify two to four priority research topics, each with three to five research questions. We will again use a modified Focus Four activity for a systematic prioritization process.




Brainstorm—Review the list of generated research questions for each identified priority topic.
Clarify—Participants will be given an opportunity to ask clarifying questions.
Advocate—You will be given time to advocate for any items about which you feel strongly.
Canvass—Indicate the questions you feel are a priority to the alliance and would make the most coherent research agenda.
18

Supporting materials for this section include:
 Slide 31.
The alliance will now discuss the results of the Focus Four prioritization. Consider the following:



How many of the top-rated questions should be included in the research agenda?
Do we have a good mix of study types (descriptive, correlational, impact evaluations)?
What is the best sequence for this work?
Use the research agenda template (pages 19–20) to summarize the priority topics and the relevant subtopics and research questions the group has identified as priorities during this workshop. In addition, record any notes you may have about the potential sequence or timing of the projects.
Remember that our alliance’s research agenda can and should be revised over time as we gain new information and other developments occur. Remember that the agenda:



Should encompass the work most important to the alliance.
Should align with the alliance goal.
Is the work that the alliance will do over the next three to five years
Ideally, the alliance will have identified two to four priority topics, each with three to five questions.
Supporting materials for this section include:
 Slides 32–33.
19

[Alliance name]
Research agenda 2012 – 17
The [alliance name] has identified a goal of [alliance goal]. To work toward that goal the alliance engaged in discussions to develop a shared research agenda that would guide current and future projects and, more specifically, define the direction of the alliance. The research agenda outlined below is designed as a roadmap for alliance work so that projects are directly useful to alliance members, supply research that informs policy and practice, and increase the capacity of alliance members to conduct and use research. To that end, the [alliance name] identified the following topics as areas of focus for their work for the next five years.
[Topic area 1]
[Provide a one-paragraph description that summarizes this topic area and the possible subtopics of interest.]
[Topic area 2]
[Provide a one-paragraph description that summarizes this topic area and the possible subtopics of interest.]
[Topic area 3]
[Provide a one -paragraph description that summarizes this topic area and the possible subtopics of interest.]
20

Research agenda for [topic area 1]
Subtopic Questions
Research agenda for [topic area 2]
Project
Subtopic Questions
Research agenda for [topic area 3]
Project
Subtopic Questions Project
21
Task
Task
Task
Data availability Timing
Data availability Timing
Data availability Timing

For the alliance:





Continued iterative review and refinement of agenda.
Collaborative involvement in study prioritization and development.
Identification of existing data or data collection for proposed studies.
Review of research literature relevant to priority topics and questions.
Oversight of research study or identification of research support.
Supporting materials for this section include:
 Slides 34–35.
Killian, J. (2008). Assessing impact: Evaluating staff development, 2nd edition. A joint publication with the National Staff
Development Council. Thousand Oaks, CA: Corwin Press.
Maynard, R., and Neild, R. C. (2012, January). Asking causal questions, meeting causal standards. A presentation at the
2012–2017 Regional Educational Laboratory program opening conference, Washington, DC.
Perez-Johnson, I., Walters, K., Puma, M., Herman, R., Garet, M., and Heppen, J., et. al. (2011). Evaluating ARRA
programs and other educational reforms: A guide for states. Resource document developed jointly by The American
Institutes for Research and Mathematica Policy Research, Inc. Washington, DC.
1. As defined in the Cochrane Collaboration Glossary of Terms ( http://www.cochrane.org/glossary) .
22