Department of Mathematics
University of Wisconsin – Madison
November, 2007
In his letter of September 17, Dean Sandefur has directed the mathematics department to prepare and endorse a self study of its Wisconsin Emerging Scholars (WES) program as the initial phase of a program review. The purpose is to present background and context for a subsequent outside review committee that will provide recommendations to the College concerning the future of
WES. The WES oversight committee was directed to prepare this self study with input from the program director, Dr. Concha Gomez, in the form of a director’s report.
Some important parameters of our charge are spelled out by the dean’s letter:
I do not question the main premise of the “Treisman model,” which is that collaboration among students in a challenging environment produces better learning of calculus and more confidence.
Rather than addressing the Emerging Scholars paradigm, the program review is intended to focus on how our implementation of the model is working at Wisconsin today,
… and what is the future. Could a broader target effectively serve more students?
… Is it time to stop, maintain the status quo, or expand?
… Among the reasons we are looking at WES today is that several of the WES calculus sections this fall were canceled due to low enrollment.
In our self study, we are asked specifically to provide information about the staffing and costs associated with WES in recent years, sources of funding, the target audience for the program and the numbers of the target audience enrolled at UW-Madison, what fraction of them are enrolled in
WES, and what the effectiveness of the program has been for those who participated. Effectiveness should include but not be limited to grade point average and drop, F, and D rates among WES and non-WES sections.
Accordingly, this report is divided into five sections:
1) Origins of WES and a brief history;
2) An overview of cumulative statistics on WES achievement;
3) A summary of recent budget, staffing, recruiting, and enrollment problems;
4) The director’s report;
5) A discussion of cost.

2
Additional data on budget, performance, and comparative cost, together with some TA testimonials about undergraduate student assistants, are presented in appendices.
An excellent account of the start of the WES program and the challenges that were faced can be found in the “WES Process and Outcomes for Faculty and Administrators” (pp. 53-63) section of the LEAD Center report “Final Evaluation Report on the Pilot Wisconsin Emerging Scholars
Program:1993-94.”The Wisconsin Emerging Scholars (WES) program is modeled after the
Emerging Scholars program created by UT-Austin Professor and MacArthur Fellow Uri
Treisman. The creation of this program in 1993 was facilitated by Prof. Treisman and the UT-
Austin Dana Center that he directs. In particular, some UW-Madison faculty and graduate students attended professional development training sessions at the Dana Center. This professional development included training and instruction on group learning, section management, and the development of Emerging Scholars styled problem sets that are an important part of the pedagogy. Training is important, because part of the success of Emerging
Scholars programs depends not only on suitable problem sets aligned with course syllabi and texts, but also on undergraduate and graduate students who are properly trained for Emerging
Scholars recitation sections. Maintaining alignment with the Emerging Scholars architecture is one of the primary responsibilities of the WES director.
According to our program’s web page, the goal of WES is “to provide motivated students with an opportunity to study calculus in a challenging, friendly, multicultural environment.” The program description continues:
WES sections are discussion sections attached to a large calculus lecture, but the
WES section meets for more hours per week than do the regular discussion sections. A WES discussion section is generally more diverse than a regular section, and students work in small groups on challenging problems designed to foster high levels of understanding and interest. The class is led by a Teaching
Assistant (TA) who has exhibited a passion for teaching; often the WES TAs receive awards recognizing their outstanding teaching abilities. The WES students also receive help and encouragement from undergraduate Student Assistants; typically a Student Assistant (SA) is a former WES student.
WES students form bonds with each other, and with the TA and SA, and this provides many different sources of techniques for solving difficult problems. It also keeps enthusiasm high, which is necessary since the class requires time and effort. WES students do earn one or two extra credits (these are non-graded credits) in addition to the graded credits of calculus.
The WES program was inaugurated at UW-Madison in the fall of 1993 by Prof. Michael
Bleicher and Dr. Melinda Certain after a planning process carried out by a committee of about a

dozen department members. Bleicher taught WES sections in the calculus sequence Math 221,
222, 223 during AY 1993-94 and fall 1994. He remained active in the program until his retirement in February 1996. Certain taught WES sections of Math 221 and 222 during AY
1993-94 and Math 211 and 213 during AY 2000-01. She continued as director of the program until her retirement in May 2003. During AY 2003-04, while the search for a new director was in progress, Visiting Van Vleck Assistant Professor John Vano coordinated the program. The current director, Dr. Concetta M. (Concha) Gomez, took up her position in the fall of 2004. During AY 2004-05 and AY 2005-06 she taught lectures of Math 171, 217 and 171,
222, respectively, with a TA-taught WES section attached to each lecture.
Excluding the current, unfinished, fall semester, a total of 1,236 students have passed a WES section course during the 15 fall terms since the program’s inception, yielding an average of 82 students per fall semester. The corresponding numbers for the 14 spring terms are 782 students with an average of 56 students per spring semester.
The first year of the program, and then the impact of the program through 1996, have been examined in two evaluations carried out and published by the LEAD Center of UW-Madison
1
.
Over the past fifteen years, the odds that a UW-Madison student received a B or better in a
Mathematics Department introductory calculus-related course (as defined in Appendix B) are less than even. UW-Madison students have taken an introductory calculus-related course 52,540 times during that period. The odds that a student who took a WES section of such a course received at least a B are better than 2 to 1. Thus, if receiving a B or better is considered doing well, then the relative odds for a WES student compared with a non-WES student of doing well in an introductory calculus-related course are better than 2 to 1. If not receiving a grade of C or better is considered doing poorly, then the WES:non-WES relative odds of not doing poorly are
2.9 to 1. For targeted minority students during the same period the results are even more skewed: the targeted minority WES:non-WES relative odds for doing well in these courses are 3 to 1 and for not doing poorly are better than 4 to 1.
The distribution of Math ACT scores among WES and non-WES students is similar, and so should not be used to account for the differences in performance. The relative odds results arise over all three categories of students with Math ACT scores of less than 28, 28-31, and greater than 31, with the greatest difference in the first two categories. There are variations by course and term, but the patterns noted above, with small variations in the actual odds, hold. The percentage of all targeted minority students taking introductory calculus-related courses who
3
1
S. B. Millar, B. B. Alexander, H. A. Lewis, J. R. Levin, “Final Evaluation Report on the Pilot
Wisconsin Emerging Scholars Program 1993-94” LEAD Center-UW-Madison, March 1995;
Steve Kosciuk , “Impact of the Wisconsin Emerging Scholars First-Semester Calculus Program on Grades and Retention from Fall 93-96” LEAD Center-UW-Madison, July 1997.

4 took such a course in a WES section dropped significantly after the first few years of WES, although there has been some fluctuation in recent semesters.
For a more detailed explanation of these findings and many additional statistics on WES student performance, see Appendix B.
Several interrelated problems concerning budget, staffing, and recruitment have caused a sharp decline in WES enrollment over the past two semesters. This sudden drop in participation is the immediate reason for the present self study and program review. Our goal here is to describe dispassionately recent events that have led to the downturn. While various possible remedies suggest themselves, specific recommendations concerning the future of WES are the charge of the outside committee to be formed by the college as the next stage of the review.
Budget Overruns
The budget for WES has been flat at $15K/yr from the college for many years now, with an additional $3K provided by the Hilliard foundation account targeted for WES. During her first year as director, AY 04-05, Dr. Gomez ran the program within the designated level of funding.
However, in 05-06 she greatly expanded the use of student assistants (SAs) in the classroom and to staff a new evening tutorial program, gave the SAs a pay raise, and also increased the number of co-curricular activities, as explained in her director’s statement below. As a consequence,
WES expenses essentially doubled that year, and during the early fall of 06 they continued to run at about the same level.
Confronted with very substantial cost overruns, our chair formed the WES oversight committee to address this situation and other program issues. Prior to that time and partially for historical reasons of college-level budgeting, WES had operated rather independently with limited departmental supervision. The previous director Melinda Certain was quite familiar not only with Mathematics Department culture, but also with the culture of the college. As the incoming director, Dr. Gomez did not fully appreciate the nature, level and frequency of ongoing administrative support that she could expect from department staff and university information systems. These circumstances contributed to the program operating outside of its targeted budget.
As a proposed remedy, the WES oversight committee and Dr. Gomez developed a monthly spreadsheet of projected expenses for future years designed to keep WES running at $18K/yr
(see appendix A). A process also was initiated whereby both the WES director and staff review expenditures on a monthly basis to make sure the program stays under budget. The steady-state plan eliminated the evening tutorial program, greatly reduced the number of student assistants, and limits food to one or two WES events per semester.
Short-term “triage” was needed for spring 07. The department budget committee designated an emergency fund of $3K to help keep the program afloat at a bare-bones level. In addition to

5 halting evening tutorials and eliminating most SAs, social activities were discontinued and the size of the program was reduced dramatically. As a result of these drastic measures, the department’s final accounting for AY 06-07 showed a WES program balance of $2781.37.
For there to be any prospect of growing WES, or even maintaining its current level, there will need to be a budget increase from the college or university.
Medical leave
Dr. Gomez became seriously ill last January and remained on sick leave for most of the spring semester. She returned to work toward the end of April, in time to do some preliminary recruiting for this year’s program during the last few weeks of the term. Admirably, WES was administered by its TAs during her absence, but this disruption exacerbated the problems already caused by previous overspending.
Recruiting breakdown
The position of WES director has always been a 9-month academic staff appointment. The previous director, Dr. Melinda Certain, has explained to us that she advised at SOAR during the summer in exchange for corresponding flex time during the academic year, adding that “I realize it is not a model that can be sustained.” There is currently no mechanism in place to compensate the WES director for work over the summer even though effective administration of the program requires a very substantial time commitment during those months.
A program review of the Mathematics Department as a whole was conducted last year. Among the principal findings concerning WES were these:
The WES director, Concha Gomez, has done a wonderful job finding and recruiting minority and rural students to the WES program. She expressed dissatisfaction, however, that she is unpaid during the summer months when much of the recruiting for WES takes place. Were she not to recruit during the summer, she pointed out, she would essentially be undermining the need for her position. …
The committee regrets the financial difficulties facing the WES program. Ideally, the director would have an assistant, would receive a summer salary, and would have a larger budget for student hourly workers and refreshments. Because no money is available for these things, WES will have to survive in a more limited fashion unless the Department chooses to cut other programs.
As explained in the director’s report that follows, knowing the critical role of summer recruitment in her work, knowing how time-consuming that recruitment is, and knowing that she would not be compensated for those three months, Dr. Gomez chose not to carry out any WESrelated activities until the beginning of AY 07-08. Consequently, three of the sections customarily offered each fall needed to be cancelled this semester due to low enrollment.

6
The Wisconsin Emerging Scholars program has been in existence for approximately 15 years.
Regrettably there was a hiatus in leadership during AY 03-04. The following year the
Department of Mathematics was fortunate to attract a director who had worked with the
Emerging Scholars program at Berkeley. Looking to the future, Dr. Concha Gomez believes that changes are essential in two key areas: recruitment, and building the WES community. Her report follows in the first person.
Recruitment
When I arrived at UW-Madison in 2004, I learned that the WES program was intended primarily for targeted minority students and students from rural Wisconsin. The original rationale for this recruitment strategy was: 1) the Treisman model required an ethnic balance in Emerging
Scholars sections; 2) women also are an underrepresented group in mathematics-based disciplines; and 3) women from rural school districts often faced transition difficulties (to a large, public university) that ethnic minority students faced. However, the program was always open to all students. In previous years, the former director Dr. Certain had sent letters to all ethnic minority students and all rural students with ACT math scores of at least 27. No special effort was made to identify students from underrepresented minority groups, or to identify those students who expressed an interest in a STEM major. Unfortunately this procedure had limitations. For instance, the rural list contained many addresses in wealthy communities near
Milwaukee. There were other problems with the system in place. My initial SOAR experience was disheartening
–
only a handful of students responded to my letter sent out earlier in the recruitment period, and only two of them came to the SOAR table. The existing SOAR recruitment strategies were evidently ineffective in recruiting targeted minority students or even rural Wisconsin students.
I started sending email messages to all candidates who were mailed an invitation, telling them that it was not too late to respond. I reminded them to come to the Math Consultant table at
SOAR, but still got only a few responses. Most students who enrolled in WES that semester were either referred by their SOAR advisors, were Students of Color who came to the Math
Consultant table looking for an open section of Math 221, or were engineering students recruited by the FIG program for a WES section of Math 221 that was part of a FIG that fall.
I realized then that our recruiting practices had to change. The next summer, I used the same criteria that Dr. Certain had, but manually removed the ostensibly rural students who I knew were not. I decided not to continue using the ineffective application form. I gave a presentation to some of the advisors who conducted SOAR training and posted WES materials on the SOAR
Advisors web site so the information could be retrieved when needed.
During the summer of 2005 I regularly sat at the Math Consultant Table in Union South, looking for potential WES students and talking to advisors about WES. I was not paid for my SOAR activity, but was able to hire a student hourly to sit at the table when I couldn’t be there. I also had access to the SOAR database, so I could look up each SOAR date and see who would be

7 coming that day and compare it to my list of invitees. Most were not coming to SOAR at all. I was surprised to find many good WES candidates by accident, only to learn that they were not on my list of people sent a letter of invitation. I later learned that the list of candidates was generated too early, and that many students — especially Students of Color — do not commit to attending UW-Madison until they’ve had a chance to weigh all of their financial aid offers. This meant that lists should be generated, and letters should go out, much later than was the practice.
Also, since new students were added to the database all summer, recruiting needed to be ongoing throughout the summer-long SOAR process.
That fall, I was still looking for a way to identify rural students. Because I went to high school in
Wisconsin, I knew there were rural high schools in the northern part of the state that had huge graduating classes because they served a large number of communities. How could I find them if the University didn’t identify them?
That year, the department hired a web applications specialist working on an application to identify candidates for our honors calculus sequence. He agreed to help me. He and I met with a consultant from the BRIO program and we devised a way to identify rural Wisconsin students using zip codes and the U.S. Census data available on the World Wide Web. He wrote a program specifically for WES that could get up-to-date information on entering students, identifying them as “rural” (using our definition) or “targeted minority.” It would display all test scores (SAT and math placement, if available, as well as ACT), high school G.P.A., number of high school math classes, intended major, and SOAR date. The data came from the Data Warehouse and was updated nightly, so I could search every few days and find new potential WES candidates, or I could look for a particular student to see if he or she had signed up for a SOAR session.
I then asked Wren Singer, the Director of New Student Programs, if I could put flyers in students’ SOAR advising folders during the summer of 2006. She said yes, but only if they were personally addressed to the student, and no earlier than the Friday before the student’s SOAR date. I therefore spent the summer of 2006 using the software program several times a week to find new candidates, mailing out new letters, and printing out flyers every Friday to be taken to
Union South by the department’s Undergraduate Assistant or a student hourly employee. I monitored the WES enrollment all summer, and checked with the SOAR Math Consultants frequently to see whether potential WES students were signing up for WES. I sat with them on a few occasions and saw students on my list trying to find a math class. When I asked if their advisors had given them a flyer about WES, they frequently said no. Clearly I still needed to be at SOAR, talking to students and reminding advisors about the WES program.
In the summer of 2007, after a long illness, I decided not to work. I sent out all the letters of invitation on the last day of my nine-month contract and I did not go to SOAR. The enrollment for WES is now so small that we are only offering two sections instead of the usual five.
I can only conclude that recruitment needs to be ongoing all summer and, because of the complicated nature of evaluating students’ academic records, the WES director must take the lead. I also believe that even stronger and dramatically more successful recruitment efforts can be made with a little bit of administrative support. Letters could be followed up with emails or phone calls, giving students advice on navigating SOAR and inviting them to talk to the director of the WES program personally while at SOAR. This is exactly what is done at other universities

8 with Emerging Scholars programs, but our department doesn’t have the funds to pay the director for this very substantial job during the summer months.
The Community of WES
One of my main roles as WES director is to advise all students in the program. In some years, that has been more than 80 students. I keep tabs on every student by keeping in touch with his or her TA. I intervene immediately if a student appears to have a problem – whether academic or non-academic in nature.
As director I am also responsible for creating the community of WES by connecting students with each other, their Student Assistants and Teaching Assistants, their professors, myself, and other resources on and off campus, in a way that allows their academic and social lives to overlap. During my first years as director I have pursued many avenues to build this community.
I organized co-curricular activities to bring students, their SAs, TAs, and professors together in academic and social settings. I invited guest speakers, organized panel discussions about summer research programs, offered presentations on study strategies, and arranged tours of laboratories on campus. I also continued a long-time WES tradition of providing pizza at review sessions.
I try to retain SAs (almost all former WES students) within the community by letting them know about conferences and summer internships. I post flyers in the WES classroom and email students directly about programs and events. For instance, I encouraged two Latino students to apply to the Society for the Advancement of Chicanos and Native Americans in Science
(SACNAS) and to the math department’s VIGRE program in order to attend the SACNAS
National Conference. One has received funding to attend the conference three years in a row, the other for two years. They are both engineering and applied math majors who are now applying to graduate programs. As another example, for each of the past two years, four different female students have attended the Nebraska Conference for Undergraduate Women in Mathematics.
Two of them met the director of the George Washington University Summer Program for
Women there and were subsequently selected to participate in that program.
In my first year as director, I started a WES drop-in tutoring program that was extremely successful. In Emerging Scholars programs at Berkeley, UT Austin, and elsewhere, the evening drop-in tutoring sessions staffed by Student Assistants are a fundamental community-building tool. Because we have a unique classroom space that is comfortable and perfect for group study,
WES students gathered there in the evenings to work on homework (something they are not allowed to do in class). The sessions were staffed by two or three undergraduate tutors who were former WES students and were often also Student Assistants during the day. The students gathered there because it was a centrally located, familiar place where they could concentrate on homework with their friends and classmates, and also get their questions answered by someone familiar and knowledgeable.
Another reason for creating the WES drop-in tutoring program was the department’s longstanding rule that WES students are not allowed to participate in the Mathematics Tutorial
Program. The reason for this rule is that the Mathematics Tutorial Program is aimed at students

9 who are struggling to get a passing grade, whereas WES is intended for students who are motivated to work hard for an A in their math course. The Mathematics Tutorial Program has limited space, and must serve the most academically at risk students. Furthermore, both programs require a significant time commitment, so it is not reasonable to expect a student to make that kind of commitment to two programs.
In 2005-06, I raised the student hourly pay scale to stay competitive with other programs on campus that hire math tutors. That year, during the second week of each semester, we had a
“Tutoring Kick-Off Party” in the WES classroom. We had pizzas and sodas and most of the SAs and TAs came to support their students and get them interested in coming to tutoring on a regular basis. At the end of each semester we had an eight-hour “Study Jam and Award Ceremony” on the day before final exams started. Again, we had refreshments. I asked each TA to sign up for a two-hour office hour “shift.” I invited the SAs and Tutors to sign up for a two-hour shift if they had the time; otherwise they could come by for pizza and say goodbye to their students. Awards were given out for things like “hidden mathematician” or “best worst answer” to individuals that
I asked the TAs to identify as good students who could use a little extra something to encourage them to keep going in math or science. The awards themselves were books about mathematicians or scientists, or gift cards from the University Bookstore.
The tutoring program ended in December 2006 due to lack of funding. By spring 2007 we no longer had pizza at review sessions and had to reduce the number of SAs in the program from two to one per class. Because of the cutback, we reduced the enrollment cap from 18 to 12.
The Future of WES
Once effective recruitment of students is achieved, I believe the future of WES is a bright one.
Students who join the program during their first year tend to stay in their intended major because they tend to do well in one of the toughest classes they will take that year, and because they are introduced to a community of people – undergraduates, graduate students, faculty and academic staff – who support their goals and want to see them succeed. This effect is magnified for
Students of Color, the students most likely to find themselves isolated their first year.
I hope the department, college, and university will find a way to provide additional funds for the
WES program, so that classrooms are fully staffed and our very successful tutoring program and co-curricular activities can be reinstated. I plan to do the necessary legwork to build, rebuild, or maintain good working relationships with other programs and advising centers around campus in order to better support our WES students. As a start, I have initiated efforts with the Chemistry
Department to coordinate the WES class schedule with their Science Scholars Program so that students may participate in both.
When the Mathematics Department hires a new Undergraduate Assistant, I hope to be able to delegate some of the administrative details of recruitment and budget reporting, as well as the catering details of some of the co-curricular events we would like to bring back. With adequate financial and administrative support, WES can once again flourish as a multicultural, rigorous academic program that creates and nurtures a community of scholars.

10
There are various accounting practices and assumptions that affect the calculation of cost per credit at the undergraduate level at UW-Madison. Since we want just an estimate of the relative cost per credit of a WES section versus a similar non-WES section in a calculus-related course, we simply have to maintain consistency in the set of assumptions that we adopt. However, we must make some such set of assumptions, since we want the comparative total cost, not just the comparative cost in categories that are different. Thus one such model shows that the UW
System analysis for the instructional cost (staff, supplies, and capital) per credit of mathematics courses for the Fall term 2006-07 was $88.09 for the freshman and sophomore level courses and
$178.40 for upper level courses. Starting with those figures but using actual lecturer costs and estimated TA and other costs, the cost per credit for non-WES sections of WES-related courses was approximately $104, and for WES sections was estimated at $145. Thus, from this perspective, WES-related sections are 39% more expensive.
There is a significant difference between the percentage of students who fail to earn a C or better in the WES sections versus the non-WES sections of WES-related courses. The percentage of such students in WES sections is 8% whereas in non-WES sections of WES-related courses it is
20%. If one treats this as an additional cost to non-WES sections (by reducing the number of credit hours but keeping the total tuition directed at those courses constant), then the adjusted cost for non-WES sections of WES-related courses is about $118. Using this figure, the WES sections are then only 23% more expensive.
There is also a difference between the WES and non-WES sections with respect to doing well – i.e., receiving a grade of B or better. The average rate for all WES sections of doing well was
68% compared to 48% for non-WES sections of WES-related courses. It is more difficult to find a way to plausibly represent this factor in a cost model, but clearly parents and taxpayers would consider it worth something.
With respect to persistence in higher level mathematics courses, WES students are more likely to take such courses than non-WES students. For two of the four courses analyzed (319 and 340), the relative odds are better than 3:2 and the results are highly significant (p<.005); for one of the four courses (541) the relative odds are about 3 to 1 and the results are significant; and for one course (521) the relative odds are about 2 to 1, but the result is not statistically significant (only 9
WES students).
A more detailed account of the cost model used for these estimates appears in Appendix C.
Submitted by the WES Oversight Committee
November 1, 2007
David Griffeath, Chair
Terry Millar, Julie Mitchell, Dietrich Uhlenbrock
Concha Gomez, WES Director

11
Student Hourlies
SAs
Tutors
Lincoln
Mentors
Total Student
Hourlies
July Aug
0
0
0
0
0
0
0
0
Sept
$2,200
0
0
$2,200
Oct
$2,400
0
0
$2,400
Nov
$2,200
0
0
$2,200
Dec
$1,400
0
0
$1,400
Jan
$800
0
0
$800
Feb
$1,700
0
0
$1,700
Mar
$2,000
0
0
$2,000
Apr
$1,500
0
0
$1,500
May
$1,100
0
0
$1,100
June
0
0
0
0
Refreshments
Review
Sessions
Guest
Speakers
Study Jam and
Awards
Total
Refreshments
Supplies
0
0
0
$0
0
0
0
$0
0
0
0
$0
0
$200
0
$200
0
0
0
$0
$400
0
$300
$700
0
0
0
$0
$0
$200
0
$200
0
0
0
$0
0
0
0
$0
$300
0
$300
0
0
0
Postage
Photocopying
Misc.
0
$20
0
0
$20
0
0
$50
$100
0
$50
0
0
$50
0
0 0
$50 $50
0 $100
0
$50
0
0
$50
0
0
$50
0
$600 $0
$160 0
$50 $20
0 0
Total Supplies $20 $20
Undergrad
Travel 0 0
Contingency
$150
0
$50
0
$50
0
$50 $150
0 0
$50
0
$50
0
$50
0
$210
0
$20
0
Totals $20 $20 $2,350 $2,650 $2,250 $2,150 $950 $1,950 $2,050 $1,550 $1,910 $20
Assumes approximately 80 to 85 students in the program in the fall and 65 to 70 students in the program in the spring.
Assumes there will be ten student hourly workers in the fall and eight in the spring.
Restricts each section to one review with food, probably the final exam review session.
Requires that the WES drop-in tutoring program be completely eliminated.
The following representation is based on 14 years of UW-Madison data and is intended to provide a perspective on the impact of the WES program on student performance.
It is important to understand the nature of a “WES section.” A WES section for a given course is typically one section among many for the lecture of the course in question. Students in a WES section take the same major exams as every other student in that lecture, and the preponderance of the grade of a student is determined by her/his performance on those exams. The WES students have more recitation hours, receive additional credits for these additional hours, and
Totals
$15,300
0
0
$15,300
$700
$400
$600
$1,700
$160
$510
$200
$870
0
$130
$18,000

12 typically work in groups during recitation on problems designed to support their understanding of the primary concepts in the large lecture. In contrast, in non-WES sections TAs usually simply solve problems related to the lecture, answer questions, and give quizzes.
In order to make the data presented more easily grasped, the language of “odds” will be used for some of the representations below. As an example, in a standard deck of playing cards the odds of picking a red card instead of a black card are 1 to 1, since the number of red cards (26) equals the number of black cards. The odds of picking a card with face value between 1 and 8 inclusive are 8 to 5. If your odds for winning a certain game are 6 to 1 and my odds are 3 to 1, then we will say that our relative odds (your odds to my odds) are 6 to 3 or 2 to 1. One reason for using the representation of odds (instead of probabilities, say) is that there is a “rule of thumb” that an odds ratio greater than 1.5 to 1 will be statistically significant with sample sizes over 200.
The introductory calculus-related courses discussed here are 171, 211, 213, 217, 221, 222, 223, and 234. The data from one of these courses for a given semester are included if and only if there was a WES section in some lecture of that course that semester. We will refer to these as WESrelated courses. Overall, there have been 52,540 such students (students counted as often as the number of courses they took), 1760 of whom were in a WES section at the time that data on them was included. Of the total of the 52,540 students, 2,767 were target minority students, and
401 of those were in a WES section at the time that data on them was included. We note that only 44,018 took the Math ACT, and only 1,617 of the WES students took the Math ACT.
The odds that a student who took a WES-related course but was not in a WES section received a grade of B or better are .9 to 1. The odds that a WES student who took a WES-related course received a grade of B or better are 2.1 to 1. The relative WES:non-WES odds of receiving a grade of B or better are 2.2 to 1 (where 2.2 is calculated from the ratio of 2.1 to .9).
The odds that a targeted minority student who took a WES-related course but was not in a WES section received a grade of B or better are 0.4 to 1. The odds that a WES targeted minority student who took a WES-related course received a grade of B or better are 1.2 to 1. The relative
WES:non-WES odds of receiving a grade of B or better are 3 to 1.
The odds that a student who took a WES-related course but was not in a WES section received a grade of C or higher are 4 to 1. The odds that a WES student who took a WES-related course received a grade of C or better are 11.6 to 1. The relative WES:non-WES odds of receiving a C or better are 2.9 to 1.
The odds that a targeted minority student who took a WES-related course but was not in a WES section received a grade of C or higher are 1.7 to 1. The odds that a WES targeted minority student who took a WES-related course received a grade of C or better are 6.9 to 1. The relative
WES:non-WES odds of receiving a C or better are 4.1 to 1.
Here is a summary of these odds:

13
Categories
All students in WES-related courses
Non-WES students
WES students
WES:Non-WES odds
WES-related targeted minority students
B or Better
0.9 to 1
2.1 to 1
2.2 to 1
C or Better
4.0 to 1
11.6 to 1
2.9 to 1
Non-WES students
WES students
WES:Non-WES odds
0.4 to 1
1.2 to 1
3.0 to 1
1.7 to 1
6.9 to 1
4.1 to 1
There are many ways one could look for differences between WES and non-WES students before they took a WES-related course. Rather than pursue the question of background in depth, in this appendix we will restrict our attention to Math ACT scores. It should be noted that the previous LEAD Center evaluations of WES did address student history more fully.
The mean Math ACT score of non-WES students in WES-related courses (among those who took the Math ACT) was 29.7. The mean for WES students was 28.3. The mean for non-WES targeted minority students in WES-related courses was 31.0. The mean for WES targeted minority students was 26.8.
Next we represent the variation in WES / non-WES student performance by ACT score. We will do this by presenting variations of the summary table above.
All students in WES-related courses
Math ACT score
B or Better C or Better
<28
WES
Non-WES
WES:Non-WES
1.4 to 1
0.6 to 1
2.5 to 1
9.4 to 1
2.8 to 1
3.4 to 1
28-31 WES
Non-WES
WES:Non-WES
>31 WES
Non-WES
WES:Non-WES
2.9 to 1
1.1 to 1
2.7 to 1
4.6 to 1
1.9 to 1
2.4 to 1
15.2 to 1
5.0 to 1
3.0 to 1
17.5 to 1
7.3 to 1
2.4 to 1

Here are plots of the distributions of Math ACT scores for all students in WES-related courses and for targeted minority students in WES-related courses, respectively:
Math ACT Score Distributions
14%
14
12%
10%
8%
6%
4%
2%
0%
15 19 23 27
ACT Math Score
31 35
WES
Non-WES
Target Minority Math ACT Score Distrutions
12%
10%
8%
WES
Non-WES
6%
4%
2%
0%
12 16 20 24
Math ACT Score
28 32 36

15
There is considerable variation by course and by semester. As an example, here are the “best”
(in terms of odds of receiving a B or better) four consecutive semesters for WES performance:
221:
C or better
WES
1002
1004
1012
1022
B or better
2.0 to 1
1.2 to 1
1.9 to 1
4.3 to 1
17.0 to 1
12.0 to 1
9.7 to 1
15.0 to 1
WES:Non-WES
1002
1004
1012
1022
2.3 to 1
2.3 to 1
5.0 to 1
2.5 to 1
5.9 to 1
5.5 to 1
2.8 to 1
4.1 to 1
And here are the “worst” four consecutive semesters:
WES B or better C or better
1042
1052
1062
1072
1.4 to 1
1.8 to 1
2.1 to 1
2.4 to 1
8.7 to 1
8.0 to 1
6.0 to 1
7.5 to 1
WES:Non-WES
1042
1052
1062
1072
1.5 to 1
2.1 to 1
2.2 to 1
2.7 to 1
1.9 to 1
2.0 to 1
1.3 to 1
1.8 to 1
Here is an example where the WES, non-WES difference is not as large. The course is 234
(7564 total; 233 WES):
WES
Non-Wes
B or better
1.4 to 1
C or better
6.8 to 1
WES:Non-WES
0.9 to 1
1.6 to 1
3.5 to 1
1.9 to 1
However, even in 234 we have the following for targeted minority students (323 total; 35 WES):
WES
Non-Wes
WES:Non-WES
B or better
0.8 to 1
C or better
6.0 to 1
0.4 to 1 1.5 to 1
1.9 to 1 4.0 to 1

20%
15%
10%
30%
25%
5%
0%
0
Here are some trends:
50%
40%
30%
20%
10%
0%
0
90%
80%
70%
60%
5
% B or better in introductory calculus related courses
10 15
Term
20
% drop or grade of less than C
25
5 10 25 15
Term
20
30
30
WES Sections non-WES sections
Difference
WES sections
Non-WES Sections
Difference
16

17
WES Percentage of Targeted Minorities in WESrelated Courses
50%
40%
30%
20%
10%
0%
0 5 10 15 20 25 30
Term
Another interesting question is whether or not there is a difference in likelihood of a WES versus non-WES (in a WES-related course) student taking a higher level course in mathematics. As one approach, we restrict to those students who took 221 as their first math course and did so before
Fall 2004 (so that they have had time to take an upper level mathematics course). There were
14,562 such students of which 449 were in WES sections. Here we will look at mathematics 319
(differential equations), 340 (linear algebra), 521 (abstract algebra), and 521 (real analysis). One must exercise caution, because of the small number of WES students and the small percentage overall of students who go on from 221 to take these courses. Therefore, in the table that follows, we will list the course, the number of WES and non-WES students who took the course from the defined population, the relative odds of a WES versus a non-WES student taking that course, and the p-value of that statistic.
Course # # Non-WES # WES WES::Non-WES odds p-value
319 988 54 1.8 <.005
340 1270 63 1.6 <.005
521
541
141
141
9
13
2.0
3.1
<.15
<.015
Thus it is almost twice as likely that a first time 221 WES student will take 319 as it is that a first time 221 non-WES student will take 319, and about 3 times more likely that the WES student will take 541. The results for 521 are not statistically significant, the results for 541 are significant, and the results for 319 and 340 are highly significant.

18
In this section we will attempt to estimate costs of the WES program. The primary differences between a WES section and a non-WES section are that a WES section has:
1) A greater number of credits;
2) More hours in sections;
3) Smaller average section size;
4) Undergraduate student assistants who help support the instruction.
In five-credit courses (such as 221), WES students receive two additional credits and meet an additional 3 hours, and students in three-credit courses (such as 234) receive one additional credit and meet one additional hour. Here are the average class sizes before Fall 2003 and after
Spring 2002. (Blanks indicate that those courses did not exit during that period or did not have
WES sections.)
Course
171
171
211
211
213
213
217
217
221
221
222
Average Section Size
Category
Non-WES
Before Fall 2003 After Spring 2002
21.4 20.4
WES
Non-WES
18.0
20.8
14.8
WES
Non-WES
17.3
17.8
WES
Non-WES
WES
Non-WES
WES
Non-WES
9.0
18.8
16.0
21.2
17.4
20.3
16.7
10.7
19.7
16.5
19.3
222
223
223
234
WES
Non-WES
WES
Non-WES
17.1
20.7
15.0
18.9
15.4
17.6
234 WES 16.5 14.9
There are one to two undergraduate Student Assistants per section, depending on the size of the section. The cost varies, depending on whether the student has work-study and how long the student has worked for WES. The rate starts at $10/hr and goes up by .50 for every two semesters of employment. The cost to the department is only half that if the student happens to have work-study. Two sections of a non-WES course are considered to be a 50% appointment for a TA, whereas one section of a WES course is considered to be a 50% appointment.

19
We now will estimate the cost per credit of the WES-related courses. This will be a crude estimate that depends on a number of assumptions that will be stated. The University of
Wisconsin System issues a Total State Instructional Budget report by term and by department.
In the Fall term 2006-07 report, the estimated cost per credit for Level 1 (freshman, sophomore) courses in the mathematics department was $88.09. This was calculated by dividing the total instructional budget (unclassified and classified staff salaries, supplies and capital) of $1,837,996 by the total number of Level 1 mathematics credits earned (20,865.5).
During that semester, the total number of credits in WES-related courses, all of which are Level
1, was approximately 12,000. We will use 50% (a slightly smaller percentage than the actual ratio of WES-related credits to total credits) of classified staff salaries, supplies and capital as
WES-related course costs. This amount divided by the total number of WES-related course credits is approximately $13. Since WES-related courses are for the most part large lectures, we do not use this method for calculating the lecturer cost. Instead we compute directly the proportional (1/3 of the 9 month salary, since the teaching load in the department is 3 courses per academic year) sum of salaries and fringe of WES-related course lecturer staff. This comes to approximately $600,000. Dividing this by the total number of WES-related course credit hours yields approximately $50.
The final part of the common cost per credit is the TA cost, although this is different for WES and non-WES sections. Since there were approximately 80 sections in the WES-related courses, we used an estimate of 40 TAs. For non-WES sections, since 2 sections constitute a 50% semester appointment, one must take the 50% appointment level academic year TA salary plus fringe (27.5%) plus tuition remission ($8,000), divide by 2 (since this is one semester), and then divide by 2 again (since 2 sections constitute a 50% appointment). Finally, one must divide by the average section size times the average number of credits per section. (To do this more precisely, one would do this by course, and then take the average, weighting by percentage of students taking each course. Since these courses are primary 5 credits, we will not do that here.)
Finally, there are different levels of TA appointment (experienced, senior, etc) – we used an approximation of $26,000 for a 50% academic year appointment. The final result is that the TA cost for non-WES sections of WES-related courses is approximately $41 per credit. Thus the total cost per credit for non-WES sections of WES-related courses was approximately $104.
To compute the TA cost for WES sections, we took the $41 and multiplied by 2.4 and then multiplied that result by 5/7. The 2.4 is an approximation to the ratio of twice the average section size of non-WES sections to the average section size of WES sections. The 5/7 adjusts for the 7 credits awarded to students in most WES sections (except 234 at present) compared to the 5 credits awarded to most students in non-WES sections of WES-related courses (except 234 at present). The result is $70 per credit for TA expenses in WES sections. Finally, WES sections have undergraduate student assistants. We estimate 1.5 per section for 6 hours per week times 14 weeks at $10/hour. (The hourly rate can be less for students hired on work-study.) This amount then has to be divided by the average number of students in a WES section and the result divided by 7 to account for the credits per student. The result is about $11 per credit in undergraduate assistant costs. The average total cost per credit of WES sections then comes to approximately
$145. Thus the cost of a WES credit is approximately 39% more expensive than a non-WES

20 credit in the same course. It is worth noting that the average cost per credit for Level II (junior, senior) mathematics courses in the cited report is $178.40.
Assume that the model added an additional cost to the non-WES sections based on the differential rate of students who do not receive a grade of at least a C (i.e., students who were not successful in the course). The percentage of such students in WES sections was 8% whereas the percentage of such students in non-WES sections of WES-related courses was 20%. One could take the position that this 12 percentage point difference is a cost of non-WES sections (certainly a perspective that a parent or a taxpayer might take.) If we reduce the number of students in
WES-related courses for Fall 2006-07 by 12% and redo the calculations, then the average cost of a non-WES section of a WES-related course would be $118 per credit and the additional percentage cost per credit of the WES sections would be 23% instead of 39%.
WES sections are normally capped at 18 students; until the past two semesters we have been enrolling 15 to 18 students in most of the sections. The WES director and TAs feel that these sections should have two Student Assistants in the classroom to keep students on task and to make sure students are not getting stuck for too long on a problem.
We asked some recent TAs with two SAs to give their impressions of the difference if sections were instead limited to one. Here are some excerpts from their responses, with enrollment numbers in parentheses.
Math 171 (16 students):
“We would be able to function with only one SA, but it wouldn't be as nice. Having two
SAs improves both the quality and quantity of attention that each group receives. If we had only one SA, we would not be able to spend as much time with each group. I would have to move around more to make sure that groups aren't getting stuck or getting offtrack. By having two SAs, we are able to give more challenging problems and can work with groups for more extended time intervals without having to worry that there are groups that are not making progress because they aren't getting the attention they need.
When students are working on problems, all three of us are busy interacting with the groups.”
Math 221 (17 students):
“In my mind, one of the main things the presence of 2 SAs accomplishes is getting the students to focus on thinking about (difficult) math for 2 hours straight (3 times per week). Having 2 SAs walking around the room providing clarification (that would otherwise stop the students from working) or asking the students ‘which problem are you working on’ does a lot to keep them focused. This is what I perceive would be the biggest loss if we each had only one SA. In my case, the SAs are also better able to know how well the groups are working and if there are students that look like they're ‘in trouble.’”

21
Math 221 (19 students):
“… In the current incarnation of the discussion, groups have serious questions about once every ten minutes; this gives me and one of the SAs time to sit at two of the tables and tease out their understanding (as well as keep them focused) while the [second] SA can act as a floater, wandering amongst the remaining four tables and ensuring that they stay on-task and giving them minor hints to keep them motivated.”
Were one of the SAs removed, one of two things would happen: either I would be forced to artificially delimit my time at each table which … would result in all the students halflearning the work and hence doing poorly on the assessed material; or I would be forced to leave multiple tables essentially untouched by authority figures through the two hours, resulting in frustration, lack of focus, and general incomprehension of the material amongst most of the students, saving only those to whom I talked, which would result in a similar problem.”
Math 222 (15 students):
“3 ‘supervisors’ seems to be sufficient to keep students working. When I had only one
SA, students tended to feel they did not need to stay on task as much. One SA is too few to keep good coverage and keep the energy up.”