Frank L. H. Wolfs
Nuclear Science Research Laboratory and the Department of Physics and Astronomy
University of Rochester, NY 14627
I. Introduction
The University of Rochester has made significant changes in its undergraduate curriculum, aimed at improving the quality of undergraduate education both for majors and non-majors. Examples include the creation of clusters, aimed at providing a coherent educational experience for students in the two divisions outside their major, and the introduction of Quest courses, which are small, problem-centered courses that employ a research-based pedagogy and introduce students to the distinctive nature of college learning, designed principally for first-year and some second-year students. In parallel to these changes in the curriculum, the College has provided funds to improve existing courses with the use of technology. This essay describes the results of one of these technology-related projects: the introduction of a Computer-Assisted Personalized Assignment system (CAPA) in introductory physics courses at the Department of Physics and Astronomy.
The introductory physics courses offered by the Department of Physics and Astronomy are in general large lecture-based courses, which are mainly taken by science majors, engineering students, and pre-meds. In parallel to the large lecture-based courses, the Department offers smaller honors sections of these introductory courses, which are mainly taken by Physics majors. In the traditional lecture courses, homework assignments are handed out on average once a week. The students have about a week to return the written solutions to their assignments, which are subsequently graded by the teaching assistants associated with the course. On average, the corrected assignment is returned to the students a week after it is handed in, and thus two weeks after the material is discussed in lecture. The long delay between assignment and return does not encourage the students to examine their mistakes and learn from them. In addition, since each teaching assistant works with between 30 and 45 students, most of their 15-hour a week workload is spend on grading homework assignments, rather than helping students.
In order to provide students with a more timely feedback on their assignments, a Computer-Assisted Personalized Assignment system (CAPA) was introduced in Physics 121 in the fall of 1994, funded with my start-up funds. This system, developed at Michigan State University, provides students with immediate feedback before the assignment is due, and as a consequence, it encourages the students to seek help if they encounter problems with their assignments. In addition, CAPA relieves the teaching assistants from the tedious job of grading the individual assignments of their students, and allows them to spend a significant fraction of their time helping the students with the problems they encounter.
In this essay I will discuss the implementation of CAPA at the University of Rochester. In Section II, the basic properties of the system are discussed. The implementation of CAPA at the University of Rochester is discussed in Section III. Section IV summarizes the feedback received from students. In Section V the educational implications of the use of computerized assignments systems are presented. A brief summary and conclusions are presented in Section VI.
II. The Computer-Assisted Personalized Assignment System (CAPA)
The CAPA system was developed in 1992 at Michigan State University by Ed Kashy et. al. (Am. J. Phys. 61 (12), 1993). The system produces personalized assignments for the students. Although the problems for different students are similar, they differ in the details (value of the variables, the graphs, etc.). The similarity of the problems foster a collaboration between students when they work on their assignments, but prevents copying of answers. Figure 1 shows an example of two different versions of the same problem that I developed for my Physics 121 course (freshmen mechanics). This problem asks the students to calculate the total displacement of a runner whose velocity versus time graph is shown in the Figure. Twenty different versions of the velocity versus time graph are being used to generate the assignment, and the time at which the total displacement is supposed to be calculated also varies from student to student. Although the majority of the problems used in CAPA assignments have a numerical answer, other types of problems are also being used. An example is shown in Figure 2, which shows a problem developed by Prof. Thoennessen at Michigan State University, used in Electricity and Magnetism courses. In this problem, the students are asked to identify the correct statements. There are subtle changes in the circuits and statements shown for different students. CAPA has also been used in non-physics courses. An example of a problem used in a biology course is shown in Figure 3.
Figure 1. Example of two different versions of the same problem used in Physics 121 (freshmen mechanics). The graphs in these two versions are different, as is the time at which the students are asked to calculate the total displacement of the runner.
Figure 2. Example of two different versions of the same problem used in Physics 122 (electricity and magnetism for sophomores). The circuits in these two versions are slightly different (note the direction of i3), as are the statements that the students are asked to consider.
The students can connect via the WEB or TELNET to the computer on which CAPA is installed, enter their answers, and are provided with immediate feedback on the correctness of their solutions. If the problem is answered incorrectly, a hint may appear to guide the students to the correct approach. Corrections to incorrect answers can be made before the due date/time, and students are not penalized for incorrect answers. However, the instructor can impose a limit on the number of tries allowed for each problem. The instructor also controls the maximum allowable deviation between the answers entered by the students and the correct answer, calculated by CAPA, in order to allow for rounding errors.
CAPA allows the instructor to monitor student progress on the assignments in great detail. Information on for example the average number of tries for the various problems, provides the instructor with feedback on the difficulty of the assignment before the due date/time. As a consequence, the instructor can discuss topics associated with the assignment in lecture in order to address some of the problems encountered by the students. A detailed summary of the performance of the students on the assignments can also be used by the teaching assistants to determine those problems on which to focus during the recitations. The instructor also has access to detailed log files showing when students connect to CAPA, the answers they enter, etc.
Figure 3. Example of a CAPA problem used in a Biology course. This problem was developed by Profs. Nadler and Kashy from Michigan State University.
III. Implementation of CAPA at the University of Rochester
III.1. History
During the fall semester of the 1994-1995 academic year, CAPA was introduced in Physics 121 (freshmen mechanics). The system was not only used for the various homework assignments during that semester, but was also used to generate individualized midterm exams and final exams. There were 43 students in the fall 1994 class and their response to CAPA was very positive. Their test performance, as compared to students who had taken the course in previous years, was also higher. The use of CAPA was continued in Physics 121 during the spring semester of the 1994-1995 academic year (with a total enrollment of 207 students). The use of CAPA was voluntarily, and students were allowed to hand in written solutions to the assignments before the due date/time. Although in the first few weeks a few students would use this option, mainly due to access problems, on average less than 1% of all students handed in written solutions.
After the successful implementation of CAPA in Physics 121, its use was expanded, and in the 1995-1996 academic year CAPA was used in Physics 113 (206 students), Physics 114 (212 students), Physics 121/121Q (253 students), and in Math 161Q (25 students). Statistics concerning the use of CAPA in later years is provided in Table 1.
The introduction of CAPA in Math 161Q (Quest) was a result of a joined proposal between the Department of Mathematics and the Department of Physics and Astronomy. A special section of Math 161 was created in which only students were enrolled who had registered simultaneously for Physics 121. The close coordination between these two courses turned out to be very beneficial, since examples from Physics 121 could be used to illustrate important mathematical concepts in Math 161, and the instructor of Physics 121 was aware of the topics in Mathematics that the students could be expected to be familiar with. CAPA was used in both Quest sections, but the limitations of CAPA for problems in Mathematics became quickly apparent (later versions of CAPA have removed some, but not all of these limitations). In order to address these limitations, the Department of Mathematics developed its own computer-assisted personalized assignment system called WEBWORKS, which is discussed elsewhere in this Volume by Prof. Gage.
| Year | Fall Semester | Enrollment | Spring Semester | Enrollment |
| 94 - 95 | PHY 121 | 43 | PHY 121 | 207 |
| 95 - 96 |
PHY 113 PHY 121(Quest) Math 161(Quest) |
206 76 25 |
PHY 114 PHY 121 |
212 177 |
| 96 - 97 |
PHY 113 PHY 121(Quest) AST 102 |
169 96 108 |
PHY 114 PHY 121 |
203 150 |
| 97 - 98 |
PHY 113 PHY 121 PHY 122 |
171 55 107 |
PHY 121 | 148 |
| 98 - 99 |
PHY 141 PHY 122 AST 102 |
32 117 77 |
III.2. CAPA Exams
The midterm exams used in Physics 121 and Physics 122 were generated using CAPA. As a result, the actual exams are different for different students. All numerical problems were multiple-choice, with 8 numerical answers listed. An example of two problems used on a Physics 122 (Electricity and Magnetism) Midterm Exam in 1997 are shown in Figure 4.
Figure 4. Two questions from a Physics 122 Midterm Exam (Electricity and Magnetism), illustrating the format of the CAPA exams. Both problems were developed by Prof. Courtney (Simon Fraser University).
After the midterm exams, the students could use CAPA to check their answers. During a period of 40 hours following the exam, the students were allowed to make corrections to those problems they answered incorrectly on the exam. Correct corrections made in this manner received 1/3 partial credit. The possibility to make corrections and receive partial credit motivated the students to learn from their mistakes on the exam. Students were given the opportunity to hand in written solutions to all exam problems, to be hand-graded for partial credit instead of obtaining partial credit using CAPA, but very few students used this possibility. One student stated that "in general I like to forget about the material after the exam, but CAPA forces me to examine my work" which illustrates the focus on the material that awarding partial credit after the exam provides.
One of the major draw backs of this type of exam is that one does not see the work of the students and that the students are not forced to write down their solutions in a format that can be understood by another human being. In order to address this issue, two of the exam questions did not have a multiple-choice answer, and the students were required to hand in written solutions to these problems, which were graded by the teaching assistants, and for which partial credit was awarded, when deserved. For these questions, the students still were able to check their answers using CAPA and make corrections for partial credit after the exam.
CAPA was also used to generate final exams, which have a format similar to that of the midterm exams. The main differences are that the written solutions are handed in and graded for partial credit. The students again could use CAPA to check their answers on the final exam, but no corrections after the exam were allowed.
III.3. Student Access
When CAPA was introduced in 1994, access to the system was only possible via TELNET. Although this mode of access ensured that all students were able to work on their assignments from any publicly accessible machine on River campus, students were required to obtain a hard copy of their assignment in order to see the graphs and schematics associated with their problems. The distribution of these hard copies turned out to be a major problem (students picking up the wrong assignment, students not being able to work on their assignment since they did not come to class to pick up their assignments, etc.). In 1995, a GOPHER system was developed to distribute the CAPA assignments. Students now have access to their assignments 24-hours a day, 7-days a week, and can print them out on any postscript printer. In 1997, WEB access was introduced, giving students the opportunity to access CAPA in different ways. The popularity of WEB access increased dramatically between 1997 and 1998. This is illustrated in Figure 5 which shows the fraction of the total number of connections on a given day that are made using the WEB as a function of the time before the due date/time for Assignment 1 of Physics 122 in the fall of 1997 and for Assignment 1 of Physics 122 in the fall of 1998. Time t = 0 corresponds to the day on which the assignment is due. In the fall of 1997, most students accessed their assignments via TELNET (on average 75% of the total number of connections were TELNET connections). In the fall of 1998, WEB access had gained significantly in popularity, and only 15% of the students used TELNET to access their assignments.
Figure 5. The fraction of the total number of connections on a given day that are made using the WEB, as a function of the time before the due date/time for Assignment 1 of Physics 122 in the fall of 1997 and in the fall of 1998.
Figure 6. The number of CAPA connections per day (shown as a fraction of the total number of CAPA connections) as a function of the time before the due date/time. The solid and dashed lines show the data for assignment 1 and assignment 9, respectively, of Physics 122 (fall 1998).
Figure 7. The number of CAPA connections per hour during the 48 hours prior to the due data/time of assignment 2 for Physics 122, fall 1998. The assignment was due at 9 am on September 19, 1998. Note that 12 am in this Figure corresponds to t = -9 hr and t = -33 hr.
Although in general students have up to seven days to complete their assignments, most students work on their assignments during the last two days before the due date. Figure 6 shows the number of CAPA connections as function of the number of days before the due date, for assignments 1 and 9 of Physics 122. Little change is observed in the solution profile between the first and the last homework assignment in this course. Between 40% and 50% of all connections are made the day before the assignment is due. Similar observations can be made in other courses where CAPA has been used.
Figure 7 shows in detail the work pattern of Physics 122 students working on Assignment 2 during the fall of 1998. Time t = 0 corresponds to the due date/time of this assignment. It is observed that most of the connections are made at night, while very few connections are made between "working hours" (between 9 am and 5 pm).
III.4. The CAPA Library
CAPA is currently being used at the 46 institutions listed in Table 2, predominately in courses offered by Physics and Chemistry Departments. Although CAPA has been successfully implemented at most of these institutions, the major drawback of any computerized assignment system is the learning curve for faculty members interested in using it. As a consequence, it is often difficult to implement these systems as a standard component of the course, since its use should be instructor independent. In order to make it easier for faculty to get started using CAPA, we initiated the development of the so-called CAPA library of Physics problems. This library contains several thousand problems used in introductory Physics courses, and is based on contributions from many faculty members at the many different institutions that are currently using CAPA. This library has been maintained by the University of Rochester, with support from the College's Deans Office and the Department of Physics and Astronomy, and is distributed to the various institutions that have contributed to the library.
| Institution | Departments |
| Allegheny College | Physics |
| Australia National University | Physics |
| Brandon University | Chemistry |
| Central Michigan University | Physics |
| Colgate University | Physics |
| Colorado School of Mines | Physics |
| Dalhousie University | Physics |
| Florida State University | Physics |
| Florida International University | Physics |
| George Washington University | Physics |
| Hope College | Chemistry, Physics |
| Idaho State | Physics |
| Iowa State | Physics |
| McGill University | Physics |
| McMaster University | Physics |
| Milwaukee School of Engineering | Physics |
| Morris College | Div. of Nat. Sci. |
| Northern Arizona University | Physics |
| Ohio University | Physics, Chemistry |
| Oregon State University | Chemistry |
| Pennsylvania State University | Chemistry |
| Rensselaer Polytechnic Institute | Physics |
| San Diego State University | Physics |
| Simon Fraser University | Physics, Chemistry |
| Suffolk Community College | Physics |
| SUNY - Binghamton | Chemistry |
| SUNY - Farmingdale | Math |
| SUNY - Geneseo | Physics |
| SUNY - Stony Brook | Physics, Chemistry |
| Texas A & M | Chemistry |
| University of Alabama (Birmingham) | Physics |
| UCLA (Pasadena City College) | Physical Sciences Div. |
| University of Colorado at Boulder | Physics |
| University of Florida | Physics |
| University of Helsinki | Physics |
| University of Houston, Downtown | Dept. of Nat. Sci. |
| University of Kansas | Physics |
| University of Kentucky | Chemistry |
| University of Notre Dame | Physics |
| University of Oregon | Physics |
| University of Rochester | Physics, Astronomy |
| University of South Carolina | Physics, Chemistry, Biology |
| University of Texas at El Paso | Physics |
| University of Washington | Physics |
| Washington State | Physics |
| Michigan State University | Physics, Chemistry, etc. |
IV. Student Feedback
The students who have used CAPA have been enthusiastic about it. During each semester surveys were carried out in order to collect feedback from the students, which was used to improve the system. The following list shows a few examples of comments made by students enrolled in courses where CAPA was used to generate the assignments:
Although in general most students found that CAPA improved their understanding of Physics since it forced them to learn from their mistakes, valid concerns were raised by some, as is illustrated in the following list which shows some of the negative comments that were made by the students:
Some of the conclusions based on the feedback provided by the students are discussed in Section V.
V. Educational Implications
In order to determine the educational impact of the use of CAPA in introductory Physics courses, we have carried out surveys of the students enrolled in these courses. Most students agree that providing immediate feedback enhances the value of the homework assignments since it forces them to ask for help when they encounter problems on the assignments, and since it promotes group learning. However, some students expressed concern about the fact that you can get the answer correct without really understanding the problem, and some admitted that they got problems correct by simply guessing. It is clear that the educational impact of any computerized assignment system depends on how it is being used by the students. CAPA provides the students with the possibility to increase the effectiveness of homework assignments by offering immediate feedback; if students do not use the system in the way it was intended to be used, they miss the opportunity to enhance their learning.
The increase in the number of office hours of the teaching assistants also has had a significant impact on the educational experience for the students enrolled in these courses. The teaching assistants were available in the so-called CAPA help rooms during the last 3 afternoons before the due date, and on average 5 to 10 students were present in the help room at any given time. Since not everyone could consult with the teaching assistants at the same time, the students started to work in teams on their assignments, with all the benefits associated with peer learning. The questions with which the teaching assistants were confronted were also different from the ones encountered in the traditional approach. Instead of getting questions like "I do not know where to start", the students would ask questions like "I tried this and that approach, and none of them work; what do I do wrong?" In almost all cases, the students had tried at least once to solve the problem before seeking help, providing the teaching assistants with the opportunity to comment on the approach used by the students, and identify the incorrect assumptions being made.
One indicator that is often used to evaluate the impact of new teaching approaches are the grades obtained. Although initially I paid attention to the grades of the students enrolled in the courses where CAPA was introduced, it was clear that besides CAPA many other aspects of the original course changed, which could have also influenced the grades received. For example, in my Physics 121 and Physics 122 courses, I provided the students with detailed information on their current standing in the course, including information on what exam scores were required to pass the course. As a result, students who were doing poorly were warned well in advance, and often dropped the course instead of receiving a failing grade. This raised the average grade level in the course, but it was not clear how much of this change in grade level was due to CAPA and how much could be attributed to students dropping out rather than failing. Obviously, a lot of information is available to study the performance of students in CAPA-based courses, but real control experiments (in which the performance of two groups of students, one using CAPA and one not using CAPA, were compared) were not carried out. Although I am convinced that CAPA has enhanced the learning process for most students enrolled in CAPA-based courses, this conviction is based on my own direct observations rather than on a statistically correct analyses of student performance.
The use of CAPA originally required TELNET access, and as a consequence most students would use their regular University computer account to connect to the CAPA computer. Since the students were connecting to the main computer at the University to work on CAPA assignments, the use of email became an efficient way of communicating with the students. As a result, all course-related information was distributed via email to the enrolled students. Information on performance on the assignments, results of midterm exams, and grade information were also distributed via email. In addition, email was used efficiently to answer course-related questions from the students, and allowed for a more one-to-one interaction between students and instructor. Although the use of email significantly improves the communication between instructor and students, especially in large lecture courses, the instructor should be prepared for a flood of email; during on of my Physics 122 courses, I received 1500 email messages in one semester
One of the main concerns related to the used of CAPA in introductory Physics courses is that too often the focus appears to be on the answers instead of on the method. Care should be taken to teach the students how to write up solutions to their problems, and this skill should be tested. In my Physics 121 and Physics 122 courses, a weekly quiz was given, which focused on a single problem and for which the answer had to be written up in terms of the variables provided. In addition, some of the problems on the exams had to be written up in terms of variables and handed in. There are of course introductory courses for which the use of CAPA is not appropriate. For example, in the honors sections of the introductory physics courses, the focus is on the theory behind the material, and there is a much bigger focus on derivations. Many of the derivations that are frequently encountered on the honors problem sets could not be easily converted into a CAPA problems. As a result, it would not be appropriate to use CAPA in these courses.
VI. Summary and Conclusions
A Computer-Assisted Personalized Assignment system (CAPA) has been introduced in various introductory physics courses at the Department of Physics and Astronomy. The immediate feedback provided by this system has enhanced the educational value of the homework assignments used in these introductory courses.
It is important to clearly determine the educational value of new teaching approaches, but although faculty introducing these new approaches are experts in their field, very few have the knowledge and skills to properly assess the impact of these new approaches. Most of us rely on circumstantial evidence to evaluate the impact of the various changes in our teaching, while instead we should seek help to evaluate the impact in a systematic manner. There would be a significant benefit if resources would be made available to help faculty to properly assess the impact of new strategies on their teaching.
VII. Acknowledgments
The introduction of CAPA at the University of Rochester was made possible by the generous support from the College Dean's Office. The development of the CAPA library was made possible with funds provided by the College Dean's Office and the Department of Physics and Astronomy.