C. Did the instruction improve the test scores?June 12, 2007by Cal LanierWould the students have done as well if they hadn't attended the class? Researchers consistently run studies that find little or no improvement. For example, Derek Brigg's study finds that: This analysis suggests unequivocally that the average effect of coaching is nowhere near the levels previously suggested by commercial test preparation companies. Private tutoring has a similarly small effect for students taking the math section of the SAT and no effect for students taking the math section of the ACT. I consider any student who a) begins below the 90th percentile and b) increases his scores by 2 points or less per section (40 points or less in SAT sections) to have failed to improve. Long before that student leaves my course for the real test, I will have talked with him and worked to identify and fix the problem (which is never a matter of more effort). If Briggs' research were accurate, even on average, I'd have long since developed a narrower definition of failure to improve. I'm not going to try here to account for the difference between research and my reality. I understand that the difference between the first and actual test scores alone aren't sufficient to demonstrate that the course helped. Ideally, a study would select two similar control groups of primarily Hispanic non-native English speakers with low family incomes and parents who didn't graduate from college. One group would take the diagnostic and the ACT, with no instruction. One group would just take the ACT. The three groups would be compared to see how much of the class students' improvement could be attributed to the test preparation. I don't have any control groups. However, thanks to the ACT national score reports, I can point out an interesting anomaly. The ACT reports section means for academic preparation, and until this year, for racial categories as well. (Only composite scores are broken down by race in 2006.) As 18 of the 21 students were Hispanic and three African American, I used the Hispanic national mean scores as the point of comparison for the entire group. Here are the class's mean ACT scores, followed by the national mean for all Hispanic students and then for all students.
Despite the fact that 18 of the 21 students speak English as a second language, their mean English score was two whole points above the mean for all Hispanics, and a bit more than a point lower than national mean for all students (approximately 20 SAT points). Their math average was even higher, less than half a point away from the overall mean. While the reading and science reasoning scores were considerably lower, they were still roughly equivalent to the mean for all Hispanic students. Taken altogether, their composite mean was nearly a point higher than the overall Hispanic mean in 2005. The ACT also reports means by student academic load, with a dividing line of "core", or college prep, and "non-core". My students provided me with all their classes, but as math courses are the most consistent from school to school, I'm using math level as a "core" indicator. Core math means completion of math beyond algebra 2; "less than core" means student has taken only algebra I, geometry, and part of algebra 2.
The group with less than core preparation has averages scores in math equal to the means for all less than core students. In fact, their math average is equal to the national Hispanic core group. Their English score is well above the Hispanic mean for their category, although not quite at the national mean for all similarly prepared students and just short of the mean for Hispanic core students. Their reading and science scores are lower in comparison Their reading and composite scores have moved up to the Hispanic mean for their category, although the science reasoning scores still lag. The core plus group means are above or roughly equal to the national means for all core plus students except in reading. In other words, a group of low income, predominantly Hispanic students (with the rest African American), most of whom have average or less academic preparation, had mean ACT scores well above the mean for their overall demographic (regardless of income) in math and English and in some cases on par or above the national mean for all students. If that happened often, the ACT wouldn't break out scores by race and income. Therefore, the performance itself, compared against the means for various controlled groups, is unusual enough to require explanation. The high performance alone isn't enough to establish that the instruction made the difference--student academic profile and prior abilities need to be sorted out--but it certainly raises the question.
2008 ACT Results | 2007 ACT Results |