Those Who Can

C2. Comparing Rates of Improvement

June 15, 2007

by Cal Lanier

This comparison of diagnostic to actual ACT scores, with FTIs broken out, illustrates the considerable difference in the two groups. I understand that these are small groups and that big differences in one or two students can distort the mean. However, these are fairly large differences that are well beyond what one or two scores could distort.

The group that showed no improvement actually started out with higher means in English, Reading, and overall composite averages on their diagnostic tests. Two months later, the improved group had 2-4 points on the FTIs in every section except science.

Moreover, while the class as a whole scored just under the national means in English and math, the improved group mean was above the national mean in both subjects, and over 20 (roughly the equivalent of a 500) in reading and composite.

The means don't tell the whole story. Pamela's English score improved nearly 50%, while Frank and Gerry saw tremendous improvement on their science tests. Likewise, despite a generally outstanding test, Larry's science score dropped 3 points from his diagnostic (in the email giving me his results, he wrote "oops!" next to the 22).

The graphs below are busy, but are as close as I can get to capturing my mental image of a class's performance. I always identify students as improvers or FTIs, and have a general sense of how each group is doing in each section. I also have the high and low improvement percentage (and scores) to ensure I'm considering the overall range of improvement.

Coincidentally, these scatterplots are a common SAT question: which student showed the greatest improvement? The answer, as taught in any SAT course, is the student whose line from the point of origin has the steepest slope. Likewise, the student with the flattest slope has the least improvement. I've labelled the high and low improvement percentages.

The trend lines show the different general slopes of improvement. In all but science, the improvers have a steeper slope. The FTI's reading trendline is nearly flat. This masks individual improvement--Gerry increased from 11 to 17--but is overall more representative than not.

The vertical lines highlight the differences in individual improvement. All of the students along a vertical line began with same diagnostic score and ended with very different ACT results.

So, for example, one of the vertical lines on the Composite chart captures four students who received a 13 on the diagnostic. The two pink dots represent two students who didn't see significant improvement; their ACTs were 14 and 15 respectively. The two blue dot students received a 17 and a 20. (Recall that each point on the composite represents between 2 and 4 points improvement per section.)

The vertical lines show the wide variance of results that the students saw despite starting from the same basic knowledge set.

Again, improvement and effort are not directly correlated. Effort helps, but there's some other factor involved.