D. Are the Practice Tests Accurate?June 13, 2007by Cal LanierThe test prep company I work with spends an enormous amount of money every year ascertaining that their practice tests are reasonable proxies for the real thing. However, experts often question whether practice tests can be used to determine improvement in comparison to the actual test. Since the practice tests weren't the "real thing", the scores may not represent the students' best effort. Even if the students are trying, the test prep company may create practice tests much harder than the actual test in order to create a perception of an increase. After four years of teaching test prep, I am personally convinced of the reliability of my employer's practice tests (as well as the tests they provide in the retail market). I've used a wide range of test prep material in my private tutoring, and consider theirs the best and most accurate. I am also certain that the majority of my students put in their best effort on the practice tests, and can tell on the occasion that they don't. For example, on Practice Test 4, it was immediately obvious that Art and Charlie did all but sleep through the last test, and they endured the Wrath of God when I returned their scores (they shamefacedly apologized, too). While I understand and agree with the skepticism about practice tests, I believe that the progression of practice test scores strongly support the argument that the higher scores represent genuine improvement. 15 students took all four tests, so I'll use their scores alone to track test progression.
Moving from student effort to practice test predictive value, the students' first diagnostic test is the closest thing available to their actual ability before the course. The "best score" is the student received on any practice test. In most cases, the student scored his best score on the third or fourth practice test, but progress is not always linear. Assuming that the ACT results represent the students' best efforts, I compared their actual results to the students' first test scores and their highest score in class to see if either reliably predicted student performance on the real test. I'm using all student scores for this comparison (using 15 students doesn't change the results significantly).
The diagnostics were below their actual test scores by 2-6 points. The best practice test scores, on average, predicted actual scores within a point. Considering the student progression from test 1 to test 4, coupled with the strong correlation between their best practice score and their ACT scores, I believe this makes a reasonable case that the practice tests are reliable proxies for the real test, and that a great deal of the improvement recorded is genuine. Skeptics might suggest that the test prep company is making the test easier each time, giving the students the impression that they are improving, when in fact the students have made no progress during the course. Using this argument, the students didn't improve because of the course and the ACT test scores represent their underlying abilities before taking the course. In that case, 25 low income Hispanic and African American students tested much higher than would otherwise be expected. For more on this, see Did the instruction improve the test scores? Throughout this website, I consider all of the practice tests as reasonable proxies for the real thing. I will occasionally point out a student's best practice performance as an indicator of actual ability, whether or not they hit that mark on the real test.
2008 ACT Results | 2007 ACT Results |