A Symbol of hope vs the Hawthorne Effect
Via Joanne Jacobs: Philadelphia Inquirer 10/30/2006 Symbol of hope:
An elementary school in the heart of North Philadelphia has grabbed the national spotlight for dramatically improving its test scores.I double checked their results and overall they are amazing, but I did notice one thing.
The initial jumps were so large the Philadelphia school district verified them by having some students retake the tests.
Over four years, the number of fifth graders scoring at the highest level - advanced - soared from 1.2 percent in math and reading to 42.1 percent in math and 29.8 percent in reading.
In reading their results improved dramatically from a 12.2% proficiency rate in 2002 to a 72.5% in 2005, but all of the sudden in 2006 they dropped back down to a 47.5%. The same pattern is reflected in math scores.
I have noted this trend in several benchmark schools that I have looked up over my brief blogging career. Could it be that the initial rises in performance are a result of the "Hawthorne effect", and that once a system becomes established there is a leveling out of performance? Unfortunately the article doesn't mention the drop off at all. Perhaps its a statistical blip, maybe there were some changes not mentioned in the article, but its something that I might keep an eye out for in the future.
Update: I found a similar but not quite as dramatic drop off in scores at Amistad Academy of Achievement First.
2006 scores weren't available, but there was a drop off in 2005 scores from 2004 after several years of improvements.
Are there any studies that look at whether this is a common phenomenon?
3 comments:
The "Hawthorne Effect" is disproven, but one thing I notice is that the scores are from 2002, 2005, and 2006. To see what's going on, you really need to see the scores from 2003-2004; there may very well be fluctuation and that's what we're seeing in the 2006 scores.
I'm just going on what you've reported. I haven't read it.
My totally off the wall guess is that the more motivated parents get their kids into these schools when they first open and their kids are more likely to have higher test scores anyway. The initial effects of quality education are then compounded by the self selecting population.
Eventually as the schools become established, they become more accessible to the general population. As more and more average (relative to the targeted population) attend, there will be a drop off of scores.
Thats not to say that the systems don't work and aren't superior to normal public schools. Even with the drop off's there is a substantial improvement from the base scores.
Disclaimer: This is mere conjecture.
Perhaps. What I'm saying, though, is that there may be significant fluctuation in the "missing" years. You'd expect it, at least from a statistical standpoint. Having said that, the difference between 2005 and 2006 does seem suspiciously large, though somebody should run tests on the data to see if it really is a statistically significant difference.
Post a Comment