Skeptical About Education Research
I’ve never hidden my skepticism about educational research, at least with regards to improving the educational system or, with a bit less ferocity, the practices of individual educators in their particular teaching and learning contexts. It’s always seemed to me that educational theory in general, and educational research in particular, has much the same relationship with actual teaching and learning as literary research and writing theory have with literary creation: the researcher wants to establish a symbiotic relationship with the art and craft, but it remains a generally parasitical enterprise.
The proof is in the pudding, and despite many decades of research we’ve ended up with a pretty thin gruel. I was struck reading this recent article about the problems endemic to medical research and have to wonder: do we really think educational research is any more reliable? Practitioners and researchers alike face many of the same pressures from institutions and demands from peer-reviewed journals, and as with medical researchers, many– if not most– of the researchers are removed from the day-to-day work they study. Sure, the dollar amounts are smaller, but if I’ve learned anything about academia while serving my life sentence there, it’s the accuracy of the truism that the fighting is most vicious because the stakes are so small.
The difference between educational and literary research is that the latter has the potential to rise to the level of art itself and, for the most part, doesn’t masquerade as an activity that is informing the practices it studies. No one writing about the dystopic vision of Gary Shteyngart’s latest novel claims their work is making Gary Shteyngart, or anyone else, a better writer. Even ordinary book critics aim to make better readers, not writers. Creative writing theory, of course, makes just these claims and look what we’ve ended up with: a workshop culture contributing to a world of artistic creation that isn’t significantly better on the whole than it was before creative writing programs and workshops existed, and is arguably worse.
Which is why the traditional field of educational research matters to me only insofar as it matters to other people– specifically those with administrative roles that put them in a position to limit or negatively effect my work. I suspect educators are better off observing and learning from the narratives and qualitative reporting from their community of practice– and reading books like Surely You’re Joking, Mr. Feynman– than scouring ERIC for the latest studies or buying the newest overpriced compendium of research results whose practical use is almost nil by the time the results are generalized in a way so as to be meaningful when considering the mythical average student.
Action based research is one useful research model, primarily because it facilitates continued attention to one’s own processes, methods, and interventions… and it seems to be the model that yields the most interesting and productive qualitative write-ups of one’s experience as a teacher, sharing what one did and what happened, rather than trying to “prove” what’s true with enough predictive value to be useful to someone else.
Themocracy
I don’t mean to target you directly, but as I read posts like this I often find myself thinking that there is an inverse relationship between the amount a person criticizes educational research and how much educational research that person does (action-based or otherwise).
I see nothing wrong in educational research attempting to find common themes across classrooms, disciplines, teaching techniques, and technologies. I think there are many people grateful that attempts are being made to isolate what works, and what doesn’t. Think of it this way: Teacher X tries Amazing New Pedagogy and it works. Teacher Y tries Amazing New Pedagogy and it fails. Teacher Z can try and compare the two to see what happened and how they can prevent the failure in their class. But the situation seems to beg for some kind of *systematic* comparison. Whether it is too generalized or not is debatable. For some studies and some educators maybe, but for others there is value in that. So if some educators are finding value in educational research, what’s the harm in them trying?
It seems
I’ve been involved in both more traditional and action-based research. But it stands to reason that those who make their careers in those areas would be quick to defend themselves and overestimate the benefits.
The harm in trying– and not a potential, but an actual one– is a) overestimating the degree to which individual classroom experiences CAN be generalized. From which comes b) devoting resources to such studies for minimal gains, if any, that could more fruitfully be used elsewhere. Educational research has been given a lot of attention for a long time… where are the gains? Where are the outcomes that are not under significant dispute?
I think what is needed is innovation, and innovation typically only depends lightly, if at all, on quantitative research.
PS. I, for one, can only be “grateful for the attempts” to isolate what works if there is some evidence that this identification actually happens. I see scant evidence of such productive (ie that has predictive value) returns, and I don’t think it’s a mystery as to WHY, given that good teaching is at least as much art as craft and the foibles of trying to generalize about people’s capacities and ways of learning in the same way medical science generalizes about their physical systems… and then expecting that to have a significant effect in the general case.
Chris, I appreciate your point of view, and am also glad that Seth challenged you, because it brought out some responses that were more satisfying. I share a number of your frustrations re. educational research. Also:
1. A lot of educational research is shoddy: poorly designed, overinterpreted, not replicated.
2. Encouraging educational research is, as you point out, not universally applicable. This is also true for medicine. And, of course, any quantitative study speaks of presumably normal samples of populations, not individuals. And you know how I feel about that.
So this aspect alone can be frustrating, and since setting up educational research studies is intensive, replicating such studies in the myriad of unique settings and for diverse audiences is daunting, if not impossible.
3. Little of the good educational research outcomes are actually applied to practice. This happens in instructional design, too. Maybe because doing so is too challenging, too time-consuming. Maybe it doesn’t align with state standards, or maybe educators are simply ignorant to it. Etc.
I’m trying not to get too into this argument, because I could spend all week on it, and I think we would end up agreeing. You made an important point that suggests the cost-effectiveness of educational research should be considered. I agree, but even so let me add that I still have significant hope for educational research, even quantitative research, but my definition of it may be broader than what’s been described so far.
Jared:
re #2- in how many respects do you think that people learn in a way that is consistent across populations in the way that, say, the human body tends to respond to antibiotics? I think the primary problem of educational research is that it is based on a foundational mistake w/r/t this question of applicability.
re #3- please provide a few examples so I can know what you are referring to. What I see across the ed research landscape are mostly confounding results from the outset or confounding results that come FROM trying to apply those “good” results from one population to the next. Where are the good practices that are proven but not being implemented?
Where does all this faith in educational research come from? Where does this assumption that learning is a close enough parallel to the physical organism that research in the same mode will be productive come from? Quantitative research is already complicated and often non-productive beyond the simplest levels of biology… do you really think that it makes sense when dealing with learning beyond the most rudimentary levels of cognition?
[...] This post was mentioned on Twitter by Alastair Creelman, Chris Lott. Chris Lott said: New at Ruminate: Skeptical About Education Research – http://chrislott.org/2010/10/23/skeptical-about-education-research/ [...]
It’s funny that you suggest I’m too quick to defend myself and “overestimate” the benefits because that’s where you think my career may lie, because I took your post the same way. It seems to me that you are less interested in research (quantitative or not) and more in narratives to deconstruct. To me, this post came across as an attempt to tend edtech research into some sort of comparative literature contest. Edtech research re-imagined in such a way would certainly (coincidentally, I’m sure) play to your strengths and preferences. In contrast, you have me defending quantitative research, something I’m only beginning to gain any sort of skill at.
It’s fascinating to me that your argument seems to hinge on this implicit dehumanizing of quantitative researchers (and me) into these knee-jerk lemmings who have blind “faith” in educational research.
I’m not saying that all edtech research should be quantitative. By all means, action-based or qualitative research have important insights into the field and should be pursued (and funded). Nor am I against innovation. If alternatives can be derived and people (practitioners or researchers) can find value in them, then they should be pursued as well. I am confident that the value in these alternative approaches would be seen soon enough. Perhaps that makes me an optimist. I think educational research is a big tent and anyone with a sound idea should be welcome.
You seem pretty intractable in your cynicism of edtech research, so I’m going to bow out. As blog owner, you’ll of course have the last word. But let me summarize on a point of common ground: New research strategies could be beneficial to the field and I would welcome any efforts in that direction.
Seth– you might wish to read more closely. You made a generalized observation that the less people were involved in research, the more likely they were to diminish the productivity of research efforts. I, in turn, observed that it stands to reason that those who are deeply involved would be most likely to overestimate that same productivity. I don’t know you, so I have no opinion on whether you are a research lemming or not (though I assume not, as I assume most researchers are not, even if I think that, for the most part, research provides very little gain at too great an expense).
What I DO maintain is that the faith in the power of research to provide significant gains for teachers and learners is just that: faith. I see very little benefit on the ground in proportion to the immense amount of effort that has gone into research, particularly quantitative research.
I don’t know that I’m seeking to turn ed research into a comparative literature game… though I’m not sure that the way art is studied, when applied to teaching and teachers, doesn’t have the power to produce gains comparable or greater than quantitative research for any individual educator and his or her ever-changing group of students.
And that really is the central dilemma: where are the similarities between teachers and learners significant enough for quantitative research results to provide useful insights? And at what cost?
If research came at zero cost, then the big tent strategy of welcoming any and all research for even the most minuscule value would make a lot more sense. As it is, institutions are in crisis, education itself is in crisis, and I don’t share the faith that the meager gains of quantitative research is the answer to that dilemma.
One other note, Seth: it’s true that as a blog owner, I have the power of the last word. But it’s a big web, if you had a blog or something to link to where you discuss this kind of thing, I’d be happy to share that with all 7 of my readers. Alternatively, I’d be happy to give you a guest post here that I will refrain from commenting on!