Hill, Ball, and Schilling created a large-scale study involving thousands of teachers. The purpose of the study was to analyze two major factors in effective teachers: knowledge of curriculum, and knowledge of student learning. Teachers analyzed student work and attempted to make sense of where they went wrong and for what reason. A large portion of the research results was based on a multiple choice platform. It was found that the wrong answers in the multiple choice were almost too wrong, making it easier for the teachers to choose the correct answer. Upon reflection, the researchers decided that further research should be accomplished through more open-ended questions. It was very difficult to differentiate what specifically allowed a teacher to answer a question correctly. It was decided that for many of the questions, teachers drew from both their content knowledge, and their knowledge of student learning. Even teachers who supposedly had minimal content knowledge of a particular strain of math, were able to effectively determine where the student went wrong, because their knowledge of how students think compensated. In the end, Hill, Ball, and Schilling were able to say that knowledge of student learning was important in being an effective teacher, however they were not able to specifically analyze solely.
One takeaway that I had from this article is that I appreciate even more so the benefit of really analyzing student work, and not just marking whether it was right or wrong. For this reason, it is very important to continue to require students to show their work. By viewing the problem-solving process, teachers are able to make adjustments to their teaching.
One problem that I had with the article, is that it was very data-dense. I understand a need for data analyses in the beginning of an article, however I did not gain a full understanding of what was achieved until the conclusion. There were many references throughout the article of the questions that the teachers had to analyze, but the researchers did present one of these said questions their paper. This made it very difficult to understand what the teachers in the study were analyzing.
My question that I have is:
If a multiple choice style of research produced inaccurate results, and a large-scale study is required to produce a desired degree of validity, how can a large-scale study be achieved through open-ended research methods?
Apologies for the lengthiness of the response.
Hi David,
ReplyDeleteThanks for your overview of the paper. Did the authors reference a paper or website with the questions? I find that sometimes authors include a reference of the questions, rather than including it as the appendix. Makes it a bit difficult for the reader, eh?
Your question is a loaded one, and one that I think people have been debating over for years. Although quantitative studies may give us a "big picture" of what is going on, as you note, the main questions were concerned with "what" was going on. Qualitative work is (arguably) more concerned with the "why," and thus require more interaction with participants, such as focus groups, interviews, etc. For me, the heart of qualitative work is the individual, so I prefer working with smaller groups, rather than large scale. Moreover, the amount of data that comes out of a qualitative study of a single person can be more than enough, let alone more than 10!
For me, the notion of large scale qualitative work seems impractical unless you have a large team of researchers who are on the exact same page in terms of interpretation and analysis. Otherwise, the validity of the study could be compromised.
To challenge your statement (and be a devil's advocate), what makes a large scale qualitative study "more valid" than the qualitative study of a single student, provided the researcher was rigorous in their methodology?
The so-called `mixed-methods' approach is probably the common ground we are looking for, no? I too, find myself surprisingly dissatisfied with purely quantitative studies: how do you determine what went wrong or if there is anything interesting happening if all you have are mostly meaningless test scores? Conducting interviews (or something) with a very small number of participants could lead to helpful insights.
ReplyDeleteI also like your point about the need to analyze student work more carefully. Students (and we!) are prone to mixing up methods and getting things confused. Having the space to recognize that there was misplaced effort is so important. I know the desire to dismiss a mistake as simply incorrect is strong, but I think recognizing not just where your students are making errors, but that they are *trying* is so critical.