Quality of Evidence
An important thinking and writing skill you must gain as a doctoral student is being able communicate the “quality of evidence” of the studies you cite. Too often, doctoral students want to spend a full paragraph on explaining the methods when really, what matters is the evaluation and analysis of the study. We expect our students to be able to interpret study results, explain why some studies weigh heavier in their findings than other studies, then discuss the impact various studies have on the research theme being explored in your analysis. This is called the “Quality of Evidence.”
In talking with our program director, Dr. Shepard, she shared with me that students should be able to address ALL of the methods in a clause introducing the study, and the rest of the text is focused on the analysis, and addressing the quality of evidence of that article. The clause will be some variation on this formula:
In this (methodology) study + methods = findings were…
Here are some examples:
In this phenomenological study, interviews were conducted with 12 teachers and 4 administrators, along with two focus groups with 6 teachers each. Findings indicated that…
In this qualitative single case study, two one-hour interviews were conducted with seven teachers along with a review of multiple artifacts of… Findings indicated that…
In this multiple case study of five teachers in three high schools, it was found that…
Essentially, you want to make sure you’re not writing large summaries of methodology, but instead exerting your energy on the interpreting of the findings and how it related to your own theme and other studies you have found. If you write more than 2 sentences explaining the methods, it is most likely too much!
Analyzing, Comparing, Contrasting, and Evaluating Research Findings
However, when analyzing, comparing, contrasting, and evaluating research results, not all studies are equal. You must consider the “power” that the participant sample paired with the methods have, before making over-reaching statements of comparison or analysis. The purpose of having you summarize the methods in one sentence is to help you focus on the quality of evidence the study has. For example, which of the two studies do you believe has a higher quality of evidence?
- A case study of 5 teachers who were interviewed at a single school
- A mixed methods study where the researcher gathered data from 450 teachers across the USA using a validated/reliable survey, and followed up with interviews of 12 teachers
How do you handle this if the first study had results that align with the research you are proposing to do for your dissertation? The mixed methods study on the other hand had results that were contradictory to the case study results. You must be careful not to place emphasis on the study whose data isn’t as powerful, just because you like the results better! Your analysis must be fair, and any comparisons you make must not ignore this quality of evidence. Your referencing the studies must be truthful and represent the data accurately.
Apply these ideas to your own writing. Have you been guilty of writing long summaries of study methods? Have you considered the methods and number of study participants when comparing and analyzing studies? Would love to hear your stories.