Learning, research, truth

It’s the pause between grades and graduation at my university. It’s also the season for many of our thesis students to present their work to outside groups.

Our students work on a variety of projects in their final year: Some create campaigns, others mount community projects, and still others create the beginnings of an innovative tool that might help with issues from education to the environment.

Instead of defending their work before a panel of academics, our students face private corporations or government agencies. The questions they get are less theoretical, more pragmatic: What does all this mean? How much will it cost in the long term? Who will this benefit?

There, too, will be criticism. We in the academe understand that sometimes, the students will not be able to draw out all the implications of their research. They might be blind to bureaucratic messes. They will not always have a ready answer for aspects that are outside the scope of their work.

Lately, however, I’ve heard critique so infuriating in its condescension.

“Your findings are invalid.”

When our students go through their thesis, especially in the social sciences, they will have completed two years of systematic, theory-driven research that addresses a specific problem in society. Our students plan their work thoroughly, from conceptualization, where they frame the problem, the methods, where they look for abstract ideas in reality, to the analysis, where they employ tools to examine their data.

If they want to look for broad ideas applicable to large groups, but with no depth, then they use quantitatively-based methods, such as surveys or quasi-experiments. They might ask about perceptions or effects. However, they cannot get to deeper concepts such as culture, upbringing, or understanding.

If they want to look for ideas that are deeply seated, whether in small or large groups, then they use qualitatively-based methods, such as interviews or focus group discussions. These include asking about or deriving culture, upbringing, or understanding. They cannot get to broad concepts such as perceptions or effects.

If they want to mix their methods, then they can get more details for their findings, or triangulate their findings; but they cannot get depth.

When are findings invalid?

When the methods do not answer the research question. If the researchers want to know people’s opinions about a film, they have to ask the viewers, not the director or producer.

When the methods do not match the objectives. If the researchers want to get in-depth perceptions about safety, then they cannot do a survey. If they insist on doing a survey to supposedly derive in-depth answers, then their findings are invalid.

When the researchers craft leading, vague, or multipronged questions. Our students spend weeks creating their questions, then pilot testing and refining them until their participants understand exactly what they mean. If questions are badly phrased, do not measure what a researcher intends to measure, or deliberately lead participants to a desired answer, then the findings become invalid.When the researchers use the wrong analysis tools. If they want to look at deep concepts, they have to do a systematic qualitative analysis of interview or discussion data. They don’t count words.

When the researchers analyze only the parts of the data that they like and leave out anything that disagrees with their opinion.So, what might the critic have meant?

Perhaps the critic felt that the research findings challenged their beliefs, told them that their efforts at solving the same problem were not working, and revealed that they had been laboring under unfounded assumptions about human behavior.

The critic felt invalidated.

Invalid findings should not equal feelings of invalidation. The first requires a closer look at data, and asking if the framework, methods, and findings logically fit together.The second is simply pride.

Research fights against norms, speaks out against injustices, questions and accepts all answers. I hope that the critic—and all outside entities—listen to our students first instead of putting up their defenses.

This is, after all, the voice of the observant, truth-telling youth.

Perhaps there really is no voice more frightening than that.

[email protected]