Last year, I conducted interviews with researchers about their attitudes towards open science. Some interviewees told me that people are easily put into camps, among others ‘the open science movement’ or ‘the critical people’. Such a label might trigger different reactions or even prejudice. Being interested in dynamics within academia, I was curious about what motivates ‘a critical person’ and how it feels to be categorized as one.
Rink Hoekstra is arguably one of these critical researchers. In his publications, he has written about how the reporting of research results might be misleading and how researchers misinterpret different statistical outcomes. His topics of interest include reflections on how research is conducted, incentivized, analyzed, read, interpreted, and which shortcuts to academic success are taken. Therefore, Rink could be put in the category of ‘the critical people’.
When I asked Rink whether he thinks that he is making a career out of being critical, he answered “yes, but I didn’t do it on purpose”. He explained that he usually just does what is interesting to him, and it somehow “ends up being a career”. When he started his Ph.D. in 2003, studying critically how research is conducted was not a popular topic and some people were explicitly doubting its necessity. Rink was surprised that his research field suddenly gained more interest in what later came to be known as the replication or credibility crisis. The resulting current popularity of meta-science also further increased the interest in his work.
“If many people make a mistake, it is probably not the individual researcher, but the system’s fault as well”.
Criticism on the System Rather Than the Individual
When being critical about science, both the individual and the system can be criticized. As Rink explained, while criticism directed at individual research output or researchers might constitute a necessary self-correcting mechanism in science, he does not focus on that. Rather, his criticism is targeted at the system as a whole and how it is functioning.
The Speed-Accuracy Tradeoff in Research Quality – Incentivised by the System
In his career, Rink has focused on criticizing the incentive system of academia, specifically how research is reinforced and (implicitly) encouraged. By incentivising quantity in a system with limited time and resources, multiple behaviors that do not necessarily produce the ‘best’ research are supported, including sloppy reading and dichotomous thinking. Put differently, cutting corners is encouraged and the individual researcher needs to decide which corners to cut. As Rink explained, “if quantity is valued, it may very well affect the quality”, as some kind of an alternative “speed-accuracy trade-off”. The quicker research output is produced, the less accurate it might be, contributing to results that may not be reproducible.
Reliability vs Validity of Incentivising
The quantity-over-quality approach of incentivizing shows in the selection procedure of permanent positions and research grants. Rink explained that “it’s a heuristic, but it’s a very bad heuristic and at a minimum, we should acknowledge how bad it is and aim for a bit higher”. He illustrates this with the concepts of reliability and validity. Incentivising quantity over quality is very reliable: if you count the publications for a job application, you always will count the same number. However, is quantity also a valid measure for job suitability in academia? What even determines the validity in this context? Figuring out the most valid measures may be very difficult and might introduce more subjectivity. Rink thinks that balancing out the advantages and disadvantages when designing new incentive structures is one of the next challenges.
Also From an Individual Perspective, We Should Aim for Slow Science
According to Rink, the incentive system might unintentionally select certain scientist personalities over others and unfavorably change the behavior of people working in academia. First, the system selects researchers that are possibly rather goal-oriented than process-oriented, but might lose potentially good researchers that do not fit within that pattern. Second, researchers may also be influenced in an unfortunate way. The ‘rat race’ encourages competition and because everybody seems to be doing it, it becomes normal to work too many hours to get there. This may not be a healthy system for the individual. Thus, Rink argues that we should put more emphasis on educating students on this and, also from the individual researcher’s perspective, aim for slower science.
The Value of a Critical Community
“I think there is a lot wrong in science. By studying it, I am not so annoyed by it anymore. If I were in the system, there would be a discrepancy between what I do and what I think. Studying these topics enables me to talk about what I actually think”.
Rink was involved in founding the Open Science Community Groningen. While he does see himself more in the periphery of the international Open Science movement, he thinks that such movements are important. There is a lot of heterogeneity of opinions within the movement, which makes the discussion very valuable. Rink expressed that that being part of a critical community gives him an excuse not to participate in the ‘rat race’.
Ways to Move Forward
“We should be careful not to be too self-congratulating.”
Rink argues that one challenge of critical communities is not to appear too exclusive. Especially a feeling of self-congratulation within the movements should be avoided. Rink explains that “we should be careful not to be too self-congratulating like ‘look how good we are, we are the cool people, we realize what’s wrong, and the other people, we have to open their eyes’”. There is some arrogance to that, which makes Rink feel uncomfortable. However, he also is careful in making these statements and does not want to present himself in a better light. What he calls meta-arrogance might be hindering a constructive discussion and should be avoided.
Rink thinks that it is important for everyone to acknowledge problems, even if there are no solutions yet. Some people think that description of what is ‘going wrong’ without offering concrete solutions is unconstructive. Rink expresses his disagreement with this argument: “Imagine not being allowed to talk about climate change until you’re able to solve it”. He explains that just describing what is going on is important and valid in itself. Recognizing and acknowledging problems should be more normal, and Rink thinks that this is actually what is already happening in both the individual researcher and policymakers.
“Let’s not use a discussion about tone to evade a discussion about substance.”
Moreover, Rink points out that spending too much time on being upset about the way criticism is communicated might hinder scientific development by avoiding the actual issue. If we do not talk about the substance anymore, but we talk about “who said what and how it was said”, that might be dangerous to advancements.
Conclusion and Conflict of Interests
Rink’s work is motivated by improving science as a whole: ”If I can make a small contribution I am already happy”. He seems optimistic about the future of science and sees that things are changing for the better. However, he does not fear his job becoming obsolete, because that would imply that a reflective perspective on science is not needed anymore. “Given that science is a human endeavor, we need a human perspective on it”, he says. Thus, critically reflecting on research methodology will always stay important.
I interviewed Rink to understand more about how it is to be actively critical about how we do science and the academic system. While I rather wanted to learn more about how it is to be (perceived as) a critical person, we ended up talking about the incentive system and the critical community. I learned that it is important to distinguish between the target of criticism, about the heterogeneity of criticism, and that science possibly always needs human reflection. Rink gave some interesting metaphors and insights about the incentive system. Being a ‘critical person’ is part of his work identity and he has an optimistic view on the development of science. In the future, I would like to find out more about these dynamics within science and therefore this will only be the first post in a series about ‘the critical people in academia’.
This post does not claim to be objective because (1) Rink is my supervisor, (2) I have similar research interests and opinions, and (3) Rink was involved in the editing process.