Trusting research
Suppose you research the arms race between liars and debunkers. You'd have to hypothesize some, predict where they will coverup next as they catch on.
One resilient strategy is false data, with which a solid methodology (even exaggeratedly solid) means nothing. Verifying data is problematic. There is a trust chain as well. Suppose the university vouches for a researcher's methodology, even sending out someone to watch the data being collected and having someone neutral input the data and record a cryptographic hash of the resulting datafile, before handing it to the researcher. Do you trust the university? This also introduces a vector of corruption, so that those who can't afford to pay an university won't be taken seriously.
Nassim Taleb suggests requiring service providers to have "skin in the game". It's best if this is not simple threat of penalty, which doesn't work in a corrupt society. The provider themselves should benefit from providing what they say they provide.
Some tricks include assigning a singular person rather than an amorphous organization such as the university, which induces a lot of biases in people from marketing and halo effects and has internal diffusion of responsibility. A singular person is something that the university can reject to protect their image. Further, researchers would have motivation to select such a person based on how many people trust that person already, as well as how unrelated they are to the researcher. In fact this whole strategy is implementable by you, today. Just do this thing, have someone who is not your friend, nor stands to benefit (e.g. you give them $100 regardless of whether or not they help you produce a confirmatory paper), vouch. Just ask them to input the data and promise to stand up for the hash they got whenever someone asks. It'll give your paper an edge, right?
Perhaps regardless of corruption, we can benefit from having a measure of "probability the paper is false". John Ioannidis did some work on this. He can give you a prior (80%), if you don't rehearse his evidence.
Pr(Dishonest) increases with:
- verbosity
- complexity
- Big Words (good studies do not need to dress up)
- grouping data in inconsistent ways
- not explaining how or why they grouped data or calculated averages
- not setting hypotheses before the study
- the Pr(Dishonest) (compared to our prior) of their other papers
On the other end, Pr(Dishonest) can only go so low. If the methodology is exaggeratedly, unusually solid, it may be a savvy researcher who chose to put the lie in the dataset so that the methods can be solid and easy to explain. Imagine that: an actual good study predicting something unexpected! The laurels and fame! We should set a floor of 10%, the double of so-called "statistical significance". Or base it on a prior motivation to lie (if this is not already in our 80% prior).
Check if you agree with this
Pr(Lying|Solid methods) ~= Pr(Lying)
IOW that the solidity of methodology is irrelevant if the researcher intends to lie anyway – it just affects the location of the lie.
Checking who funded a study is a common way to check for bias today. More and more funders will catch on, creating front organizations. In Merchants of Doubt, they show that many front groups sound like grassroots organizations e.g.