The iatrogenics of having a model

"Stan uses No U-Turn sampling, JAGS uses Gibbs sampling". Great. How many people understand that? Is the method proven? What is it proven to do? What are the assumptions that sneak aboard, if any?

The more subcomponents a piece of research is built on, the more people we need to trust. I want to know that the mathematicians/economists/etc that have popularized each of these components knew what they were doing, and that those who get into the gritty math anew also see nothing amiss.

When it comes to using pure math lemma to prove another lemma, there's little to worry about, as math is unusual in being provable as a whole. When a theorem has been proven once, you never worry about it again.

This is one aspect of Trusting research: trusting the implementation correctness. It overlaps a bit with my idea of having an explicit Chain of verifiability, so that you can Measure payoff from increased complexity.

…

But when I wrote this title, I had something else in mind with "iatrogenics". Basically the harm done when people have a mental framework they trust too much, when they'd be better served having no framework at all.

Example 1: Categorizing has consequences.

Example 2: The finance bubble of 2008 was propped up on unreasonable beliefs in that the economists and financial analysts were modeling the world well. You could call it a species of when "a scientist's mindset oversimplifies reality" but I'd place the fault on the fact that the analysts involved didn't have high epistemic standards and everybody else believed they did. Perhaps I'd prefer to fault the deceptive practices of big banks, but if the analysts were more intimately aware of the "iatrogenics of creating a model", more aware of what they didn't know, perhaps they'd never have gone out and made claims others would bet on.

Sometimes nerds think that people's behavior ought to be reducible to a small set of rules just like physics, and that's an error we could talk about more in high school: it's possible to do real science in social sciences, but due to people being people, it's much harder to extract useful truth than in fields of study that don't involve people (Double hermeneutic), and there is never gonna be a small set of rules. (For more on this: History is not a science)